What is known about invariants in self-modifying systems?

post by mishka · 2023-12-02T05:04:19.299Z · LW · GW · 2 comments

This is a question post.



I have been trying to find out what is known about invariants in self-modifying systems. This might become a rather acute topic if we end up moving towards self-modifying AIs or self-modifying ecosystems of AIs.

But it seems that not much has been done. For example, I have found a 1995 Chinese paper, "S-and T-Invariants in Cyber Net Systems" by Yuan Chongyi, Google Scholar page, PDF available which is doing a study of invariants in self-modifying nets (a natural extension of Petri nets), but it only has 4 references to it known to Google Scholar.

I wonder if people know about more examples of this kind of research (or about researchers or organizations currently trying to look at this topic)...



Comments sorted by top scores.

comment by Charlie Steiner · 2023-12-03T06:03:16.688Z · LW(p) · GW(p)

Yeah, not a ton. For I think the obvious reason that real-world agents are complicated and hard to reason about.

Though search up "tiling agents" for some MIRI work in this vein.

Replies from: mishka
comment by mishka · 2023-12-03T07:49:55.865Z · LW(p) · GW(p)

Yes, thanks!

I am familiar with some work from MIRI about that which focuses on Loebian obstacle, e.g. this 2013 paper: Tiling Agents for Self-Modifying AI, and the Löbian Obstacle.

But I should look closer at other parts of those MIRI papers; perhaps there might be some material which actually establishes some invariants, at least for some simple, idealized examples of self-modification...