What is known about invariants in self-modifying systems?
post by mishka · 2023-12-02T05:04:19.299Z · LW · GW · 2 commentsThis is a question post.
Contents
2 comments
I have been trying to find out what is known about invariants in self-modifying systems. This might become a rather acute topic if we end up moving towards self-modifying AIs or self-modifying ecosystems of AIs.
But it seems that not much has been done. For example, I have found a 1995 Chinese paper, "S-and T-Invariants in Cyber Net Systems" by Yuan Chongyi, Google Scholar page, PDF available which is doing a study of invariants in self-modifying nets (a natural extension of Petri nets), but it only has 4 references to it known to Google Scholar.
I wonder if people know about more examples of this kind of research (or about researchers or organizations currently trying to look at this topic)...
Answers
2 comments
Comments sorted by top scores.
comment by Charlie Steiner · 2023-12-03T06:03:16.688Z · LW(p) · GW(p)
Yeah, not a ton. For I think the obvious reason that real-world agents are complicated and hard to reason about.
Though search up "tiling agents" for some MIRI work in this vein.
Replies from: mishka↑ comment by mishka · 2023-12-03T07:49:55.865Z · LW(p) · GW(p)
Yes, thanks!
I am familiar with some work from MIRI about that which focuses on Loebian obstacle, e.g. this 2013 paper: Tiling Agents for Self-Modifying AI, and the Löbian Obstacle.
But I should look closer at other parts of those MIRI papers; perhaps there might be some material which actually establishes some invariants, at least for some simple, idealized examples of self-modification...