Embedded Agents

I think it would be useful to give your sense of how Embedded Agency fits into the more general problem of AI Safety/Alignment. For example, what percentage of the AI Safety/Alignment problem you think Embedded Agency represents, and what are the other major chunks of the larger problem?

Very nice! I like the colour-coding scheme, and the way it ties together those bullet points in MIRI's research agenda.

Looks like these sequences are going to be a great (content-wise and aesthetically) introduction to a lot of the ideas behind agent foundations; I'm excited.

I actually have some understanding of what MIRI's Agent Foundations work is about

Promoted to curated: I think this post (and the following posts in the sequence) might be my favorite posts that have been written in the last year. I think this has a lot to do with how this post feels like it is successfully doing one of the things that Eliezer sequences did, which is to combine a wide range of solid theoretical results, with practical implications for rationality as well as broad explanations that improve my understanding not just of the specific domain at hand, but of a range of related domains as well.

I also really like the change of pace with the format, and quite like how easy certain things become to explain when you use a more visual format. I think it's a exceptionally accessible post, even given its highly technical nature. I am currently rereading GEB, and this post reminds me of that book in a large variety of ways, in a very good way.

The text is beautifully condensed. And the handwritten style does help it look casual/inviting.

But the whole thing loaded significantly slower than I read. How many megabytes is this post? I haven't waited this long for a website to load for years.

Insofar as the AI Alignment Forum is part of the Best-of-2018 Review, this post deserves to be included. It's the friendliest explanation to MIRI's research agenda (as of 2018) that currently exists.

This post (and the rest of the sequence) was the first time I had ever read something about AI alignment and thought that it was actually asking the right questions. It is not about a sub-problem, it is not about marginal improvements. Its goal is a gears-level understanding of agents, and it directly explains why that's hard. It's a list of everything which needs to be figured out in order to remove all the black boxes and Cartesian boundaries, and understand agents as well as we understand refrigerators.

Embedded Agency sounds like an essential property of the world to acknowledge, as we humans naturally think in terms of Cartesian dualism.

Excellent timing! Enough about this has been leaking onto LessWrong from Intelligent Agent Foundations that I have been wondering about it more intensely of late.

I nominate this post for two reasons.

One, it is an excellent example of providing supplemental writing about basic intuitions and thought processes, which is extremely helpful to me because I do not have a good enough command of the formal work to intuit them.

Two, it is one of the few examples of experimenting with different kinds of presentation. I feel like this is underappreciated and under-utilized; better ways of communicating seems like a strong baseline requirement of the rationality project, and this post pushes in that direction.

This post has significant changed my mental model of how to understand key challenges in AI safety, and also given me a clearer understanding of and language for describing why complex game-theoretic challenges are poorly specified or understood. The terms and concepts in this series of posts have become a key part of my basic intellectual toolkit.

From the version of this post on the MIRI blog:

Bayesian reasoning works by starting with a large collection of possible environments, and as you observe facts that are inconsistent with some of those environments, you rule them out. What does reasoning look like when you’re not even capable of storing a single valid hypothesis for the way the world works? Emmy is going to have to use a different type of reasoning, and make updates that don’t fit into the standard Bayesian framework.

I think maybe this paragraph should say "Solomonoff induction" instead of Bayesian reasoning. If I'm reasoning about a coin, and I have a model with a single parameter representing the coin's bias, there's a sense in which I'm doing Bayesian reasoning and there is some valid hypothesis for the coin's bias. Most applied Bayesian ML work looks more like discovering a coin's bias than thinking about the world at a sufficiently high resolution for the algorithm to be modeling itself, so this seems like an important distinction.

Thank you for explaining this in a way in which a visual learner like me can understand. Please make more comic style posts in the future!

This illustrated style is fantastic; the pictures and the colors really added something. Having all four topics named and bundled together was very helpful.

This sounds similar in effect to what philosophy of mind calls “embodied cognition”, but it takes a more abstract tack. Is there a recognized background link between the two ideas already? Is that a useful idea, regardless of whether it already exists, or am I off track?

In the comments of this post, Scott Garrabrant says:

I think that Embedded Agency is basically a refactoring of Agent Foundations in a way that gives one central curiosity based goalpost, rather than making it look like a bunch of independent problems. It is mostly all the same problems, but it was previously packaged as "Here are a bunch of things we wish we understood about aligning AI," and in repackaged as "Here is a central mystery of the universe, and here are a bunch things we don't understand about it." It is not a coincidence that they are the same problems, since they were generated in the first place by people paying close to what mysteries of the universe related to AI we haven't solved yet.

This entire sequence has made that clear for me. Most notably it has helped me understand the relationship between the various problems in decision theory that have been discussed on this site for years, along with their proposed solutions such as TDT, UDT, and FDT. These problems are a direct consequence of agents being embedded in their environments.

Furthermore, it's made me think more clearly about some of my high level models of ideal AI and RL systems. For instance, the limitations of the AIXI framework and some of it's derivatives has become more clear to me.

Just saw this linked on SlateStarCodex. I love this style. I also think it's valuable to demonstrate that it's possible to have fun without dumbing down the content at all.

This sequence was the first time I felt I understood MIRI's research.

(Though I might prefer to nominate the text-version that has the whole sequence in one post [? · GW].)

Thanks for also sharing this as text (on the MIRI blog); makes it much easier to read.

Read sequence as research for my EA/rationality novel, this was really good and also pretty easy to follow despite not having any technical background

Is this the first post in the sequence? It's not clear.

Bayesian reasoning works by starting with a large collection of possible environments, and as you observe facts that are inconsistent with some of those environments, you rule them out. What does reasoning look like when you’re not even capable of storing a single valid hypothesis for the way the world works? Emmy is going to have to use a different type of reasoning, and make updates that don’t fit into the standard Bayesian framework.

I think maybe this paragraph should say "Solomonoff induction" instead of Bayesian reasoning. If I'm reasoning about a coin, and I have a model with a single parameter representing the coin's bias, there's a sense in which I'm doing Bayesian reasoning and there is some valid hypothesis for the coin's bias. Most applied Bayesian ML work looks more like discovering a coin's bias than thinking about the world at a sufficiently high resolution for the algorithm to be modeling itself, so this seems like an important distinction.

I don't understand why being an embedded agent makes Bayesian reasoning impossible. My intuition is that an hypothesis doesn't have to be perfectly correlated with reality to be useful. Furthermore suppose you conceived of hypotheses as being a conjunction of elementary hypothesis, then I see no reason why you cannot perform Bayesian reasoning of the form "hypothesis X is one of the consituents of the true hypothesis", even if the agent can't perfectly describe the true hypothesis.

Also, "the agent is larger/smaller than the environment" is not very clear, so I think it would help if you would clarify what those terms mean.

At least some of these ideas have been pondered by anthropologists over the years.

I don't see the point in adding so much complexity to such a simple matter. AIXI is an incomputable agent who's proofs of optimality require a computable environment. It requires a specific configuration of the classic agent-environment-loop where the agent and the environment are independent machines. That specific configuration is only applicable to a sub-set of real-world problems in which the environment can be assumed to be much "smaller" than the agent operating upon it. Problems that don't involve other agents and have very few degrees of freedom relative the agent operating upon them.

Marcus Hutter already proposed computable versions of AIXI like AIXI_lt. In the context of agent-environment loops, AIXI_lt is actually more general than AIXI because AIXI_lt can be applied to all configurations of the agent-environment loop including the embedded agent configuration. AIXI is a special case of AIXI_lt where the limits of "l" and "t" go to infinity.

Some of the problems you bring up seem to be concerned with the problem of reconciling logic with probability while others seem to be concerned with real-world implementation. If your goal is to define concepts like "intelligence" with mathematical formalizations (which I believe is necessary), then you need to delineate that from real-world implementation. Discussing both simultaneously is extremely confusing. In the real world, an agent only has is empirical observations. It has no "seeds" to build logical proofs upon. That's why scientists talk about theories and evidence supporting them rather than proofs and axioms.

You can't prove that the sun will rise tomorrow, you can only show that it's reasonable to expect the sun to rise tomorrow based on your observations. Mathematics is the study of patterns, mathematical notation is a language we invented to describe patterns. We can prove theorems in mathematics because we are the ones who decide the fundamental axioms. When we find patterns that don't lend themselves easily to mathematical description, we rework the tool (add concepts like zero, negative numbers, complex numbers, etc.). It happens that we live in a universe that seems to follow patterns, so we try to use mathematics to describe the patterns we see and we design experiments to investigate the extent to which those patterns actually hold.

The branch of mathematics for characterizing systems with incomplete information is probability. If you wan't to talk about real-world implementations, most non-trivial problems fall under this domain.

I had a thought about this that was too long for a comment, so I've posted it separately [LW · GW]. Bottom line:

When thinking about embedded agency it might be helpful to drop the notion of ‘agency’ and ‘agents’ sometimes, because it might be confusing or underdefined. Instead one could think of [sub-]processes running according to the laws of physics. Or of algorithms running on a stack of interpreters running on the hardware of the universe.

I have been thinking about this for a while and I'd like to propose a different viewpoint on this same idea. It's equivalent but personally I prefer this perspective. Instead of saying 'embedded agency' which implies an embedding of the agent in the environment. Perhaps we could say 'embedded environment'. Which implies that the environment is within the agent. To illustrate, the same way 'I' can 'observe' 'my' 'thoughts', I can observe my laptop. Are my thoughts me, or is my laptop me? I think the correct way way took look at it that my computer is just as much a part of me as are my thoughts. Even though surely everyone is convinced that this dualistic notion between agent and environment does not make sense, there seems to be a difference between what one believes on a level of 'knowledge', and what you can phenomenologically observe. I believe this phrasing allows people to more easier experience for themselves the truthness of the embedded environment/agent idea, which is important for further idea generation! :)