“embedded self-justification,” or something like that

It seems to me that there are roughly two types of "boundary" to think about: ceilings and floors.

• Floors are aka the foundations. Maybe a system is running on a basically Bayesian framework, or (alternately) logical induction. Maybe there are some axioms, like ZFC. Going meta on floors involves the kind of self-reference stuff which you hear about most often: Gödel's theorem and so on. Floors are, basically, pretty hard to question and improve (though not impossible).
• Ceilings are fast heuristics. You have all kinds of sophisticated beliefs in the interior, but there's a question of which inferences you immediately make, without doing any meta to consider what direction to think in. (IE, you do generally do some meta to think about what direction to think in; but, this "tops out" at some level, at which point the analysis has to proceed without meta.) Ceilings are relatively easy to improve. For example, the AlphaGo move proposal network and evaluation network (if I recall the terms correctly). These have cheap updates which can be made frequently, via observing the results of reasoning. These incremental updates then help the more expensive tree-search reasoning to be even better.

Both floors and ceilings have a flavor of "the basic stuff that's actually happening" -- the interior is built out of a lot of boundary stuff, and small changes to boundary will create large shifts in interior. However, floors and ceilings are very different. Tweaking floor is relatively dangerous, while tweaking ceiling is relatively safe. Returning to the AlphaGo analogy, the floor is like the model of the game which allows tree search. The floor is what allows us to create a ceiling. Tweaks to the floor will tend to create large shifts in the ceiling; tweaks to the ceiling will not change the floor at all.

(Perhaps other examples won't have as clear a floor/ceiling division as AlphaGo; or, perhaps they still will.)

What remains unanswered, though, is whether there is any useful way of talking about doing this (the whole thing, including the self-improvement R&D) well, doing it rationally, as opposed to doing it in a way that simply “seems to work” after the fact.

[...] Is there anything better than simply bumbling around in concept-space, in a manner that perhaps has many internal structures of self-justification but is not known to work as a whole? [...]

Can you represent your overall policy, your outermost strategy-over-strategies considered a response to your entire situation, in a way that is not a cartoon, a way real enough to defend itself?

My intuition is that the situation differs, somewhat, for floors and ceilings.

• For floors, there are fundamental logical-paradox-flavored barriers. This relates to MIRI research on tiling agents.
• For ceilings, there are computational-complexity-flavored barriers. You don't expect to have a perfect set of heuristics for fast thinking. But, you can have strategies relating to heuristics which have universal-ish properties. Like, logical induction is an "uppermost ceiling" (takes the fixed point of recursive meta) such that, in some sense, you know you're doing the best you can do in terms of tracking which heuristics are useful; you don't have to spawn further meta-analysis on your heuristic-forming heuristics. HOWEVER, it is also very very slow and impractical for building real agents. It's the agent that gets eaten in your parable. So, there's more to be said with respect to ceilings as they exist in reality.

I can never tell whether they’ve never thought about the things I’m thinking about, or whether they sped past them years ago.  They do seem very smart, that’s for sure.

Whenever I have a great idea, it turns out that someone at MIRI considered it five years earlier. This simultaneously makes me feel very smart and rather dissapointed. With that being said, here are some relevant things:

Thing #1:

Oh, you’re so very clever!  By now you’ve realized you need, above and beyond your regular decision procedure to guide your actions in the outside world, a “meta-decision-procedure” to guide your own decision-procedure-improvement efforts.

This is a nitpick but it's an important one in understanding how meta-stuff works here: If you've decided that you need a decision procedure to decide when to update your decision procedure, then whatever algorithm you used to make that decision is already meta. This is because your decision procedure is thinking self-referentially. Given this, why would it need to build a whole new procedure for thinking about decision procedures when it could just improve itself?

This has a number of advantages because it means that anything you learn about how to make decisions can also be directly used to help you make decisions about how you make decisions--ad infinitum.

Thing #2:

You are a business.  You do retrospectives on your projects.  You’re so very clever, in fact, that you do retrospectives on your retrospective process, to improve it over time.  But how do you improve these retro-retros?  You don’t.  They’re in your boundary.

This case reminded me a lot of Eliezer on Where Recursive Justification Hits Rock Bottom [LW · GW] except placed in a context where you can modify your level of recursion.

You need to justify that your projects are good so you do retrospectives. But you need to justify why your retrospectives are good so you do retrospectives on those. But you need to justify why your retro-retros are good too right? To quote Eliezer:

Should I trust my brain?  Obviously not; it doesn't always work.  But nonetheless, the human brain seems much more powerful than the most sophisticated computer programs I could consider trusting otherwise.  How well does my brain work in practice, on which sorts of problems?

So there are a couple questions here. The easy question:

Q: How do I justify the way I'm investing my resources?

A: You don't. You just invest them using the best of your ability and hope for the best

And the more interesting question:

Q: What is the optimal level of meta-justification I use in investing my resources?

A1: This still isn't technically knowable information. However, there are plenty of unjustified priors that might be built into you which cause you to make a decision. For instance, you might keep going up the meta-levels enough rounds until you see diminishing returns and then stop. Or you might just never go above three levels of meta because you figure that's excessive. Depends on the AI.

A2: Given that Thing #1 is true, you don't need any meta-decision algorithms--you just need a self-referential decision algorithm. In this case, we just have the answer to the easy question: You use the full capabilities of your decision algorithm and hope for the best (and sometimes your decision algorithm makes decisions about itself instead of decisions about physical actions)

Just a quick note: Sometimes there is a way out of this kind of infinite regress by implementing an algorithm that approximates the limit. Of course, you can also be put back into an infinite regress by asking if there is a better approximation.

A lot of what you write here seems related to my notion of Turing Reinforcement Learning [AF(p) · GW(p)]. In Turing RL we consider an AI comprising of a "core" RL agent and an "envelope" which is a computer on which the core can run programs (somewhat similarly to neural Turing machines). From the point of the view of the core, the envelope is a component of its environment (in addition to its usual I/O), about which it has somewhat stronger priors than about the rest. Such a system learns how to make optimal use of the envelope's computing resources. Your "boundary" corresponds to the core, which is the immutable part of the algorithm that produces everything else. Regarding the "justification" of why a particular core algorithm is correct, the justification should come from regret bounds we prove about this algorithm w.r.t. some prior over incomplete models [AF · GW]. Incomplete models are the solution to "even if you could obtain a perfect model of your world and beings like you, you wouldn’t be able to fit it inside your own head". Instead of obtaining a perfect model, the agent learns all patterns (incomplete models) in the world that it can fit into its head, and exploits these patterns for gain. More precisely, in Turing RL the agent starts with some small class of patterns that the core can fit into its head, and bootstraps from those to a larger class of patterns, accounting for a cost-benefit analysis of resource use. This way, the regret bound satisfied by the core algorithm should lead to even stronger guarantees for the system as a whole (for example this [AF(p) · GW(p)]).

'Doing it well' seems to be very load bearing there. I think you're sneaking in an 'all' in the background? Like, in order to be defined as superintelligent it must do better at all domains than X or something?

My current answer is something hand wavy about the process just trying to ungoodhart itself (assuming that the self and world model as given start off goodharted) and the chips fall where they may.

Why did you spend it as you did, then?  You cannot answer, ever, without your answer invoking something on the boundary.

It seems like if the O(N^(n-1)) algorithm (algorithm 2) is better than the O(N^n) algorithm (algorithm 1), then there is an amount of time, such that, after that time has elapsed (after adapting the new algorithm), the reduction in resource consumption will equal the cost spent finding the new algorithm. This might be called "breaking even", and doesn't seem to invoke something on the boundary.

I think that this infinite regress can be converted into a loop. Given an infinite sequence of layers, in which the job of layer $n+1$ is to optimise layer $n$. Each layer is a piece of programming code. After the first couple of layers, these layers will start to look very similar. You could have layer 3 being a able to optimize both layer 2 and layer 3.

One model is that your robot just sits and thinks for an hour. At the end of that hour, it designs what it thinks is the best code it can come up with, and runs that. To the original AI, anything outside the original hour is external, it is answering the question "what pattern of bits on this hard disk will lead to the best outcome." It can take all these balances and tradeoffs into account in whatever way it likes. If it hasn't come up with any good ideas yet, it could copy its code, add a crude heuristic that makes it run randomly when thinking (to avoid the preditors) and think for longer.