What's Up With Confusingly Pervasive Goal Directedness?

Suppose we took this whole post and substituted every instance of "cure cancer" with the following:
Version A: "win a chess game against a grandmaster"
Version B: "write a Shakespeare-level poem"
Version C: "solve the Riemann hypothesis"
Version D: "found a billion-dollar company"
Version E: "cure cancer"
Version F: "found a ten-trillion-dollar company"
Version G: "take over the USA"
Version H: "solve the alignment problem"
Version I: "take over the galaxy"

And so on. Now, the argument made in version A of the post clearly doesn't work, the argument in version B very likely doesn't work, and I'd guess that the argument in version C doesn't work either. Suppose I concede, though, that the argument in version I works: that searching for an oracle smart enough to give us a successful plan for taking over the galaxy will very likely lead us to develop an agentic, misaligned AGI. Then that still leaves us with the question: what about versions D, E, F, G and H? The argument is structurally identical in each case - so what is it about "curing cancer" that is so hard that, unlike winning chess or (possibly) solving the Riemann hypothesis, when we train for that we'll get misaligned agents instead?

We might say: well, for humans, curing cancer requires high levels of agency. But humans are really badly optimised for many types of abstract thinking - hence why we can be beaten at chess so easily. So why can't we also be beaten at curing cancer by systems less agentic than us?

Eliezer has a bunch of intuitions which tell him where the line of "things we can't do with non-dangerous systems" should be drawn, which I freely agree I don't understand (although I will note that it's suspicious how most people can't do things on the far side of his line, but Einstein can). But insofar as this post doesn't consider which side of the line curing cancer is actually on, then I don't think it's correctly diagnosed the place where Eliezer and I are bouncing off each other.

This dialog was much less painful for me to read than i expected, and I think it manages to capture at least a little of the version-of-this-concept that I possess and struggle to articulate!

(...that sentence is shorter, and more obviously praise, in my native tongue.)

A few things I'd add (epistemic status: some simplification in attempt to get a gist across):

If there were a list of all the possible plans that cure cancer, ranked by "likely to work", most of the plans that might work route through "consequentalism", and "acquire resources."

Part of what's going on here is that reality is large and chaotic. When you're dealing with a large and chaotic reality, you don't get to generate a full plan in advance, because the full plan is too big. Like, imagine a reasoner doing biological experimentation. If you try to "unroll" that reasoner into an advance plan that does not itself contain the reasoner, then you find yourself building this enormous decision-tree, like "if the experiments come up this way, then I'll follow it up with this experiment, and if instead it comes up that way, then I'll follow it up with that experiment", and etc. This decision tree quickly explodes in size. And even if we didn't have a memory problem, we'd have a time problem -- the thing to do in response to surprising experimental evidence is often "conceptually digest the results" and "reorganize my ontology accordingly". If you're trying to unroll that reasoner into a decision-tree that you can write down in advance, you've got to do the work of digesting not only the real results, but the hypothetical alternative results, and figure out the corresponding alternative physics and alternative ontologies in those branches. This is infeasible, to say the least.

Reasoners are a way of compressing plans, so that you can say "do some science and digest the actual results", instead of actually calculating in advance how you'd digest all the possible observations. (Note that the reasoner specification comprises instructions for digesting a wide variety of observations, but in practice it mostly only digests the actual observations.)

Like, you can't make an "oracle chess AI" that tells you at the beginning of the game what moves to play, because even chess is too chaotic for that game tree to be feasibly representable. You've gotta keep running your chess AI on each new observation, to have any hope of getting the fragment of the game tree that you consider down to a managable size.

Like, the outputs you can get out of an oracle AI are "no plan found", "memory and time exhausted", "here's a plan that involves running a reasoner in real-time" or "feed me observations in real-time and ask me only to generate a local and by-default-inscrutable action". In the first two cases, your oracle is about as useful as a rock; in the third, it's the realtime reasoner that you need to align; in the fourth, all whe word "oracle" is doing is mollifying you unduly, and it's this "oracle" that you need to align.

(NB: It's not obvious to me that cancer cures require routing through enough reality-chaos that plans fully named in advance need to route through reasoners; eg it's plausible that you can cure cancer with a stupid amount of high-speed trial and error. I know of no pivotal act, though, that looks so easy to me that nonrealtime-plans can avoid the above quadlemma.)

And it's not obvious to me whether this problem gets better or worse if you've tried to train the oracle to only output "reasonable seeming plans"

My point above addresses this somewhat, but I'm going to tack another point on for good measure. Suppose you build an oracle and take the "the plan involves a realtime reasoner" fork of the above quadlemma. How does that plan look? Does the oracle say "build the reasoner using this simple and cleanly-factored mind architecture, which is clearly optimizing for thus-and-such objectives?" If that's so easy, why aren't we building our minds that way? How did it solve these alignment challenges that we find so difficult, and why do you believe it solved them correctly? Also, AIs that understand clean mind-architectures seem deeper in the tech tree than AIs that can do some crazy stuff; why didn't the world end five years before reaching this hypothetical?

Like, specifying a working mind is hard. (Effable, transparent, and cleanly-factored minds are hander still, apparently.) You probably aren't going to get your first sufficiently-good-reasoner from "project oracle" that's training a non-interactive system to generate plans so hard that it invents its own mind architectures and describes their deployment, you're going to get it from some much more active system that is itself a capable mind before it knows how to design a capable mind, like (implausible detail for the purpose of concrete visualization) the "lifelong learner" that's been chewing through loads and loads of toy environments while it slowly acretes the deep structures of cognition.

Maybe once you have that, you can go to your oracle and be like "ok, you're now allowed to propose plans that involve deploying this here lifelong learner", but of course your lifelong learner doesn't have to be a particularly alignable architecture; its goals don't have to be easily identifiable and cleanly separable from the rest of its mind.

Which is mostly just providing more implausible detail that makes the "if your oracle emits plans that involve reasoners, then it's the reasoners you need to align" point more concrete. But... well, I'm also trying to gesture at why the "what if we train the oracle to only output reasonable plans?" thought seems, to me, to come at it from a wrong angle, in a manner that I still haven't managed to precisely articulate.

(I'm also hoping this conveys at least a little more of why the "just build an oracle that does alignment research" looks harder than doing the alignment research our own damn selves, and I'm frustrated by how people give me a pitying look when I suggest that humanity should be looking for more alignable paradigms, and then turn around and suggest that oracles can do that no-problem. But I digress.)

While writing this, I was reminded of an older (2017) conversation between Eliezer and Paul on FB. I reread it to see whether Paul seemed like he'd be making the set of mistakes this post is outlining.

It seems like Paul acknowledges the issues here, but his argument is that you can amplify humans without routing through "the hard parts" that are articulated in this post. i.e. it seems like you can use current ML to build something that helps a human effectively "think longer", and he thinks one can do this without routing through the dangerous-plan-searchspace. I don't know if there's much counterargument beyond "no, if you're building an ML system that helps you think longer about anything important, you already need to have solved the hard problem of searching through plan-space for actually helpful plans."

But, here were some comments from that thread I found helpful to reread.

Eliezer:

I have a new oversimplied straw way to misdescribe the disagreement between myself and Paul Christiano, and I'm interested to hear what Paul thinks of it:

Paul sees "learn this function" or "learn and draw from this probability distribution" as a kind of primitive op that modern machine learning is increasingly good at and can probably give us faithful answers to. He wants to figure out how to compose these primitives into something aligned.

Eliezer thinks that what is inside the black box inexorably kills you when the black box is large enough, like how humans are cognitive daemons of natural selection (the outer optimization process operating on the black box of genes accidentally constructed a (sapient) inner optimization process inside the black box) and this is chicken-and-egg unavoidable whenever the black box is powerful enough to do something like predict complicated human judgments, since in this case the outer optimization was automatically powerful enough to consider and select among multiple hypotheses the size of humans, and the inner process is automatically as powerful as human intelligence.

Paul thinks it may be possible to do something with this anyway given further expenditures of cleverness and computation, like taking a million such black boxes and competing them to produce the best predictions.

Eliezer expects an attempt like this to come with catastrophically exponential sample-complexity costs in theory, and to always fail in practice because you are trying to corral hostile superintelligences which means you've already lost. E.g. we can tell stories about how inner predictors could take over AIXI-tl by using bad off-policy predictions to manipulate AIXI-tl into a position where only that predictor (or LDT-symmetrized predictor class) can predict the answer to a one-way-hash problem it set up; and this isn't an isolated flaw, it faithfully reflects the fact that once you are trying to outwit a hostile superintelligence you're already screwed. Plus maybe it can do the equivalent of Rowhammering you, or using a bad but "testable" prediction just once that gets somebody to let it out of the box, etcetera. Only it doesn't do any of those things, it does something cleverer, etcetera. Eliezer thinks that once there's a hostile superintelligence running anywhere inside the system you are almost surely screwed *in practice*, which means Eliezer thinks Paul never gets to the point of completing one primitive op of the system before the system kills him.

Paul:

I think this is a mostly correct picture of the disagreement. I *would* agree with "what is inside the black box inexorably kills you when the black box is large enough," if we imagine humans interacting with an arbitrarily large black box. This is a key point of agreement.

I am optimistic anyway because before humanity tries to produce "aligned AI with IQ 200" we can produce "aligned AI with IQ 199." Indeed, if we train our systems with gradient descent then the intelligence will almost necessarily increase continuously. The goal is to maintain alignment as an inductive invariant, not to abruptly inject it into an extremely powerful system. So the gap between "smartest aligned intelligence we have access to" and "AI we are currently trying to train" is always quite small. This doesn't make the problem easy, but I do think it's a central feature of the problem that isn't well accounted for in your arguments for pessimism.

Buck:

My guess of Eliezer's reply is:

If we had an IQ 199 aligned AGI, that would indeed be super handy for building the IQ 200 one. But that seems quite unlikely.

Firstly, the black box learner required to build an AI that is aligned at all (eg the first step of capability amplification), even if that learned AI is very dumb, must itself be a really powerful learner, powerful enough that it is susceptible to scary internal unaligned optimizers.

Secondly, building an IQ 200 aligned agent via imitation of a sequence of progressively smarter aligned agents seems quite unlikely to be competitive, so without unlikely amounts of coordination someone will just directly build the IQ 200 agent.

Paul:

The first step of capability amplification is a subhuman AI similar in kind to the AI we have today; so if this is someone's objection then they ought to be able to stick their neck out today (e.g. by saying that we can't solve the alignment problem for systems we build today, or by saying that systems we can build today definitely won't be able to participate in amplification).

The AlphaGo Zero example really seems to take much of the wind out of the concerns about feasibility. It's the most impressive example of RL to date, and it was literally trained as a sequence of increasingly good go players learning to imitate one another.

I think the worst concerns are daemons (which are part of the unreasonably-innocuous-sounding "reliability" in my breakdown) and the impossibility alignment-preserving amplification. Setting up imitation / making informed oversight work also seems pretty hard, but I think it's less related to Eliezer's concerns.

Buck:

My Eliezer-model says that systems we can build today definitely won't be able to participate that helpfully in amplification. Amplification of human tasks seems like an extremely hard problem--our neural nets don't seem like they'd be up to it without adding in a bunch of features like attention, memory, hierarchical planning systems, and so on. The daemons come in once you start adding in all of that, and if you don't add in all that, your neural nets aren't powerful enough to help you.

Skipping ahead a bit, to the next Paul rejoinder that felt relevant:

Paul

I think the most important general type of amplification is "think longer." I think breaking a question down into pieces is a relatively easy case for amplification that can probably work with existing models. MCTS is a lot easier to get working than that.

> My Eliezer-model says that systems we can build today definitely won't be able to participate that helpfully in amplification.

To the extent that this is actually anyone's view, it would be great if they could be much clearer about what exactly they think can't be done.

I'm having some trouble phrasing this comment clearly, and I'm also not sure how relevant it is to the post except that the post inspired the thoughts, so bear with me...

It seems important to distinguish between several things that could vary with time, over the course of a plan or policy:

1. What information is known.
   • This is related to Nate's comment here [LW(p) · GW(p)]: it is much more computationally feasible to specify a plan/policy if it's allowed to contain terms that say "make an observation, then run this function on it to decide the next step," rather than writing out a lookup table pairing every sequence of observations to the next action.
2. What objective function is being maximized.
   • This is usually assumed (?) to be static in this kind of discussion, but in principle the objective could vary in response to future observations.
     
     In principle, this is equivalent to a static objective function with terms for "how it would respond" to each possible sequence of observations (ignoring subtleties about orders over world-states vs. world-histories).  But this has exactly the same structure as the previous point: it's more feasible to say "make an observation, then run this function to update the objective" than to unroll the same thing into a lookup table known entirely at the start.
3. Object-level features of the actions that are chosen.
   • Some of the properties under discussion, like Nate's "lasing," are about this stuff changing equivariantly / "in the right way" as other things change.

The recent discussions about consequentialism seem to be about the case where we have a task that takes a significant amount of real-world time, over which many observations (1) will be made with implications for subsequent decisions -- but over which the objective (2) is approximately unchanging.  This setup leads to various scary properties of what the policies actually do (3).

But, I don't understand the rationale for focusing on this case where the objective (2) doesn't change.  (In the sense of "doesn't change" specified above -- that we can specify it simply over long time horizons, rather than incurring an exp(T) cost for unrolling its updates on observations sequences.)

One reason to care about this case is a hope for oracle AI, since oracle AI is something that receives "questions" (objectives simple enough for us to feel we understand) and returns "answers" (plans that may take time).  This might produce a good argument that oracle AI is unsafe, but it doesn't apply to systems with changing objectives.

In the case of human intelligence, it seems to me that (2) evolves not too much more slowly than (1), and becomes importantly non-constant for longer-horizon cases of human planning.

If I set myself a brief and trivial goal like "make the kitchen cleaner over the next five minutes," I will spend those five minutes acting much like a clean-kitchen-at-all-costs optimizer, with all my subgoals pointing coherently in that direction ("wash this dish," "pick up the sponge").  If I set myself a longer-term goal like "get a new job," I may well find my preferences about the outcome have evolved substantially well before the task is complete.

This fact seems orthogonal to the fact that I am "good at search" relative to all known things that aren't humans.  Relative to all non-humans, I'm very good at finding policies that are high-EV for the targets I'm trying to hit.  But my targets evolve over time.

Indeed, I imagine this is why the complexity of human value doesn't create more of a problem for human action than it does.  I don't have a simply-specifiable constant objective with a term for "make people happy" (or whatever); I have an objective with an update rule that reacts to human feedback over time.  The update rule may have been optimized for something on an evolutionary timescale, but it's not obvious its application in an individual human can be modeled as optimizing anything.

(For a case that has the intelligence gap of humans/AGI, consider human treatment of animals.  I've heard this brought up as an analogy for misaligned AI, and it's an interesting one.  But the shape of the problem is not "humans are good at search, and have an objective which omits 'animal values,' or includes them in the wrong way."  Sometimes people just decide to become vegan for ethical reasons!  Sometimes whole cultures do.

This looks like a real case of individual values being updated, i.e. I don't think the right model of someone who goes vegan at age 31 is "this person is maximizing an objective which gives them points for eating animals, but only until age 31, and negative points thereafter.")

If we think of humans as a prototype case of an "inner optimizer," with evolution the outer optimizer, we have to note that the inner optimizer doesn't have a constant objective, even though the outer one does.  The inner optimizer is very powerful, has the lasing property, and all of that, but it gets applied to a changing objective, which seems to produce qualitatively different results in terms of corrigibility, Goodhart, etc.  The same thing could be true of an AGI, if it's the product of something like gradient descent rather than a system with an internal objective we explicitly wrote.  This is not strong evidence that it will be true, but it at least motivates asking the question.

(It seems noteworthy, here, that when people talk about the causes of human misery / "non-satisfaction of human values," they typically point to things like scarcity, coordination problems, and society-level optimization systems with constant objectives.  If we're good at search, and human value is complex, why aren't we constantly harming each other by executing incorrigibly on misaligned plans at an individual level?  Something fitting this description no doubt happens, but it causes less damage that a naive application of AI safety theory would lead one to expect.)

Curated. This is a great instance of someone increasing clarity on a load-bearing concept (at least in some models) through an earnest attempt to improve their own understanding.

I'm not sure I understand why it's important that the fraction of good plans is 1% vs .00000001%. If you have any method for distinguishing good from bad plans, you can chain it with an optimizer to find good plans even if they're rare. The main difficulty is generating enough bits--but in that light, the numbers I gave above are 7 vs 33 bits--not a clear qualitative difference. And in general I'd be kind of surprised if you could get up to say 50 bits but then ran into a fundamental obstacle in scaling up further.

Or rather, the part where alignment is hard is precisely when the thing I'm trying to accomplish is hard. Because then I need a powerful plan, and it's hard to specify a search for powerful plans that don't kill everyone.

This is a helpful sentence.

This post seems to be using a different meaning of "consequentialism" to what I am familiar with (that of moral philosophy). Subsequently, I'm struggling to follow the narrative from "consequentialism is convergently instrumental" onwards.

Can someone give me some pointers of how I should be interpreting the definition of consequentialism here? If it is just the moral philosophy definition, then I'm getting very confused as to why "judge morality of actions by their consequences" is a useful subgoal for agents to optimize against...

Thanks, I found this helpful!

Consequentialism is convergently instrumental. Consequentialism is a (relatively) simple, effective process for accomplishing goals, so things that efficiently optimize for goals tend to approximate it.

I think this is the most important premise. I don't have a solid justification for it yet, but I'm groping towards a non-solid justification at least in my agency sequence [? · GW]. I think John Wentworth's stuff on the good regulator theorem [LW · GW] is another line of attack that could turn into a solid justification. TurnTrout also has relevant work IIRC.

Is there a book out there on instrumental convergence? I have not come across this idea before and would like to learn more about it.

Here is a couple of "hard" things you can easily do with hypercompute, without causing dangerous consequentialism.

Given a list of atom coordinates, run a quantum accuracy simulation of those atoms. (Where the atoms don't happen to make a computer running a bad program).

Find the smallest arrangement of atoms that forms a valid Or gate by brute forcing over the above simulator. 

Brute forcing over large arrangements of atoms could find a design containing a computer containing an AI. But brute forcing over arrangements of 100 atoms should be fine, and can do a lot of interesting chemistry. Note that a psychoactive that makes humans care less about AI risk won't be preferentially selected. Its not simulating the simple molecule in the world, that would be dangerous. Its simulating a simple molecule in a vacuum. (Or a standard temp and pressure 80% N2 + 20% O2 atmosphere, or some other simple hardcoded test setup.) 

I think there's some confusion going on with "consequentialism" here, and that's at least a part of what's at play with "why isn't everyone seeing the consequentialism all the time".

One question I asked myself reading this is "does the author distinguish 'consequentialism' with 'thinking and predicting' in this piece?" and I think it's uncertain and leaning towards 'no'.

So, how do other people use 'consequentialism'?

It's sometimes put forward as a moral tradition/ethical theory, as an alternative to both deontology and virtue ethics.  I forget which philosopher decided this was the trifecta but these are often compared and contrasted to each other.  In particular, the version used here seems to not fit well with this article.

Another might be that consequentialism is an ethical theory that requires prediction (whereas others do not) -- I think this is an important feature of consequentialism, but it seems like 'the set of all ethical theories which have prediction as a first class component' is bigger than just consequentialism.  I do think that ethical theories that require prediction as a first class component are important for AI alignment, specifically intent alignment (less clear if useful for non-intent-alignment alignment research).

A different angle to this would be "do common criticisms of consequentialism apply to the concept being used here".  Consequentialism has had a ton of philosophical debate over the last century (probably more?) and according to me there's a bunch of valid criticisms.^[1]

Finally I feel like this is missing a huge step in the recent history of ethical theories, which is the introduction of Moral Uncertainty.  I think Moral Uncertainty is a huge step, but the miss (in this article) is a 'near miss'.  I think a similar argument could have been made for AI researchers / Alignment researchers, using the framing of Moral Uncertainty, should be updating on net in the direction of consequentialism being useful/relevant for modeling systems (and possibly useful for designing alignment tech).

1. ^{^}

   I'm not certain that the criticisms will hold, but I think that proponents of consequentialism have insufficiently engaged with the criticisms; my net current take is uncertain but leaning in the consequentialists favor.  (See also: Moral Uncertainty)

I agree that non-agentic AI is a fools errand when it comes to alignment, but there's one point where I sort of want to defend it as not being quite as bad as this post suggests:

Me: Okay, so partly you're pointing out that hardness of the problem isn't just about getting the AI to do what I want, it's that doing what I want is actually just really hard. Or rather, the part where alignment is hard is precisely when the thing I'm trying to accomplish is hard. Because then I need a powerful plan, and it's hard to specify a search for powerful plans that don't kill everyone.

This depends heavily on the geometry/measure of the space you search over, which depends heavily on the "interface" you have for interacting with the world.

Consider the case of getting rid of cancer. If your interface is a chemical mixture that you inject into the cancer patient(s), it would technically be a valid solution for that chemical mixture to contain nanobots that seize broad power and uses this for extensive research that eventually generates a cure for cancer. But this is a complex solution, which requires enormously many coordinated pieces. It probably occupies a much smaller part of the search space than genuine solutions, and even unaligned solutions would tend to be stuff like "this kills the cancer but it also kills the patient". Further, just from a computational point of view, the power-grabbing solution would be much easier to "catch" because it suddenly requires modelling huge parts of society, which might be orders of magnitude more expensive than anything staying within a person.

On the other hand, if your interface to the world isn't a chemical mixture that gets injected into patients, but is instead a computer program that gets uploaded to some server, then a great deal of power seeking is necessary to even get an arrangement where you are able to medically affect the cancer patients. The jump from a direct medical solution to even further power seeking becomes much smaller.

This isn't just an informal argument; you can take a look at e.g. Alex Turner's proofs, and see that they are deeply dependent on the measure on the goals (which is formalized by the symmetry group chosen).

That said, this doesn't necessarily solve things. There are some tasks that can be very satisfactorily solved with a narrow window like this, so that power-seeking isn't a problem. But there is enormous economic value in more generalized interaction with the world, so there will inevitably be pressure to building genuine agents.

I wonder if the confusion isn't about implications of consequentialism, but about the implications of independent agents.  Related to the (often mentioned, but never really addressed) problem that humans don't have a CEV, and we have competition built-in to our (inconsistent) utility functions.

I have yet to see a model of multiple agents WRT "alignment".  The ONLY reason that power/resources/self-preservation is instrumental is if there are unaligned agents in competition.  If multiple agents agree on the best outcomes and the best way to achieve them, then it doesn't matter which agent does what, or even which agent(s) exist.  

Fully-aligned agents are really just multiple processing cores of one agent.  

It's when we talk about partial-alignment that we go off the rails.  In this case, we should address competition and tradeoffs as actual things the agent(s) have to consider.

Let's assume for a moment that consequentialism in Eliezer's sense is the most pervasive thing in the problem space (this is not a claim anyone has made as far as I can tell). What does leaning into consequentialism super hard look like in terms of approaches? The only line of attack l know of which seems to meet the description is the convergent power-seeking sequence [? · GW].

Not only that, most of the plans route through "acquire resources in a way that is unfriendly to human values." Because in the space of all possible plans, while consequentialism doesn't take that many bits to specify, human values are highly complexand take a lot of bits to specify.

1) It's easier to build a moon base with money. And*, it's easier to steal money than earn it.

*This is a hypothetical

2) Even replacing that plan with a one that 'human values' says works, is tricky. What is an acceptable way to earn money?

Just listing the plans.

One does not enumerate all of possibility.

Okay, but if I imagine a researcher who is thoughtful but a bit too optimistic, what they might counterargue with is: "Sure, but I'll just inspect the plans for whether they're unfriendly, and not do those plans."

And here you swap out 'a plan' for 'plans'.

Me: Okay, so partly you're pointing out that hardness of the problem isn't just about getting the AI to do what I want, it's that doing what I want is actually just really hard. Or rather, the part where alignment is hard is precisely when the thing I'm trying to accomplish is hard. Because then I need a powerful plan, and it's hard to specify a search for powerful plans that don't kill everyone.

The fact that this is being used as a metaphor, disconnects it from the problem.

Suppose, tomorrow, a 'cure for cancer' was created. And the solution was surprisingly simple.

It seems clear that say, 'beating you at chess' isn't that hard to plan. Why would 'cure cancer' be so very, very hard?

It seems like the tricky bit about a plan is that...maybe a plan wouldn't work?

You might have to do experiments, and learn from them, and come up with new ideas...you are not sailing somewhere that is on a map, or doing something that has been done before.

Rather than saying that most likely-to-work plans for curing cancer route through consequentialism, I think it would be more precise to say that most simple likely-to-work plans route through consequentialism.

For every plan that can be summarized as "build a powerful consequentialist and then delegate the problem to it", it seems like there should be a corresponding (perhaps very complicated) plan that can be summarized as "directly execute the plan that that consequentialist would have used if you had built it."

The size of that complexity penalty varies depending on the nature of the problem.  There's maybe a useful sense in which the "hard problems" are exactly the ones where the complexity penalty for avoiding the consequentialist is large.