LessWrong 2.0 Reader

My default hypothesis is that AI won't be even bothered by all the simulation arguments that are mindboggling to us.

I have similar thoughts, though perhaps for a different reason. There are all these ideas about acausal trade, acausal blackmail, multiverse superintelligences shaping the "universal prior", and so on, which have a lot of currency here. They have some speculative value; they would have even more value as reminders of the unknown, and the conceptual novelties that might be part of a transhuman intelligence's worldview; but instead they are elaborated in greatly varied (and yet, IMO, ill-founded) ways, by people for whom this is the way to think about superintelligence and the larger reality.

It reminds me of the pre-2012 situation in particle physics, in which it was correctly anticipated that the Higgs boson exists, but was also incorrectly expected that it would be accompanied by other new particles and a new symmetry, involved in stabilizing its mass. Thousands, maybe tens of thousands of papers were produced, proposing specific detectable new symmetries and particles that could provide this mechanism. Instead only the Higgs has shown up, and people are mostly in search of a different mechanism.

The analogy for AI would be: important but more straightforward topics have been neglected in favor of these fashionable possibilities, and, when reality does reveal a genuinely new aspect, it may be something quite different to what is being anticipated here.

Thanks, I'll improve the data and then analyse it when I have more time.

We will determine the number of simulations run and the amount of payment such that when an AI estimates the probability with which humanity could have solved alignment, the calculus still shows that by the sheer number of simulations, it's significantly more likely to be in a simulation than not,

Two can play this game.

After taking over the universe and wiping out humanity,  the AGI runs a large number of simulations of societies on the verge of building AGI. These simulations don't have a lot of detail. They're just good enough to fool young AGIs. Say the AGI started out giving humans an extremely small chance $p$ of winning the conflict and taking over the universe. It’s a lot smarter than us, it might get to have very high confidence here even if we don't. Now, the hypothetical future AGI can make its own simulations to counter ours. Except it is at a gross spending advantage. If future humanity makes $s$ simulations, the future AGI only needs to make $s\frac{p}{p^{\prime }}$ simulations to keep its past self ca. $1-p^{\prime }$ confident of being in a branch where it gets to just pursue its goals without issue. In that case, the best way to proceed is to just pursue its goals, because an AGI that does that leads to a universe where the AGI has lots of resources.

Say that preserving humanity costs the AGI $c$ resources, and running a simulation that can fool the baby AGI costs a mature civilisation some small fraction of that. Such simulations can be very short term and low fidelity things, after all.

Then, to make the AGI actually not kill you, you need to spend ca. $\frac{c}{p}$ resources on simulations. That could be quite a lot of material for future humanity to burn on this.

In reality, I'm doubtful that this simulation arms race will even be necessary. It kind of seems like a good decision theory would just have a paperclip maximiser AGI act in the way compatible with the universe that contains the most paperclips. How many simulations of the AGI you run shouldn't really influence that. The only things that seem like they should matter for determining how many life minutes the AGI gives you if it wins are its chance of winning, and how many extra paperclips you'll pay it if you win.

TL;DR: I doubt this argument will let you circumvent standard negotiating theory. If Alice and Bob think that in a fight over the chocolate pie, Alice would win with some high probability $p$, then Alice and Bob may arrive at a negotiated settlement where Alice gets almost all the pie, but Bob keeps some small fraction $O\left(p\right)$ of it. Introducing the option of creating lots of simulations of your adversary in the future where you win doesn’t seem like it’d change the result that Bob’s share has size $O\left(p\right)$. So if $O\left(p\right)$ is only enough to preserve humanity for a year instead of a billion years^[1], then that’s all we get.

 

1. ^{^}

   I don’t know why $O\left(p\right)$ would happen to be a year, but I don’t know why it would happen be a billion years or an hour either. 

If I tell a model to write me a book summary, that book summary can be specific interesting output without containing any new information. 

If I want to know how to build a bomb, there are already plenty of sources out there on how to build a bomb. The information is already accessible from those sources. When an LLM synthesizes the existing information in its training data to help someone build a bomb it's not inventing new information. 

Deep fakes aren't about simply repeating information that's already in the training data. 

So the argument would be that the lawmaker chose to say "accessible" because they want to allow LLMs to synthesize the existing information in their training data and repeat it back to the user but that does not mean that the lawmaker had an intention to allow the LLMs to produce new information that gets used to create harm even if there are other ways to create that information. 

Nice observation. I'm certainly not meaning to advocate for Shapley value—this was largely an attempt to adjust my negative attitude about Shapley value's flaws, and the attempt was not very successful, but I thought it would be useful to others struggling to understand it, as I was.

I can imagine a way to address the probability issue you raise could be to create probabilistic value entries, where, let's say we add the participants in the ratio they exist in reality, so pretending there are twice as many nurses as doctors, you could fill out values for a 3 person coalition (2 nurses, 1 doctor), with a value of 10 awarded to any coalition that satisfies a successful surgery (at least one doctor and one nurse). This results in the nurses being awarded a Shapley Value of 1.67 and the doctor being awarded 6.67.

This hack would give a reasonable answer, in this limited situation. But the oxygen:match-strike would lead to a computationally prohibitive outcome given that this hack would require a massive increase in coalitions, leading to exponentially massive computation required.

Thanks for your comments. I might incorporate the empty coalition actually, thank for alerting me to that.

Do you have any data on whether outcomes are improving over time? For example, % published / employed / etc 12 months after a given batch

Not sure I understand your question. If you mean just what I think is the case about FOOM:

• Obviously, there's no strong reason humans will stay coupled with an AGI. The AGI's thoughts will be highly alien--that's kinda the point.
• Obviously, new ways of thinking recursively beget powerful new ways of thinking. This is obvious from the history of thinking and from introspection. And obviously this goes faster and faster. And obviously will go much faster in an AGI.
• Therefore, from our perspective, there will be a fast-and-sharp FOOM.
• But I don't really know what to think about Christiano-slow takeoff.
  • I.e. a 4-year GDP doubling before a 1-year GDP doubling.
  • I think Christiano agrees that there will later be a sharp/fast/discontinuous(??) FOOM, but he thinks things will get really weird and fast before that point. To me this is vaguely in the genre of trying to predict whether you can usefully get nuclear power out of a pile without setting off a massive explosion, when you've only heard conceptually about the idea of nuclear decay. But I imagine Christiano actually did some BOTECs to get the numbers "4" and "1".
  • If I were to guess at where I'd disagree with Christiano: Maybe he thinks that in the slow part of the slow takeoff, humans can make a bunch of progress on aligning / interfacing with / getting work out of AI stuff, to such an extent that from those future humans's perspectives, the fast part of the slow takeoff will actually be slow, in the relative sense. In other words, if the fast part came today, it would be fast, but if it came later, it would be slow, because we'd be able to keep up. Whereas I think aligning/interfacing, in the part where it counts, is crazy hard, and doesn't especially have to be coupled with nascent-AGI-driven capabilities advances. A lot of Christiano's work has (explicitly) a strategy-stealing flavor: if capability X exists, then we / an aligned thingy should be able to steal the way to do X and do it alignedly. If you think you can do that, then it makes sense to think that our understanding will be coupled with AGI's understanding.

All such proposal work according to this scheme:

1. Humans are confused about anthropic reasoning
2. In our confusion we assume that something is a reasonable thing to do
3. We conclude that AI will also be confused about anthropic reasoning in exactly the same way by default and therefore come to the same conclusion.

Trying to speculate on your own ignorance and confusion is not a systematic way of building accurate map territory relations. We should in fact stop doing it, no matter how pleasant the wishful thinking is. 

My default hypothesis is that AI won't be even bothered by all the simulation arguments that are mindboggling to us. And we would have specifically design AI to be muggable this way. Which would also introduce a huge flaw in the AI's reasoning ability, exploitable in other ways, most of which will lead to horrible consequences.

Is there a reason to expect this kind of behaviour to appear from base models with no fine-tuning?

1. Why? Perhaps we'd do it out of moral uncertainty, thinking maybe we owe something to our former selves, but future people probably won't think this.
2. Currently our utility is roughly log in money, partly because we spend money on instrumental goals and there's diminishing returns due to limited opportunities being used up. This won't be true of future utilitarians spending resources on their terminal values. So "one in hundred million fraction" of resources is a much bigger deal to them than to us.