LessWrong 2.0 Reader

Okay, I read to the end and I'm a little skeptical that you properly understand the incompleteness theorem. Are you aware, for instance, that the incompleteness theorem prohibits us from even proving all first-order statement the natural numbers using any consistent mathematical framework (regardless of how powerful it is)? And that it was later shown that no consistent mathematical framework can even prove the existence/non-existence of solutions to diophantine equations? The reason I ask is that you brought up the necessity of abstract categories and I'm not really sure what you meant by that. It also seemed that you might be unaware that no mathematical framework can resolve the halting problem for any Turing complete system (this is essentially a tautology). Is this correct?

Hey i'm from YouTube (Art of the Problem) and working on a new series on Economics. Thinking about exploring it via evolutionary lens (making decisions about value)...if interested please reach out

To be clear, I think the exact scheme in A proposal for humanity in the future [LW · GW] probably doesn't work as described because the exact level of payment is wrong and more minimally we'll probably be able to make a much better approach in the future.

This seemed important to explicitly call out (and it wasn't called out explicitly in the post), though I do think it is reasonable to outline a concrete baseline proposal for how this can work.

In particular, the proposal randomly picks 10 planets per simulation. I think the exact right amount of payment will depend on how many sims/predictions you run and will heavily depend on some of the caveats under Ways this hope could fail [LW · GW]. I think you probably get decent results if the total level of payment is around 1/10 million, with returns to higher aggregate payment etc.

As far as better approaches, I expect that you'll be doing a bunch of stuff more efficient than sims and this will be part of a more general acausal trade operation among other changes.

I was expecting an argument like "most of the probability measure for a given program, is found in certain embeddings of that program in larger programs". Has anyone bothered to make a quantitative argument, a theorem, or a rigorous conjecture which encapsulates this claim?

Yeah, I currently disagree on the competent aliens bailing us out, but I haven't thought super hard about it. It does seem good to think about (though not top priority).

I agree that in as much as you have an AI that somehow has gotten in a position to guarantee victory, then leaving humanity alive might not be that costly (though still too costly to make it worth it IMO), but a lot of the costs come from leaving humanity alive threatening your victory. I.e. not terraforming earth to colonize the universe is one more year for another hostile AI to be built, or for an asteroid to destroy you, or for something else to disempower you.

Disagree on the critique of Nate's posts. The two posts seem relatively orthogonal to me (and I generally think it's good to have debunkings of bad arguments, even if there are better arguments for a position, and in this particular case due to the multiplier nature of this kind of consideration debunking the bad arguments is indeed qualitatively more important than engaging with the arguments in this post, because the arguments in this post do indeed not end up changing your actions, whereas the arguments Nate argued against were trying to change what people do right now).

If you are doing evals for CBRN capabilities, you are very much in the zone of terrorists killing billions of innocent people. Indeed, that's practically the definition. There's no citation, it's just my personal experience while doing evals that are much to spicy to publish.

Of course, if you're only doing evals for relatively tame proxy skills (e.g. WMDP) then probably you get less of this effect. I don't have a quantification of the rates or specific datasets, just anecdata.

acausal trade framework rest on the assumption that we are in a (quantum or Tegmark) multiverse

I think the argument should also go through without simulations and without the multiverse so long as you are a UDT-ish agent with a reasonable prior.

I guess I was not clear enough in defining what I was talking about. While it is possible to stretch the definition of "nuclear world war" to include WW2 and Little Boy and Fat Man were certainly strategic weapons at their time, this is not at all what I meant. I was talking about modern strategic weapons, i.e. MIRVed ICBMs shot from hardened silos or ballistic missile submarines, used by a modern nuclear superpower to defeat a near peer opponent. I.e. the scenario Petrov faced.

If e.g. the US in Petrov's time had managed to pull off a perfect nuclear first strike (a pretty bold assumption), destroying the whole USSR's and Chinese nuclear triad without any counter strike at all, the economic (supply chain disruption, Europe and Middle East overrun with refugees...) and political repercussions (everyone thinks the US is run by complete psychopaths) alone would have been enough to ensure in expectation a precipitous drop in quality of life for nearly all US citizens, including generals and politicans. This is true even if the whole nuclear winter idea is complete bunk.

Re "It's costly for AI to leave humans alive", I think the best thing written on this is Paul's comment here [LW(p) · GW(p)], the most relevant part of which is:

• First, I think you should talk quantitatively. How many more resources can an AI get by killing humans? I'd guess the answer is something like 1 in a billion to 1 in a trillion.
  • If you develop as fast as possible you will wreck the human habitat and incidentally kill a lot of people. It's pretty complicated to figure out exactly how much "keep earth livable enough for human survival" will slow you down, since it depends a lot on the dynamics of the singularity. I would guess more like a month than a year, which results in a miniscule reduction in available resources. I think that (IMO implausible) MIRI-style views would suggest more like hours or days than months.
    • Incidentally, I think "byproducts of rapid industrialization trash Earth's climate" is both much more important than the dyson sphere as well as much more intuitively plausible.
  • You can get energy from harvesting the biosphere, and you can use it to develop slightly faster. This is a rounding error compared to the last factor though.
  • Killing most humans might be the easiest way to prevail in conflict. I think this is especially plausible for weak AI. For very powerful AI, it also seems like a rounding error. Even a moderately advanced civilization could spend much less than 1 in a trillion of its resources to have much less than 1 in a billion chance of being seriously inconvenienced by humanity.