LessWrong 2.0 Reader

Here is the promised comment on what kind of "commitment" I want to make given all the responses. 

I agree with Buck that no one should make very direct commitment about this sort of thing, as there might be blackmail related scary things lurking in the shadows when one does acausal trade. I think we will probably figure out how to handle that, but we shouldn't make any strong promises of specific actions until we figure that out. 

However, the promise I'm intending to keep is that if humanity wins and I'm alive to see it, I will remember how scary it was to live in the world before we knew we'd win, and if there is no currently unknown reason not to do something like the simulation scheme described in the post, and the only cost is not utilizing some small fraction of the conquered Universe, I will advocate for doing it and spend a significant fraction of my resources on it if necessary. I think it would be good for sending future AIs a signal about the seriousness of this, if some other people who are worried about everyone dying from AI, but are not strongly longtermist absolute utilitarians, also left a comment making a similar vague promise that if they ever get to control full star systems, they will contribute to this plan if no currently unknown serious counterargument comes up. Or at least leave an Agreed sing on this sentence.

Other updates from the comments: If I wrote the post now, I would more strongly emphasize a framing that this plan is basically about entering into an acausal insurance contract with other young civilizations who don't yet know what their chances are for aligning AI, neither how big their Universe is compared to the usual Universe-size in the great prior over possible worlds. Then the civilizations who align their AIs and find that they live in a relatively big Universe, bail out everyone else in the insurance contract.

But I think that, while this acausal insurance framing might be more satisfying to the people who are already thinking a lot about acausal trade, in practice the way we implement this "insurance" will likely be very similar to the scheme described in the post. So I maintain that for most people it's better not to think in terms of acausal trade, but just think about the simulation proposal described in the post.

Exactly. You can't make the kid read something, but if he doesn't know the book exists he's not going to read it for sure.

Conversational signposts are just one technique to improve social interactions. For more advanced techniques, I would recommend checking out:

• How to Talk to Anyone: 92 Little Tricks for Big Success in Relationships by Leil Lowndes
• How to Win Friends & Influence People by Dale Carnegie

I agree it seems unlikely that we'll see coordination on slowing down before one actor or coalition has a substantial enough lead over other actors that it can enforce such a slowdown unilaterally, but I think it's reasonably likely that such a lead will arise before things get really insane.

A few different stories under which one might go from aligned "genius in a datacenter" level AI at time t to outcomes merely at the level of weirdness in this essay at t + 5-10y:

• The techniques that work to align "genius in a datacenter" level AI don't scale to wildly superhuman intelligence (eg because they lose some value fidelity from human-generated oversight signals that's tolerable at one remove but very risky at ten). The alignment problem for serious ASI is quite hard to solve at the mildly superintelligent level, and it genuinely takes a while to work out enough that we can scale up (since the existing AIs, being aligned, won't design unaligned successors).
• If people ask their only-somewhat-superhuman AI what to do next, the AIs say "A bunch of the decisions from this point on hinge on pretty subtle philosophical questions, and frankly it doesn't seem like you guys have figured all this out super well, have you heard of this thing called a long reflection?" That's what I'd say if I were a million copies of me in a datacenter advising a 2024-era US government on what to do about Dyson swarms!
• A leading actor uses their AI to ensure continued strategic dominance and prevent competing AI projects from posing a meaningful threat. Having done so, they just... don't really want crazy things to happen really fast, because the actor in question is mostly composed of random politicians or whatever. (I'm personally sympathetic to astronomical waste arguments, but it's not clear to me that people likely to end up with the levers of power here are.)
• The serial iteration times and experimentation loops are just kinda slow and annoying, and mildly-superhuman AI isn't enough to circumvent experimentation time bottlenecks (some of which end up being relatively slow), and there are stupid zoning restrictions on the land you want to use for datacenters, and some regulation adds lots of mandatory human overhead to some critical iteration loop, etc.
  • This isn't a claim that maximal-intelligence-per-cubic-meter ASI initialized in one datacenter would face long delays in making efficient use of its lightcone, just that it might be tough for a not-that-much-better-than-human AGI that's aligned and trying to respect existing regulations and so on to scale itself all that rapidly.
• Among the tech unlocked in relatively early-stage AGI is better coordination, and that helps Earth get out of unsavory race dynamics and decide to slow down.
• The alignment tax at the superhuman level is pretty steep, and doing self-improvement while preserving alignment goes much slower than unrestricted self-improvement would; since at this point we have many fewer ongoing moral catastrophes (eg everyone who wants to be cryopreserved is, we've transitioned to excellent cheap lab-grown meat), there's little cost to proceeding very cautiously.
  • This is sort of a continuous version of the first bullet point with a finite rather than infinite alignment tax.

All that said, upon reflection I think I was probably lowballing the odds of crazy stuff on the 10y timescale, and I'd go to more like 50-60% that we're seeing mind uploads and Kardashev level 1.5-2 civilizations etc. a decade out from the first powerful AIs.

I do think it's fair to call out the essay for not highlighting the ways in which it might be lowballing things or rolling in an assumption of deliberate slowdown; I'd rather it have given more of a nod to these considerations and made the conditions of its prediction clearer.

Resonating with you here!  Yes, I think autonomous corporations (and other organisations) would result in society-wide extraction, destabilisation and totalitarianism.

I'm not that concerned with lobbyists ruining the deliberative proceedings. I think you're underestimating normal people a bit. They have state power to shut down annoying and undesired feedback if they wish. I also think the assembly will tend to trust their own advisors, whom they hired themselves, over outside self-proclaimed expert lobbyists. 

My bigger concern is with corruption and bribery. Because we're dealing with very normal people, we also ought to expect normal criminal behavior. We ought to expect assembly members getting arrested from time to time, and doing all the normal things we expect from 500 random people. 

I think bribery is a sufficiently high concern that a police force should constantly operate to perform sting operations and monitor elicit behavior from assembly members. IMO, this should already be happening with elected officials too. 

Another big concern is whether a purely lottocratic assembly would self-regulate its own corruption. It has some interest to, in that the lottocrats help their future selves, after their term has ended, by creating future rules that would regulate corruption. Terril Bouricus attempts to create a system where layers on layers of assemblies check and re-check the work of other assemblies to mitigate corruption concerns. 

I can't easily conclude whether election or sortition would be better at corruption mitigation. With elections, opposition parties have an incentive to investigate their enemies to root out corruption. HOWEVER, the same opposition parties have an incentive to lie about the results of investigations, leading to an environment of fake news, where voters cannot distinguish between a political attack and actual corruption. In the American context, bribery is about already legalized with campaign donations. 

Sortition could possibly lead to a ridiculous scenario:

Imagine the public is outraged at the insane level of corruption of the sortition-assembly. However as a new assembly is formed by lottery, these anti-corruption sentiments are suddenly rotated into office. The members of the public hate corruption, as does this new assembly! The question is, would the members of the assembly be able to do the Machiavellian about-face and suddenly embrace corruption? I have a hard time believing they would, though I have doubts. In my opinion, normal people being utterly normal, would rather do the easy thing and yes, go ahead and regulate the corruption while enjoying their government salary. Getting to serve in office is already a win-win, why not win and also be declared heroes? Alternatively they can "Go Breaking Bad", embrace corruption and pilfer the state coffers. They can win big (for now) but will become despised. What do you think normal people would do? High risk high reward, or low risk medium reward? I don't think going "Breaking Bad" is the best of ideas. Elected politicians use their offices to protect themselves from legal challenge. Obvious example, Donald Trump using the presidency to overcome his legal problems. He's obviously not the first politician to cling to office in order to protect themselves. Lottocrats can't do the same. Lottocrats soon lose their powers and become vulnerable. 

I mostly agree on current margins — the more I mistrust a lab, the more I like transparency.

I observe that unilateral transparency is unnecessary on the view-from-the-inside if you know you're a reliably responsible lab. And some forms of transparency are costly. So the more responsible a lab seems, the more sympathetic we should be to it saying "we thought really hard and decided more unilateral transparency wouldn't be optimific." (For some forms of transparency.)

let's not kill the messenger, lest we run out of messengers.

Unfortunately we're a fair way into this process, not because of downvotes^[1] but rather because the comments are often dominated by uncharitable interpretations that I can't productively engage with.^[2]. I've had researchers and policy people tell me that reading the discussion convinced them that engaging when their work was discussed on LessWrong wasn't worth the trouble.

I'm still here, sad that I can't recommend it to many others, and wondering whether I'll regret this comment too.

Thank you for the response <3

"...I would like to prove to the Court Philosopher that I'm right and he's wrong."

This part of the story tickles me more, reading it a second time.

I like to write stories that mean different things to different people ...this story isn't a puzzle at all. It is a joke about D&D-style alignment systems.

And it kinda resonates with this bit. In both cases there's a certain flexibility. The flexibility itself is unexpected, but reasonable safe... which is often a formula for comedy? It is funny to see the flexibility in Phil as he "goes social", and also funny to see it in you as you "go authorial" :-)

How could one control AI without access to the hardware/software? What would stop one with access to the hardware/software from controlling AI?

One would gain control by renting access to the model, i.e., the same way you can control what an instance of ChatGPT currently does. Here, I am referring to practical control over the actual behavior of the AI, when determining what the AI does, such as what tasks it performs, how it is fine-tuned, or what inputs are fed into the model.

This is not too dissimilar from the high level of practical control one can exercise over, for example, an AWS server that they rent. While Amazon may host these servers, and thereby have the final say over what happens to the computer in the case of a conflict, the company is nonetheless inherently dependent on customer revenue, implying that they cannot feasibly use all their servers privately for their own internal purposes. As a consequence of this practical constraint, Amazon rents these servers out to the public, and they do not substantially limit user control over AWS servers, providing for substantial discretion to end-users over what software is ultimately implemented.

In the future, these controls could also be determined by contracts and law, analogously to how one has control over their own bank account, despite the bank providing the service and hosting one's account. Then, even in the case of a conflict, the entity that merely hosts an AI may not have practical control over what happens, as they may have legal obligations to their customers that they cannot breach without incurring enormous costs to themselves. The AIs themselves may resist such a breach as well.

In practice, I agree these distinctions may be hard to recognize. There may be a case in which we thought that control over AI was decentralized, but in fact, power over the AIs was more concentrated or unified than we believed, as a consequence of centralization over the development or the provision of AI services. Indeed, perhaps real control was always in the hands of the government all along, as they could always choose to pass a law to nationalize AI, and take control away from the companies.

Nonetheless, these cases seem adequately described as a mistake in our perception of who was "really in control" rather than an error in the framework I provided, which was mostly an attempt to offer careful distinctions, rather than to predict how the future will go.

If one actor—such as OpenAI—can feasibly get away with seizing practical control over all the AIs they host without incurring high costs to the continuity of their business through loss of customers, then this indeed may surprise someone who assumed that OpenAI was operating under different constraints. However, this scenario still fits nicely within the framework as I've provided, as it merely describes a case in which one was mistaken about the true degree of concentration along one axis, rather than one of my concepts intrinsically fitting reality poorly.