LessWrong 2.0 Reader

Yes, that was a pretty terrible take. Markets quite clearly do not price externalities well, and never have done. So long as any given investor rates their specific investment as being unlikely to tip the balance into doom, they get the upside of directly financially benefiting from major economic growth due to AI, and essentially the same downside risk as if they didn't invest. Arguments like "short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends" are obviously not even trying to seriously reflect the widespread investment decisions that affect real markets.

I think he's making a bet that, if people could literally get tasty steak without killing cows for the same price (or cheaper), most of them would not pay for factory farmed cow.

alas not easily

Here is the promised comment on what kind of "commitment" I want to make given all the responses. 

I agree with Buck that no one should make very direct commitment about this sort of thing, as there might be blackmail related scary things lurking in the shadows when one does acausal trade. I think we will probably figure out how to handle that, but we shouldn't make any strong promises of specific actions until we figure that out. 

However, the promise I'm intending to keep is that if humanity wins and I'm alive to see it, I will remember how scary it was to live in the world before we knew we'd win, and if there is no currently unknown reason not to do something like the simulation scheme described in the post, and the only cost is not utilizing some small fraction of the conquered Universe, I will advocate for doing it and spend a significant fraction of my resources on it if necessary. I think it would be good for sending future AIs a signal about the seriousness of this, if some other people who are worried about everyone dying from AI, but are not strongly longtermist absolute utilitarians, also left a comment making a similar vague promise that if they ever get to control full star systems, they will contribute to this plan if no currently unknown serious counterargument comes up. Or at least leave an Agreed sing on this sentence.

Other updates from the comments: If I wrote the post now, I would more strongly emphasize a framing that this plan is basically about entering into an acausal insurance contract with other young civilizations who don't yet know what their chances are for aligning AI, neither how big their Universe is compared to the usual Universe-size in the great prior over possible worlds. Then the civilizations who align their AIs and find that they live in a relatively big Universe, bail out everyone else in the insurance contract.

But I think that, while this acausal insurance framing might be more satisfying to the people who are already thinking a lot about acausal trade, in practice the way we implement this "insurance" will likely be very similar to the scheme described in the post. So I maintain that for most people it's better not to think in terms of acausal trade, but just think about the simulation proposal described in the post.

Exactly. You can't make the kid read something, but if he doesn't know the book exists he's not going to read it for sure.

Conversational signposts are just one technique to improve social interactions. For more advanced techniques, I would recommend checking out:

• How to Talk to Anyone: 92 Little Tricks for Big Success in Relationships by Leil Lowndes
• How to Win Friends & Influence People by Dale Carnegie

I agree it seems unlikely that we'll see coordination on slowing down before one actor or coalition has a substantial enough lead over other actors that it can enforce such a slowdown unilaterally, but I think it's reasonably likely that such a lead will arise before things get really insane.

A few different stories under which one might go from aligned "genius in a datacenter" level AI at time t to outcomes merely at the level of weirdness in this essay at t + 5-10y:

• The techniques that work to align "genius in a datacenter" level AI don't scale to wildly superhuman intelligence (eg because they lose some value fidelity from human-generated oversight signals that's tolerable at one remove but very risky at ten). The alignment problem for serious ASI is quite hard to solve at the mildly superintelligent level, and it genuinely takes a while to work out enough that we can scale up (since the existing AIs, being aligned, won't design unaligned successors).
• If people ask their only-somewhat-superhuman AI what to do next, the AIs say "A bunch of the decisions from this point on hinge on pretty subtle philosophical questions, and frankly it doesn't seem like you guys have figured all this out super well, have you heard of this thing called a long reflection?" That's what I'd say if I were a million copies of me in a datacenter advising a 2024-era US government on what to do about Dyson swarms!
• A leading actor uses their AI to ensure continued strategic dominance and prevent competing AI projects from posing a meaningful threat. Having done so, they just... don't really want crazy things to happen really fast, because the actor in question is mostly composed of random politicians or whatever. (I'm personally sympathetic to astronomical waste arguments, but it's not clear to me that people likely to end up with the levers of power here are.)
• The serial iteration times and experimentation loops are just kinda slow and annoying, and mildly-superhuman AI isn't enough to circumvent experimentation time bottlenecks (some of which end up being relatively slow), and there are stupid zoning restrictions on the land you want to use for datacenters, and some regulation adds lots of mandatory human overhead to some critical iteration loop, etc.
  • This isn't a claim that maximal-intelligence-per-cubic-meter ASI initialized in one datacenter would face long delays in making efficient use of its lightcone, just that it might be tough for a not-that-much-better-than-human AGI that's aligned and trying to respect existing regulations and so on to scale itself all that rapidly.
• Among the tech unlocked in relatively early-stage AGI is better coordination, and that helps Earth get out of unsavory race dynamics and decide to slow down.
• The alignment tax at the superhuman level is pretty steep, and doing self-improvement while preserving alignment goes much slower than unrestricted self-improvement would; since at this point we have many fewer ongoing moral catastrophes (eg everyone who wants to be cryopreserved is, we've transitioned to excellent cheap lab-grown meat), there's little cost to proceeding very cautiously.
  • This is sort of a continuous version of the first bullet point with a finite rather than infinite alignment tax.

All that said, upon reflection I think I was probably lowballing the odds of crazy stuff on the 10y timescale, and I'd go to more like 50-60% that we're seeing mind uploads and Kardashev level 1.5-2 civilizations etc. a decade out from the first powerful AIs.

I do think it's fair to call out the essay for not highlighting the ways in which it might be lowballing things or rolling in an assumption of deliberate slowdown; I'd rather it have given more of a nod to these considerations and made the conditions of its prediction clearer.

"'it seems pretty plausible that AI will be at least somewhat nice', similar to how humans are somewhat nice to animals.". He must not know about factory farming - that thing where humans systematically hold harm and slaughter billions yearly. The following paragraph discusses if AI will care at all about humans and see us as valuable agents. The author then discusses how we do care about the wellbeing of cows. "We won't kill them for arbitrarily insignificant reasons" because clearly the taste of a good steak being more enjoyable than a plant based burger is anything but arbitrary. It's objectively more important than their life. To not understand the irony and ridiculousness here says something about the quality of the entire passage and the author's reasoning ability.

Resonating with you here!  Yes, I think autonomous corporations (and other organisations) would result in society-wide extraction, destabilisation and totalitarianism.

I'm not that concerned with lobbyists ruining the deliberative proceedings. I think you're underestimating normal people a bit. They have state power to shut down annoying and undesired feedback if they wish. I also think the assembly will tend to trust their own advisors, whom they hired themselves, over outside self-proclaimed expert lobbyists. 

My bigger concern is with corruption and bribery. Because we're dealing with very normal people, we also ought to expect normal criminal behavior. We ought to expect assembly members getting arrested from time to time, and doing all the normal things we expect from 500 random people. 

I think bribery is a sufficiently high concern that a police force should constantly operate to perform sting operations and monitor elicit behavior from assembly members. IMO, this should already be happening with elected officials too. 

Another big concern is whether a purely lottocratic assembly would self-regulate its own corruption. It has some interest to, in that the lottocrats help their future selves, after their term has ended, by creating future rules that would regulate corruption. Terril Bouricus attempts to create a system where layers on layers of assemblies check and re-check the work of other assemblies to mitigate corruption concerns. 

I can't easily conclude whether election or sortition would be better at corruption mitigation. With elections, opposition parties have an incentive to investigate their enemies to root out corruption. HOWEVER, the same opposition parties have an incentive to lie about the results of investigations, leading to an environment of fake news, where voters cannot distinguish between a political attack and actual corruption. In the American context, bribery is about already legalized with campaign donations. 

Sortition could possibly lead to a ridiculous scenario:

Imagine the public is outraged at the insane level of corruption of the sortition-assembly. However as a new assembly is formed by lottery, these anti-corruption sentiments are suddenly rotated into office. The members of the public hate corruption, as does this new assembly! The question is, would the members of the assembly be able to do the Machiavellian about-face and suddenly embrace corruption? I have a hard time believing they would, though I have doubts. In my opinion, normal people being utterly normal, would rather do the easy thing and yes, go ahead and regulate the corruption while enjoying their government salary. Getting to serve in office is already a win-win, why not win and also be declared heroes? Alternatively they can "Go Breaking Bad", embrace corruption and pilfer the state coffers. They can win big (for now) but will become despised. What do you think normal people would do? High risk high reward, or low risk medium reward? I don't think going "Breaking Bad" is the best of ideas. Elected politicians use their offices to protect themselves from legal challenge. Obvious example, Donald Trump using the presidency to overcome his legal problems. He's obviously not the first politician to cling to office in order to protect themselves. Lottocrats can't do the same. Lottocrats soon lose their powers and become vulnerable.