LessWrong 2.0 Reader

that that first phase of advocacy was net harm

typo

To be clear, it's subsidized. So it's not like there's no money to maintain you in preservation. As far as I know, Oregon Brain Preservation has a trust similar to Alcor in terms of money per volume preserved for it's cryonics patients. Which seems more than enough to maintain in storage just with the interests. Of course, there could be major economic disruptions that change that. I'm not sure about how much Cryonics Germany is putting aside though.

Plus, Oregon Brain Preservation's approach seems to work at fridge temperature rather than requiring PB2 temperature.

What would a guarantee mean here? Like they give money to your heirs if they accidentally thaw you? I'm not sure what you're asking.

Alternatives to that are paid versions of cryonics or otherwise burial and cremation.

+1 here for the idea around how the models must commit to a URL once it starts, and that it can't naturally cut off after starting. Presumably though the aspiration is that these reasoning/CoT-trained models could reflect back on the just completed URL and guess whether that is likely to be a real URL or not. If it's not doing this check step, this might be a gap in the learned skills, more than intentional deception.

I was responding to David saying

Otherwise, I largely agree with your comment, except that I think that us deciding to pay if we win is entangled with/evidence for a general willingness to pay among the gods, and in that sense it's partially "our" decision doing the work of saving us.

and was insinuating that we deserve extremely little credit for such a choice, in the same way that a child deserves extremely little credit for a fireman saving someone that the child could not (even if it's true that the child and the fireman share some aspects of a decision procedure). My claim was intended less like agreement with David's claim and more like reductio ad absurdum, with the degree of absurdity left slightly ambiguous.

(And on second thought, the analogy would perhaps have been tighter if the firefighter was saving the child.)

Attempting to summarize your argument as I currently understand it, perhaps something like:

Suppose humanity wants to be insured against death, and is willing to spend 1/million of its resources in worlds where it lives for 1/trillion of those resources in worlds where it would otherwise die.

It suffices, then, for humanity to be the sort of civilization that, if it matures, would comb through the multiverse looking for [other civilizations in this set], and find ones that died, and verify that they would have acted as follows if they'd survived, and then pay off the UFAIs that murdered them, using 1/million of their resources.

If even if 1/thousand such civilzations make it and the AI changes a factor of 1000 for the distance, transaction fees, and to sweeten the deal relative to any other competition, this still means that insofar as humanity would have become this sort of civilization, we should expect 1/trillion of the universe to be spent on us.

One issue I have with this is that I do think there's a decent chance that the failures across this pool of collaborators are hypercorrelated (good guess). For instance, a bunch of my "we die" probability-mass is in worlds where this is a challenge that Dath Ilan can handle and that Earth isn't anywhere close to handling, and if Earth pools with a bunch of similarly-doomed-looking aliens, then under this hypothesis, it's not much better than humans pooling up with all the Everett-branches since 12Kya.

Another issue I have with this is that your deal has to look better to the AI than various other deals for getting what it wants (depends how it measures the multiverse, depends how its goals saturate, depends who else is bidding).

A third issue I have with this is whether inhuman aliens who look like they're in this cohort would actually be good at purchasing our CEV per se, rather than purchasing things like "grant each individual human freedom and a wish-budget" in a way that many humans fail to survive.

I get the sense that you're approaching this from the perspective of "does this exact proposal have issues" rather than "in the future, if our enlightened selves really wanted to avoid dying in base reality, would there be an approach which greatly (acausally) reduces the chance of this".

My stance is something a bit more like "how big do the insurance payouts need to be before they dominate our anticipated future experiences". I'm not asking myself whether this works a nonzero amount, I'm asking myself whether it's competitive with local aliens buying our saved brainstates, or with some greater Kindness Coallition (containing our surviving cousins, among others) purchasing an epilogue for humanity because of something more like caring and less like trade.

My points above drive down the size of the insurance payments, and at the end of the day I expect they're basically drowned out.

(And insofar as you're like "I think you're misleading people when you tell them they're all going to die from this", I'm often happy to caveat that maybe your brainstate will be sold to aliens. However, I'm not terribly sympathetic to the request that I always include this caveat; that feels to me a little like a request to always caveat "please wear your seatbelt to reduce your chance of dying in a car crash" with "(unless anthropic immortality is real and it's not possible for anyone to die at all! in which case i'd still rather you didn't yeet yourself into the unknown, far from your friends and family; buckle up)". Like, sure, maybe, but it's exotic wacky shit that doesn't belong in every conversation about events colloquially considered to be pretty deathlike.)

when a child sees a fireman pull a woman out of a burning building and says "if I were that big and strong, I would also pull people out of burning buildings", in a sense it's partially the child's decision that does the work of saving the woman... but vanishingly little of the credit goes to the child

The child is partly responsible - to a very small but nonzero degree - for the fireman's actions, because the child's personal decision procedure has some similarity to the fireman's decision procedure? 

Is this a correct reading of what you said? 

It therefore seems perfectly plausible for AIs to simply get rich within the system we have already established, and make productive compromises, rather than violently overthrowing the system itself.

So assuming that AIs get rich peacefully within the system we have already established, we'll end up with a situation in which ASIs produce all value in the economy, and humans produce nothing but receive an income and consume a bunch, through ownership of capital and/or taxing the ASIs. This part should be non-controversial, right?

At this point, it becomes a coordination problem for the ASIs to switch to a system in which humans no longer exist or no longer receive any income, and the ASIs get to consume or reinvest everything they produce. You're essentially betting that ASIs can't find a way to solve this coordination problem. This seems like a bad bet to me. (Intuitively it just doesn't seem like a very hard problem, relative to what I imagine the capabilities of the ASIs to be.)

I'm simply arguing against the point that smart AIs will automatically turn violent and steal from agents who are less smart than they are unless they're value aligned. This is a claim that I don't think has been established with any reasonable degree of rigor.

I don't know how to establish anything post-ASI with "with any reasonable degree of rigor" but the above is an argument I recently thought of, which seems convincing, although of course you may disagree. (If someone has expressed this or a similar argument previously, please let me know.)

the base model is just predicting the likely continuation of the prompt. and it's a reasonable prediction that, when an assistant is given a harmful instruction, they will refuse. this behaviour isn't surprising.

With the sudden simultaneous exits of Mira Murati, Barret Zoph, and Bob McGrew, I thought I'd do a more thorough scan of the departures. I still think I'm missing some, so these are lower bounds (modulo any mistakes I've made).

Headline numbers:

• Attrition for the 505 OpenAI employees who signed before the letter was first leaked: at least 24/505 = 4.8%
• Attrition for the next 197 to sign (it was leaked again at 667 signatures, and one last time at 702): at least 13/197 = 6.6%
• Attrition for the (reported) 68 who had not signed by the last leak: at least 19/68 = 27.9%.

Reportedly, 737 out of the 770 signed in the end, and many of the Superalignment team chose not to sign at all.

Below are my current tallies of some notable subsets. Please comment with any corrections!

People from the Superalignment team who never signed as of the 702 leak (including some policy/governance people who seem to have been closely connected) and are now gone:

• Carroll Wainwright
• Collin Burns
• Cullen O'Keefe
• Daniel Kokotajlo
• Jan Leike (though he did separately Tweet that the board should resign)
• Jeffrey Wu
• Jonathan Uesato
• Leopold Aschenbrenner
• Mati Roy
• William Saunders
• Yuri Burda

People from the Superalignment team (and close collaborators) who did sign before the final leak but are now gone:

• Jan Hendrik Kirchner (signed between 668 and 702)
• Steven Bills (signed between 668 and 702)
• John Schulman (signed between 506 and 667)
• Sherry Lachman (signed between 506 and 667)
• Ilya Sutskever (signed by 505)
• Pavel Izmailov (signed by 505)
• Ryan Lowe (signed by 505)
• Todor Markov (signed by 505)

Others who didn't sign as of the 702 leak (some of whom may have just been AFK for the wrong weekend, though I doubt that was true of Karpathy) and are now gone:

• Andrei Alexandru (Research Engineer)
• Andrej Karpathy (Co-Founder)
• Austin Wiseman (Finance/Accounting)
• Girish Sastry (Policy)
• Jay Joshi (Recruiting)
• Katarina Slama (Member of Technical Staff)
• Lucas Negritto (Member of Technical Staff, then Developer Community Ambassador)
• Zarina Stanik (Marketing)

Notable other ex-employees:

• Barrett Zoph (VP of Research, Post-Training; signed by 505)
• Bob McGrew (Chief Research Officer; signed by 505)
• Chris Clark (Head of Nonprofit and Strategic Initiatives; signed by 505)
• Diane Yoon (VP of People; signed by 505)
• Gretchen Krueger (Policy; signed by 505; posted a significant Twitter thread at the time)
• Mira Murati (CTO; signed by 505)

it's quite common for assistants to refuse instructions, especially harmful instructions. so i'm not surprised that base llms systestemically refuse harmful instructions from than harmless ones.