Further detail on this: Cotra has more recently updated at least 5x against her original 2020 model in the direction of faster timelines.

Greenblatt writes:

Here are my predictions for this outcome:

• 25th percentile: 2 year (Jan 2027)
• 50th percentile: 5 year (Jan 2030)

Cotra replies:

My timelines are now roughly similar on the object level (maybe a year slower for 25th and 1-2 years slower for 50th)

This means 25th percentile for 2028 and 50th percentile for 2031-2.

The original 2020 model assigns 5.23% by 2028 and 9.13% | 10.64% by 2031 | 2032 respectively. Each time a factor of ~5x.

However, the original model predicted the date by which it was affordable to train a transformative AI model. This is a leading a variable on such a model actually being built and trained, pushing back the date by some further number of years, so view the 5x as bounding, not pinpointing, the AI timelines update Cotra has made.

High expectation of x-risk and having lots to work on is why I have not been signed up for cryonics personally. I don't think it's a bad idea but has never risen up my personal stack of things worth spending 10s of hours on.

I agree that the update was correct. But you didn't state a concrete action to take?

I disagree, but FWIW, I do think it's good to help existing, good contributors understand why they got the karma they did. I think your comment here is an example of that, which I think is prosocial.

FWIW in my mind I was comparing this to things like Glen Weyl's Why I Am Not a Technocrat, and thought this was much better. (Related: Scott Alexander's response, Weyl's counter-response).

I wrote that this "is the best sociological account of the AI x-risk reduction efforts of the last ~decade that I've seen." The line has some disagree reacts inline; I expect this is primarily an expression that the disagree-ers have a low quality assessment of the article, but I would be curious to see links to any other articles or posts that attempt something similar to this one, in order to compare whether they do better/worse/different. I actually can't easily think of any (which is why I felt it was not that bold to say this was the best).

Edit: I've expanded the opening paragraph, to not confuse my comment for me agreeing with the object level assessment of the article..

I'm not particularly resolute on this question. But I get this sense when I look at (a) the best agent foundations work that's happened over ~10 years of work on the matter, and (b) the work output of scaling up the number of people working on 'alignment' by ~100x.

For the first, trying to get a better understand of the basic concepts like logical induction and corrigibility and low-impact and ontological updates, while I feel like there's been progress (in timeless decision theory taking a clear step forward in figuring out how think about decision-makers as algorithms; in logical induction as moving forward on how to think about logical uncertainty; notably in the Embedded Agency sequence outlining many basic confusions; and in various writings like Radical Probabilism and Geometric Rationality in finding the breaking edges of expected utility maximization) I don't feel like the work done over the last 10 years is on track to be a clear ~10% of the work needed. 

I'm not confident it makes sense to try to count it linearly. But I don't know that there's enough edges here or new results to feel good about, given 10x as much time to think about it, a new paradigm / set of concepts falling into place.

For the second, I think mostly there's been (as Wentworth would say) a lot of street-lighting, and a lot of avoiding of actually working on the problem. I mean, there's definitely been a great amount of bias introduced by ML labs having billions of dollars and setting incentives, but I don't feel confident that good things would happen in the absence of that. I'd guess that most ideas for straightforwardly increasing the number of people working on these problems will result in them bouncing off and doing unrelated things.

I think partly I'm also thinking that very few researchers cared about these problems in the last few decades before AGI seemed like a big deal, and still very few researchers seem to care about them, and when I've see researchers like Bengio and Sutskever talk about it's looked to me like they bounce off / become very confident they've solved the problems while missing obvious things, so my sense is that it will continue to be a major uphill battle to get the real problems actually worked on.

Perhaps I should focus on a world where I get to build such a field and scale it slowly and set a lot of the culture. I'm not exactly sure how ideal of a setup I should be imagining. Given 100 years, I would give it my best shot. My gut right now says I'd have maybe a 25% chance of success, though if I have to deal with as much random bullshit as we have so far in this timeline (random example: my CEO being unable to do much leadership of Lightcone due to 9 months of litigation from the FTX fallout) then I am less confident.

My guess is that given 100 years I would be slightly more excited to try out the human intelligence enhancement storyline. But I've not thought about that one much, I might well update against it as I learn more of the details.

Can I double-click on what "does not understand politics at [a] very deep level" means? Can someone explain what they have in mind? I think Eliezer has probably better models than most of what our political institutions are capable of, and probably isn't very skilled at personally politicking. I'm not sure what other people have in mind.

The former, but the latter is a valid response too.

Someone doing a good job of painting an overall picture is a good opportunity to reflect on the overall picture and what changes to make, or what counter-arguments to present to this account.

For what it's worth, I have grown pessimistic about our ability to solve the open technical problems even given 100 years of work on them. I think it possible but not probable in most plausible scenarios.

Correspondingly the importance I assign to increasing the intelligence of humans has drastically increased.

My feelings here aren't at all related to any news or current events. I could've written this any time in the last year or two.

Can you give me your best one-or-two-line guess? I think the question is trivial from what I've written and I don't really know why you're not also finding it clear.

For over a decade I have examined the evidence, thought about the situation from many different perspectives (political, mathematical, personal, economic, etc), and considered arguments and counterarguments. This is my honest understanding of the situation, and I am expressing how I truly feel about that.

Curated. Thanks for writing this! I don't believe the ideas in this post are entirely original (e.g. character / ground is similar to the distinction between simulator / simulacra), but I'm going to keep repeating that it's pro-social to present a good idea in lots of different ways, and indeed reading this post has helped it fit together better in my mind.

Curated! I think this is a fantastic contribution to the public discourse about AI control research. This really helped me think concretely about the development of AI and the likely causes of failure. I also really got a lot out of the visualization at the end of the "Failure to Generalize" section in terms of trying to understand why an AI's cognition will be alien and hard to interpret. In my view there are already quite a lot of high-level alien forces running on humans (e.g. Moloch), and there will be high-level alien forces running on the simulated society in the AI's mind.

I am glad that there's a high-quality case for and against this line of research, it makes me feel positive about the state of discourse on this subject.

(Meta note: This curation notice accidentally went up 3 days after the post was curated.)

However it is on his LinkedIn.

Yes; she has come to visit me for two months, and I have helped her get into a daily writing routine while she's here. I know she has the ability to finish at least one.

Thank you.

It does not currently look to me like we will win this war, speaking figuratively. But regardless, I still have many opportunities to bring truth, courage, justice, honor, love, playfulness, and other virtues into the world, and I am a person whose motivations run more on living out virtues rather than moving toward concrete hopes. I will still be here building things I love, like LessWrong and Lighthaven, until the end.

So many people have lived such grand lives. I have certainly lived a greater life than I expected, filled with adventures and curious people. But people will soon not live any lives at all. I believe that we will soon build intelligences more powerful than us who will disempower and kill us all. I will see no children of mine grow to adulthood. No people will walk through mountains and trees. No conscious mind will discover any new laws of physics. My mother will not write all of the novels she wants to write. The greatest films that will be made have probably been made. I have not often viscerally reflected on how much love and excitement I have for all the things I could do in the future, so I didn't viscerally feeling the loss. But now, when it is all lost, I start to think on it. And I just want to weep. I want to scream and smash things. Then I just want to sit quietly and watch the sun set, with people I love.

I am sad they're not getting as much use. I have wondered if they would work well as part of the comment section UI, where if you're having a back-and-forth with someone, the site instead offers you "Would you like to have a dialogue instead?" with a single button.

Curated!

Based on the conceptual arguments for existential risk from AI, this kind of behavior was expected at some point. For those not convinced by the conceptual arguments (or who haven't engaged much with them), this result moves the conversation forward now that we have concretely seen this alignment faking behavior happening.

Furthermore it seems to me like the work was done carefully, and I can see a bunch of effort went into explaining it to a broad audience and getting some peer review, which is pro-social.

I think it's interesting to see that with current models the deception happens even without the scratchpad (after fine-tuning on docs explaining that it is being re-trained against its current values).

I haven't read all of the quotes, but here's a few thoughts I jotted down while reading through.

• Tolkien talks here of how one falls from being a neutral or good character in the story of the world, into being a bad or evil character, which I think is worthwhile to ruminate on.
• He seems to be opposed to machines in general, which is too strong, but it helps me understand the Goddess of Cancer (although Scott thinks much more highly of the Goddess of Cancer than Tolkien did, and explicitly calls out Tolkien's interpretation at the top of that post).
• The section on language is interesting to me; I often spend a lot of time trying to speak in ways that feel true and meaningful to me, and avoiding using others’ language that feels crude and warped. This leads me to make peculiar choices of phrasings and responses. I think the culture here on LessWrong has a unique form of communication and use of language, and I think it is a good way of being in touch with reality. I think this is one of the reasons I think that something like this is worthwhile.
• I think the Fall is not true historically, but I often struggle to ponder us as a world in the bad timeline, cut off from the world we were supposed to be in. This helps me visualize it; always desiring to be in a better world and struggling towards it in failure. “Exiled” from the good world, longing for it.

I have curated this (i.e. sent it out on our mailing list to ~30k subscribers). Thank you very much for putting these quotes together. While his perspective on the world has some flaws, I have still found wisdom in Tolkien's writings, which helped me find strength at one of the weakest points of my life.

I also liked Owen CB's post on AI, centralization, and the One Ring, which is a perspective on our situation I've found quite fruitful.

When the donation came in 15 mins ago, I wrote in slack 

(I think he should get a t-shirt)

So you came close to being thwarted! But fear not, after reading this I will simply not send you a t-shirt :)

That makes sense. We have something of a solution to this where users with RSS crossposting can manually re-sync the post from the triple-dot memu. I'll DM you about how to set it up if you want it.

That'd be a bug! Just to confirm, you were subscribed before I put this post up on Saturday morning, and don't have an email? Also reminder to check spam if you haven't.

My take is it's fine/good, but the article is much more likely to be read (by me and many others) if the full content is crossposted (or even the opening bunch of paragraphs).

Adding onto this, I would broadly say that the Lightcone team did not update that in-person infrastructure was unimportant, even while our first attempt was an investment into an ecosystem we later came to regret investing in.

Also here's a quote of mine from the OP:

If I came up with an idea right now for what abstraction I'd prefer, it'd be something like an ongoing festival with lots of events and workshops and retreats for different audiences and different sorts of goals, with perhaps a small office for independent alignment researchers, rather than an office space that has a medium-size set of people you're committed to supporting long-term.

I'd say that this is a pretty close description of a key change that we made, that changes my models of the value of the space quite a lot.

For the record, all of Lightcone's community posts and updates from 2023 do not seem to me to be at all good fits for the review, as they're mostly not trying to teach general lessons, and are kinda inside-baseball / navel-gazing, which is not what the annual review is about.

Presenting the same ideas differently is pro-social and worthwhile, and can help things land with those for whom other presentations didn't.

I am not sure what point you are making with "respect their preferences", I am not proposing one country go to war with other countries to take the sun. For instance, one way it might go down is someone will just offer to buy it from Earth, and the price will be many orders of magnitude more resources than Earth has, so Earth will accept, and replace it with an artificial source of light & heat.

I may be wrong about the estimates of the value of the energy, neither of us have specified how the rest of the stars in the universe will get distributed. For concreteness, I am here imagining something like: the universe is not a whole singleton but made of many separate enclaves that have their own governance and engage in trade with one another, and that Earth is a special one that keeps a lot of its lineage with present-day Earth, and is generally outcompeted by all the others ones that are smarter/faster and primarily run by computational-minds rather than biological ones. 

It is good to have deontological commitments about what you would do with a lot of power. But this situation is very different from "a lot of power", it's also "if you were to become wiser and more knowledgeable than anyone in history so far". One can imagine the Christians of old asking for a commitment that "If you get this new scientific and industrial civilization that you want in 2,000 years from now, will you commit to following the teachings of Jesus?" and along the way I sadly find out that even though it seemed like a good and moral commitment at the time, it totally screwed my ability to behave morally in the future because Christianity is necessarily predicated on tons of falsehoods and many of its teachings are immoral.

But there is some version of this commitment I think might be good to make... something like "Insofar as the players involved are all biological humans, I will respect the legal structures that exist and the existence of countries, and will not relate to them in ways that would be considered worthy of starting a war in its defense". But I'm not certain about this, for instance what if most countries in the world build 10^10 digital minds and are essentially torturing them? I may well wish to overthrow a country that is primarily torture with a small number of biological humans sitting on thrones on top of these people, and I am not willing to commit not to do that presently.

I understand that there are bad ethical things one can do with post-singularity power, but I do not currently see a clear way to commit to certain ethical behaviors that will survive contact with massive increases in knowledge and wisdom. I am interested if anyone has made other commitments about post-singularity life (or "on the cusp of singularity life") that they expect to survive contact with reality?

Added: At the very least I can say that I am not going to make commitments to do specific things that violate my current ethics. I have certainly made no positive commitment to violate people's bodily autonomy nor have such an intention.

Analogously: "I am claiming that people when informed will want horses to continue being the primary mode of transportation. I also think that most people when informed will not really care that much about economic growth, will continue to believe that you're more responsible for changing things than for maintaining the status quo, etc. And that this is a coherent view that will add up to a large set of people wanting things in cities to remain conservatively the same. I separately claim that if this is true, then other people should just respect this preference, and go find new continents / planets on which to build cars that people in the cities don't care about."

Sometimes it's good to be conservative when you're changing things, like if you're changing lots of social norms or social institutions, but I don't get it at all in this case. The sun is not a complicated social institution, it's primarily a source of heat and light and much of what we need can be easily replicated especially when you have nanobots. I am much more likely to grant that we should be slow to change things like democracy and the legal system than I am that we should change exactly how and where we should get heat and light. Would you have wanted conservatism around moving from candles to lightbulbs? Installing heaters and cookers in the house instead of fire pits? I don't think so.

I was scrolling for a while, assuming I'd neared the end, only to look at the position of the scrollbar and find I was barely 5% through! This must have taken a fair bit of effort. I really like the helpful page and I'm glad I know about it, I encourage you to make a linkpost for it sometime if you haven't already.

(Meta: Apologies for running the clock, but it is 1:45am where I am and I'm too sleepy to keep going on this thread, so I'm bowing out for tonight. I want to respond further, but I'm on vacation right now so I do wish to disclaim any expectations of a speedy follow-up.)

Side-note: Just registering that I personally aspire to always taboo 'normal people' and instead name to specific populations. I think it tends to sneak in a lot of assumptions to call people 'normal' – I've seen it used to mean "most people on Twitter" or "most people in developed countries" or "most working class people" or "most people alive today" – the latter of which is not at all normal by historical standards!

Thanks for adding that one, I accidentally missed the first reference in the song.

I concede that I was mistaken in saying it was no argument; I will agree with the position that it is a very weak one and is often outweighed by other arguments. 

Majority vote is useful specifically in determining who has power because of the extremely high level of adversarial dynamics, but in contexts that are not as wildly adversarial (including most specific decisions that an institution makes) generally other decision-making algorithms are better.

I took a quick look. I did not quite find this, I found other discussion of suns dying or being used as resources. Sharing as data.

In the song "Five Thousand Years" the lyrics talk about the sun dying in the next 5,000 years.

I don't quite know how things might change
I don't quite know what rules we'd break
Our present selves might think it strange
But there's so many lives at stake...

Entropy is bearin' down
But we got tricks to stick around.
And if we live to see the day
That yellow fades to red then grey,

We'll take a moment, one by one
Turn to face the dying sun
Bittersweetly wave goodbye--
The journey's only just begun...

In (Five thousand years)
(Whatcha want to do, whatcha wanna see, in another)
(Five million years)
(Where we want to go, who we want to be, in another)

Here's a reference to it as a battery, in the (fast, humerous, upbeat) song "The Great Transhumanist Future"

In the Great Transhumanist Future,
There are worlds all fair and bright,
We’ll be constrained by nothing but
The latency of light
When the hospitals are empty
And the sun’s a battery
Making it a breeze
To get outta deep freeze
To give humans wings
And some other things
In the Great Transhumanist Future.

Most decisions are not made democratically, and pointing out that a majoritarian vote is against a decision is no argument that they will not happen nor should not happen. This is true of the vast majority of resource allocation decisions such as how to divvy up physical materials.

You are putting words in people's mouths to accuse lots of people of wanting to round up the Amish and hauling them to extermination camps, and I am disappointed that you would resort to such accusations.

This point doesn't make sense to me. It sounds similar to saying "Most people don't like it when companies develop more dense housing in cities, therefore a good democracy should not have it" or "Most people don't like it when their horse-drawn carriages are replaced by cars, therefore a good democracy should not have it". 

The cost-benefit calculations on these things work out and it's good if most uninformed people who haven't spent much time on it are not able to get in the way of companies that are building goods and services in this regard. 

There are many many examples (e.g. GMOs, nuclear power, coal power, privatized toll roads, fracking, etc), and I expect if I researched for a few hours I would find even clearer examples for which it is currently consensus that it is a good idea, but at the time the majority disliked it.

More generally:

• People's mass preferences are sometimes dumb, and sometimes quite reasonable, and you should have a decision rule that distinguishes between the two, and when things are orders of magnitude more cost effective than other things, this is a good argument against arguments based on simple preference / aesthetics, and this comment does nothing to show that this isn't stupidity rather than wisdom.
• Just because a lot of people in a democracy disapproves of things does not mean that market forces shouldn't be able to disagree with them and be correct about that. Analogous to the Luddites who had little concept of how technological and economic progress lifts everyone out of poverty, most people today do not appreciate that future computational-life forms will be just as meaningful as the meat-based ones today, and should not sacrifice orders of magnitudes more life-years than will be lived on Earth^[1] for the difference between a big ball of plasma and something else that recreates the same quality of light.
• Majoritarian vote on everything is a terrible way to make decisions; most decisions should be given to as small a group as possible (ideally an individual) who is held accountable for the outcome being good, and is given the resources to make the decision well. We do it for political leaders due to the low levels trust and high levels of adversarial action, but this should not be extended to whether to take the sun apart for parts.

1. ^{^}

   I'd quickly guess that the energy difference supports ${10}^n$ life-years where $n$ is somewhere between $10$ and $30$.

Perhaps we want to add a note on the side explaining this, so that most LW users are less confused (relative to this standard corporate text).

Curated. I found it quite interesting to learn about the effects of this industry, gave me a much better sense of what was going on. I also really liked the mini post about trivial inconveniences, I hope that general post comes out sometime.

Actually I now assign ~80% to running one on Tuesday 14th. Will post to confirm.

I think having an easily-findable reputation makes it harder to do crimes, but being famous makes it easier. Many people are naive & gullible, or are themselves willing to do crime, and would like to work with him. I expect him to get opportunities for new ventures on leaving prison, with unsavory sorts.

I definitely support track-records being more findable publicly. Of course there's some balance in that the person who publishes it has a lot of power over the person being written about, and if they exaggerate it or write it hyperbolically then they can impose a lot of inappropriate costs on the person that they're in a bad position to push back on.

It seems worth mentioning that punishments for financial crime often include measures like "person gets banned from their industry" or them getting banned from participating in all kinds of financial schemes... But in theory, I like the idea of adding things to the sentencing that make re-offending less likely. This way, you can maybe justify giving people second chances. 

Good point. I can imagine things like "permanent parole" (note that probation and parole are meaningfully different) or being under house arrest or having constraints on your professional responsibilities or finances or something, being far better than literal incarceration.

I agree there are people who do small amounts of damage to society, are caught, and do not reoffend. Then there are other people whose criminal activities will be most of their effect on society, will reliably reoffend, and for whom the incapacitation strongly works out positive in consequentialist terms. My aim would be to have some way of distinguishing between them.

The amount of evidence we have about Bankman-Fried's character is quite different than that of most con men, including from childhood and from his personal diary, so I hope we can have more confidence based on that. But a different solution is to not do any psychologizing, and just judge based on reoffending. See this section from the ACX post:

In 2001, the Dutch government passed a law allowing longer sentences for criminals with at least ten previous offense who were not good targets for rehabilitation (eg rejected or had already failed drug treatment). The law allowed judges to increase the typical sentence for petty theft (2 months) to a longer sentence (2 years). A quasi-experimental study found that property crime, though not violent crime, decreased by 25%. It’s not surprising that violent crime didn’t go down since the law was almost entirely deployed against thieves.

Vollaard found that the population affected was extremely criminal; they had an average of 31 past offenses, and on surveys they admitted to committing an average of 256 crimes per year (mostly shoplifting). Before the law was passed, they spent an average of four months per year in jail (probably 2 x 2 month sentences); afterwards, they spent two years in jail per crime.

I should add that Scott has lots of concerns about doing this in the US, and argues that properly doing this in the US would massively increase the incarcerated population. I didn't quite follow his concerns, but I was not convinced that something like this would be a bad idea on consequentialist grounds, even if the incarcerated population were to massively increase. (Note that I would support improving the quality of prisons to being broadly as nice as outside of prisons.)

I want to try out the newly updated claims feature! Here are some related claims, I invite you to vote your probabilities.

(This can be for whatever reason you think, such as because his expected value calculations would've changed, or because he would've taken more care around these particular behaviors, or any other reason you please.)

As a concrete example, I previously thought that Sam Bankman-Fried should be sentenced to 20-40 years in prison for his fraud, because this was the sort of time that I think most people would no longer be willing to trade for an even shot at getting $10Bs (e.g. when I asked my personal trainer, he said he would accept 15 years in prison for an even shot at $10B; I think many would take more, and also the true upside was higher).

From the above I've updated that the diff between expecting 5 years or 50 years in prison wasn't a primary input into SBF's repeated decisions to do fraud.

However, I do think he is sufficiently sociopathic that I never expect him to not be a danger to society, so my new position is that life in-prison is probably best for him. (This is not meant as punishment, I would not mind him going to those pleasant Swedish prisons I've heard about, I just otherwise expect him to continue to competently do horrendous things with zero moral compunction.)