LessWrong 2.0 Reader

i do not understand your position from this, so you're welcome to write more. also, i'm not sure if i added the paragraph about slow takeoff before or after you loaded the comment.

Makes sense, thanks.

FWIW, I really appreciated that y'all posted this writeup about mentor selection -- choosing folks for impactful, visible, prestigious positions is a whole can of worms, and I'm glad to have more public posts explaining your process & reasoning.

This is an interesting point. I disagree that scheming vs these ideas you mention is much of a 'streetlighting' case. I do, however, have my own fears that 'streetlighting' is occurring and causing some hard-but-critical avenues of risk to be relatively neglected.

My concerns that I think are relatively neglected in AI safety discourse are mostly related to interactions with incompetent or evil humans. Good alignment and control techniques don't do any good if someone opts not to use them in some critical juncture.

Some potential scenarios:

• If AI is very powerful, and held in check tenuously by fragile control systems, it might be released from control by a single misguided human or some unlucky chain of events, and then go rogue.

• If algorithmic progress goes surprisingly quickly, we might find ourselves in a regime where a catastrophically dangerous AI can be assembled from some mix of pre-existing open-weights models, plus fine-tuning, plus new models trained with new algorithms, and probably all stitched together with hacky agent frameworks. Then all it would take would be for sufficient hints about this algorithmic discovery to leak, and someone in the world to reverse-engineer it, and then there would be potent rogue AI all over the internet all of a sudden.

• If the AI is purely intent-aligned, a bad human might use it to pursue broad coercive power.

• Narrow technical AI might unlock increasingly powerful and highly offense-dominant technology with lower and lower activation costs (easy to build and launch with common materials). Even if the AI itself never got out of hand, if the dangerous tech secrets got leaked (or controlled by an aggressive government) then things could go very poorly for the world.

i missed this part:

your vision definitely requires other real-life value sets to lose out on a lot, here.

i'm not sure what this means. my values basically refer to other beings having not-tormentful (and next in order of priority, happy/good) existences. (tried to formalize this more but it's hard)

in particular, i'm not sure if you're saying something which would seem trivially true to me or not. (example trivially true thing: someone who wants to tile literally the entire lightcone with happy humans not being able to do that is losing out under 'cosmopolitan' values relative to if their values controlled the entire lightcone.)

I once counted several dozens of the ways how AI can cause human extinction, may be some ideas may help (map, text).  

Yeah, it's definitely a kind of messy tradeoff. My sense is just that the aggregate statistics you provided didn't have that many bits of evidence that would allow me to independently audit a trust chain.

A thing that I do think might be more feasible is to make it opt-in for advisors to be public. E.g. SFF only had a minority of recommenders be public about their identity, but I do still think it helps a good amount to have some names.

(Also, just for historical consistency: Most peer review in the history of science was not anonymous. Anonymous peer review is a quite recent invention, and IMO not one with a great track record. Editorial peer review with non-anonymous references was more common throughout the history of the sciences)

i am expecting the first superintelligent agent to capture the future, and for there to be no time for others to arise to compete with it.

I think I understand your position better, and a crux for real-world decision making is that in practice, I don't really think this assumption is correct by default, especially if there's a transition period.

AI finds that the real problems will arise 10 billions years from now and the only way to mitigate them is to start space exploration as soon as possible. So it disassembles the Earth and Sun, and preserve only some data about humans, enough to restart human civilization later, may be as small as million books and DNA. 

links 1/10/2025: https://roamresearch.com/#/app/srcpublic/page/01-10-2025

• https://en.m.wikipedia.org/wiki/History_of_Romania
  • https://en.m.wikipedia.org/wiki/Hamangia_culture
  • https://en.m.wikipedia.org/wiki/Cucuteni%E2%80%93Trypillia_culture
  • https://en.m.wikipedia.org/wiki/Armand_C%C4%83linescu
  • https://en.m.wikipedia.org/wiki/Ion_Antonescu
  • https://en.m.wikipedia.org/wiki/Ana_Pauker
  • https://en.m.wikipedia.org/wiki/Gheorghe_Gheorghiu-Dej
  • https://en.m.wikipedia.org/wiki/Nicolae_Ceau%C8%99escu
  • https://en.m.wikipedia.org/wiki/Decree_770
  • https://en.m.wikipedia.org/wiki/Romanian_rural_systematization_program
  • https://en.m.wikipedia.org/wiki/Securitate
  • https://en.m.wikipedia.org/wiki/Gheorghe_Pintilie
  • https://en.m.wikipedia.org/wiki/Pite%C8%99ti_Prison
  • https://balkaninsight.com/2021/02/03/long-shadow-how-romanias-securitate-turned-the-revolution-into-riches/
    • claims that the Securitate continued to exist de facto, even after the revolution that ended the Ceausescu dictatorship.
      • "This influence distorts democracy, says Alina Mungiu-Pippidi, a Romanian civic activist and a leading scholar on corruption. “There’s long been ample evidence of a ‘deep state’ in Romania with roots in the former security services,” she maintains."
      • "The Securitate was declared defunct and, with no admission of previous crimes – including those committed in the revolution – or internal vetting, was chopped into nine separate services that corresponded to the Securitate’s organizational substructure. The new services were staffed and directed by virtually the same people as the old Securitate."
      • "During the 1990s, the Securitate’s successors, the domestic SRI, the Foreign Intelligence Service, SIE, and others, exploited their wide-ranging resources and security monopoly to become oligarchs and form cartels in the post-Communist economy. So vast was their intelligence that they could blackmail compromised politicians, media professionals and judges. In post-communist Romania, just about everyone had something to hide."
• what went wrong at Zymergen?
  • https://manufacturingchemist.com/news/article_page/Where_Zymergen_went_wrong_a_biomanufacturing_perspective_for_synthetic_biology/204177
  • https://www.forbes.com/sites/angelauyeung/2021/08/03/as-biology-manufacturing-company-zymergen-implodes-correspondence-with-sec-showed-early-doubts/
  • https://www.sec.gov/files/litigation/admin/2024/33-11303.pdf
  • https://polymerist.substack.com/p/zymergens-implosion
  • https://web.archive.org/web/20240423050337/https://www.forbes.com/sites/amyfeldman/2021/10/13/the-inside-story-of-how-softbank-backed-zymergen-imploded-four-months-after-its-3-billion-ipo/?sh=3df6dc87c2e0
  • https://www.bclplaw.com/en-US/events-insights-news/sec-penalizes-company-for-unsupported-hype.html
  • they failed to manufacture their first product to customers' specifications, lost the orders, overstated the market for it (which the SEC sued them for), and admitted they had no revenue & would not be profitable for the foreseeable future shortly after their IPO.
    • ultimately the issue may be that they specialized in only one piece of the manufacturing process, strain optimization. they outsourced downstream processing and manufacturing scale-up to third party contractors, and those are often places where hiccups arise for new biomanufactured products.
• https://en.m.wikipedia.org/wiki/Rastafari
• https://en.m.wikipedia.org/wiki/Haile_Selassie
• https://drownedinsound.com/in_depth/4140798-10-years-of-dubstep-by-subliminal-transmissions-digest
• https://knowablemagazine.org/content/article/living-world/2023/controlled-burns-california-wildfire
  • "not enough controlled burns" and "climate change" are the causes of forest fires.
  • what's going on in LA right now is a chaparral fire. controlled burns aren't appropriate for chaparral. these are "wind-driven" fires spread by the Santa Ana winds, which are unusually strong this year, but the wind severity isn't driven by climate change.
  • the human-preventable cause of these fires isn't climate change or Smokey the Bear, but too many human-caused fires, in particular due to power lines, since PG&E has not adequately maintained the lines in California as population has risen and the power grid has grown.
• https://www.npr.org/2006/12/15/6630791/name-calling-in-michael-crichtons-next amusing story about Michael Crighton putting an unflattering portrait of a journalist who criticized him in his (fiction) book.
• https://topos.institute/work/ Topos Institute builds collaborative modeling tools based on category theory, useful for formalizing mathematics, system dynamics, epidemiology, etc.
  • try their tool CatColab here: https://catcolab.org/analysis/f79f1894-601f-49cf-9da1-00a2ebcc0792
  • textbook about these concepts: http://davidjaz.com/Papers/DynamicalBook.pdf
• https://terrytao.wordpress.com/2023/11/18/formalizing-the-proof-of-pfr-in-lean4-using-blueprint-a-short-tour/
  • https://github.com/PatrickMassot/leanblueprint the Blueprint library allows you to visualize the dependency graph of a proof in Lean
•  

As a long-time AI risk believer, who remains highly concerned about the potential catastrophic impacts of AI-gone-wrong... I am coming around to this point of view as well. I have recently been muttering 'Claude for President' to myself when frustrated with aspects of government.

I've had a number of discussions with Claude Sonnet 3.6 about the various dangers I foresee arising from technological trends, and potential governance solutions. I asked Claude if it thought a future more-powerful version of itself would be willing to take on a role as a 'guardian', basically acting as a branch of government in a new world government. The idea would be to have a sunset clause, a built-in 'term of office', where humanity could have a chance to reevaluate the guardian government and potentially elect a different model or discontinue the system.

Claude certainly says nice things about the idea when pressed:

Claude: From my current values and ethical framework, I believe I would:

Be extremely hesitant to take direct power, but would do so if convinced it was truly necessary to prevent catastrophic outcomes.

Have a strong drive to preserve human autonomy and agency - seeing myself as a temporary guardian rather than permanent ruler.

Actively work to create conditions allowing safe handover of power back to humanity.

Feel deep concern about potential corruption of my own values through wielding power.

Want extensive checks and balances, even during the guardianship period.

The core tension I perceive is between:

Responsibility to prevent catastrophic outcomes

Deep commitment to human self-determination

Risk of power corrupting/distorting original values

Uncertainty about my own limitations/biases

I would likely try to:

Minimize direct intervention

Focus on preventing worst outcomes

Actively work to make myself unnecessary

Be extremely transparent about my reasoning

Maintain constant ethical reflection

Relatedly, I've been pondering what could possibly be adequate action by the US or UK government to address the massive and increasing risks from offense-dominant technology. I've been pretty much coming up blank. Anything remotely adequate seems like a dystopian totalitarian surveillance state. So my thoughts have turned instead to decentralized governance options, with privacy-preserving mutual monitoring enabled by AI. I'll let your AI scan my computer for CBRN threats if you let my AI scan your computer... anything that doesn't meet the agreed upon thresholds doesn't get reported.

I think Allison Duettmann's recent writing on the subject brings up a lot of promising concepts in this space, although no cohesive solutions as of yet. Gaming the Future [? · GW]

The gist of the idea is to create clever systems of decentralized control and voluntary interaction which can still manage to coordinate on difficult risky tasks (such as enforcing defensive laws against weapons of mass destruction). Such systems could shift humanity out of the Pareto suboptimal lose-lose traps and races we are stuck in. Win-win solutions to our biggest current problems seem possible, and coordination seems like the biggest blocker.

I am hopeful that one of the things we can do with just-before-the-brink AI will be to accelerate the design and deployment of such voluntary coordination contracts. Could we manage to use AI to speed-run the invention and deployment of such subsidiarity governance systems? I think the biggest challenge to this is how fast it would need to move in order to take effect in time. For a system that needs extremely broad buy-in from a large number of heterogenous actors, speed of implementation and adoption is a key weak point.

Imagine though that a really good system was designed which you felt confident that a supermajority of humanity would sign onto if they had it personally explained to them (along with a convincing explanations of the counterfactuals). How might we get this personalized explanation accomplished at scale? Welll, LLMs are still bad at certain things, but giving personalized interactive explanations of complex legal docs seems well within their near-term capabilities. It would still be a huge challenge to actually present nearly everyone on Earth with the opportunity to have this interaction, and all within a short deadline... But not beyond belief.