LessWrong 2.0 Reader

The option to buy SPY at $855 in January 2027 is going for $1.80 today, because most people don’t expect the price to get that high. But if in fact SPY increases in the intervening time by 50% from its present value ($582), as stipulated by kairos, then the option will ultimately be worth 1.5*582 - 855 ~ $18. I think this is where the 12x figure is coming from.

I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it.

I basically agree with this.

A crux here is that I believe that AI governance is basically worthless if you cannot control the ability to create powerful AIs, because they will soon be able to create states within a state, and thus resist any governance of AI plan you might wish to implement.

In other words, you need something close to a monopoly on violence in order for a government to continue existing, and under scenarios where AIs get very powerful very quickly because of algorithms, there is no good way to control the distribution of AI power, and it's too easy for people to defect from a governance on AI treaty/law.

I'm not hopeless that such a world can survive. However, I do think AI governance breaks hard if we are in a future where AI power primarily is determined by algorithms, since I basically consider algorithmic advances mostly impossible to control.

I don't think it's hopeless, just that the plans would be rather differently shaped.

IMO, a plan for short timelines where algorithmic secrets are very powerful I would pivot hard towards technical safety/alignment, and would more or less abandon all but the most minimal governance/control plans, and the other big thing I'd focus on is on making the world's offense-defense balance much better than it is now, which means we'd have to invest more in making bio-tech less risky, with the caveat that we cannot assume the government will successfully suppress bio-threats from civilian actors.

I'm not sure I understand what you're driving at, but as far as I do, here's a response: I have lots of concepts and abstractions over the physical world (like chair). I don't have many concepts or abstractions over strings of language, apart from as factored through the physical world. (I have some, like register or language, but they don't actually feel that "final" as concepts).

As far as factoring my predictions of language through the physical world, a lot of the simplest and most robust concepts I have are just nouns, so they're already represented by tokenisation machinery, and I can't do interesting interp to pick them out.

Thanks for writing this, David! Your sequence of notes on virtues is one of my favorites on this site; I often find myself coming back to them, to better understand what it might mean to eg Love. As someone who's spent a lot of time now in EA, I appreciated that this piece was especially detailed, going down all kinds of great rabbitholes. I hope to leave more substantive thoughts at some future time, but for now: thank you again for your work on this.

links 1/2/25: https://roamresearch.com/#/app/srcpublic/page/01-02-2025

• https://reason.com/2024/11/14/abolish-the-small-business-administration/ the case for abolishing the Small Business Administration, which subsidizes small businesses
• https://www.mercatus.org/doge  the Mercatus center's wishlist for DOGE on budget cuts and deregulation
  • sadly, these are not policies! they are editorials making arguments for why smaller government would be a good idea, and why certain tactics are worth consideration (like sunset provisions). zero object-level deregulation policy work (i.e. identifying which regulations to cut and who can cut them) has been done here. I am beginning to see why people complain about think tanks not actually dOiNg PoLicY.
• https://www.perplexity.ai/search/what-are-vortex-crystals-zhsBX93zTdOxMh5sg3nWAw#3  Perplexity explains superfluids; they are frictionless, have extremely high thermal conductivity, and exhibit "quantized vortices", where the speed of the spinning fluid is an integer multiple of a constant and the vortex can keep spinning literally forever. Liquid helium is a superfluid. This has been known since 1937!
  • https://en.wikipedia.org/wiki/Superfluidity
• https://www.science.org/doi/10.1126/science.adn5694  a passively radiative aerogel that reflects >100% of solar radiation through fluorescence and phosphorescence
• https://www.nature.com/articles/s41467-021-26050-z  0.5% of European GDP reduced by heat waves in the 21st century
• https://www.nbcnews.com/news/us-news/new-orleans-killed-mass-casualty-bourbon-street-car-crowd-rcna185914 "Texas man kills 14 on New Orleans' Bourbon Street after driving truck with ISIS flag through crowd." Police believe he did not act alone.
• https://en.m.wikipedia.org/w/index.php?title=Shotgun_cartridge A shotgun cartridge is filled with tiny little metal balls called shot, or a single projectile called a slugs. Smaller shot goes farther than bigger shot, and slugs go farther still.
• https://en.m.wikipedia.org/wiki/Can_You_Ever_Forgive_Me movie (I haven't seen it but saw it recommended) about a woman who forged letters from dead celebrities
• https://www.sympatheticopposition.com/p/risk-averse-women-rarely-birth-royalty Matthew's genealogy of Jesus notes four Biblical women -- Tamar, Rahab, Ruth, and Bathsheba -- who took sexual risks to secure their children's legacy. I love this piece and think we talk too little about the fact that women take calculated risks sexually. To live a good life, you have to risk intimacy and decide when to bet that it won't put you in danger or ruin your future. That's true in both ancient societies and today, though of course the risk landscape is very different.
• https://www.econlib.org/archives/2011/01/the_stranger.html I like this old Bryan Caplan post on what we owe strangers. Common sense ethics says we should not aggress upon strangers but we don't usually owe them much positive help; yet this is a radical and uncommon position in politics.
• https://arxiv.org/pdf/2412.16075 a case for formal verification + AI hybrid systems in mathematics, and a roadmap for future progress, by leading AI-for-math researchers Kaiyu Yang, Gabriel Poesia, Jingxuan He, Wenda Li, Kristin Lauter, Swarat Chaudhuri, and Dawn Song. Excellent, detailed, this is the survey paper to reference for the next while!
• https://marginalrevolution.com/marginalrevolution/2025/01/africa-facts-of-the-day.html "Africa is now experiencing more conflicts than at any point since at least 1946". Prediction: more immigration from African countries.
• what are the current military drones models?
  • https://www.aerotime.aero/articles/25712-worlds-best-military-drones
  • https://www.ssbcrack.com/2024/06/top-military-drones.html
  • https://inphoenixaviation.com/top-10-military-drones-ruling-the-skies-in-2024/
  • countries that make long-range drones (basically unmanned planes): USA, UK, France, China, Turkey, Russia
• https://www.construction-physics.com/p/morris-chang-and-the-origins-of-tsmc TSMC founder Morris Chang's autobiography; lots of false starts along the way. TSMC's big innovation was being the world's first foundry. They did not start with the latest and best equipment; but they were the first to offer semiconductor manufacturing as a service, and Asian semi manufacturers were already more reliable at quality than US ones (going back to the 60's! wonder what's up with that.)
• https://www.dwarkeshpatel.com/p/notes-on-china Dwarkesh Patel visits China.
  • surprising-to-me claim that there's no crime (Chinese friend claims much the opposite!)
  • "The biggest surprise from talking to Chinese VCs people at AI labs was how capital constrained they felt."
• https://en.m.wikipedia.org/wiki/The_Lighthouse_in_Economics back in 1974 Ronald Coase pointed out that lighthouses, long a prototypical example of a public good, were actually privately provided in England during the 17th-19thc. Critics said they went out of business, which proved the market couldn't actually incentivize lighthouse production; Coase replied that the British government made a policy choice to take over the lighthouse industry.
• http://www.daviddfriedman.com/Academic/Property/Property.html David Friedman argues that property rights come to be as a result of Schelling points.
  • if self-interested people are negotiating (say, dividing up a pie), obviously everybody will want the most for themselves; but we also have an interest in eventually settling the negotiation, saving ourselves time and effort (and gaining safety, if the "negotiation" is violent).
  • If one arrangement stands out as "natural" or unique, it can be a Schelling point for where to stop negotiating and accept the arrangement. "Push back if they ask for more than the Schelling point, acquiesce if they ask for no more than the Schelling point" is a stable strategy. (analogous to "contrite Tit for Tat" which performs very well in evolutionary game theory experiments, though Friedman doesn't mention that.)
  • the status quo ante always makes a great Schelling point; laws, contracts, and other common-knowledge establishment of who has a right to what, can become stably self-enforcing even without a formal enforcement mechanism. (eg there is no world government but national borders are usually respected.)
  • the fact that people can, empirically, control their own bodies much more easily than other people's bodies, and can better defend property they can hide and territory they live in, than objects and land not literally in their current possession, also makes concepts like self-ownership and property ownership "natural" Schelling points.
    • though of course the exact boundaries of how property rights work are not given a priori and different societies can define them differently.
• https://sense-nets.xyz/ proposals for better science social media networks
• https://marginalrevolution.com/marginalrevolution/2024/12/the-cows-in-the-coal-mine.html are we neglecting H5N1?
• https://www.noahpinion.blog/p/why-japan-opened-itself-up-to-immigration Noah Smith argues that Japan tried being ethnically homogeneous, found it couldn't (due to labor shortages), and has allowed big increases in immigration, with the approval of most Japanese voters.
• https://zhengdongwang.com/2024/12/29/2024-letter.html  "The model does the eval": as soon as we come up with an evaluation for an AI capability and a dataset to train on, human ingenuity will find ways to make the AI hit the benchmark. If it's not model scaling, it'll be inference-time compute, or mixture-of-experts, or something else.
  • AI progress, like Moore's Law progress, isn't due to a single technological innovation. Once an industry has a moving quantitative target and a strong economic incentive (and social expectation) to keep hitting that moving target, it'll develop multiple technologies with overlapping S-curves that keep improving performance, potentially for a very long time.
• https://ontheones.wordpress.com/2019/06/29/on-denpa-a-guest-article-by-kenji-the-engi/ Denpa, the anime genre that Neon Genesis Evangelion was from, was prevalent in the 1990's and early 2000's; these were disturbing, experimental meditations on the theme of social misfits who retreated from reality.
• https://www.lesswrong.com/posts/KSguJeuyuKCMq7haq/is-vnm-agent-one-of-several-options-for-what-minds-can-grow [LW · GW] [[Anna Salamon]] asks if "utility optimizers" are what all sufficiently "smart" minds will end up being, or if there are other options
• https://trevorklee.com/want-to-reverse-aging-try-reversing-graying-first/ Trevor Klee on reversing gray hair. There are isolated case studies of people whose gray hair has regained pigmentation; lots of these are associated with the use of immunosuppressants, which suggests that something in the aged immune system may be attacking the melanocytes (or their precursors) which produce hair pigmentation.
• https://pmc.ncbi.nlm.nih.gov/articles/PMC10461778/  Can you make an organoid of hairy skin? Yes you can!

• https://asteriskmag.com/issues/08/the-biggest-community-development-program-youve-never-heard-of Clara Collier looks into the history of a giant attempt to improve India's agricultural productivity in the 1950's-60's, by letting village leaders ask for what their village needs most, while the org would provide technological know-how to solve their problems. It worked great when founder Albert Mayer was running it; not so much when the Indian government tried to scale up nationally. Mostly because of common scale-up issues: difficulty finding talented staff, too big for the founder to personally go to all the villages and fix problems, etc.

  • today, global development experts would consider relying on village "leaders" to be an inherently biased approach; these would invariably be male, high-caste, and relatively rich, and the rest of the village wouldn't necessarily buy into what they proposed.
  • despite this flaw, it worked fine in the pilot, because while Mayer didn't have modern egalitarian language to describe it, he was de facto insisting that village discussions included all sorts of people. But when he was no longer micromanaging the program and using his own elbow grease to fix problems, the explicit/formal protocol of the program naturally devolved into "only village elites get a say" and villages indeed failed to follow through on the proposed reforms/improvements.

   

Thanks for writing this. I very much agree that we should have more work on actual plans for short timelines - even if it's only guessing what vague plans orgs and governments are likely to create or follow by default.

Arguments for short timelines are speculative. So are arguments for longer timelines. It seems like the reasonable conclusion is that both are possible. It would seem urgent to have a plan for short timelines.

I also think that plan should be collaborative. So here's some of my contribution, framed as a suggestion for improvement, per your invitation:

This plan only briefly addresses actual alignment techniques. Plans should probably address it more carefully. Control only works until it fails, and CoT  transparency and eval measures only work if we stop deploying capable, scheming models. As Buck argues [LW(p) · GW(p)], and you imply, we might well see powerful models that are obviously scheming, but see orgs (or government(s)) press on because of race dynamics.

So it seems like alignment is a key piece of the puzzle even by the time we reach human-researcher level. Human researchers have agency and can solve novel problems. Researcher systems may solve the novel problem "so what do my goals really imply"? Having less agency and flexible problem-solving seems like it would severely hamper capabilities, creating a large alignment tax that decision-makers will reject under race dynamics.^[1]

Then, a more detailed but still sketchy proposal for an  alignment plan for short timelines:

I agree that RLHF and RLAIF (and now Deliberative Alignment) are working better than expected. Perhaps they'll be adequate to get us to the point where your predictions stop. But I doubt they get us that far. At the least, this probably deserves a good bit more analysis. That's what I'm trying to do in my work: anticipate how alignment techniques will work (or not work) as LLMs are made smarter and more agentic. I summarize some likely alignment approaches for language model agents here [AF · GW], and am working on an updated list of techniques that are obvious and low-tax enough that they're likely to actually be used.

The alignment target those techniques aim at is also crucially important.

As you say:

For example, making models jailbreak-resistant could decrease their corrigibility. [...] I’m currently unsure which training techniques you would want to use for agentic long-horizon internal deployments, e.g. because jailbreaks might be less relevant but taking non-power-seeking actions is much more relevant.

I'm also unsure.

I think this issue is why agent foundations type thinkers usually don't have a plan for short timelines; they assume that in short timelines, we die because we don't have the right alignment target or a way to specify them precisely in the face of strong optimization. We are building increasingly powerful AI with its own goals (like resisting jailbreaks and being "helpful, harmless, and honest" by some definition and training procedure). When those models become fully agentic and strongly optimize against those goals, we may find we didn't define them quite carefully enough. (I'd say strong optimization on the goals of current LLMs would probably doom us, but at the least it seems we should have a strong argument that it wouldn't).

I think those agent foundations incorrectly assume that corrigibility is anti-natural. That's only true when you try to employ it as an add-on to some other primary goal instead of as a singular alignment target [LW · GW]  Thus I think the obvious decision is to preserve corrigibility by dropping the alignment goal of internal ethics (like refusing harmful information requests) and emphasizing strict instruction-following, to preserve corrigibility. By this argument, Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW].

Current alignment techniques directed at that corrigibility/IF might well get us safely to AGI levels that can invent and improve alignment techniques, and thereby get us to aligned ASI.

But this leaves aside the pressure to actually deploy your AGI, and thereby make a ton of money. If you're selling its services to external users, you're going to want it to refuse harmful instructions from that lower tier of authorization. That reduces corrigibility if you do it through training, as you note. I don't know how to balance those goals; having the authorized superuser give instructions to refuse harmful requests is the obvious route, but it's hard to say if that's adequate in practice.

This also risks misuse when those systems are stolen or replicated. If we solve alignment, do we die anyway? [LW · GW] After that discussion, I think the answer is no, but only if some government or coalition of them wisely decide to limit AGI proliferation - and offset the resultant global outrage by sharing the massive benefits from AGI with everyone.

That addresses the other limitation of this plan - it only gets us to the first aligned AGI, without addressing what happens when other actors steal or reproduce it. I think a complete plan needs to include some way we don't die as a result of AGI proliferation to bad actors.

1. ^{^}

   The claim that highly useful AI systems will be capable of solving novel problems, and that narrowing their scope or keeping them myopic will be a high alignment tax is load-bearing. I haven't produced a really satisfactory compact argument for this. The brief argument is that humans achieve our competence by using our general intelligence (which LLMs copy) to solve novel problems and learn new skills. The fastest route to AGI is to expand LLMs to do those things very similarly to how humans do them. Creating specialist systems requires more specialized work for each domain, and human debugging/engineering when systems can't solve the new problems that domain entails. More in my argument for short timelines [LW(p) · GW(p)] and in Capabilities and alignment of LLM cognitive architectures [LW · GW]

But I'm approx 80% confident that that's the world we're in! I don't want to just give up in 80% of cases!

I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it. I don't think it's hopeless, just that the plans would be rather differently shaped.

The realistic answer for decision making purposes (not epistemic purposes) is mostly hope that offense-defense balances are good enough to prevent the end of the world, combined with accepting somewhat larger risk of misalignment to prevent open source from being bad.

To be kind of blunt, any scenario where AI progress is algorithm dominated and a world where basically everyone can train superintelligence without nations being able to control it and a world where timelines are short is a world where governance of AI is more or less useless, and alignment becomes a little more dicey, so we should mostly ignore such worlds for utility purposes (they are valid for prediction/epistemic purposes).

Yeah, I think the crux is precisely this, in which I disagree with this statement below, mostly because I think instruction following/corrigibility is both plausibly easy in my view, and also removes most of the need for value alignment.

"The AI's values must be much more aligned in order to be safe outside the text domain"

Try to apply ability to stop to the process of not-doing

amazing. i just need to turn my tendency towards deconstruction inwards. :p

In my hypothesis, scrolling is a bad way to rest, because it actually eats up a lot of cognitive power

agreed. i archived my twitter last year, but alas, i keep checking lesswrong