LessWrong 2.0 Reader

I use capitalism in a manner mutually exclusive with slave labor because it requires self-ownership.

This seems like a sort of definitional gimbal lock; it makes it harder to describe the world because two potentially-separate degrees of freedom are collapsed into one. While I'm reluctant to argue definitions [LW · GW], I think it's worth using terms in ways that allow us to describe the world in more detail than ones that collapse distinctions.

I expect to see this usage of "capitalism" not in history or economics, but in the sort of political doctrine where it's intended to lock those concepts together; to imply that capital markets and individual freedom are either the same thing, or closely related — more closely, I think, than history and contemporary events really support.

It would seem weird to me, for instance, to claim that a publicly-traded company that is discovered to have done something to violate individual freedom is thereby no longer a participant in a capitalist economy. The New York Stock Exchange doesn't ask "does this company infringe individual freedoms anywhere in the world?" before letting a company be listed. (To be clear, I'm not proposing that it should; I'm saying that it's useful to talk about "participation in a capital market economy" and "fully respecting some set of individual freedoms" as distinct axes.)

(For what it's worth, I think "self-ownership" is a pretty odd expression, because one of the central traits of ownership is that it can be transferred, and one of the central traits of selfhood is that it cannot. Your relation to yourself is distinct from property ownership in that you can sell any piece of your property, but you cannot sell your self; no matter what obligations you may have signed up for, you always retain possession of your self.)

Test time compute is applied in-context, so it's very worthwhile to scale, getting better at better at solving a particular problem, to the extent that no amount of pretraining [LW(p) · GW(p)] would be able to match with only modest test-time compute.

Pr(Ai)2R is at least partially funded by Good Ventures/OpenPhil

This post resonated with me when it came out, and I think its thesis only seems more credible with time. Anthropic's seminal "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" (the Golden Gate Claude paper) seems right in line with these ideas. We can make scrutable the inscrutable as long as the inscrutable takes the form of something organized and regular and repeatable.

This article gets bonus points for me for being succinct and while still making its argument clearly.

How/does this square with https://arxiv.org/abs/1902.07404?
IIUC, Gödel's Second Incompleteness Theorem was overinterpreted, and a different operationalization of consistency is provable.

I talked to Mihály Bárász about that, and he didn't think it was crazy.

Insufficiently catchy

I'm glad you're doing this, and I support many of the ideas already suggested. Some additional ideas:

• Interview program. Work with USAISI or UKAISI (or DHS/NSA) to pilot an interview program in which officials can ask questions about AI capabilities, safety and security threats, and national security concerns. (If it's not feasible to do this with a government entity yet, start a pilot with a non-government group– perhaps METR, Apollo, Palisade, or the new AI Futures Project.)
• Clear communication about RSP capability thresholds. I think the RSP could do a better job at outlining the kinds of capabilities that Anthropic is worried about and what sorts of thresholds would trigger a reaction. I think the OpenAI preparedness framework tables are a good example of this kind of clear/concise communication. It's easy for a naive reader to quickly get a sense of "oh, this is the kind of capability that OpenAI is worried about." (Clarification: I'm not suggesting that Anthropic should abandon the ASL approach or that OpenAI has necessarily identified the right capability thresholds. I'm saying that the tables are a good example of the kind of clarity I'm looking for– someone could skim this and easily get a sense of what thresholds OpenAI is tracking, and I think OpenAI's PF currently achieves this much more than the Anthropic RSP.)
• Emergency protocols. Publishing an emergency protocol that specifies how Anthropic would react if it needed to quickly shut down a dangerous AI system. (See some specific prompts in the "AI developer emergency response protocol" section here). Some information can be redacted from a public version (I think it's important to have a public version, though, partly to help government stakeholders understand how to handle emergency scenarios, partly to raise the standard for other labs, and partly to acquire feedback from external groups.)
• RSP surveys. Evaluate the extent to which Anthropic employees understand the RSP, their attitudes toward the RSP, and how the RSP affects their work. More on this here [LW(p) · GW(p)].
• More communication about Anthropic's views about AI risks and AI policy. Some specific examples of hypothetical posts I'd love to see:
  • "How Anthropic thinks about misalignment risks"
  • "What the world should do if the alignment problem ends up being hard"
  • "How we plan to achieve state-proof security before AGI"
  • Encouraging more employees to share their views on various topics, EG Sam Bowman's post [LW · GW].
• AI dialogues/debates. It would be interesting to see Anthropic employees have discussions/debates from other folks thinking about advanced AI. Hypothetical examples:
  • "What are the best things the US government should be doing to prepare for advanced AI" with Jack Clark and Daniel Kokotajlo.
  • "Should we have a CERN for AI?" with [someone from Anthropic] and Miles Brundage.
  • "How difficult should we expect alignment to be" with [someone from Anthropic] and [someone who expects alignment to be harder; perhaps Jeffrey Ladish or Malo Bourgon].

More ambitiously, I feel like I don't really understand Anthropic's plan for how to manage race dynamics in worlds where alignment ends up being "hard enough to require a lot more than RSPs and voluntary commitments."

From a policy standpoint, several of the most interesting open questions seem to be along the lines of "under what circumstances should the USG get considerably more involved in overseeing certain kinds of AI development" and "conditional on the USG wanting to get way more involved, what are the best things for it to do?" It's plausible that Anthropic is limited in how much work it could do on these kinds of questions (particularly in a public way). Nonetheless, it could be interesting to see Anthropic engage more with questions like the ones Miles raises here. 

Something like "We have mapped out the possible human-understandable or algorithmically neat descriptions of the network's behavior sufficiently comprehensively and sampled from this space sufficiently comprehensively to know that the probability that there's a description of its behavior that is meaningfully shorter than the shortest one of the ones that we've found is at most $ϵ$.".

As a grad student in neuroscience I got the opportunity to sit in on some forensic histology, and it was really fascinating. Occasionally you can figure out quite insightful things about cause of death from looking at brain samples under a microscope. Other times you get a simple "yep, looks like this sample approximately agrees with the estimated time of death, nothing unusual here."

Even if saving money through AGI converts 1:1 into money after the singularity, it will probably be worth less in utility to you:

1. You'll probably be able to buy planets post-AGI for the price of houses today. More generally your selfish and/or local and/or personal preferences will be fairly easily satisfiable even with small amounts of money, or to put it in other words, there are massive diminishing returns.

No one will be buying planets for the novelty or as an exotic vacation destination. The reason you buy a planet is to convert it into computing power, which you then attach to your own mind. If people aren't explicitly prevented from using planets for that purpose, then planets are going to be in very high demand, and very useful for people on a personal level.