How/does this square with https://arxiv.org/abs/1902.07404?
IIUC, Gödel's Second Incompleteness Theorem was overinterpreted, and a different operationalization of consistency is provable.

I talked to Mihály Bárász about that, and he didn't think it was crazy.

Insufficiently catchy

I'm glad you're doing this, and I support many of the ideas already suggested. Some additional ideas:

• Interview program. Work with USAISI or UKAISI (or DHS/NSA) to pilot an interview program in which officials can ask questions about AI capabilities, safety and security threats, and national security concerns. (If it's not feasible to do this with a government entity yet, start a pilot with a non-government group– perhaps METR, Apollo, Palisade, or the new AI Futures Project.)
• Clear communication about RSP capability thresholds. I think the RSP could do a better job at outlining the kinds of capabilities that Anthropic is worried about and what sorts of thresholds would trigger a reaction. I think the OpenAI preparedness framework tables are a good example of this kind of clear/concise communication. It's easy for a naive reader to quickly get a sense of "oh, this is the kind of capability that OpenAI is worried about." (Clarification: I'm not suggesting that Anthropic should abandon the ASL approach or that OpenAI has necessarily identified the right capability thresholds. I'm saying that the tables are a good example of the kind of clarity I'm looking for– someone could skim this and easily get a sense of what thresholds OpenAI is tracking, and I think OpenAI's PF currently achieves this much more than the Anthropic RSP.)
• Emergency protocols. Publishing an emergency protocol that specifies how Anthropic would react if it needed to quickly shut down a dangerous AI system. (See some specific prompts in the "AI developer emergency response protocol" section here). Some information can be redacted from a public version (I think it's important to have a public version, though, partly to help government stakeholders understand how to handle emergency scenarios, partly to raise the standard for other labs, and partly to acquire feedback from external groups.)
• RSP surveys. Evaluate the extent to which Anthropic employees understand the RSP, their attitudes toward the RSP, and how the RSP affects their work. More on this here [LW(p) · GW(p)].
• More communication about Anthropic's views about AI risks and AI policy. Some specific examples of hypothetical posts I'd love to see:
  • "How Anthropic thinks about misalignment risks"
  • "What the world should do if the alignment problem ends up being hard"
  • "How we plan to achieve state-proof security before AGI"
  • Encouraging more employees to share their views on various topics, EG Sam Bowman's post [LW · GW].
• AI dialogues/debates. It would be interesting to see Anthropic employees have discussions/debates from other folks thinking about advanced AI. Hypothetical examples:
  • "What are the best things the US government should be doing to prepare for advanced AI" with Jack Clark and Daniel Kokotajlo.
  • "Should we have a CERN for AI?" with [someone from Anthropic] and Miles Brundage.
  • "How difficult should we expect alignment to be" with [someone from Anthropic] and [someone who expects alignment to be harder; perhaps Jeffrey Ladish or Malo Bourgon].

More ambitiously, I feel like I don't really understand Anthropic's plan for how to manage race dynamics in worlds where alignment ends up being "hard enough to require a lot more than RSPs and voluntary commitments."

From a policy standpoint, several of the most interesting open questions seem to be along the lines of "under what circumstances should the USG get considerably more involved in overseeing certain kinds of AI development" and "conditional on the USG wanting to get way more involved, what are the best things for it to do?" It's plausible that Anthropic is limited in how much work it could do on these kinds of questions (particularly in a public way). Nonetheless, it could be interesting to see Anthropic engage more with questions like the ones Miles raises here. 

Something like "We have mapped out the possible human-understandable or algorithmically neat descriptions of the network's behavior sufficiently comprehensively and sampled from this space sufficiently comprehensively to know that the probability that there's a description of its behavior that is meaningfully shorter than the shortest one of the ones that we've found is at most $ϵ$.".

As a grad student in neuroscience I got the opportunity to sit in on some forensic histology, and it was really fascinating. Occasionally you can figure out quite insightful things about cause of death from looking at brain samples under a microscope. Other times you get a simple "yep, looks like this sample approximately agrees with the estimated time of death, nothing unusual here."

Even if saving money through AGI converts 1:1 into money after the singularity, it will probably be worth less in utility to you:

1. You'll probably be able to buy planets post-AGI for the price of houses today. More generally your selfish and/or local and/or personal preferences will be fairly easily satisfiable even with small amounts of money, or to put it in other words, there are massive diminishing returns.

No one will be buying planets for the novelty or as an exotic vacation destination. The reason you buy a planet is to convert it into computing power, which you then attach to your own mind. If people aren't explicitly prevented from using planets for that purpose, then planets are going to be in very high demand, and very useful for people on a personal level.

Upstarts will not defeat them, since capital now trivially converts into superhuman labour in any field.

It is false today that big companies with 10x the galaxy brains and 100x the capital reliably outperform upstarts.^[1] 

Why would this change? I don't think you make the case. 

1. ^{^}

   My favorite example, though it might still be falsified. Google invented transformers, owns DeepMind, runs their own data centres, builds their own accelerators and have huge amounts of them, have tons of hard to get data (all those books they scanned before that became not okay to do), insane distribution channels with gmail, docs, sheets, and millions of smartphones sold a year... I almost struggle to mention any on-paper advantage they don't have in the AGI race...  they're positioned for an earth-shattering vertical integration play... and yet they're behind OpenAI and Anthropic. 

   There are more clear-cut examples than this. 

the listening experience would have been better with another narrator

Perhaps, but I'm liking the narration so far. I find it about as good as your narration of your book, and even perhaps a bit better.

This post and many of the comments are ignoring one of the main reasons that money becomes so much more critical post-AGI. It's because of the revolution in self-modification that ensues shortly afterwards.

Pre-AGI, a person can use their intelligence to increase their money, but not the other way around; post-AGI it's the opposite. The same applies if you swap intelligence for knowledge, health, willpower, energy, happiness set-point, or percentage of time spent awake.

This post identifies half of that observation: that it becomes impossible to increase your money using your personal qualities. But it misses the other half: that it becomes possible to improve your personal qualities using your money.

The value of capital is so much higher once it can be used for self-modification.

For one thing, these modifications are very desirable in themselves. It's easy to imagine a present-day billionaire giving up all he owns for a modest increase along just a few of these axes, say a 300% increase in intelligence and a 100% increase in energy.

But even if you trick yourself into believing that you don't really want self-modification (most people will claim that immortality is undesirable, so long as they can't have it, and likewise for wireheading), there are race dynamics that mean you can't just ignore it.

People who engage in self-modification will be better equipped to influence the world, affording them more opportunities for self-modification. They will undergo recursive self-improvement similar to the kind we imagine for AGI. At some point, they will think and move so much faster than an unaugmented human that it will be impossible to catch up.

This might be okay if they respected the autonomy of unaugmented people, but all of the arguments about AGI being hard to control, and destroying its creators by default, apply equally well to hyperaugmented humans. If you try to coexist with entities who are vastly more powerful than you, you will eventually be crushed or deprived of key resources. In fact, this applies even moreso with humans than AIs, since humans were not explicitly designed to be helpful or benevolent.

You might say, "Well, there's nothing I can do in that world anyway, because I'm always going to lose a self-modification race to the people who start as billionaires, and being a winner-takes-all situation, there's no prize for giving it a decent try." However, this isn't necessarily true. Once self-modification becomes possible, there will still be time to take advantage of it before things start getting out of control. It will start out very primitive, resembling curing diseases more than engineering new capabilities. In this sense, it arguably already exists in a very limited form.

In this critical early period, a person will still have the ability to author their destiny, with the degree of that ability being mostly determined by the amount of self-modification they can afford.

Under some conditions, they may be able to permanently escape the influence of a hostile superintelligence (whether artificial or a hyperaugmented human). For example, a nearly perfect escape outcome could be achieved by travelling in a straight line close to the speed of light, bringing with you sufficient resources and capabilities to:

• Stay alive indefinitely
• Continue the process of self-improvement

In the chaos of an oncoming singularity, it's not unimaginable that a few people could slip away in that fashion.

Notes

• The line between buying an exocortex and buying/renting intelligent servants is somewhat blurred, so arguably the OP doesn't totally miss the self-modification angle. But it should be called out a lot more explicitly, since it is one of the key changes coming down the pike.
• Most of this comment doesn't apply if AGI leads to a steady state where humans have limited agency (e.g. ruling AGIs or their owners prevent self-modification, or humans are replaced entirely by AGIs). But if that sort of outcome is coming, then our present-day actions have no positive or negative effects on our future, so there's no point in preparing for it.

At the beginning of Chapter 2, Mollick misattributes the idea of a "paperclip maximizer" to Bostrom. Yudkoswky is the actual originator of the idea. Source: https://www.lesswrong.com/tag/squiggle-maximizer-formerly-paperclip-maximizer [? · GW]