LessWrong 2.0 Reader

I agree! I contributed to and endorse this Corrigibility plan by Max Harms (MIRI researcher): Corrigibility as Singular Target [? · GW]

(See also posts by Seth Herd)

I think CAST offers much better safety under higher capabilities and more agentic workflows.

In regards to:

Give more access to orgs like Redwood, Apollo, and METR (I don't know how much access you currently give, but I suspect the globally-optimal thing would be to give more)

I agree, and I also think that this would be better implemented by government AI Safety Institutions.

Specifically, I think that AISIs should build (and make mandatory the use of) special SCIF-style [AF · GW] reading rooms where external evaluators would be given early access to new models. This would mean that the evaluators would need permission from the government, rather than permission from AI companies. I think it's a mistake to rely on the AI companies voluntarily giving early access to external evaluators.

I think that Anthropic could make this a lot more likely to happen if they pushed for it, and that then it wouldn't be so hard to pull other major AI companies into the plan.

When you say "true probability", what do you mean?

The current hypotheses I have about what you mean are (in part non-exclusive):

1. You think some notion of objective, non-observer dependent probability makes sense, and that's the true probability.
2. You do not think "true probability" exists, you are referencing to it to say the market price is not anything like that.
3. You define "true probability" a probability that observers contextually agree on (like a coin flip observed by humans who don't know the thrower).

Reminder that this is happening tonight! Last one of the year :)

Also tonight's might be the last one for a while, Lighthaven is getting rented out for MATS in the New Year, so we might be needing to find a new space. I'll be adding folks who come tonight to a Signal group for more chat/updates (you can also text me to be added).

Yes, Bryan Caplan is not noticeably differentiated from other libertarian economists.

I'd be curious to hear if you see something deeper or more totalising in these people?

My answer might contain a frustratingly small amount of detail, because answering your question properly would require a top-level post for each person just to get the main ideas across.

Paul Graham is special because he has a proven track record of accurately calibrated confidence. He has an entire system for making progress at unknown unknowns. Much of that system is about knowing what you don't know, which results in him carefully restricting claims about his narrow domain of specialization. However, because that domain of specialization is "startups", its lightcone has already had (what I consider to be) a totalising impact.

Asimov's turned The Decline and Fall of the Roman Empire into his first popular novel. He eventually extended the whole thing into a future competition between psychohistory and a gaia planet. He didn't just create one Dath Ilan. He created two of them (albeit at much lower resolution).

As to the other authors you mention:

• I haven't read enough Greg Egan or Vernor Vinge comment on them.
• Heinlein absolutely has "his own totalising and self-consistent worldview/philosophy". I love his writing, but I just don't agree with him enough for him to make the list. I prefer Saturn's Children (and especially Neptune's Brood) by Charles Stross. Saturn's Children is basically Heinlein + Asimov fanfiction that takes their work in a different direction. Neptune's Brood is its sequel about interstellar cryptocoin markets.
• Clarke was mostly boring to me, except for 3001: The Final Odyssey.
• Neal Stephenson is definitely smart, but I never got the feeling he was trying to mind control me. Maybe that's just because he's so good at it.

The range of capabilities between what can be gained at a reasonable test-time cost and at an absurd cost (but in reasonable time) can remain small, with most improvements to the system exceeding this range, likely to move what could only be obtained at an absurd cost before into the reasonable range. This is true right now (for general intelligence), and it could well remain true until the intelligence explosion.

to hate something is the origin of my work

I like that quote.

Yes! 100%. I too have noticed that stating these outright doesn't work at all. It's also bad for developing one too.

When I'm trying to sell ideas I do so more indirectly than this. The reason I wrote this post is because I felt I did have one, and wanted to verify to myself that this was true.

All the infra for fiat currency exists; I don't see why the AIs would need to reinvent that

Because using an existing medium of exchange (that's not based on the value of a real commodity) involves transferring real wealth to the current currency holders. Instead, they might, for example, start up a new bitcoin blockchain, and use their new bitcoin, rather than transfer wealth to present bitcoin holders.

Maybe they'd use gold, although the current value of gold is mostly due to its conventional monetary value (rather than its practical usefulness, though that is non-zero).

I am focused here on short-term politics in the US, which ordinarily would matter less, if it wasn't likely that world-changing AI would be built in the US, but given that it might, it becomes way more important than normal.