LessWrong 2.0 Reader

Correction: "is that you experienced was real" -> "is that what you experienced was real"

> Now I knew how to not trigger those defense mechanisms.
The linked video looks like rhetorical aikido. If that's what you're talking about, link it. If you meant something else, what did you learn to do?

I've often heard it said that doing RL on chain of thought will lead to 'neuralese' (e.g. most recently in Ryan Greenblatt's excellent post on the scheming). This seems important for alignment. Does anyone know of public examples of models developing or being trained to use neuralese?

A very heavy and dense body on an elliptical orbit that touches the Sun's surface at each perihelion would collect sizable chunks of the Sun's matter. The movement of matter from one star to another nearby star is a well-known phenomenon.

When the body reaches aphelion, the collected solar matter would cool down and could be harvested. The initial body would need to be very massive, perhaps 10-100 Earth masses. A Jupiter-sized core could work as such a body.

Therefore, to extract the Sun's mass, one would need to make Jupiter's orbit elliptical. This could be achieved through several heavy impacts or gravitational maneuvers involving other planets.

This approach seems feasible even without ASI, but it might take longer than 10,000 years. 

• The interaction with users used for o1 (where the AI thinks for a while prior to sending a response) is consistent with neuralese.
• RL adding substantial additional capabilities means there might be enough RL for this to work.
• o3 is a substantial leap over o1 seemingly.

(Based on public knowledge, it seems plausible (perhaps 25% likely) that o3 uses neuralese which could put it in this category.)

What public knowledge has led you to this estimate?

a particular technique doesn’t immediately solve a problem

I remember a story that got coverage on the state radio in New Zealand years ago.  It said that multiple people have parts of the solution to some problem, and there is progress when there is an accident that introduces them to each other.  There was a book about it, but I'm failing to find the details.

I've put it on our list of possible future pages, and added some of the things from that doc to our Funders page. Thanks Chris!

I've read and appreciate the entire series, but this was the chapter that really stood out to me. I think that improved systems of voluntary contracts with decentralized enforcement might be our best hope out of the multiple Pareto-suboptimal races-to-doom that humanity seems stuck in.

I think this is potentially a place where we could use our biggest risk factor (AI) to save us before it dooms us. Could we manage to use AI to speed-run the invention and deployment of such subsidiarity governance systems? I think the biggest challenge to this is how fast it would need to move in order to take effect in time. For a system that needs extremely broad buy-in from a large number of heterogenous actors, speed of implementation and adoption is a key weak point.

Imagine though that a really good system was designed which you felt confident that a supermajority of humanity would sign onto if they had it personally explained to them (along with a convincing explanations of the counterfactuals). How might we get this personalized explanation accomplished at scale? Welll, LLMs are still bad at certain things, but giving personalized interactive explanations of complex legal docs seems well within their near-term capabilities. It would still be a huge challenge to actually present nearly everyone on Earth with the opportunity to have this interaction, and all within a short deadline... But not beyond belief.

This is the first non-destructive non-coercive pivotal act that I've considered plausibly sufficient to save us from the current crisis. I've thought up and heard of a lot of different bad plans over the past ten years, so it's really nice to finally find a good one!

Existential Hope indeed!

Some small corrections/additions to my section ("Altair agent foundations"). I'm currently calling it "Dovetail research". That's not publicly written anywhere yet, but if it were listed as that here, it might help people who are searching for it later this year.

Which orthodox alignment problems could it help with?: 9. Humans cannot be first-class parties to a superintelligent value handshake

I wouldn't put number 9. Not intended to "solve" most of these problems, but is intended to help make progress on understanding the nature of the problems through formalization, so that they can be avoided or postponed, or more effectively solved by other research agenda. 

Target case: worst-case

definitely not worst-case, more like pessimistic-case

Some names: Alex Altair, Alfred Harwood, Daniel C, Dalcy K

Add "José Pedro Faustino"

Estimated # FTEs: 1-10

I'd call it 2, averaged throughout 2024.

Some outputs in 2024: mostly [LW · GW] exposition [LW · GW] but it’s early days

Basically right; I'd add this post [LW · GW] and this post [LW · GW].

Richard said "I don't think my priors on that are very different from yours but the thing that would have made this post valuable for me is some object-level reason to upgrade my confidence in that." He didn't say it'd be a longterm project, I think he just meant he didn't change his beliefs about it due to thist post.