LessWrong 2.0 Reader

They spend more time thinking about the concrete details of the trip, not because they know the trip is happening soon, but because some think the trip is happening soon. Disagreement and attention to concrete details is driven by only some people saying that the current situation looks like, or is starting to look like the event occurring according to their interpretation. If the disagreement had happened at the start, they would soon have started using different words.

In the New York example, it could be that when someone says “Guys, we should really buy those Broadway tickets. February is next month already.” they prompt the response “What? I thought we were going in March!”, hence the disagreement.

Likewise, in the case of AGI, some people’s alarm bells are currently going off for AGI, while others are saying more capabilities are required to satisfy their interpretation. However, whether or not AGI exists depends only marginally on any one person’s interpretation. Words are a communicative tool and therefore depend on others’ interpretations.

The meaning of words doesn’t fall out of the sky. It doesn’t pass through a membrane from another reality. We collectively—and often unconsciously—define it. For example, “What is intelligence?” is a question of how that word is in practice interpreted by other people. “How should it be interpreted (according to me personally)?” is a valid but different question.

What seems to be happening with AGI specifically is that people at one point latched on to the concept of AGI, thinking that their interpretation was virtually the same as those of others because of its lack of definition. Again, if they had disagreed with the definition at the start, they would have used a different word altogether. Now that some people are claiming that AGI is here or soon here, it turns out that those interpretations do in fact differ. The most obnoxious cases are when people disagree with their own past interpretation once that interpretation is threatened to be satisfied, on the basis of some deeper, undefined intuition (or in the case of OpenAI and Microsoft, ulterior motives). This of course is known as “moving the goalposts”.

Once upon a time, not that long ago, AGI was interpreted by many as “it can beat anyone at chess”, or “it can beat anyone at go” or “it can pass the Turing test”. We are there now, according to those interpretations.
 

Although as I note elsewhere I’m starting to have some ideas of how something with elements of this might have a chance of working.

I've missed where you discussed this. Does anyone have a link or can anyone expound?

Makes sense. Perhaps we'll know more when o3 is released.  If the model doesn't offer a summary of CoT it makes neuralese more likely. 

Correction: "is that you experienced was real" -> "is that what you experienced was real"

> Now I knew how to not trigger those defense mechanisms.
The linked video looks like rhetorical aikido. If that's what you're talking about, link it. If you meant something else, what did you learn to do?

I've often heard it said that doing RL on chain of thought will lead to 'neuralese' (e.g. most recently in Ryan Greenblatt's excellent post on the scheming). This seems important for alignment. Does anyone know of public examples of models developing or being trained to use neuralese?

A very heavy and dense body on an elliptical orbit that touches the Sun's surface at each perihelion would collect sizable chunks of the Sun's matter. The movement of matter from one star to another nearby star is a well-known phenomenon.

When the body reaches aphelion, the collected solar matter would cool down and could be harvested. The initial body would need to be very massive, perhaps 10-100 Earth masses. A Jupiter-sized core could work as such a body.

Therefore, to extract the Sun's mass, one would need to make Jupiter's orbit elliptical. This could be achieved through several heavy impacts or gravitational maneuvers involving other planets.

This approach seems feasible even without ASI, but it might take longer than 10,000 years. 

• The interaction with users used for o1 (where the AI thinks for a while prior to sending a response) is consistent with neuralese.
• RL adding substantial additional capabilities means there might be enough RL for this to work.
• o3 is a substantial leap over o1 seemingly.

(Based on public knowledge, it seems plausible (perhaps 25% likely) that o3 uses neuralese which could put it in this category.)

What public knowledge has led you to this estimate?

a particular technique doesn’t immediately solve a problem

I remember a story that got coverage on the state radio in New Zealand years ago.  It said that multiple people have parts of the solution to some problem, and there is progress when there is an accident that introduces them to each other.  There was a book about it, but I'm failing to find the details.

I've put it on our list of possible future pages, and added some of the things from that doc to our Funders page. Thanks Chris!