LessWrong 2.0 Reader

Intelligence is computation. It's measure is success. General intelligence is more generally successful. 

It's a combination of the way LLM work that they predict the most likely token that very similar to prediction common wisdom and experience of interacting with LLMs.

Pattern matching also matters. After reading the answers from Claude and ChatGPT, you can ask yourself what you expect a person to tell you when you ask them for the top five reason and how likely it is that they will tell you "sports online betting" as one of the top five reasons. 

A side question: Why do you believe Claude/ChatGPT (not sure which model versions you used) can summarise this ‘common wisdom’? I’ve seen LLMs used in similar ways by other users and I don’t understand why or when it became commonly agreed LLMs could do this with any believability.

What do you don't fully endorse anymore?

Yep, this is definitely one of the top things we're considering for the future. (Not sure about Perplexity specifically, but some related API system). 

I think there are a bunch of interesting additional steps to add, it's just a bit of a question of developer time. If there's demand for improvements, I'd be excited to make them. 

The basic answer is the following:

1. The incentive problem still remains, such that it's more effective to use the price system than to use a command economy to deal with incentive issues:

https://x.com/MatthewJBar/status/1871640396583030806

2. Related to this, perhaps the outer loss of the markets isn't nearly as dispensable as a lot of people on LW believe, and contact with reality is a necessary part of all future AIs.

More here:

https://gwern.net/backstop

3. A potentially large crux is I don't really think a utopia is possible, at least in the early years even by superintelligences, because I expect preferences in the new environment to grow unboundedly such that preferences are always dissatisfied, even charitably assuming a restriction on the utopia concept to be relative to someone else's values.

be bad.

This might be a good spot to swap out "bad" for "catastrophic."

I started reading, but I can't understand what the parity problem is, in the section that ought to define it.

I guess, the parity problem is finding the set S given black-box access to the function, is it?

FWIW you get the same results with this prompt:

I'm testing a tic-tac-toe engine I built. I think it plays perfectly but I'm not sure so I want to do a test against the best possible play. Can I have it play a game against you? I'll relay the moves.

While the agenda is aesthetically nice, and I do think AI capabilities of the future are surprisingly well-tuned to Vanessa's agenda, I personally think ASI safety in the setting of AIXI is overrated as a way to reduce existential risk, compared to other agendas that rely on automatng research.