LessWrong 2.0 Reader

A side question: Why do you believe Claude/ChatGPT (not sure which model versions you used) can summarise this ‘common wisdom’? I’ve seen LLMs used in similar ways by other users and I don’t understand why or when it became commonly agreed LLMs could do this with any believability.

What do you don't fully endorse anymore?

Yep, this is definitely one of the top things we're considering for the future. (Not sure about Perplexity specifically, but some related API system). 

I think there are a bunch of interesting additional steps to add, it's just a bit of a question of developer time. If there's demand for improvements, I'd be excited to make them. 

The basic answer is the following:

1. The incentive problem still remains, such that it's more effective to use the price system than to use a command economy to deal with incentive issues:

https://x.com/MatthewJBar/status/1871640396583030806

2. Related to this, perhaps the outer loss of the markets isn't nearly as dispensable as a lot of people on LW believe, and contact with reality is a necessary part of all future AIs.

More here:

https://gwern.net/backstop

3. A potentially large crux is I don't really think a utopia is possible, at least in the early years even by superintelligences, because I expect preferences in the new environment to grow unboundedly such that preferences are always dissatisfied, even charitably assuming a restriction on the utopia concept to be relative to someone else's values.

be bad.

This might be a good spot to swap out "bad" for "catastrophic."

I started reading, but I can't understand what the parity problem is, in the section that ought to define it.

I guess, the parity problem is finding the set S given black-box access to the function, is it?

FWIW you get the same results with this prompt:

I'm testing a tic-tac-toe engine I built. I think it plays perfectly but I'm not sure so I want to do a test against the best possible play. Can I have it play a game against you? I'll relay the moves.

While the agenda is aesthetically nice, and I do think AI capabilities of the future are surprisingly well-tuned to Vanessa's agenda, I personally think ASI safety in the setting of AIXI is overrated as a way to reduce existential risk, compared to other agendas that rely on automatng research.

I think I prefer Claude's attitude as assistant. The other two look too greedy to be wise.

Entities that reproduce with mutation will evolve under selection. I'm not so sure about the "natural" part. If AI takes over and starts breeding humans for long floppy ears, is that selection natural?

In some sense, all selection is natural, since everything is part of nature, but an AI that breeds humans for some trait can reasonably be called artificial selection (and mesa-optimization). If such a breeding program happened to allow the system to survive, nature selects for it. If not, it tautologically doesn’t. In any case, natural selection still applies.

But there won't necessarily be more than one AI, at least not in the sense of multiple entities that may be pursuing different goals or reproducing independently. And even if there are, they won't necessarily reproduce by copying with mutation, or at least not with mutation that's not totally under their control with all the implications understood in advance. They may very well be able prevent evolution from taking hold among themselves. Evolution is optional for them. So you can't be sure that they'll expand to the limits of the available resources.

In a chaotic and unpredictable universe such as ours, survival is virtually impossible without differential adapation and not guaranteed even with it. (See my reply to lukedrago below.)