LessWrong 2.0 Reader

That's extremely cool, seems worth adding to the main post IMHO!

Yes, thank you, it's less than 1000 parameters for both

We have been able to scale to 79% accuracy on a balanced dataset of n119 and non-n119, with networks each less than three convolution layers and less than 1000 neurons, compared to pure deep-learning which does 92% on 1000 parameters and three convolution layers

Is the "1000 parameters" a typo, should it be "1000 neurons"? Otherwise, this would be a strange comparison (since 1000 parameters is a much smaller network than 1000 neurons)

I agree that contrarians 'round these parts are often wrong more often than academic consensus, but the success of their predictions about AI, crypto, and COVID prove to me its still worth listening to them, trying to be able to think like them, and probably taking their investment advice. That is, when they're right, they're right big-time.

Why DEX though? Like, conceptually it's absolutely unpredictable, this is one of the most useful scores in most TTRPGs.

So the first image is based on AI control, which is indeed part of their strategies, and you could see constructability as mainly leading to this kind of strategy applied to plain code for specific subtasks. It's important to note constructability itself is just a different approach to making understandable systems.

The main differences are :

1. Instead of using a single AI, we use many expert-like systems that compose together which we can see the interaction of (for instance, in the case of a go player, you would use KataGo to predict the best move and flag moves that lost the game, another LLM to explain the correct move, and another one to factor this explanation into the code)

2. We use supervision, both automatic and human, to overview the produced code and test it, through simulations, unit tests, and code review, to ensure the code makes sense and does its task well.

it seems to me that disentangling beliefs and values are important part of being able to understand each other

and using words like "disagree" to mean both "different beliefs" and "different values" is really confusing in that regard

What does it mean to claim that these people are contrarians?

Is there a consensus position at all? For any existing policy, you could claim that there is some kind of centrist compromise that it's a good policy, so people who propose changing policy, like Hanson and Caplan, are defying that compromise. But there is not really any explicit consensus goal of most policies, so claiming existing institutions are a bad compromise because they pursue multiple goals and separating those goals is not in defiance of any consensus. Caplan, Hanson, and Sailer are offensive because they feel we should try to understand the world and try steer it. They may be wrong, but the people opposed to them rarely offer an opposing position, but are rather opposed to any position. It seems to me that the difference between true and false is much smaller than the gap between argument and pseudoscience. Maybe Sailer is wrong, but the consensus position that he is peddling pseudoscience is much more wrong and much more dangerous.

Sailer rarely argues for genetic causes, but leaves that to the psychologists. He believes it and sometimes he uses the hypothesis, but usually he uses the hypotheses 1-4 that Turkheimer, Harden, and Nisbett concede. Spelling out the consequences of those claims is enough to unperson him. Maybe he's wrong about these, but he's certainly not claiming to be a contrarian. And people who act like these are false rarely acknowledge an academic consensus. Or compare Jay: it's very hard to distinguish genetic effects from systemic effects, so when Jay argues that racial IQ gaps aren't genetic, he is (explicitly!) arguing that they are caused by racial differences in parenting. Sailer often claims this (he thinks it's half the effect), but people hate this just as much as anything else he says. Calling him a contrarian and focusing attention on one claim seem like an attempt to mislead.

That is a very clear example, but I think something similar is going on in the rest. Guzey seems to have gone overboard in reaction to Matthew Walker's book Why We Sleep. Did that book represent a consensus? I don't know, but it was concrete enough to be wrong, which seems to me much better than an illusion of a consensus.

Is the "cure cancer goal ends up as a nuke humanity action" hypothesis valid and backed by evidence?

My understanding is that the meaning of the "cure cancer" sentence can be represented as a point in a high-dimensional meaning space, which I expect to be pretty far from the "nuke humanity" point. 

For example "cure cancer" would be highly associated with saving lots of lives and positive sentiments, while "nuke humanity" would have the exact opposite associations, positioning it far away from "cure cancer".

A good design might specify that if the two goals are sufficiently far away they are not interchangeable. This could be modeled in the AI as an exponential decrease of the reward based on the distance between the meaning of the goal and the meaning of the action.

Does this make any sense? (I have a feeling I might be mixing concepts coming from different types of AI)

Haven't read the entire post, but my thoughts on seeing the first image: Pretty sure this is priced into Anthropic / Redwood / OpenAI cluster of strategies where you use an aligned boxed (or 'mostly aligned) generative LLM-style AGI to help you figure out what to do next.