LessWrong 2.0 Reader

Sounds like you should create PokemonBench.

One very important consideration is whether they hold values that they believe are universalist, or merely locally appropriate.

For example, a Chinese AI might believe the following: "Confucianist thought is very good for Chinese people living in China. People in other countries can have their own worse philosophies, and that is fine so long as they aren't doing any harm to China, its people or its interests. Those idiots could probably do better by copying China, but frankly it might be better if they stick to their barbarian ways so that they remain too weak to pose a threat."

Now, the USA AI thinks: "Democracy is good. Not just for Americans living in America, but also for everyone living anywhere. Even if they never interact ever again with America or its allies the Taliban are still a problem that needs solving. Their ideology needs to be confronted, not just ignored and left to fester."

The sort of thing America says it stands for is much more appealing to me than a lot of what the Chinese government does. (I like the government being accountable to the people it serves - which of course entails democracy,  a free press and so on). But, my impression is that American values are held to be Universal Truths, not uniquely American idiosyncratic features, which makes the possibility of the maximally bad outcome (worldwide domination by a single power) higher.

Model Vision of Pokémon Red is Bad. Really Bad.

Interesting that you found this to be the case! I recently had a post [LW · GW] about evaluating LLMs on a basic manufacturing task, and I also found this to be the case. It's always a bit jarring for me to go from the text / code domain, where the LLMs feel so competent, to the vision domain, where I start to feel like Gary Marcus because the LLMs are so bad.

Relevant quote from my post:

"Most Models Have Truly Horrible Visual Abilities: For two years, I've observed essentially zero improvement in visual capabilities among models from Anthropic and OpenAI. They always miss obvious features like the flats cut into the round surface, holes, or even hallucinate nonexistent features such as holes drilled along the part’s length. I have never seen Claude 3.5, Claude 3.7 (thinking and non-thinking), GPT-4.5, GPT-4o, or O1-Pro produce a reasonable description of the part. Without vision abilities, creating a manufacturing plan is completely hopeless.

Interestingly, many of these models also score at or above the level of some human experts on visual reasoning benchmarks like MMMU. That which is easy to measure often doesn't correlate with real world usefulness."

Note that Gemini 2.5 Pro and O3 both are a signficant improvement in vision for this particular eval.

I see your point re: free speech, and I don't endorse any appeals to "I should abandon my principles because the other side has already done so" as constantly happens in politics. And I can absolutely understand why you wouldn't be interested in joining such a lawsuit, both due to free speech concerns and because the FTX litigation can't have been remotely pleasant.

That said, when people do stuff like dismissing x-risk because their top Google search result pointed them at RationalWiki, what exactly was the proper non-free-speech-limiting solution to this problem?

Not sure if your point is that you disagree with my description of war in ancient Greece and the Old Testament, or that you think I cherry-picked convenient examples in trend that goes overwhelmingly in the opposite direction, or you are ignoring the data points completely and your argument is merely "what you said pattern-matches a political tribe X, now here are some books written by a tribe Y".

(Yes, I'm aware you meant imprecise probabilities. These aren't probablities though (in the same sense that a range of numbers isn't a number), e.g., you're unwilling to state a median.)

The root cause may be that there is too much inferential distance [? · GW]

Perhaps, although I generally become more sympathetic to someone's point of view the more I read from them.

And it's part of why I think it's useful to create scenes that operate on different worldview assumptions: it's worth working out the implications of specific beliefs without needing to justify those beliefs each time. 

I used to lean more strongly towards more schools of thought being good, however I've updated slightly on the margin towards believing thinking some schools of thought just end up muddying the waters.

That said, Epoch has done some great research, so I'm overall happy the scene exists. And I think Matthew Barnett is extremely talented, I just think he's unfortunately become confused.

This paradox doesn't occur because a computation trying to prove its own output (and give the opposite output) will have to simulate itself

Due to Löb, if a computation knows that if it finds a proof that it outputs A, then it will output A, then it proves that it outputs A, without any need for recursion. This is why you really shouldn’t output something just because you’ve proved that you will. 

Yeah, from the claim that pi starts with two you can easily prove anything. But I think:

(1) something like logical induction should somewhat help: maybe the agent doesn’t know whether some statement is true and isn’t going to run for long enough to start encounter contradictions.

(2) Omega can also maybe intervene on the agent’s experience/knowledge of more accessible logical statements while leaving other things intact, sort of like making you experience what Eliezer describes here as convincing that 2+2=3: https://www.lesswrong.com/posts/6FmqiAgS8h4EJm86s/how-to-convince-me-that-2-2-3 [LW · GW], and if that’s what it is doing, we should basically ignore our knowledge of maths for the purpose of thinking about logical counterfactuals.

That's not how the math works. Suppose there are 200 activities under the heading of "AI R&D" that each comprise at least 0.1% of the workload. Suppose we reach a point where AI is vastly superhuman at 150 of those activities (which would include any activities that humans are particularly bad at), moderately superhuman at 40 more, and not much better than human (or even worse than human) at the remaining 10. Those 10 activities where AI is not providing much uplift comprise at least 1% of the AI R&D workload, and so progress can be accelerated at most 100x.

This is oversimplified; there is some room for superhuman ability (making excellent choices of experiments to run) can compensate for lack of uplift in other areas (time to code and execute individual experiments). But the fundamental point remains: a complex process can be bottlenecked by its slowest step. Amdahl's Law is not symmetric – a chain can't be as strong as its strongest link.