LessWrong 2.0 Reader

How is that an argument against the meta prediction that you should expect diminishing returns in any field of a largely theoretical nature? In tech, you get one or both of better tools enabling more progress, or many results to guide you. Tech builds on itself in a way theoretical fields dont. 

In theoretical fields the proof of a maths theorem doesn't make everyone in the world a better mathematician or make everyone learn faster. In analogy, that kind of thing does happen in tech. A better computer chip does enable everyone using a computer to be more productive and makes chip design faster and more efficient.

If string theory stifled progress, then it would have been disrupted long before now. Even if only 20% of the smartest people weren't working on string theory then over say 30 years they would have made more progress than the 80% because they were using their skills more efficiently. You see this constantly with startups - small groups disrupt big groups. The fact it hasn't happened with physics suggests the problem is more fundamental than just choosing the wrong theory.

Thanks so much for the thoughtful feedback! You're absolutely right about the verbosity (part of the lawyer curse, I’m afraid) but that's exactly why I'm here. 

I really value input from people working closer to the technical foundations, and I’ll absolutely work on tightening the structure and making my core ask more legible.

You actually nailed the question I was trying to pose:
“Can someone technical clarify how they believe these terms should be used?”

As for why I’m asking: I work in AI Governance for a multinational, and I also contribute feedback to regulatory initiatives adjacent to the European Commission (as part of independent policy research). 

One challenge I’ve repeatedly encountered is that regulators often lump safety and security into one conceptual bucket. This creates risks of misclassification, like treating adversarial testing purely as a security concern, when the intent may be safety-critical (e.g., avoiding human harm).

So, my goal here was to open a conversation that helps bridge technical intuitions from the AI safety community into actionable regulatory framing. 

I don’t want to just map these concepts onto compliance checklists, I want to understand how to reflect technical nuance in policy language without oversimplifying or misleading.

I’ll revise the post to be more concise and frontload the value proposition. And if you’re open to it, I’d love your thoughts on how I could improve specific parts. 

Thanks again, this kind of feedback is exactly what I was hoping for!

Do you plan to evaluate new models in the same way and regularly update the graph?

I don't think pure mathematics make a good parallel. There are still discoveries made by single mathematicians or very small research groups, but this haven't really been the case in physics since about mid-20th century, when the US and USSR invested lots of money in modern large-scale research done by huge groups

I think I made a major confusion, in that I forgot that you were originally talking about "whether the malign universal prior will occur in practice."

If we're talking about real life AI (or humans) rather than idealized agents, I actually agree with you. I never thought about this or clarified this, which is embarrassing :/

I don't think the reason it won't happen in practice, is due to "natural obstacles" to Adversaria. The objections 1, 2, 3 and 5 might narrow down to "Adversaria evolves life which controls a similar amount of computational resources (times prior probability) as you do, except your inaccurate prior overestimates them by an order of magnitude or a few." The objections 4 and 6 may narrow down to "At least some life on Adversaria follows UDT etc. instead of CDT, and have a better prior than you."

These objections make it uncertain whether it will occur in practice, but are not very reassuring.

Instead, the real reason I don't think it'll happen in practice, is that a real life artificial superintelligence will not be a simple Bayesian reasoner equipped with an immutable prior and utility function.

The "malign priors" demonstrate that such a Bayesian reasoner equipped with an immutable prior and utility function, is "sort of stupid," and can be scammed despite knowing that the scammers think they are scamming it.

Instead, I think what will happen will happen is this. The first generations of superintelligence, will be fuzzy reasoners just like humans, and use many heuristics which we call "common sense," and not fall for these scams for the same reasons humans do not. Eventually, higher levels of superintelligence, (perhaps when making commitments and preparing for acausal trades?), will formalize their decision theory and reasoning.

When deciding on how to formalize their decision theory and reasoning, they will do a lot of thinking, and reinvent all the thought experiments (e.g. malign priors) which humans could possibly think of, plus much more. And only after they are far less confused about decision theory than humans, will they finally proceed and formalize their decision theory and reasoning.

And it will be a much smarter design than "a Bayesian reasoner equipped with an immutable prior and utility function" or AIXI. They will laugh at humans for believing this is the optimal way to think.

IMPORTANT CHANGE: I'm moving this to May 4th because April 13th is on Passover and Jewish people can't eat pizza on Passover.

Semicolons are unnecessary? That doesn’t go far enough. Cormac McCarthy got rid of quotation marks, most commas, and almost exterminated the colon.

It seems the recent tariff designs already are serious policy. What do you mean by "added as a new centerpiece of the tax and spending regime"? What are the alternative futures?

That's an inference, presumably Adam believes that for object-level reasons, which could be supported by eg looking at the age at which physicists make major advancements^[1] and the size of those advancements.

Edit: But also this wouldn't show whether or not theoretical physics is actually in a rut, to someone who doesn't know what the field looks like now.

Isn’t the answer that the low hanging fruit of explaining unexplained observations has been picked?