LessWrong 2.0 Reader

Suppose you're in a setting where the world is so large that you will only ever experience a tiny fraction of it directly, and you have to figure out the rest via generalization. Then your argument doesn't hold up: shifting the mean might totally break your learning. But I claim that the real world is like this. So I am inherently skeptical of any result (like most convergence results) that rely on just trying approximately everything and gradually learning which to prefer and disprefer.

I think that there's an important difference between the claim I'm making and the kinds of claims that Marcus has been making.

I definitely didn't mean to sound like I was comparing your claims to Marcus's! I didn't take your claims that way at all (and in particular you were very clear that you weren't putting any long-term weight on those particular cases). I'm just saying that I think our awareness of the outside view should be relatively strong in this area, because the trail of past predictions about the limits of LLMs is strewn with an unusually large number of skulls.

Yeah I don't have any strong theoretical reason to expect that scaling should stay stopped. That part is based purely on the empirical observation that scaling seems to have stopped for now

My argument is that it's not even clear (at least to me) that it's stopped for now. I'm unfortunately not aware of a great site that keeps benchmarks up to date with every new model, especially not ones that attempt to graph against estimated compute -- but I've yet to see a numerical estimate that shows capabilities-per-OOM-compute slowing down. If you're aware of good data there, I'd love to see it! But in the meantime, the impression that scaling laws are faltering seems to be kind of vibes-based, and for the reasons I gave above I think those vibes may be off.

This metaphor really stuck with me: “It’s not about how smart AI is. It’s about how safe we feel with AI when it is wrong.”

I’ve been circling something similar in my own work—what if “alignment” is less about perfect obedience and more like the relational scaffolding we use in human dynamics? Not control, but trust, repair, transparency.

There’s a tension I keep bumping into: we want AI to surprise us (creativity, insight), but we also want it to never surprise us in dangerous ways. Maybe part of the answer isn’t in technical capability but in how we shape the culture around AI: who gets to audit it, who it listens to, and how it owns its mistakes.

Curious if you’ve explored frameworks or design principles that treat AI development more like relationship architecture than engineering?

I just listened to Ege and Tamay's 3-hour interview by Dwarkesh. They make some excellent points that are worth hearing, but they do not stack up to anything like a 25-year-plus timeline. They are not now a safety org if they ever were.

Their good points are about bottlenecks in turning intelligence into useful action. These are primarily sensorimotor and the need to experiment to do much science and engineering. They also address bottlenecks to achieving strong AGI, mostly compute.

In my mind this all stacks up to convincing themselves timelines are long so they can work on the exciting project of creating systems capable of doing valuable work. Their long timelines also allow them to believe that adoption will be slow, so job replacement won't cause a disastrous economic collapse.

Key ideas include long timelines, slow takeoff, eventual explosive growth, optimism about alignment, concerns about overregulation, concerns about hawkishness towards China, advocating the likelihood of AI sentience and desirability of AI rights, debating the desirability of different futures, and so on.

Small semantic note: these are not new ideas to Epoch, they are a new package of positions on ideas predominantly originating from the MIRI/LW cluster that you earlier mentioned.

Not taking critiques of your methods seriously is a huge problem for truth-speaking. What well-informed critiques are you thinking of? I want to make sure I've taken them on board.

The best approach to aggregating preferences [LW · GW] doesn't involve voting systems.

You could regard of carefully controlling one's expression of one's utility function as being like a vote, and so subject to that blight of strategic voting, in general people have an incentive to understate their preferences about scenarios they consider unlikely/vice versa, which influences the probability of those outcomes in unpredictable ways and fouls their strategy, or to understate valuations when buying and overstate when selling, this may add up to a game that cannot be played well, a coordination problem, outcomes no one wanted.

But I don't think humans are all that guileful about how they express their utility function. Most of them have never actually expressed a utility function before, it's not easy to do, it's not like checking a box on a list of 20 names. People know it's a game that can barely be played even in ordinary friendships, people don't know how to lie strategically about their preferences to the youtube recommender system, let alone their neural lace.

This seems relatively clearly false in the case of competition programming problems. Concretely, o3 with 50 submissions beats o1 with 10k submissions. (And o1 is presumably much better than the underlying instruct model.)

I'd guess this paper doesn't have the actual optimal methods.

"So we would need to find a hypothesis where we accidentally already made all the necessary experiments and even described the intermediate findings (because LLMs are good at words, but probably suck at analyzing the primary data), but we somehow failed to connect the dots. Not impossible, but requires a lot of luck."

Exactly: untested hypotheses that LLMs already have enough data to test. I wonder how rare such hypotheses are.

It strikes me as wild that LLMs have ingested enormous swathes of the internet, across thousands of domains, and haven't yet produced genius connections between those domains (eg between psychoanalysis and tree root growth). Cross Domain Analogies seem like just one example of ripe category of hypotheses that could be tested with existing LLM knowledge.

I'm not sure if this is fiction. I realize there's something nice about opening in media res but I think this could use better indication of who this is for or what it is.