LessWrong 2.0 Reader

Product manager, non-technical counterpart to a team lead in a development team

A conversation I had in 2021

Them: Man I wish I could do X, it's so much more valuable then what I'm doing right now, but I can't.
Me: But could you though?

I'd forgotten this, but in 2023 they came up to thank me because they were doing X, were pleased with the choice, and assigned me some credit for it.  I've heard other people spontaneously praise project X, without knowing I'd been involved in any way. 

It's also very common for seeing someone at a conference to move them along the path of hiring me. Rarely from 0-> hired, but maybe from 0-> idea, or from "been meaning to"->hired, or something in the middle.

Anthropic's computer use model and Google's Deep Research both do this. Training systems like this to work reliably has been a bottleneck to releasing them

I feel like this sub-thread is going in circles; perhaps we should go back to the start of it. I said:

I don't think this distinction between old-paradigm/old-concepts and new-paradigm/new-concepts is going to hold up very well to philosophical inspection or continued ML progress; it smells similar to ye olde "do LLMs truly understand, or are they merely stochastic parrots?" and "Can they extrapolate, or do they merely interpolate?"

You replied:

I find this kind of pattern-match pretty unconvincing without more object-level explanation. Why exactly do you think this distinction isn't important? (I'm also not sure "Can they extrapolate, or do they merely interpolate?" qualifies as "ye olde," still seems like a good question to me at least w.r.t. sufficiently out-of-distribution extrapolation.)

Now, elsewhere in this comment section, various people (Carl, Radford) have jumped in to say the sorts of object-level things I also would have said if I were going to get into it. E.g. that old vs. new paradigm isn't a binary but a spectrum, that automating research engineering WOULD actually speed up new-paradigm discovery, etc. What do you think of the points they made?

Also, I'm still waiting to hear answers to these questions: "Can you say more about this distinction -- is it a binary, or a dimension? If it's a dimension, how can we measure progress along it, and are we sure there hasn't been significant progress on it already in the last few years, within the current paradigm? If there has indeed been no significant progress (as with ARC-AGI until 2024) is there another explanation for why that might be, besides your favored one (that your distinction is super important and that because of it a new paradigm is needed to get to AGI)"

In my extrapolation, going from $3,000 to $1,000,000 for one task would move one from 175th to 87th position on the CodeForces leaderboard, which seems to be not that much. 

O1 preview: $1.2 -> 1258 ELO

O1: $3 -> 1891

O3 low $20 -> 2300

O3 high: $3,000 -> 2727

O4: $1,000,000 -> ? Chatgpt gives around 2900 ELO 

2,800 H800 GPUs

Perhaps they accomplished this bang-per-compute by training it on outputs of GPT4, o1 and the like.

Sticky trap outrage is covert factory farm apologetics. Pretending the suffering time integral of a single mouse held still exeeds the suffering of billions of chicken and cow "instant" deaths is a way to culturally launder factory farming as humane.

This is a great question!

Point one:

The computational capacity of the brain used to matter much more than it matters now.  The AIs we have now are near-human or superhuman at many skills, and we can measure how skill capacity varies with resources in the near-human range.  We can debate and extrapolate and argue with real data.

But we spent decades where the only intelligent system we had was the human brain, so it was the only anchor we had for timelines.  So even though it’s very hard to make good estimates from, we had to use it.

Point two:

Most information that gives rise to the human mind is learned, not evolved.

The information encoded by evolution is less than a hundred megabytes.  It’s limited by the size of the genome (1 gigabytes).  Moreover, we know that much of the genome is unimportant for mental development.  About 40% is parasitic (viruses and transposons).  Much of the remaining DNA is not under evolutionary control, varying randomly between individuals.  Of expressed genes, only about a quarter appear to be expressed in the brain.  And some of them encode things AI doesn’t need, like the high-reliability plumbing of the circle of Willis, or the mysteries of love, or the biochemical pickiness of the blood-brain barrier, or wanting to pee when you hear running water.  So the “program” contributed by evolution is no more than the size of a largish program like a compiler.  (I would claim it’s probably even less.  I think the important instincts plus the learning algorithms are only a few thousand lines of code.  But that’s debatable.)

On the other hand, the amount learned in a lifetime is on the order of one or a few gigabytes.

Point three:

Most of the information accumulated by evolution has been destroyed.  All of the information accumulated in a species is lost when that species goes extinct. And most species have gone extinct, leaving no descendants.  The world of the Permian period was (as far as we know) just as busy as today, with hundreds of thousands of animal species.  Just one of those species, a little burrowing critter among many other types of little burrowing critters, was the ancestor of all mammals.  All the other little burrowing critters lost out.  All their evolutionary innovations have been lost.

This doesn’t apply to species with horizontal transmission of genes, like bacteria.  But it applies to animals, who are the only creatures with brains.

I wonder if you mean arguments against AGI x-risk being a concern right now? You've included some timeline and safety arguments.

You might want to edit the post to clarify what arguments you're talking about.

There’s an easy way to turn any mathematical answer-based benchmark into a proof-based benchmark and it doesn’t require coq or lean or any human formalization of the benchmark design: just let the model choose whether or not to submit an answer for each question, and score the model zero for the whole benchmark if it submits any wrong answers.