LessWrong 2.0 Reader

That's an interesting argument. However, something similar to your hypothetical explanation in footnote 6 suggests the following hypothesis: Most humans aren't optimized by evolution to be good at abstract physics reasoning, while they easily could have been, with evolutionary small changes in hyperparameters. After all Einstein wasn't too dissimilar in training/inference compute and architecture from the rest of us. This explanation seems somewhat plausible, since highly abstract reasoning ability perhaps wasn't very useful for most of human history.

(An argument in a similar direction is the existence of Savant syndrome, which implies that quite small differences in brain hyperparameters can lead to strongly increased narrow capabilities of some form, which likely weren't useful in the ancestral environment, which explains why humans generally don't have them. The Einstein case suggests a similar phenomenon may also exists for more general abstract reasoning.)

If this is right, humans would be analogous to very strong base LLMs with poor instruction tuning, where the instruction tuning (for example) only involved narrow instruction-execution pairs that are more or less directly related to finding food in the wilderness, survival and reproduction. Which would lead to bad performance at many tasks not closely related to fitness, e.g. on Math benchmarks. The point is that a lot of the "raw intelligence" of the base LLM couldn't be accessed just because the model wasn't tuned to be good at diverse abstract tasks, even though it easily could have been, without a big change in architecture or training/inference compute.

But then it seems unlikely that artificial ML models (like LLMs) are or will be unoptimized for highly abstract reasoning in the same way evolution apparently didn't "care" to make us all great at abstract physics and math style thinking. Since AI models are indeed actively optimized in diverse abstract directions. Which would make it unlikely to get a large capability jump (analogous to Einstein or von Neumann) just from tweaking the hyperparameters a bit, since those are probably pretty optimized already.

If this explanation is assumed to be true, it would mean we shouldn't expect sudden large (Einstein-like) capability gains once AI models reach Einstein-like ability.

The (your) alternative explanation is that there is indeed at some point a phase transition at a certain intelligence level, which leads to big gains just from small tweaks in hyperparameters. Perhaps because of something like the "grokking cascade" you mentioned. That would mean Einstein wasn't so good at physics because he happened to be, unlike most humans, "optimized for abstract reasoning", but because he reached an intelligence level where some grokking cascade, or something like that, occurs naturally. Then indeed a similar thing could easily happen for AI at some point.

I'm not sure which explanation is better.

See also discussion here [LW(p) · GW(p)].

I think you are misjudging the mental attributes that are conducive to scientific breakthroughs. 

My (not very well informed) understanding is that Einstein was not especially brilliant in terms of raw brainpower (better at math and such than the average person, of course, but not much better than the average physicist). His advantage was instead being able to envision theories that did not occur to other people. What might be described as high creativity rather than high intelligence.

Other attributes conducive to breakthroughs are a willingness to work on high-risk, high-reward problems (much celebrated by granting agencies today, but not actually favoured), a willingness to pursue unfashionable research directions, skepticism of the correctness of established doctrine, and a certain arrogance of thinking they can make a breakthrough, combined with a humility allowing them to discard ideas of theirs that aren't working out. 

So I think the fact that there are more high-IQ researchers today than ever before does not necessarily imply that there is little "low hanging fruit".

I think this depends a whole lot on the domain/product, the scalability vs locality question (cookies get worse if more are made in the same place and then distributed, most software doesn't), and the network effect (software that depends on many people using the same thing).

I love my Kindle (and have since the ugly angular V1).  It would be very hard to argue that Amazon is particularly good at "one thing well", though.  Almost none of it's other products are that focused, and it's only because serious readers spend so much on books that they've kept the Kindle fairly pure.  

More generally, focusing on a single thing is HARD.  You need to find a thing that people are willing to pay for (either a few paying a lot or a lot paying a bit), and that you can make more cheaply and better than your competitors.  For most enterpreneurs, the path to that is to do a lot of things, then cut the ones which aren't that successful.  This is a SEARCH strategy, not a long-term octopus vision.  They're trying to find the one (or few very related) things that they can do well (and do well for themselves).   Many of them don't actually find it, so they either just get used to "do lots of stuff pretty badly", or give up.   

What are your goals when you teach?

What gives you pleasure when teaching?

"This very clearly does not" apply to X and "I have an argument that it doesn't apply to X" are not the same thing.

(And it wouldn't be hard for a court to make some excuse like "these specific harms have to be $500m, and other harms 'of similar severity' means either worse things with less than $500m damage or less bad things with more than $500m damage". That would explain the need to detail specific harms while putting no practical restriction on what the law covers, since the court can claim that anything is a worse harm.

Always assume that laws of this type are interpreted by an autistic, malicious, genie.)

Also my impression is that business or political assassinations exist to this day in many countries; a little searching suggests Russia, Mexico, Venezuela, possibly Nigeria, and more.

Oh definitely. In Mexico in particular business pairs up with organized crime all of the time to strong-arm competitors. But this happens when there's an "organized crime" tycoons can cheaply (in terms of risk) pair up with. Also, OP asked about why companies don't assassinate whistlebowers all the time specifically.

a lot of hunter-gatherer people had to be able to fight to the death, so I don't buy that it's entirely about the human constitution

That was not criminal murder by the standards of the time. Arguably a lot of gang murders committed in the United States are committed by people not capable or willing to go out and murder people on their own.

Update: there's now a YouTube link

In brief: large amounts of high quality process based RL might result in AI being more useful earlier (prior to them becoming much smarter). This might be expensive and annoying (e.g. it might require huge amounts of high quality human labor) such that by default labs do less of this relative to just scaling up models than would be optimal from a safety perspective.

Most people do not have the constitution or agency for criminal murder

I think my model of people is that people are very much changed by the affordances that society gives them and the pressures they are under. In contrast with this statement, a lot of hunter-gatherer people had to be able to fight to the death, so I don't buy that it's entirely about the human constitution. I think if it was a known thing that you could hire an assassin on an employee and unless you messed up and left quite explicit evidence connecting you, you'd get away with it, then there'd be enough pressures to cause people in-extremis to do it a few times per year even in just high-stakes business settings. Also my impression is that business or political assassinations exist to this day in many countries; a little searching suggests Russia, Mexico, Venezuela, possibly Nigeria, and more.

I generally put a lot more importance on tracking which norms are actually being endorsed and enforced by the group / society as opposed to primarily counting on individual ethical reasoning or individual ethical consciences.

(TBC I also am not currently buying that this is an assassination in the US, but I didn't find this reasoning compelling.)