LessWrong 2.0 Reader

It's hard to see what a novel insight is exactly. Any example can be argued against. Can you give an example of one? Or of one you've personally had?

Various LLMs can spot issues in code bases that are not public. Do all of these count?

I think a classic example of an efficient market is one where goods are mostly fungible, e.g. the market for grain, or screws of a particular specification, or copper.

I imagine that inside those markets it feels a lot less like there's any good deals to sniff out. There's definitely bad ones like fraudsters or subpar quality, or someone selling holy screws for 10 times the price, or someone just praying on newcomers to the market who aren't yet calibrated to the standard price, but these are fairly easy to filter out with a bit of due diligence.

Agreed, tone and framing are crucial. The populist framing might work for conservatives, but it will also set off the enemy rhetoric detectors among liberals. So coding it to either side is prone to backfire. Based on that logic, I'm leaning toward thinking that it needs to be framed to carefully avoid or walk the center line between the terms and framings of both sides.

It would be just as bad to have it polarized as conservative, right? Although we've got four years of conservatism, so it might be worth thinking seriously about whether that trade might be worth it. I'm not sure a liberal administration would undo restrictions on AI even if they had been conservative-coded...

Interesting. I'm feeling more like saying "the elites want to make AI that will make them rich while putting half the world out of a job". That's probably true as far as it goes, and it could be useful.

I would love to see an analysis and overview of predictions from the Dwarkesh podcast with Leopold. One for Situational awareness would be great too.

I think it's working on one part of the problem, while other parts remain. If I were to be equally uncharitable, I'd say you seem to assume that if you can't solve everything all at once, you shouldn't say anything.

I don't actually think you assume that.

What I do think is that Instruction-following AGI is easier and more likely than value aligned AGI [LW · GW], and that's a route to solving goodharting and deception. It's complex and unfinished, like every other proposed approach to avoiding death by AGI. You might like more meticulous detail; if so see Max Harms' admirably detailed corrigibility as singular target (CAST) sequence [LW · GW] on a very similar alignment target and approach to solving goodharting and deception.

I think the way the issue is framed matters a lot. If it's a "populist" framing ("elites are in it for themselves, they can't be trusted"), that frame seems to have resonated with a segment of the right lately. Climate change has a sanctimonious frame in American politics that conservatives hate.

Biggest disagreement between the average worldview of people I met with at EAG and my own is something like "cluster thinking vs sequence thinking," where people at EAG were often like "but even if we get this specific policy/technical win, doesn't it not matter unless you also have this other, harder thing?" and I was often more like, "Well, very possibly we won't get that other, harder thing, but still seems really useful to get that specific policy/technical win, here's a story where we totally fail on that first thing and the second thing turns out to matter a ton!"

I'm sort of confused by the image you posted? Von Neumann existed, and there are plenty of very smart people well beyond the "Nerdy programmer" range.

But I think I agree with your overall point about IQ being under stabilizing selection in the ancestral environment. If there was directional selection, it would need to have been weak or inconsistent; otherwise I'd expect the genetic low hanging fruit we see to have been exhausted already. Not in the sense of all current IQ-increasing alleles being selected to fixation, but in the sense of the tradeoffs becoming much more obvious than they appear to us currently. I can't tell what the tradeoffs even were: apparently IQ isn't associated with the average energy consumption of the brain? The limitation of birth canal width isn't a good explanation either since IQ apparently also isn't associated with head size at birth (and adult brain size only explains ~10% of the variance in IQ).

My understanding when I last looked into it as that the efficient updating of the NNUE basically doesn't matter, and what really matters for its performance and CPU-runnability is its small size.

these are also alignment failures we see in humans.

Many of them have close analogies in human behaviour. But you seem to be implying "and therefore those are non-issues"???

There are many groups of humans (or groups of humans), that if you set them on the task of solving alignment, will at some point decide to do something else. In fact, most groups of humans will probably fail like this.

How is this evidence in favour of your plan ultimately resulting in a solution to alignment???

but these systems empirically often move in reasonable and socially-beneficial directions over time

Is this the actual basis of your belief in your plan to ultimately get a difficult scientific problem solved? 

and i expect we can make AI agents a lot more aligned than humans typically are

Ahh I see. Yeah this is crazy, why would you expect this? I think maybe you're confusing yourself by using the word "aligned" here, can we taboo it? Human reflective instability looks like: they realize they don't care about being a lawyer and go become a monk. Or they realize they don't want to be a monk and go become a hippy (this one's my dad). Or they have a mid-life crisis and do a bunch of stereotypical mid-life crisis things. Or they go crazy in more extreme ways.

We have a lot of experience with the space of human reflective instabilities. We're pretty familiar with the ways that humans interact with tribes and are influenced by them, and sometimes break with them.

But the space of reflective-goal-weirdness is much larger and stranger than we have (human) experience with. There are a lot of degrees of freedom in goal specification that we can't nail down easily through training. Also, AIs will be much newer, much more in progress, than humans are (not quite sure how to express this, another way to say it is to point to the quantity of robustness&normality training that evolution has subjected humans to). 

Therefore I think it's extremely, wildly wrong to expect "we can make AI agents a lot more [reflectively goal stable with predictable goals and safe failure-modes] than humans typically are".

but, Claude sure as hell seems to

Why do you even consider this relevant evidence?