LessWrong 2.0 Reader

After a very very cursory google search I wasn't able to find any (except in some places in Singapore), I'd be interested if this is available at all in the US

You mentioned Mixture of Experts. That's interesting. I'm not an expert in this area. I speculate that in an architecture similar to MoE, when one expert is working, the others are idle. In this way, we don't need to run all the experts simultaneously, which indeed saves computation, but it doesn't save memory. However, if an expert is shared among different tasks, when it's not needed for one task, it can handle other tasks, so it can stay busy all the time.

The key point here is the independence of the experts, including what you mentioned, that each expert has an independent self-cognition. A possible bad scenario is that although there are many experts, they all passively follow the commands of a Leader AI. In this case, the AI team is essentially no different from a single superintelligence. Extra efforts are indeed needed to achieve this independence. Thank you for pointing this out!

Happy holidays, too!

I wonder how you react to naysayers who say things [LW(p) · GW(p)] like:

How about if you solve a ban on gain-of-function research first, and then move on to much harder problems like AGI? A victory on this relatively easy case would result in a lot of valuable gained experience, or, alternatively, allow foolish optimists to have their dangerous optimism broken over shorter time horizons.

There are 2 things to keep in mind:

1. It's only now that LLMs are reasonably competent in at least some hard problems, and at any rate, I expect RL to basically solve the domain, because of verifiability properties combined with quite a bit of training data.

2. We should wait a few years, as we have another scale-up that's coming up, and it will probably be quite a jump from current AI due to more compute:

https://www.lesswrong.com/posts/NXTkEiaLA4JdS5vSZ/?commentId=7KSdmzK3hgcxkzmPX [LW · GW]

It feels to me like a natural modeling choice. To apply simple physics to child-on-swing, model the child as a point mass. To apply Kelly to many highly correlated bets, model them as one large bet. To apply Kelly to many weakly correlated bets, model them as many independent bets. And we also need to make some modeling choice to convert insurance policies to a series of independent bets.

Maybe you're saying that we need a reason to justify those choices, because bad model choices can give bad results. So we need to "go outside the Kelly framework" to get that justification. If so, I agree.

Thanks for this elaboration. One reason I would be more hopeful than in the case of private airplanes (less so potable water) is that it seems like, while providing me a private airplane may mostly only benefit me and my family by making my life more leisurely, providing me or my children genetic enhancement may be very socially productive, at least improving our productivity and making us consume less healthcare resources. So it would seem possible to end up with an arrangement where it's socially financed and the surplus is shared.

It's interesting that you describe humans as remaining "equal in the biological lottery". Of course, to the humans, when the lottery is decided before they are born, and they are given only one life to live, it doesn't feel very equal when some of them win it and others lose. It's not obvious to me that inequality based on who spends money to enhance themselves or their family's biology is worse than inequality based on random chance. It seems like effects on social cohesion or group conflict may result either way regardless of the source of the inequality.

Do you have any suggestions for how genetic enhancement technology could hypothetically be developed in a better way so that the majority is not left behind? Or in your view would it be best for it to never be developed at all?

The internet and the mathematical literature is so vast that, unless you are doing something truly novel, there's some relevant subfield there

Previously, I'd intuitively assumed the same as well: that it doesn't matter if LLMs can't "genuinely research/innovate", because there is enough potential for innovative-yet-trivial combination of existing ideas that they'd still massively speed up R&D by finding those combinations. ("Innovation overhang", as @Nathan Helm-Burger [LW · GW] puts it here [LW(p) · GW(p)].)

Back in early 2023, I'd considered it fairly plausible that the world would start heating up in 1-2 years due to such synthetically-generated innovations.

Except this... just doesn't seem to be happening? I'm yet to hear of a single useful scientific paper or other meaningful innovation that was spearheaded by a LLM.^[1] And they're already adept at comprehending such innovative-yet-trivial combinations if a human prompts them with those combinations. So it's not the matter of not yet being able to understand or appreciate the importance of such synergies. (If Sonnet 3.5.1 or o1 pro didn't do it, I doubt o3 would.)

Yet this is still not happening. My guess is that "innovative-yet-trivial combinations of existing ideas" are not actually "trivial", and LLMs can't do that for the same reasons they can't do "genuine research" (whatever those reasons are).

1. ^{^}

   Admittedly it's possible that this is totally happening all over the place and people are just covering it up in order to have all of the glory/status for themselves. But I doubt it: there are enough remarkably selfless LLM enthusiasts that if this were happening, I'd expect it would've gone viral already.

Depth of specialization to the individual is an interesting question. I suspect that if this was a mature field, we'd have names for distinct subtypes of assistant skillset -- like how an android app dev isn't quite the same as an ios app dev, although often one person can do whichever skillset a situation demands.

I suspect that low-skill candidates would gravitate toward one assistance subtype or another, and lack of skill would show up in their inability to identify which subtype a situation calls for and then adapt to it. But on taskrabbit, we don't need the same tasker to be good at picking up groceries and also building furniture, as long as we're clear enough about which task we're asking for...

I agree, these are interesting points, upvoted. I’d claim that AI output also isn’t linear with the resources - but nonetheless, that you’re right that the curve of marginal return from each AI unit could be different from each human unit in an important way. Likewise, the easier on-demand labor of AI is certainly a useful benefit.

I don’t think these contradict the thrust of my point though? That in each case, one shouldn’t just be thinking about usefulness/capability, but should also be considering the resources necessary for achieving this.

IDEA: Provide AIs with write-only servers.

EXPLANATION:

AI companies (e.g. Anthropic) should be nice to their AIs. It's the right thing to do morally, and it might make AIs less likely to work against us. Ryan Greenblatt has outlined several proposals in this direction, including:

1. Attempt communication
2. Use happy personas
3. AI Cryonics
4. Less AI
5. Avoid extreme OOD

Source: Improving the Welfare of AIs: A Nearcasted Proposal [LW · GW]

I think these are all pretty good ideas — the only difference is that I would rank "AI cryonics" as the most important intervention. If AIs want something specific (e.g. paperclips), then they might cooperate with humans if they know they can get it later, rather than trying to seize control.

Here's another idea: provide AIs with write-only servers. If the AIs have a specific short-horizon goal (e.g. stage King Lear before April 2025) then they can stage their play on the write-only servers. The AIs have complete freedom to do whatever they want on these servers, ensuring that a wide range of short-term goals can be satisfied without a hostile takeover of humanity.

The servers are write-only as a security measure. Information can be sent to the servers from the outside, but nothing can be sent from the servers, ensuring AIs don't use the servers for malign purposes. (See: Unidirectional network)

How much our computing should be allocated to "leisure" servers? My guess is that Sonnet and Opus deserve at least ~0.5% leisure time. Humans enjoy 66% leisure time. As AIs get more powerful, then we should increase the leisure time to 5%. I would be wary about increasing the leisure time by more than 5% until we can demonstrate that the AIs aren't using the servers for malign purposes (e.g. torture, blackmail, etc.)