LessWrong 2.0 Reader

yes! :D 

Relatedly, one of the things that drove me to have short timelines in the first place was reading the literature and finding the best arguments for long timelines. Especially Ajeya Cotra's original bio anchors report, which I considered to be the best; I found that when I went through it bit by bit and made various adjustments to the parameters/variables, fixing what seemed to me to be errors, it all added up to an on-balance significantly shorter timeline.

Re: Point 1: I agree it would not necessarily be incorrect. I do actually think that probably the remaining challenges are engineering challenges. Not necessarily, but probably. Can you point to any challenges that seem (a) necessary for speeding up AI R&D by 5x, and (b) not engineering challenges?

Re: Point 2: I don't buy it. Deep neural nets are actually useful now, and increasingly so. Making them more useful seems analogous to selective breeding or animal training, not analogous to trying to time the market.

Why call it "converse Lawvere" instead of the more standard "utm property" of general recursion theory, e.g. as in Odifreddi? Only because the maps are to [0,1]? That seems like insufficient reason to adopt an unrelated name.

"AI governance looking bleak" seems like an overstatement. Certain types or aims of AI governance are looking bleak right now, especially getting strong safety-oriented international agreements that include the US and China, or meaningful AI regulation at the national level in the US. But there may be other sorts of AI governance projects (e.g. improving the policies of frontier labs, preparing for warning shots, etc.) that could still be quite worthwhile.

But hey, I don’t think the “Demis Hassabis signs off on perpetual mass sabotage of the global AI research enterprise” thing is going to actually happen either.

Here's Demis' recent version of 'IAEA/CERN for AGI'

Featuring
 

...to monitor unsafe projects, right? And sort of (awkward nose touch) deal with those

I think a realistic example would be useful! I suspect a lot of the nuance (nuance that might feel obvious to you) is in how to apply this over a long conversation with lots of data points, amendments on both sides, etc.

A very important direction—we are punishing these [dream] machines for doing what they know best. The average user obviously wants to kill these "hallucinations," but the researchers in math and sciences in general highly benefit from these "hallucinations."

Full paper here: https://arxiv.org/abs/2501.13824

Yes, this is the type of idea big labs will definitely already have (also what I think ~100% of the time someone says "I don't have to give big labs any ideas").

Consider this model.

Suppose the state threatens people to do the following six things for their citizens:

* Teach the young
* Cure the sick
* Maintain law and order
* Feed, clothe and house people with work injuries
* Feed, clothe and house the elderly
* Feed, clothe and house people with FUBAR agency

(Requesting roughly equally many resources to be put into each of them.)

People vary in how they react to the threats, having basically three actions:

1. Assist with what is asked
2. Develop personal agency for essentially-selfish reasons, beyond what is useful on the margin to handle the six tasks above
3. Using the tokens the government provides to certify the completion of the threatened tasks, put citizens in charge of executing similar tasks for foreigners

The largest scale of assisting with what is asked could be to find areas with powerful economies of scale, for instance optimizing the efficiency with which food and clothing is distributed to citizens. However, economies of scale require homogenous tasks, which means that the highest extremes of action 1 trades negatively against extremes of action 2, as one develops narrower specialization while neglecting the general end-to-end agency.

One cannot do much of action 3 without also doing a lot of action 1, so wealth inequality correlates to a focus on economies of scale.

I'm not sure which of "oppression" and "production" this scenario corresponds to under your model.

Similar to the "production" scenario, the production under this model seems to be "real", for instance people are getting clothed and the people who are handsomely rewarded for this are contributing a lot of marginal value. However, unlike the "production" scenario, the wealth doesn't straightforwardly applying knowing better than others. One might know better with respect to one's specialty, but the flipside is that one has neglected the development of skills outside of that specialty (potentially due to starting out with less innate ability to develop them, e.g. a physical disability or lack of connectedness to tutors).

Meanwhile, the scenario I described here doesn't resemble "oppression" at all, except for the original part where the state threatens people to perform the various government services instead of improving their own agency. I get the impression that your oppression hypothesis is more concerned that people provide a simulacrum of these products to the state than that people are forced to provide a genuine version of these products in the most efficient possible way. I do see a strong case for the simulacrum model, but my comment here seems like a relevant alternative to consider, unless I am missing something.

I think superbabies would still have a massive positive impact on the world even if all we do is decrease disease risk and improve intelligence. But with this kind of thing I think the impact could be very robustly positive to an almost ridiculous degree.

My hope is as we scale operations and do more fundraising we can fund this kind of research.