LessWrong 2.0 Reader

We don't have a live count, but we have a one-time analysis from late 2023: https://www.lesswrong.com/posts/WYqixmisE6dQjHPT8/2022-and-all-time-posts-by-pingback-count [LW · GW] 

My guess is not much has changed since then, so I think that's basically the answer.

That’s an important point I neglected. I mean something like “the top LW post on the list would have the most links from other LW posts”

For example, I’d expect “More Dakka” would be high up on the list. Since it is mentioned in LW posts quite often.

This seems to be arguing that the big labs are doing some obviously-inefficient R&D in terms of advancing capabilities, and that government intervention risks accidentally redirecting them towards much more effective R&D directions.  I am skeptical.

1. If such training runs are not dangerous then the AI safety group loses credibility. 
2. It could give a false sense of security when a different arch requiring much less training appears and is much more dangerous than the largest LLM. 
3. It removes the chance to learn alignment and safety details from such large LLM

1. I'm not here for credibility.  (Also, this seems like it only happens, if it happens, after the pause ends.  Seems fine.)
2. I'm generally unconvinced by arguments of the form "don't do [otherwise good thing x]; it might cause people to let their guard down and get hurt by [bad thing y]" that don't explain why they aren't a fully-general counterargument.
3. If you think LLMs are hitting a wall and aren't likely to ever lead to dangerous capabilities then I don't know why you expect to learn anything particularly useful from the much larger LLMs that we don't have yet, but not from those we do have now.

This seems non-reponsive to arguments already in my post:

If we institute a pause, we should expect to see (counterfactually) reduced R&D investment in improving hardware capabilities, reduced investment in scaling hardware production, reduced hardware production, reduced investment in research, reduced investment in supporting infrastructure, and fewer people entering the field.

We ran into a hardware shortage during a period of time where there was no pause, which is evidence that the hardware manufacturer was behaving conservatively.  If they're behaving conservatively during a boom period like this, it's not crazy to think they might be even more conservative in terms of novel R&D investment & ramping up manufacturing capacity if they suddenly saw dramatically reduced demand from their largest customers.

For example, suppose we pause now for 3 years and during that time NVIDIA releases the RTX5090,6090,7090 which are produced using TSMC's 3nm, 2nm and 10a processes.

This and the rest of your comment seems to have ignored the rest of my post (see: multiple inputs to progress, all of which seem sensitive to "demand" from e.g. AGI labs), so I'm not sure how to respond.  Do you think NVIDIA's planning is totally decoupled from anticipated demand for their products?  That seems kind of crazy, but that's the scenario you seem to be describing.  Big labs are just going to continue to increase their willingness-to-spend along a smooth exponential for as a long as the pause lasts?  What if the pause lasts 10 years?

If you think my model of how inputs to capabilities progress are sensitive to demand for those inputs from AGI labs is wrong, then please argue so directly, or explain how your proposed scenario is compatible with it.

Thank you !

The links to the report are now fixed.

The 4 blog posts cover most of the same ground as the report. The report goes into more detail, especially in sections 5 & 6.

I wrote up some of my thoughts on Bengio's agenda here [LW(p) · GW(p)].

TLDR: I'm excited about work on trying to find any interpretable hypothesis which can be highly predictive on hard prediction tasks (e.g. next token prediction).^[1] From my understanding, the bayesian aspect of this agenda doesn't add much value.

I might collaborate with someone to write up a more detailed version of this view which engages in detail and is more clearly explained. (To make it easier to argue against and to exist as a more canonical reference.)

As far as Davidad, I think the "manually build an (interpretable) infra-bayesian world model which is sufficiently predictive of the world (as smart as our AI)" part is very likely to be totally unworkable even with vast amounts of AI labor. It's possible that something can be salvaged by retreating to a weaker approach. It seems like a roughly reasonable direction to explore as a possible hail mary to that we automate researching using AIs, but if your not optimistic about using safely using vast amounts of AI labor for safety work via some other safety techniques, you should discount accordingly.

For an objection along these lines, see this comment [LW(p) · GW(p)].

(The fact that we can be conservative with respect to the infra-bayesian world model doesn't seem to buy much, most of the action is in getting something which is at all good at predicting the world. For instance, in Fabien's example, we would need the infrabayesian world model to be able to distinguish between zero-days and safe code which requires as much intelligence as our AI has.)

Proof checking to this world model also seems likely to be unworkable, though I have less confidence. And, the more the infra-bayesian world model is computationally intractible to run, the harder it is to proof check. E.g., if running the world model on many inputs is intractable, I'm also very skeptical about proving anything about what it predicts.

I'm not an expert on either agenda and it's plausible that this comment gets some important details wrong.

This could also be the reason behind the issue mentioned in footnote 5. 

Hindsight is 20/20. I think you're underemphasizing how our current state of affairs is fairly contingent on social factors, like the actions of people concerned about AI safety.

For example, I think this world is actually quite plausible, not incongruent:

A world where AI capabilities progressed far enough to get us to something like chat-gpt, but somehow this didn’t cause a stir or wake-up moment for anyone who wasn’t already concerned about AI risk.

I can easily imagine a counterfactual world in which:

• ChatGPT shows that AI is helpful, safe, and easy to align
• Policymakers are excited about accelerating the benefits of AI and unconvinced of risks
• Industry leaders and respectable academics are not willing to make public statements claiming that AI is an extinction risk, especially given the lack of evidence or analysis
• Instead of the UK AI Safety Summit, we get a summit which is about driving innovation
• AI labs play up how AIs can help with safety and prosperity and dismiss anything related to AI risk