LessWrong 2.0 Reader

Thanks Lucius, 
You have a good point and I've removed that section to not give the wrong impression. 

This doesn't matter for predicting the outcome of a hypothetical war between 16th century Britain and 21st century USA.

If AI systems can make 500 years of progress before we notice it's uncontrolled, it's already assuming it's a insanely strong superintelligence.
 

We could probably understand how a von Neumann probe or an anti-aging cure worked too, if someone taught us.

Probably, if it's of a type we can imagine and is comprehensible in those terms - but that's assuming the conclusion! As Gwern noted, we can't understand chess endgames. Similarly, in the case of a strong ASI, the ASI- created probe or cure could look more like a random set of actions that aren't explainable in our terms which cause the outcome than it does like an engineered / purpose driven system that is explainable at all.

Right, yeah. But you could also frame it the opposite way

Ha, very fair point!

From the most recent ACX linkpost:

More new-ish AI policy substacks potentially worth your time:

• You may remember Helen Toner from the OpenAI board drama, but she’s also an experienced and thoughtful scholar on AI policy and now has a Substack, Rising Tide. I especially appreciated Nonproliferation Is The Wrong Approach To AI Misuse.
• You may remember Miles Brundage from OpenAI Safety Team Quitting Incident #25018 (or maybe 25019, I can’t remember). He’s got an AI policy Substack too, here’s a dialogue with Dean Ball.
• You may remember Daniel Reeves from Beeminder, but he has an AI policy Substack too, AGI Fridays. Here’s his post on AI 2027.
• If you’re at all familiar with AI policy you already know Dean Ball (Substack here), but congratulate him on being named White House senior policy advisor.

It seems like there are a bunch of people posting about AI policy on Substack, but these people don't seem to be cross-posting on LW. Is it that LW doesn't put much focus on AI policy, or is it that AI policy people are not putting much focus on LW?

, I simply don’t experience anything ‘requiring a metaphysical explanation

Meaning that you think only certain things need explanation? Or meaning that you already have the explanati on?

AI for improving human reasoning seems promising; I'm uncertain whether it makes sense to invest in new custom applications, as maybe improvements in models are going to do a lot of the work.

I'm more bullish on investing in exploration of promising workflows and design patterns. As an example, a series of youtube videos and writeups on using O3 as a forecasting aid for grantmaking, with demonstrations. Or a set of examples of using LLMs to aid in prodcutive meetings, with a breakdown of the tech used and social norms that the participants agreed to. 
- I think these are much cheaper to do in terms for time and money.
- A lot of epistemics seems to be HCI bottlenecked.
- Good design patterns are easily copyable, which also means they're probably underinvested in relative to their returns.
- Social diffusion of good epistemic practices will not necessarily hapepn as fast as AI improvements.
    - Improving the AIs themselves to be more truth seeking and provide good advice - with good benchmarks - is another avenue.

I imagine a fellowship for prompt engineers and designers, prize competitions, or perhaps retroactive funding for people who have already developed good paterns.
 

*nods*, yeah, your team does seem competent and truth-seeking enough to get a lot of stuff right, despite what I model as shortcomings.

That experience was an in-person conversation with Jaime some years ago, after an offhand comment I made expecting fairly short timelines. I imagine there are many contexts where Epoch has not had this vibe.

I bet that we will not see a model released in the future that equals or surpasses the general performance of Chinchilla while reducing the compute (in training FLOPs) required for such performance by an equivalent of 3.5x per year.

FWIW I think much of software progress comes from achieving better performance at a fixed or increased compute budget rather than making a fixed performance level more efficient, so I think this underestimates software progress.

The main justification for having compute efficiency be approximately equal to compute in terms of progress given in the timeline supplement and main dropdown is the Epoch AI measurements which are specifically about fixed-performance and lower compute. At the very least this concedes that the estimates are not based on trend-extrapolation and are conjecture.

I agree that it's harder to quantify software improvements at the same or higher levels of compute in a way that can be easily compared against compute increases, but we can totally measure some part of it by looking at performance increasing given thes same compute budget (it's quite hard to measure the metric of "how much compute would it have taken 2015 agorithms/data to reach 2025 performance" though, for obvious reasons).

Something being harder to measure is not an excuse for ignoring it.

Something being unfalsifiable forward-looking and unmeasurable backwards-looking is a justification for not treating it with high credence, so I think this is also a core disagreement.

To be clear, I agree that there will be some slowdown due to complementarity of software and hardware, and ideally this would be measured in the model. One can think that there will be multiple effects in different directions. I think that at the levels of research speedup observed in the timelines supplement, the magnitude is likely to be low enough to not change the overall takeaways from the model, but maybe you disagree. I might get around to adding this in as it would be nice.

Here are two charts demonstrating that small changes in estimates of current R&D contribution and changes in R&D speedup change the model massively in the absence of a singularity. I know we're just going to go straight back to "well the real model is the even-more-unfalsifiable benchmarks and gaps model," but I think that is unreasonable.

Figure 1: R&D is 50% of current progress, with and without speedups, exponential only

Figure 2: R&D is 33% of current progress, with and without speedups, exponential only

I do not understand how "I think this variable doesn't matter (without checking)" is a good defense about questionably implemented variables that do overdetermine the model, but "this variable doesn't matter to outcomes" is not a valid critique w.r.t. things like "what are current capabilities/time horizon"

Maybe a crux here is maybe how big the speedup is?

What you describe are good reasons why companies are unlikely to want to release this information unilaterially, but from a safety perspective, we should instead consider how imposing such a policy alters the overall landscape.

From this perspective, the main question seems to me to be whether it is plausible that US AI companies would spend more on safety in worlds where other US AI companies are further behind such that having a closer race between different US companies reduces the amount spent on safety. And, how this compares to the chance of this information being helpful in other ways (e.g., making broader groups than just AI companies get involved).

It also seems quite likely to me that in practice people in the industry and investors basically know what is happening, but is harder to trigger a broader response because without more credible sources you can just dismiss it as hype.

How do you expect to deal with misuse worries? Do you just eat the risk?

The proposal is to use monitoring measures, similar to e.g. constitutional classifiers.

Also, don't we reduce misuse risk a bunch by only deploying to 10k external researchers?

(I'm skeptical of any API misuse concerns at this scale except for bio and maybe advancing capabilities at competitors, but this is a stretch given the limited number of tokens IMO.)

Formatting is off for most of this post.