LessWrong 2.0 Reader

In the benchmark gaps timelines forecast there are two "doubling rate" parameters modeled with log-normal uncertainty. Log-normal is inappropriate as a prior on doubling times (frequency) and massively inflates extremely low values of the CDF relative to a more reasonable inverse Gaussian prior (Note 1) with equivalent mean and variance (Note 2), creating an impression of much higher probabilities on super-fast doubling times.

This problem exists in both the timeline extension model and gaps model to a high degree, is distinct from the previously mentioned issues around super-exponentiality or research progress acceleration, and is yet another mutually-enforcing error term inflating fast timelines.

Note 1: https://www.tandfonline.com/doi/pdf/10.1080/07362994.2015.1010124

Note 2: ratio of LogNormal/InverseGaussian CDFs analytically here for equivalent mean of e^1.5 and variance: https://www.wolframalpha.com/input?i=plot+%281%2F2%29+*%281+%2B+erf%28%28-1+%2B+log%28x%29%29%2Fsqrt%282%29%29%29+%2F+%280.5+*%28erfc%28-%280.919989+%28-4.48169+%2B+x%29%29%2Fsqrt%28x%29%29+%2B+3.88584%C3%9710%5E6+erfc%28%280.919989+%284.48169+%2B+x%29%29%2Fsqrt%28x%29%29%29%29%2C+x+from+0+to+20

It wouldn't surprise me if people having a narrative like this make it less likely to suffer from akrasia.

I'm not commenting anything about positive suggestions in this post, but it has some really strange understanding of what it argues with.

an 'Axiom of Rational Convergence'. This is the powerful idea that under sufficiently ideal epistemic conditions – ample time, information, reasoning ability, freedom from bias or coercion – rational agents will ultimately converge on a single, correct set of beliefs, values, or plans, effectively identifying 'the truth'.

There is no reason to "agree on values" (in a sense of actively seeking agreement instead of observing it if it is here) and I don't think many rationalists would argue with that? Values are free parameters in rational behavior, they can be arbitrary. Different humans have different values and this is fine. CEV is about acting where human values are coherent, not about imposing coherence out of nowhere. 

I'm not using this is a prior, I'm using it to update my existing prior (whatever that was). I believe the posterior will be well defined, so long as the prior was.

It would also update you towards 1600 over 2000.

The basic arguments are that (a) becoming fully superhuman at something which involves long-horizon agency across a diverse range of situations seems like it requires agency skills that will transfer pretty well to other domains (b) once AIs have superhuman data efficiency, they can pick up whatever domain knowledge they need for new tasks very quickly.

I agree we didn't justify it thoroughly in our supplement, the reason it's not justified more is because we didn't get around to it.

> Bukele could release Abrego Garcia to the US

Could he? I don't know El Salvador's legal system very well, but I do know that just because someone is incarcerated in an American prison it does not follow that Trump could release them. The US President has no legal authority to release a person from state prison. Do we actually know that Bukele has authority to release Garcia?

and the US could bring him back, if the administration wanted to do so. As a point of reference, the US has routinely negotiated for the return or release of citizens and even of non-citizens considered at risk in other countries.

That is one hell of an unwarranted assumption. The US does sometimes try to get people released from foreign prisons, sometimes successfully, sometimes not. For the not case, see here for an example. I don't know if the Trump administration even could get Garcia released if it wanted to. Neither do you. Neither do any of American judges issuing orders in this case. Even if there is something that El Salvador would accept in exchange for Garcia, (1) there is necessarily a weighing to be done of the cost of that thing versus the benefit of bringing Garcia back. Judges are not in a position to do that weighing, and are not attempting to do it. (2) If El Salvador can see that Trump is under a court order to get Garcia back, that puts Trump in a terrible negotiating position. It would essentially allow El Salvador to coerce whatever it wants from our government. That would be a terrible outcome. This is why negotiation with foreign governments is a function of the executive branch, not the judicial.

As a prerequisite, it will be necessary to enumerate the set of activities that are necessary for "AI R&D"

As I think you're aware, Epoch took a decent stab at this IMO here. I also spent a bunch of time thinking about all the sub-tasks involved in AI R&D early on in the scenario development. Tbh, I don't feel like it was a great use of time compared to thinking at a higher level, but perhaps I was doing it poorly or am underestimating its usefulness.

What is the profile of acceleration across all tasks relating to AI R&D? What percentage of tasks are getting accelerated by 1.1x, 1.5x, 2x?

A late 2024 n=4 survey of frontier AI researchers estimated a median of a 1.15x AI R&D progress multiplier relative to no post-2022 AIs. I'd like to see bigger surveys here but FWIW my best guess is that we're already at a ~1.1x progress multiplier.

Readers are likely familiar with Hofstadter's Law:

It always takes longer than you expect, even when you take into account Hofstadter's Law.

It's a good law. There's a reason it exists in many forms (see also the Programmer's Credo^[9], the 90-90 rule, Murphy's Law, etc.) It is difficult to anticipate all of the complexity and potential difficulties of a project in advance, and on average this contributes to things taking longer than expected. Constructing ASI will be an extremely complex project, and the AI 2027 attempt to break it down into a fairly simple set of milestones and estimate the difficulty of each milestone seems like fertile territory for Hofstadter's Law.

One reason I don't put much weight on this for timelines forecasts is that to the extent I might have done so before, I would have been more wrong from my current view. For example, my AGI timelines median 3 years ago was 2060ish, and over the last I've updated toward an AGI median of more like 2031 due to reasons including underpredicting benchmark scores [LW · GW], underpredicting real-world impacts, and the model we built for AI 2027.

(wow, I didn't remember that my median 3 years ago was 2060ish, wild)