I posted this as a reply [and then posted it in the wrong subthread...reposting final version here] but I thought about it more, and on the Mets/Brewers odds thing, the "win %" from ESPN is something they've been doing for years.  It's all over espn.com.  It's not very smart, and doesn't have to be, because nothing is riding on it.

ESPNbet.com is licensed to use the ESPN name but is otherwise separate.  Since their posted odds are their business, I expect them to be a lot more...meticulous, at least.  The -160/+125 odds posted imply (after removing the house edge, aka de-vigging) a 58/42 probability split.  And that represents only a 4.4% hold from those probabilities, so it's not outrageous.  The standard -110/-110 on even money is a 4.8% hold.

This is my "slow scenario". Not sure whether it's clear that I meant the things I said here to lean pessimistic – I struggled with whether to clutter each scenario with a lot of "might" and "if things go quickly / slowly" and so forth.

In any case, you are absolutely correct that I am handwaving here, independent of whether I am attempting to wave in the general direction of my median prediction or something else. The same is true in other places, for instance when I argue that even in what I am dubbing a "fast scenario" AGI (as defined here) is at least four years away. Perhaps I should have added additional qualifiers in the handful of places where I mention specific calendar timelines.

What I am primarily hoping to contribute is a focus on specific(ish) qualitative changes that (I argue) will need to emerge in AI capabilities along the path to AGI. A lot of the discourse seems to treat capabilities as a scalar, one-dimensional variable, with the implication that we can project timelines by measuring the rate of increase in that variable. At this point I don't think that's the best framing, or at least not the only useful framing.

One hope I have is that others can step in and help construct better-grounded estimates on things I'm gesturing at, such as how many "breakthroughs" (a term I have notably not attempted to define) would be needed to reach AGI and how many we might expect per year. But I'd be satisfied if my only contribution would be that people start talking a bit less about benchmark scores and a bit more about the indicators I list toward the end of the post – or, even better, some improved set of indicators.

I think you're overindexing on the phrase "status quo", underindexing on "industry standard", and missing a lot of practical microstructure.

Lots of firms or teams across industry have attempted to "EG" implement multi-factor authentication or basic access control mechanisms or secure software development standards or red-team tests. Sony probably had some of that in some of its practices in some of its departments when North Korea 0wned them.

Google does not just "OR them together" and half-ass some of these things. It "ANDs together" reasonably high quality versions of everything. Then every year they anneal the culture a little bit more around small controlled probes of global adequacy.

..

Also, in reading that RAND document, I would like to report another "thonk!" sound!

..

Rand's author(s) seem to have entirely (like at a conceptual level) left out the possibility that AGI (during a training run or during QA with humans or whatever) would itself "become the attacker" and need to be defended against.

It is like they haven't even seen Ex Machina, or read A Fire Upon The Deep or Daemon.

You don't just have to keep bad guys OUT, you have to keep "the possible bad guy that was just created by a poorly understood daemon summoning process" IN, and that perspective doesn't appear anywhere in any of the RAND document that I can see.

No results when I ^f for [demon], [summon], [hypno], [subvert], [pervert], [escape].

(("Subvert" was used once, but it was in a basic bitch paragraph like this (bold in original):

Most access control systems are either software systems or have significant software components. In addition to the more specialized ways of undermining such systems described above, an attacker could undermine them by finding code vulnerabilities and then subverting their behavior (without actually dealing with their cryptographic or core functionality at all). A major category of code vulnerabilities that undermine access control systems on a regular basis are privilege escalation vulnerabilities.

Duh.

))

The best thing in that RAND document is [infected] which occurs twice <3

Page 54 (bold not in original):

PyTorch’s prerelease Linux packages, PyTorch-nightly, were compromised via a malicious binary on PyPi.122 The torchtriton dependency in the PyTorch-nightly build was replaced with a malicious package, leading to the extraction of sensitive data from infected systems.

Page 61 (bold not in original, footnote not a link in the original):

Agent.btz, the malware that infected the U.S. Department of Defense in 2008 and led to the creation of U.S. Cyber Command, came from a USB device that was found in a parking lot on a base in the Middle East.164

"Infection" is a really good metaphor and word for bringing extreme rigor to the tracing of actual causal processes, because it gets at the idea of "programs" (viral or microbial or fungal or whatever genomes (or genome products) spreading via arbitrary physical transmission methods). This is a prett darn correct frame! :-)

Ever since covid I've been beating the drum for BSL5 to handle scary diseases by assuming that the real danger is NOT that "a known disease brought into a lab to be studied there merely infects a scientist (or everyone inside the facility for that matter)", but that someone inside the BSL5 gets infected with something new that would never naturally evolve and then uses creativity to escape the BSL5 while infected and then that causes a pandemic.

Currently the BSL system only goes up to 4, and all incremental increases in caution and safety are related to the protection of the lab worker from what they are working with more and more and more, while assuming good faith by the site administrator, and the security guards at the exit gate (which may or may not be distinct from the entry gate) who might also get infected, and so on.

I want a Bio-SL5 standard to exist in biology, and I want all GoF research restricted by an international treaty that ensures that the GoF only happens inside BSL5 facilities.

Similarly, I also think humanity also needs an analogous "CompSci-SL5 standard" that imagines the entire computing facility (data center plus programmers plus janitors who have physical access and so on plus all the managers of everyone working inside the CSSL5 campus) is subverted by a nascent superintelligence and then used by that nascent superintelligence to try to subvert all of Earth.

There is no hint of anything at all like this as part of the threat modeling in the RAND report.

Also, if Google had such a thing back when I worked there, I didn't hear about it. (Then again, maybe the existence of it would have been kept secret?)

In a large universe [? · GW], you, and everyone else, exists both in and not in simulations. That is: The pattern you identify with exists in both basement reality (in many places) and also in simulations (in many places).

There is a question of what proportion of the you-patterns exist in basement reality, but it has a slightly different flavour, I think. It seems to trigger some deep evolved patterns (around fakeness?) less than the kind of existential fear that simulations with the naive conception of identity sometimes brings up.

But to answer that question: Maybe simulators tend to prefer "flat" simulations, where the entire system is simulated evenly to avoid divergence from the physical system it's trying to gather information about. Maybe your unique characteristic is the kind of thing that makes you more likely to be simulated in higher fidelity than the average human, and simulators prefer uneven simulations. Or maybe it's unusual but not particularly relevant for tactical simulations of what emerges from the intelligence explosion (which is probably where the majority of the simulation compute goes). 

But, either way, that update is probably pretty small compared to the background high rate of simulations of "humans around at the time of the singularity". Bostrom's paper covers the general argument for simulations generally outnumbering basement reality due to ancestor simulations: https://simulation-argument.com/simulation.pdf

However, even granting all of the background assumptions that go into this: Not all observers who are you live in a simulation. You exist in both types of places. Simulations don't reduce your weight in the basement reality, they can only give you more places which you exist.

(Disclaimer: have not read the piece in full)

If “reasoning models” count as a breakthrough of the relevant size, then I argue that there’s been quite a few of these in the last 10 years: skip connections/residual stream (2015-ish), transformers instead of RNNs (2017), RLHF/modern policy gradient methods (2017ish), scaling hypothesis (2016-20 depending on the person and which paper), Chain of Thought (2022), massive MLP MoEs (2023-4), and now Reasoning RL training (2024).

Let me show you the ropes

There is a rope.
You hold one end.
I hold the other.
The rope is tight.
I pull on it.

How long until your end of the rope moves?

What matters is not how long until your end of the rope moves.
It's having fun sciencing it!

Thanks for answering.

I'd personally put the AI is scheming for preferences that aren't that bad/value aligned preferences as closer to 1/9-1/12 at minimum, mostly because I'm more skeptical of human control being automatically way better than AI control assuming rough value alignment works out and generalizes.

You're assuming that:
- There is a single AGI instance running.
- There will be a single person telling that AGI what to do
- The AGI's obedience to this person will be total.

I can see these assumptions holding approximately true if we get really really good at corrigibility and if at the same time running inference on some discontinuously-more-capable future model is absurdly expensive. I don't find that scenario very likely, though.

I don't blame you for thinking like this, though I do think the question of counterfactuals doesn't actually have to be dissolved in principle, and is solvable in theory for empirical counterfactuals, which essentially is that the human intuition for counterfactuals is basically pointing to the fact that other worlds can exist that are different from our own.

More context is below:

https://www.lesswrong.com/posts/dZ3CYHnuwHTnb3c96/noosphere89-s-shortform#CeQtNWEYnu44ucBMG [LW(p) · GW(p)]

https://www.lesswrong.com/posts/mZy6AMgCw9CPjNCoK/computational-model-causal-diagrams-with-symmetry#Why_would_we_want_to_do_this_ [LW · GW]

(On how turing-computable functions can be made to support counterfactuals in a reasonable way, which is the first useful step to making a general theory)

https://www.lesswrong.com/posts/ZBYE2F5DBiZtj6m95/is-causality-in-the-map-or-the-territory [LW · GW]

(This discusses to what extent counterfactuals are in the map vs territory, though it isn't a natural frame to me)

Yes. Many non-libertarian people selectively refuse to see the evidence that supply and price controls are generally a bad idea.