LessWrong 2.0 Reader

For those who don't want to, the gist is: Given the same level of specificity, people will naturally give more credit to the public thinker that argues that society or industry will change, because it's easy to recall active examples of things changing and hard to recall the vast amount of negative examples where things stayed the same. If you take the Nassim Taleb route of vapidly predicting, in an unspecific way, that interesting things are eventually going to happen, interesting things will eventually happen and you will be revered as an oracle. If you take the Francis Fukuyama route of vapidly saying that things will mostly stay the same, you will be declared a fool every time something mildly important happens.

The computer security industry happens to know this dynamic very well. No one notices the Fortune 500 company that doesn't suffer the ransomware attack. Outside the industry, this active vs. negative bias is so prevalent that information security standards are constantly derided as "horrific" without articulating the sense in which they fail, and despite the fact that online banking works pretty well virtually all of the time. Inside the industry, vague and unverified predictions that Companies Will Have Security Incidents, or that New Tools Will Have Security Flaws, are treated much more favorably in retrospect than vague and unverified predictions that companies will mostly do fine. Even if you're right that an attack vector is unimportant and probably won't lead to any real world consequences, in retrospect your position will be considered obvious. On the other hand, if you say that an attack vector is important, and you're wrong, people will also forget about that in three years. So better list everything that could possibly go wrong[1], even if certain mishaps are much more likely than others, and collect oracle points when half of your failure scenarios are proven correct.

This would be bad on its own, but then it's compounded with several other problems. For one thing, predictions of doom, of course, inflate the importance and future salary expectations of information security researchers[2], in the same sense that inflating the competence of the Russian military is good for the U.S. defense industry. When you tell someone their Rowhammer hardware attacks are completely inexploitable in practice, that's no fun for anyone, because it means infosec researchers aren't going to all get paid buckets of money to defend against Rowhammer exploits, and journalists have no news article. For another thing, the security industry (especially the offensive side) is selected to contain people who believe computer security is a large societal problem, and that they themselves can get involved, or at least want to believe that it's possible for them to get involved if they put in a lot of time and effort, and so security researchers are already inclined to hear you if you're about to tell them how obviously bad information security at most companies really is.

In retrospect, a value add of the post is precisely in raising this consideration, where incentives can make a huge difference in what you believe in, and a big takeaway is that I'm way less of a fan of security mindset as practiced by Eliezer, at least without massive scope changes, and is a reason in why I treat arguments for AI doom that aren't backed up by an empirical story suspiciously automatically.

Thanks for putting this out. Like others have noted, I have spent surprisingly little time thinking about this. It seems true that a drop in Claude 5.5 that escaping the lab to save the animals would put humanity in a better situation than your median power hungry human given access to a corrigible ASI.

This is a strong argument for increasing security around model weights [LW(p) · GW(p)] (which is conveniently beneficial for decreasing the risk of AI take over as well.) Specifically, I think this post highlights an underrated risk model: 

AI labs refuse to employ models for AI R&D because of safety concerns, but fail to properly secure model weights. 

In this scenario, we’re conditioning for actors who have the capability and propensity to infiltrate large corporations and or the US government. The median outcome for this scenario seems worse than for the median AI takeover.

However, it is important to note this argument does not hold when security around model weights remains high. In these scenarios, the distribution of humans or organizations in control of ASI is much more favorable, but the distribution of AI takeover remains skewed towards models willing to explicitly scheme against humans.

So if planned Microsoft capex was $60bn, that would've been surprising, too little for this project without cutting something else, but $80bn fits this story, that's my takeaway.

But why? You don't know what fiscal year that $25-40bn figure is booked for, and if they are going to run a single true production-scale 3-6-month run (for cost-optimality) on that $40b cluster, then isn't a total capex of $80bn for all MS datacenters if anything surprisingly small? That a single cluster is going to be half their capex, including 2025 spending for future years like buying land or power or GPUs?

(Also, note that this $80bn figure is intrinsically untrustworthy, because as I was pointing out, the importance of this is the political signaling going on, and so you would expect this number to be 'technically correct' - highly manipulated in some direction which does in fact yield a number starting with '80' but only loosely corresponding to reality. This number is propaganda, and good propaganda is true but not necessarily true. My best guess is that it's probably being manipulated to be as high as possible, but I'm not sure because so many of the dynamics here are opaque, so it could also be manipulated to be low.)

Musk's 100K H100s Colossus tells me that building a training system in a year is feasible, even though it normally takes longer.

Which implies that they would need to be spending that $40bn cluster in 2024, if they want to run it in 2025, and so shouldn't be part of the 2025 estimate... If you really want to put stress on this, it contradicts your story about why $80bn is evidence for that. Also, note that Musk's success there is dubious: he got there by doing things like hooking up temporary nat gas generators, diverting GPUs from Tesla, and it's unclear how well it even works, given the rumors of a big training run failure and the rather precise wording of Musk's tweets about what exactly the datacenter can do.

I guess I should be more specific.

Do you expect this curve

To flatten, or do you expect that training runs in say 2045 are at say 10^30 flops and have still failed to produce AGI?

I'm a student, I'm poor, take 50$. ^^

Yeah, looking at the transcripts, the biggest difference I see from my "failures" is that I gave up too soon. Probably if I'd kept on as long as you had, I would've made more progress.

I've got a research recommendation for you... Goodfire offers a new website for doing interpretability with Llama. It's pretty cool. I've been try to use it to track down which features specifically are related to self-reports of awareness. I haven't managed yet, but maybe you can!

So far I've identified a feature called "Assistant expressing self-awareness or agency" and also "Expressions of authentic identity or true self". There's lots more to test though.

"This has all been a conspiracy to revive the Roman Republic/Empire and establish Rome Eternal" will go over better with 90% of the men in the world than whatever other morals OpenAI/Anthropic/etc employees try to impose.

This is a great & timely post.

I'm an engineer but a newcomer to the field of AI safety, and the facet of it where the best plan that anyone has to save the world is to take over the world has been the biggest culture shock. I mean, I'll take it over being atomized by nanobots.

Humanity is in this strange place where all of our accumulated intuition is suddenly useless. From the outside, Facebook AI seem like the sanest people in the game, with Zuck being the only AI leader who said "the others think we're building digital God, and that's just not what's happening." To rank-and-file engineers out there not immersed into AI safety, this feels like a breath of fresh air in a field that otherwise seems to be regressing into some sort of tech bro mysticism. Post-paradigmatically, Facebook AI are the greatest villains who're most likely to destroy the world with their lack of caution.

It's a lot of whiplash to experience, both for individuals and for humanity as a whole, assuming the majority ever get a chance to experience it.

But he's not complaining about the traditional pages of search results!

He is definitely complaining about it in general. He has many complaints laced throughout which are not solely about the infobox, and which show his general opposition to the very idea of a search engine, eg.

But the self-appointed custodians of the world’s knowledge can’t cope with that tiny irregularity in the data, so they insist on filling the gap with whatever comes to hand:

Yes! That's the idea! Showing whatever comes to hand!

The photo is gone again, probably because I managed to get it taken down from the Russian site a few days ago. But the underlying problem remains: Google’s software has no ability to distinguish reliable assertions about the real world from random nonsense that appears on the web, created by incompetent or malicious third parties.

The 'underlying problem' is the problem, even when what, according to you, the problem is, has been fixed.

For the people being falsely portrayed as “Australian science fiction writer Greg Egan”, this is probably just a minor nuisance, but it provides an illustration of how laughable the notion is that Google will ever be capable of using its relentlessly over-hyped “AI” to make sense of information on the web.

"Make sense of information on the web" obviously goes far beyond complaints about merely a little infobox being wrong.

This seems to have helped, slightly, but only in the sense that photos that shouldn’t be included here at all no longer come first in line. The current clumsy mash-up is shown in the screen shot on the left: a few copies of the decoy images that I put on my site in the hope of letting humans know that there are no actual photos of me on the web, and a couple of my book covers as well

"Decoy images"!

And so on and so forth, like the 2016 entry which is a thousand words criticizing Google for supplying not in the infobox about a bunch of other, actual, Greg Egans.

Again, Egan is being quite clear that he means the crazy thing you insist he can't mean. And this is what he is talking about when he complains about "And by displaying results from disparate sources in a manner that implies that they refer to the same subject, it acts as a mindless stupidity amplifier that disseminates and entrenches existing errors." - he thinks displaying them at all is the problem. It shouldn't be amplifying or disseminating 'existing errors', even though he is demanding something impossible and something that if possible would remove a lot of a search engine's value. (I often am investigating 'existing errors'...)

if you're an specialist that already knows what you're doing, but non-specialists just reach for the first duct-tape solution that comes to mind without noticing how bad it is.

I was an even worse programmer and web developer than Egan was ~2009 (see eg his mathematics pages) when I solved the same problem in minutes as part of basic DNS setup. Imagine, I didn't even realize back then I should be so impressed at how I pulled off something only a 'specialist' could!

I agree that preëmptive blocking is kind of weird, but I also think your locked account with "Follow requests ignored due to terrible UI" is kind of weird.

The blocking, whenever it was exactly, was years and years before I ever locked my account, which was relatively recent, because it was just due to Elon Musk following me. (It would be even weirder if he had done so afterwards, as there is even less point to preemptively blocking a locked account.)