LessWrong 2.0 Reader

As far as I understand what Satya, was saying that 80 billion is what Microsoft wants to invest per year into Azure infrastructure. 

That does not seem to be a part of the money that goes to Stargate.

The 0.5 is planned, but it's not clear that the funding is actually there.

Can you say more about how alignment is crucial for usefulness of AIs? I'm thinking especially of AIs that are scheming / alignment faking / etc.; it seems to me that these AIs would be very useful -- or at least would appear so -- until it's too late.

I kind of envy that you figured this out yourself — I learned the parallelipiped hypervolume interpretation of the determinant from browsing forums (probably this MSE question's responses). Also, please do write that blog article.

And if I keep doing that, hypothetically speaking, some of those discoveries might even be original.

Yeah, I hope you will! I'm reminded of what Scott Aaronson said recently:

When I was a kid, I too started by rediscovering things (like the integral for the length of a curve) that were centuries old, then rediscovering things (like an efficient algorithm for isotonic regression) that were decades old, then rediscovering things (like BQP⊆PP) that were about a year old … until I finally started discovering things (like the collision lower bound) that were zero years old. This is the way.

As of today (26 Jan) this isn't reproducing for me. It's possible that DeepSeek changed their model API to serve a distilled model. 

Seems very sensitive to the type of misalignment right? As an extreme example suppose literally all AIs have long run and totally inhuman preferences with linear returns. Such AIs might instrumentally decide to be as useful as possible (at least in domains other than safety research) for a while prior to a treacherous turn.

2. Following instructions never to design a chemical weapon with probability at least 99.999% is also a capability.

This requires a capability, but also requires a propensity. For example, smart humans are all capable of avoiding doing armed robbery with pretty high reliability, but some of them do armed robbery despite being told not to do armed robbery at a earlier point in their life. You could say these robbers didn't have the capability to follow instructions, but this would be an atypical use of these (admittedly fuzzy) words.

I agree, a lot of outcomes are possible and there's no reason to think only fast takeoffs are dangerous+likely.

Also I went too far saying that it "needs only tiny amounts of compute to reach superintelligence" without caveats. The $6 million is disputed by a video arguing that DeepSeek used far more compute than they admit to.

Can you give me your best one-or-two-line guess? I think the question is trivial from what I've written and I don't really know why you're not also finding it clear.

Obviously there is similarity, but if you rounded character / ground to simulator / simulacra [LW · GW], it's a mistake. About which I do not care because wanting to claim originality, but because I want people to get the model right.

The models are overlapping but substantially different as we are explaining in this comment [LW(p) · GW(p)] and sometimes have very different implications - i.e. it is not just the same good idea presented in a different way.

If the long-term impact of the simulators post would be for LW readers to round every similar model in this space to simulator / simulacra, it would be pretty bad. I do understand it is difficult for people to hold partially overlapping frames/ontologies in mind, but please do try. If not for other seasons, because simulator / simulacra were written before Character-trained models were a thing; now they are, and they make some claims of simulators obsolete. 

(Btw also the ideas in simulators are not entirely original. Simulators are independent but mostly overlapping reinvention of concepts from active inference / predictive processing [LW · GW])