LessWrong 2.0 Reader

Just watched it upon your recommendation. Thanks! It is indeed a fantastic film, and a great example of (epistemic) rationality.

I am slightly worried about the rate at which LW is shipping new features. I'm not convinced they are net positive.

I think LW provides huge value to the community via just being an open platform supporting community feedback for AI safety research (and other content). The interface has been getting busier whereas I think the modal reader would benefit from having as few distractions from them as possible while reading.

Donation doesn't have much to do with this, and I am glad to see that people are donating, but I would have preferred this post to carefully signpost the difference between existing value of LW (immense) from the marginal value of paying for more features for LW (possibly negative), and from your other enterprises.

Couldn't you imagine that you use philosophical reasoning to derive accurate facts about consciousness,

My imagination is pretty good, and while I can imagine that, it's not about this universe or my experience in reasoning and prediction.

Can you give an example in another domain where philosophical reasoning about a topic led to empirical facts about that topic?  Not meta-reasoning about science, but actual reasoning about a real thing?

I'm not sure the o1 model has even been benchmarked yet, let alone used, so that's why they are focused on o1 preview.

I expect Lightcone to be my primary or maybe only x-risk-related donation this year—see my manifund comment here for my endorsement:

As a full-time AGI safety / alignment researcher (see my research output), I can say with confidence that I wouldn’t have been able to get into the field in the first place [LW(p) · GW(p)], and certainly wouldn’t have made a fraction as much progress, without lesswrong / alignment forum (LW/AF). I continue to be extremely reliant on it for my research progress. … [much more here]

Wish I had more to give, but I’ll send something in the mid four figures at the beginning of January (for tax reasons).

Donated. Lighthaven is incredible.

I started working on PhD applications about 12 days ago. I expect to have fairly polished applications for the first deadline on December 1, despite not working on this full time. So I think it's quite possible to do applications for the December 15 deadlines. You would need to contact your referees (and potential supervisors for UK universities) in the next couple of days.

Disclaimer: I am not sure I've done what you think of as Looking, but all your metaphors make sense to me.

If I "get" the general thing, then would you agree that aside from Fake Frameworks, experience with Focusing must help? Especially for people who haven't yet meditated much or find the idea of a "non-verbal thought" elusive.

I'm thinking of Focusing as targeting something that can also happen in meditation, but could take some beginner meditators a long time until they get direct experience with. It's the way that your mind can suddenly produce a new awareness or new knowledge, without any conscious chain-of-thought, any verbal reasoning behind it.

Focusing hammers that home again and again, yes, there's a way and it's right there. It gave me a lot of confidence to try the mental move of "step back and wait until I See Something" in a variety of contexts.

PS: Thank you for pointing out the purpose of koans. I had "dissolved" them, but now I see, that perhaps I can try to answer them anyway!

Because the USA has always looked at the cost of using that 'robust military superiority', which would entail the destruction of Seoul and possibly millions of deaths and the provoking of major geopolitical powers - such as a certain CCP - and decided it was not worth the candle, and blinked, and kicked the can down the road, and after about three decades of can-kicking, ran out of road.

I can't explicitly speak for the China Hawks (not being one myself), but I believe one of the working assumptions is that AGI will allow the "league of free nations" to disarm China without the messiness of millions of deaths.  Probably this is supposed to work like EY's "nanobot swarm that melts all of the GPUs".  

I agree that the details are a bit fuzzy, but from an external perspective "we don't publicly discuss capabilities" and "there are no adults in the room" are indistinguishable.  OpenAI openly admits the plan is "we'll as the AGI what to do".  I suspect NATSEC's position is more like "amateurs discuss tactics, experts discuss logistics" (i.e. securing decisive advantage is more important that planning out exactly how to melt the GPUs)

To believe that the same group that pulled of Stuxnet and this lack the imagination or will to use AGI enabled weapons strikes me as naive, however.

The USA, for example, has always had 'robust military superiority' over many countries it desired to not get nukes, and yet, which did get nukes.

It's also worth nothing AGI is not a zero-to-one event but rather a hyper-exponential curve.  Theoretically it may be possible to always stay far-enough-ahead to have decisive advantage (unlike nukes where even a handful is enough to establish MAD).

I find this argument highly compelling. I think it's necessary to actually think through those 100 ways to prevent rivals from gaining AGI if you already have one. And to be realistic about the rate of progress that AGI. We will not immediately have unstoppable nanobots. To be safe, you'd need some way to not only stop the use of Chinese and Russian nukes, but reliably keep them disabled. To prevent massive bloodshed, you'd also probably need to do the same with conventional military assets - and probably without causing massive casualties.

Diplomatic solutions are probably going to be part of any realistic plan to use AGI to prevent rival AGI - but as you say they won't be enough.

Nonproliferation efforts for nukes slowed down proliferation but didn't stop it. AGI is different in that it will fairly quickly allow nearly universal surveillance - if you can stomach deploying it, and if you don't trigger a nuclear exchange by deploying it.

The other possibly-important difference between this scenario and the history of nuclear proliferation is the presence of a smarter-than-human advisor who can say "no really human, if you fail to follow through, these will be the very likely results, and you won't like them".

I also hope that smarter-than-human advisor will say something like "look guys, you can all get vastly wealthier and longer-lived if you can just not freak out and fight each other" - and be so obviously right and convincing that humans will actually listen. The win-win solutions may just be compelling. I fully agree that no amount of sharing will prevent others from pursuing AGI - but generous sharing of technological benefits would reduce the priority of those efforts and the animosity when they're thwarted.

Now is the time to think this through carefully, before the US commits to a race.