LessWrong 2.0 Reader

I just skimmed but just wanted to flag that I like Bengio's proposal of one coordinated coalition that develops several AGIs in a coordinated fashion (e.g. training runs at the same time on their own clusters), which decreases the main downside of having one single AGI project (power concentration). 

I still agree with a lot of that post and am still essentially operating on it. 

I also think that it's interesting to read the comments because at the time the promise of those who thought my post was wrong was that Anthropic's RSP would get better and that this was only the beginning. With RSP V2 being worse and less specific than RSP V1, it's clear that this was overoptimistic. 

Now, risk management in AI has also gone a lot more mainstream than it was a year ago, in large parts thanks to the UK AISI who started operating on it. People have also started using more probabilities, for instance in safety cases paper, which this post advocated for.  

With SaferAI, my organization, we're still continuing to work on moving the field closer from traditional risk management and ensuring that we don't reinvent the wheel when there's no need to. There should be releases going in that direction over the coming months. 

Overall, if I look back on my recommendations, I think they're still quite strong. "Make the name less misleading" hasn't been executed on but other names than RSPs have started being used, such as Frontier AI Safety Commitments, which is a strong improvement from my "Voluntary safety commitments" suggestion.

My recommendation about what RSPs are and aren't are also solid. My worry that the current commitments in RSPs would be pushed in policy was basically right: it's been used in many policy conversations as an anchor for what to do and what not to do. 

Finally, the push for risk management in policy that I wanted to see happen has mostly happened. This is great news. 

The main thing that misses from this post is the absence of prediction of RSP launching the debate about what should be done and at what levels. This is overall a good effect which has happened, and would probably have happened several months after if not for the publication of RSPs. The fact that it was done in a voluntary commitment context is unfortunate, because it levels down everything, but I still think this effect was significant.

I think this paper was great. I'm very proud of it. It's a big hard to separate out this paper from the follow-up post arguing for control [LW · GW], but I'll try to.

This paper led to a bunch of research by us and other people; it helped AI control become (IMO correctly) one of the main strategies discussed for ensuring safety from scheming AIs. It was accepted as an oral at ICML 2024. AI companies and other researchers have since built on this work (Anthropic’s “Sabotage Evaluations”, Mathew et al “Hidden in Plain Text”), and AI control has been framed by (Grosse, Buhl, Balesni, Clymer) as a bedrock approach for mitigating catastrophic risk from misaligned AI.

My main regrets about this paper is that we didn't use defer-to-resample, a technique where you replace suspicious actions with a resample from the untrusted model (as discussed e.g. here [LW · GW]). This probably would have been better than the other defer techniques we tried.

I have more regrets about the follow-up post ("The case for ensuring...") than about this post; this post was more straightforward and less ambitious, and so gave us fewer opportunities to stick our necks out making arguments or introducing concepts that we'd later regret.

I'm very excited for more follow-up work on this paper, and I'm working on mentoring such projects and sourcing funding for them.

Oliver specifically wanted me to include the word "naive" because obviously there are sensible things people could mean by this but they phrase things overly strongly and the Lightcone Team's Autism is Powerful.

Yes I think your equation looks right.

things are almost never greater than the sum of their parts Because Reductionism

Isn't it more like, the value of the sum of the things is greater than the sum of the value of each of the things? That is, $f(a+b)>f\left(a\right)+f\left(b\right)$ (where perhaps $f$ is a utility function). That seems totally normal and not-at-all at odds with Reductionism.

Also, I don't claim there's another major grant maker that's less constrained like this.)

I think the SFF appears less constrained like this

In theory, possibly, but it's not clear how to save the world given such restricted access. See e.g. https://www.lesswrong.com/posts/NojipcrFFMzNx6Grc/sudo-s-shortform?commentId=onKfTrunn2Q2Gc4Pw [LW(p) · GW(p)]

In practice no, because you can't deal with a superintelligence safely. E.g.

• You can't build a computer system that's robust to auto-exfiltration. I mean, maybe you can, but you're taking on a whole bunch more cost, and also hoping you didn't screw up.
• You can't develop this tech without other people stealing it and running it unsafely.
• You can't develop this tech safely at all, because in order to develop it you have to do a lot more than just get a few outputs, you have to, like, debug your code and stuff.
• And so forth. Mainly and so forth.

I'm not sure that AI boxing is a live debate anymore.  People are lining up to give full web access to current limited-but-unknown-capabilities implementations, and there's not much reason to believe there will be any attempt at constraining the use or reach of more advanced versions.

Here's the text with improved grammar:

I think there is one more level at which natural abstraction can occur: the level just "beneath" consciousness.

For example, we can create an LLM that almost perfectly matches my internal voice dialogue's inputs and outputs. For me – internally – there would be no difference if thoughts appearing in my mind were generated by such an LLM, rather than by real biological neurons or even cortical columns. The same applies to the visual cortex and other brain regions.

Such an LLM for thoughts would be no larger than GPT-4 (as I haven't had that many new ideas). In most cases, I can't feel changes in individual neurons and synapses, but only the high-level output of entire brain regions.

I think we can achieve 99 percent behavioral and internal thought mimicry with this approach, but a question arises: what about qualia? However, this question isn't any easier to answer if we choose a much lower level of abstraction.

If we learn that generating qualia requires performing some special mathematical operation F(Observations), we can add this operation to the thought-LLM's outputs. If we have no idea what F(Observations) is, going to a deeper level of abstraction won't reassure us that we've gone deep enough to capture F(O).

Similar ideas here: https://medium.com/@bablulawrence/cognitive-architectures-and-llm-applications-83d6ba1c46cd

For those interested in Chinese philosophy, I'd suggest 韩非子 (Han Feizi), which offers a thoughtful meta-analysis of earlier philosophers like Laozi and contemporaries such as Xunzi, in so far as their thoughts applied to statecraft. (The first Emperor was a big fan of the work.)

This recommendation assumes some basic knowledge of Chinese history. 

For those new to the subject, [Recommendation to come, I am trying to find the English version for a children's book to Chinese Philosophy and History] might be a better starting point.

My recommendations are based on several key considerations:

• Quality of available translations
• Relevance to Western readers
• Focus on state-building as both an entry point and a way to understand how philosophy shaped Chinese governance
• Historical impact on actual governance

Chinese statebuilding is a very relevant lens because how early the Chinese started to concentrate executive power. The legalist school makes sense because the state was able to codify law staffed with court officers instead of relying on customary law enforced by local notables (thou the latter still happened a lot). And for Confucianism, for almost all of its existence, saw the best way to enact their worldly vision was through influencing the Emperor/King and their imperial/monarchial apparatus; Chinese philosophy is very much Chinese Political Philosophy. (Besides things like Xuanxue, ofc.)

While Mozi and Xunzi are often recommended here, I've found they might not be the most accessible entry points. Mozi, while interesting, had relatively limited historical influence compared to other schools of thought - being more well known to lay Chinese audience by his portrayals in historical dramas than his works. It would be somewhat like introducing Western philosophy through a lesser-known PreSocratic philosopher – potentially interesting but perhaps not the most representative starting point.

I've found that Chinese philosophy becomes more accessible when viewed through the lens of practical governance, as this provides concrete examples of how these ideas were implemented. Governance, being a universal concept, offers familiar ground for readers from any background and which ever cardinal direction. Additionally, many philosophical classics about governance have been reinterpreted over time, making them more approachable for modern readers.

TLDR: Chinese philosophy is firmly attached to Chinese history just like Western philosophy to the history of the Church. Gonna know both and let them bounce off each other.