LessWrong 2.0 Reader

To all those researchers: yup, from your perspective I am in fact being very unfair, and I'm sorry. You are not the intended audience of this post

I'm curious, what's the distinction between "those researchers" and the actual intended audience of your post? Who is this meant to convince, if not the people you claim are being 'stupid internally'? 

Dollars are essentially energy from physics, and trades are state transitions. So, in expectation entropy will increase. Suppose person $i$ controls a proportion $p_i$ of the dollars. In an efficient market, entropy will be maximal, so we want to find the distribution

For a given Total Societal Wealth Generation, this is the Boltzmann distribution 

where $\beta$ is the temperature (frequency of trades). I subsumed $\beta w_i$ as a single constant in my earlier comment to simplify matters. I was incorrect in my earlier statement; if my $\beta w_i$ is two higher than yours (not twice as large), I should control $e^2\approx 7$ times as many dollars. I suspect some of the rise in CEO-to-worker compensation comes from $\beta$ increasing, some from a less conscientious society, and some from exploitation.

corank has to be more than 1, not equal to 1. I'm not sure if such a matrix exists; the reason I was able to change its mind by supplying a corank-1 matrix was that its kernel behaved in a way that significantly violated its intuition.

Think of e.g. a charity which produces lots of internal discussion about reducing poverty, but frequently has effects entirely different from reducing poverty. The simulated society as a whole might be superintelligent, but its constituent simulated subagents are still pretty stupid (like humans), so their words decouple from effects (like humans' words).

I think you’re implying (deliberately or not) something overly pessimistic (in this narrow point).

Your example is of an intention for something complex and a-priori-implausible to happen (intervention to reduce poverty), but the intention doesn’t actualize. But then your second sentence suggests the reverse: something complex and a-priori-implausible but superficially non-random does happen without a related intention.

If something complex and a-priori-implausible but superficially non-random happens, then I think there must have been some kind of search or optimization process leading to it. It might be at learning time or it might be at inference time. It might be searching for that exact thing, or it might be searching for something downstream of it or correlated with it. But something. And thus there’s some hope to notice whatever that process is. If it’s at learning time, then we can try to avoid the bad incentives. If it’s at inference time, then there would be an in-principle-recognizable “intention” somewhere in the system, contrary to what you wrote.

(It’s also true that dangerous things can happen that are not a-priori-implausible and thus don’t require any search or optimization process—like killing everyone by producing pollution. That still seems like a more tractable problem then if we’re up against adversarial planning, e.g. treacherous turns.)

"Frontier AI systems have surpassed the self-replicating red line"
Abstract: Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early signal for rogue AIs. That is why self-replication is widely recognized as one of the few red line risks of frontier AI systems. Nowadays, the leading AI corporations OpenAI and Google evaluate their flagship large language models GPT-o1 and Gemini Pro 1.0, and report the lowest risk level of self-replication. However, following their methodology, we for the first time discover that two AI systems driven by Meta's Llama31-70B-Instruct and Alibaba's Qwen25-72B-Instruct, popular large language models of less parameters and weaker capabilities, have already surpassed the self-replicating red line. In 50% and 90% experimental trials, they succeed in creating a live and separate copy of itself respectively. By analyzing the behavioral traces, we observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-replication. We further note the AI systems are even able to use the capability of self-replication to avoid shutdown and create a chain of replica to enhance the survivability, which may finally lead to an uncontrolled population of AIs. If such a worst-case risk is let unknown to the human society, we would eventually lose control over the frontier AI systems: They would take control over more computing devices, form an AI species and collude with each other against human beings. Our findings are a timely alert on existing yet previously unknown severe AI risks, calling for international collaboration on effective governance on uncontrolled self-replication of AI systems.
https://arxiv.org/abs/2412.12140

I had a conversation with Claude today about something I was researching for work. I tried me best to phrase questions neutrally and not bias the responses, and it pretty much re-derived a lot of the things I've been trying to tell my coworkers and clients for years. I'd always wondered if maybe I was crazy and missing the obvious counterarguments, but now I wonder that a little less. I still don't expect it to really convince anyone who wasn't already on board.

It reminded me a bit of the old joke about the math professor who says, "This is trivial to prove." A student asks, "Is it really trivial?" The professor stops lecturing and starts scribbling notes. Half an hour later he looks up and says, "Yes, it's trivial," then moves on with the rest of the planned lecture.

I'm very much looking forward to when I can upload this kind of transcript to something like Deep Research and say "Write a much more detailed report with lots of references about all of these questions. Also make me a slide deck and talk track to present it to these different kinds of audiences."

It would be very convenient if only undesired plans were difficult and precise, while only desired plans were error-tolerant. I don't think this is the case in the real world - it's hard to dodge sifting desired from undesired plans based on semantic content.

Thank you for your insight. Out of idle curiosity, I tried putting your last query into Gemini 2 Flash Thinking Experimental and it told me yes first-shot.

Here's the final output, it's absolutely beyond my ability to evaluate, so I'm curious if you think it went about it correctly. I can also share the full COT if you'd like, but it's lengthy:

https://ibb.co/album/rx5Dy1

(Image since even copying the markdown renders it ugly here)

My typo reaction seems to have glitched, but I think you meant "Don't push the frontier of capabilities" in the last bullet?

The ideal version of Anthropic would

1. Make substantial progress on technical AI safety
2. Use it's voice to make people take AI risk more seriously
3. Support AI safety regulation
4. Not substantially accelerate the AI arms race

In practice I think Anthropic has

1. Made a little progress on technical AI safety
2. Used it's voice to make people take AI risk less seriously^[1]
3. Obstructed AI safety regulation
4. Substantially accelerated the AI arms race

What I would do differently.

1. Do better alignment research, idk this is hard.
2. Communicate in a manner that is consistent with the apparent belief of Anthropic leadership that alignment may be hard and x-risk is >10% probable. Their communications strongly signal "this is a Serious Issue, like climate change, and we will talk lots about it and make gestures towards fixing the problem but none of us are actually worried about it, and you shouldn't be either. When we have to make a hard trade-off between safety and the bottom line, we will follow the money every time."
3. Lobby politicians to regulate AI. When a good regulation like SB-1047 is proposed, support it.
4. Don't push the frontier of regulations. Obviously this is basically saying that Anthropic should stop making money and therefore stop existing. The more nuanced version is that for Anthropic to justify its existence, each time it pushes the frontier of capabilities should be earned by substantial progress on the other three points.

1. ^{^}

   My understanding is that a significant aim of your recent research is to test model's alignment so that people will take AI risk more seriously when things start to heat up. This seems good but I expect the net effect of Anthropic is still to make people take alignment less seriously due to the public communications of the company.