LessWrong 2.0 Reader

Current model of why property prices remain high in many 'big' cities in western countries despite the fix being ostensibly very simple (build more homes!) 

• Actively expanding supply requires (a lot of) effort and planning. Revising zoning restrictions, dealing with NIMBYs, expanding public infrastructure, getting construction approval etc.
• In a liberal consultative democracy, there are many stakeholders, and any one of them complaining can stifle action. Inertia is designed into the government at all levels.
• Political leaders usually operate for short terms, which means they don't reap the political gains of long-term action, so they're even less inclined to push against this inertia.
• (There are other factors I'm ignoring, like the role of private equity in buying up family homes, or all demand-side factors) 

Implication: The housing crunch is a result of political malaise / bad incentives, and as such we shouldn't expect government to solve the issue without political reform.

To all those researchers: yup, from your perspective I am in fact being very unfair, and I'm sorry. You are not the intended audience of this post

I'm curious, what's the distinction between "those researchers" and the actual intended audience of your post? Who is this meant to convince, if not the people you claim are being 'stupid internally'? 

Dollars are essentially energy from physics, and trades are state transitions. So, in expectation entropy will increase. Suppose person $i$ controls a proportion $p_i$ of the dollars. In an efficient market, entropy will be maximal, so we want to find the distribution

For a given Total Societal Wealth Generation, this is the Boltzmann distribution 

where $\beta$ is the temperature (frequency of trades). I subsumed $\beta w_i$ as a single constant in my earlier comment to simplify matters. I was incorrect in my earlier statement; if my $\beta w_i$ is two higher than yours (not twice as large), I should control $e^2\approx 7$ times as many dollars. I suspect some of the rise in CEO-to-worker compensation comes from $\beta$ increasing, some from a less conscientious society, and some from exploitation.

corank has to be more than 1, not equal to 1. I'm not sure if such a matrix exists; the reason I was able to change its mind by supplying a corank-1 matrix was that its kernel behaved in a way that significantly violated its intuition.

Think of e.g. a charity which produces lots of internal discussion about reducing poverty, but frequently has effects entirely different from reducing poverty. The simulated society as a whole might be superintelligent, but its constituent simulated subagents are still pretty stupid (like humans), so their words decouple from effects (like humans' words).

I think you’re implying (deliberately or not) something overly pessimistic (in this narrow point).

Your example is of an intention for something complex and a-priori-implausible to happen (intervention to reduce poverty), but the intention doesn’t actualize. But then your second sentence suggests the reverse: something complex and a-priori-implausible but superficially non-random does happen without a related intention.

If something complex and a-priori-implausible but superficially non-random happens, then I think there must have been some kind of search or optimization process leading to it. It might be at learning time or it might be at inference time. It might be searching for that exact thing, or it might be searching for something downstream of it or correlated with it. But something. And thus there’s some hope to notice whatever that process is. If it’s at learning time, then we can try to avoid the bad incentives. If it’s at inference time, then there would be an in-principle-recognizable “intention” somewhere in the system, contrary to what you wrote.

(It’s also true that dangerous things can happen that are not a-priori-implausible and thus don’t require any search or optimization process—like killing everyone by producing pollution. That still seems like a more tractable problem then if we’re up against adversarial planning, e.g. treacherous turns.)

I had a conversation with Claude today about something I was researching for work. I tried me best to phrase questions neutrally and not bias the responses, and it pretty much re-derived a lot of the things I've been trying to tell my coworkers and clients for years. I'd always wondered if maybe I was crazy and missing the obvious counterarguments, but now I wonder that a little less. I still don't expect it to really convince anyone who wasn't already on board.

It reminded me a bit of the old joke about the math professor who says, "This is trivial to prove." A student asks, "Is it really trivial?" The professor stops lecturing and starts scribbling notes. Half an hour later he looks up and says, "Yes, it's trivial," then moves on with the rest of the planned lecture.

I'm very much looking forward to when I can upload this kind of transcript to something like Deep Research and say "Write a much more detailed report with lots of references about all of these questions. Also make me a slide deck and talk track to present it to these different kinds of audiences."

It would be very convenient if only undesired plans were difficult and precise, while only desired plans were error-tolerant. I don't think this is the case in the real world - it's hard to dodge sifting desired from undesired plans based on semantic content.

Thank you for your insight. Out of idle curiosity, I tried putting your last query into Gemini 2 Flash Thinking Experimental and it told me yes first-shot.

Here's the final output, it's absolutely beyond my ability to evaluate, so I'm curious if you think it went about it correctly. I can also share the full COT if you'd like, but it's lengthy:

https://ibb.co/album/rx5Dy1

(Image since even copying the markdown renders it ugly here)

My typo reaction seems to have glitched, but I think you meant "Don't push the frontier of capabilities" in the last bullet?

The ideal version of Anthropic would

1. Make substantial progress on technical AI safety
2. Use it's voice to make people take AI risk more seriously
3. Support AI safety regulation
4. Not substantially accelerate the AI arms race

In practice I think Anthropic has

1. Made a little progress on technical AI safety
2. Used it's voice to make people take AI risk less seriously^[1]
3. Obstructed AI safety regulation
4. Substantially accelerated the AI arms race

What I would do differently.

1. Do better alignment research, idk this is hard.
2. Communicate in a manner that is consistent with the apparent belief of Anthropic leadership that alignment may be hard and x-risk is >10% probable. Their communications strongly signal "this is a Serious Issue, like climate change, and we will talk lots about it and make gestures towards fixing the problem but none of us are actually worried about it, and you shouldn't be either. When we have to make a hard trade-off between safety and the bottom line, we will follow the money every time."
3. Lobby politicians to regulate AI. When a good regulation like SB-1047 is proposed, support it.
4. Don't push the frontier of regulations. Obviously this is basically saying that Anthropic should stop making money and therefore stop existing. The more nuanced version is that for Anthropic to justify its existence, each time it pushes the frontier of capabilities should be earned by substantial progress on the other three points.

1. ^{^}

   My understanding is that a significant aim of your recent research is to test model's alignment so that people will take AI risk more seriously when things start to heat up. This seems good but I expect the net effect of Anthropic is still to make people take alignment less seriously due to the public communications of the company.