LessWrong 2.0 Reader

It sure doesn't seem to generalize in GPT-4o case. But what's the hypothesis for Sonnet 3.5 refusing in 85% of cases? And CoT improving score and o1 being better in browser suggests the problem is in models not understanding consequences, not in them not trying to be good. What's the rate of capability generalization to agent environment? Are we going to conclude that Sonnet is just demonstrates reasoning, instead of doing it for real, if it solves only 85% of tasks it correctly talks about?

Also, what's the rate of generalization of unprompted problematic behaviour avoidance? It's much less of a problem if your AI does what you tell it to do - you can just don't give it to users, tell it to invent nanotechnology, and win.

LASR (https://www.lasrlabs.org/) is giving a £11,000 stipend for a 13 week program, assuming 40h/week it works out to ~$27

VCs are already doing this. They have offered to buy both the oral surgery practice and the dental practice I use in town.

Investors have offered to buy both, but why do you believe those investors were VCs? It seem very unlikely to me that they were.

The realist in me says that tyrannical souls/tyrannical governments seem likely to be the default state of governance, because the forces that power democracy and liberty will be gone with the rise of advanced AI, so we should start planning to make the future AIs we build, and the people that control AI, and the future AIs that do control the government.

More generally, I expect value alignment to be much more of a generator of outcomes in the 21st century than most other forces with the rise of AI, and this is not just about the classical AI alignment problem, compared to people selfishly doing stuff that generates positive externalities as a side effect.

Why not generate it after it's posted publically?

Any time I am faced with this kind of shocking inefficiency, I ask myself a simple question: why was no one doing this before?

Well, as I understand it, the general belief is that...

1. The "scaled up" practices are relatively unpleasant to work in, and make people (who went through a lot of education expecting to get "prestige" jobs, mind you...) feel deprived of agency, deprived of choices about the when-where-and-how of their work, and just generally devalued.

2. The "non business savvy" people who actually generate the value believe, probably entirely correctly, that somewhere between most and actually-more-than-all of the increased income from that kind of scale-up will end up going to MBAs (or to the one or two theoretically-practitioners who actually own of a "medium-sized" practice), and not to them^[1].

3. Healthcare facilities operated by private equity are widely believed, both based on industry rumor and based on actual measurement, to reduce quality of care, and people don't like to be forced to do a bad job if they don't have to?

Why would you voluntarily make your daily life actually unpleasant just to increase an already high income that you'll probably have less time to enjoy anyway?

I continue to think this is a great post. Part of why I think that is that I haven't forgotten it; it keeps circling back into my mind.

Recently this happened and I made a fun connection: What you call wrapper-minds seem similar to what Plato (in The Republic) calls people-with-tyrannical-souls. i.e. people whose minds are organized the way a tyrannical city is organized, with a single desire/individual (or maybe a tiny junta) in total control, and everything else subservient.

I think the concepts aren't exactly the same though -- Plato would have put more emphasis on the single bit, whereas for your concept of wrapper-mind it doesn't matter much if it's e.g. just paperclips vs. some complicated mix of lots of different things, for the concept of wrapper-mind the emphasis is on immutability and in particular insensitivity to reasoned discussion / learning / etc.

"there was a model that worked ok, and there weren't enough businesses savvy people who understood enough of the details to really scale the DSO model."

This applies to a lot of the enshittification of the world.  There used to be tons of small/family businesses, where "successful" for the owner was defined as "make a decent living, by working harder than average".  There was tons of value left on the table (or rather, lots of unmeasured surplus went to consumers).  When things started getting moneyballed - optimized financially and reframed in terms of capital and returns, that surplus got squeezed out.

1. VCs are already doing this. They have offered to buy both the oral surgery practice and the dental practice I use in town.

2. The care they provide turns worse and worse because the model you envision turns a professional (someone who should have a fiduciary responsibility to the patient's best interest above their own) into an employee of a non-professional corporation. All of the pre-and postoperative care that you envision being done by less highly paid individuals in order to free up the surgeon to "generate profit" gets done cheaply and more slapdash resulting in worse and worse patient care. Either the oral surgeon fights back and attempts to maintain the physician patient relationship and gets fired from their own practice that they sold out (pretty common already with Derm and Optho) or they don't and you get the actual medical version of the plastic surgery chop shops common in Miami. This ethical problem is why non-lawyers cannot own a legal practice and yet we failed to recognize the same destruction of the professional relationship when it comes to physicians.

3. Aspen dental is a franchise based venture capital funded organization that already does this.

4. This is where rationalists fall apart. Everything you say makes sense, but it doesn't take into account the sociocultural aspects that make a physician patient relationship different than the value extractive relationship that you propose.

On the same day I posted my original comment I later realized what I said was wrong, and I'll soon edit it to reflect that.

Regarding your response: I think I have a guess on the important difference you're referring to. They both seem to be equivalent to an Incubator Sleeping Beauty, but see consideration 2 bellow.

1

I think another useful (at least to me) way of seeing/stating what is happening here is that all of the following sentences are true, in an ISB and your two experiments:

• The probability (from an external POV) that the coin was Heads or Tails is 1/2.
• Each individual "me" (however many there are) will experience the coin being Heads or Tails one half of the time.
• If every "me" always predicts Heads, all of my mes will be correct 1/3 of the time and wrong 2/3 of the time. Each individual me will only be able to notice this if we get together after the experiments to compare notes.

I think this is equivalent to the difference in scoring methods you used in Anthropical Motte and Bailey in two versions of Sleeping Beauty.

2

With the two experiments in your response, the only significant difference I can see is that, in experiment 1, there are two identical copies of me, and in 2, there are two different people. I don't know if you're implying that this changes any probabilities, and I'm not sure that it does. What I can say is that experiment 2 is, AFAICT, equivalent to the Doomsday argument in it's setup: two theories on the amount of people that will come to be, with 1:1 prior odds between them, and the question is "should you update on your existing". I have more reflection to make before I can give any firm answer here, but I'm inclined toward "no".

3

I have a feeling that, even though we agree with the final probabilities, we disagree on some of the internal details of how these experiments work. What would you say is the significant difference between the experiments, and does it change the numbers?