LessWrong 2.0 Reader

I get the impression that you're conflating two meanings of «personal» - «private» and «individual». The fact that I might feel uncomfortable discussing this in a public forum doesn’t mean it «only works for me» or that it «doesn’t work, but I’m shielded from testing my beliefs due to privacy». There are always anonymous surveys, for example. Perhaps you meant something else?

I meant to say that private values/things are unlikely to coincide between different people, though now I'm a bit less sure.

Moreover, even if I were to provide yet another table of my own subjective experience ratings, like the ones here, you likely wouldn’t find it satisfactory — such tables already exist, with far more respondents than just myself, and you aren’t satisfied. Probably because you disagree with the methodology — for instance, since measuring «what people call pleasurable» is subject to distortions like the compulsions mentioned earlier.

I was unfamiliar with those, thanks for pointing! I have an idea to estimate if "optimization-power" could replace existing currencies, and those surveys' data seems like it might be useful.

AFAIK the only info about the (lack of) release of o3 comes from this tweet - and it does seem like o3 will be available explicitly via the API. 
I think it's what you say - pricing causing bad PR. It seems that even for pro users, if you use o1 pro too much they will silently switch you to using o3 mini without telling you even if you select o1 pro. 
So I think they want to continue doing this while not looking so bad doing it. 

And I think you believe others to experience this extra thing because you have failed to understand what they're talking about when they discuss qualia.

Does this still work? I've often heard it referred to as the "shit sandwich method" (by STEMish non-rationalists), so I wonder if people are sufficiently inoculated to it for it to no longer work

Key constraints are memory for storing KV-caches, scale-up world size (a smaller collection of chips networked at much higher bandwidth than outside such collections), and the number of concurrent requests. A model needs to be spread across many chips to fit in memory, leave enough space for KV-caches, and run faster. If there aren't enough requests for inference, all these chips will be mostly idle, but the API provider will still need to pay for their time. If the model is placed on fewer chips, it won't be able to process too many requests at all because otherwise the chips will run out of memory for KV-caches, and also each request will be processed slower.

So there is a minimal threshold in the number of users needed to serve a model at a given speed with a low cost. GB200 NVL72 is going to change this a lot, since it ups scale-up world size from 8 to 72 GPUs, and a B200 chip has 192 GB of HBM to H100's 80 GB (though H200s have 141 GB). This allows to fit the same model on fewer chips while maintaining high speed (using fewer scale-up worlds) and processing many concurrent requests (having enough memory for many KV-caches), so the inference prices for larger models will probably collapse and Hoppers will become more useful for training experiments than inference (other than for the smallest models). It's a greater change than between A100s and H100s, since both had 8 GPUs per scale-up world.

I wonder what the crisis will be.

I think it's quite likely that if there is a crisis that leads to beneficial response, it'll be one of these three:

• An undeployed privately developed system, not yet clearly aligned nor misaligned, either:
  • passes the Humanity's Last Exam benchmark, demonstrating ASI, and the developers go to congress and say "we have a godlike creature here, you can all talk to it if you don't believe us, it's time to act accordingly."
  • Not quite doing that, but demonstrating dangerous capability levels in red-teaming, ie, replication ability, ability to operate independently, pass the hardest versions of the turing test, get access to biolabs etc. And METR and hopefully their client go to the congress and say "This AI stuff is a very dangerous situation and now we can prove it."
• A deployed military (beyond frontier) system demonstrates such generality that, eg, Palmer Luckey (possibly specifically Palmer Luckey) has to go to congress and confess something like "that thing we were building for coordinating military operations and providing deterrence, turns out it can also coordinate other really beneficial tasks like disaster relief, mining, carbon drawdown, research, you know, curing cancer? But we aren't being asked to use it for those tasks. So, what are we supposed to do? Shouldn't we be using it for that kind of thing?" And this could lead to some mildly dystopian outcomes, or not, I don't think the congress or the emerging post-prime defence research scene is evil, I think it's pretty likely they'd decide to share it with the world (though I doubt they'd seek direct input from the rest of the world on how it should be aligned)

Some of the crises I expect, I guess, wont be recognized as crises. Boiled frog situations.

• A private system passes those tests, but instead of doing the responsible thing and raising the alarm, the company just treats it like a normal release and sells it. (and the die is rolled and we live or we don't.)

Or crises in the deployment of AI that reinforce the "AI as tool" frame so deeply that it becomes harder to discuss preparations for AI as independent agents:

• Automated invasion: a country is successfully invaded, disarmed, controlled and reshaped with almost entirely automated systems, minimal human presence from the invading side. Probable in gaza or taiwan.
  • It's hard to imagine a useful policy response to this. I can only imagine this leading to reactions like "Wow. So dystopian and oppressive. They Should Not have done that and we should write them some sternly worded letters at the UN. Also let's build stronger AI weapons so that they can't do that to us."
• A terrorist attack or a targeted assassination using lethal autonomous weapons.
  • I expect this to just be treated as if it's just a new kind of bomb.

Instead of "Goodharting", I like the potential names "Positive Alignment" and "Negative Alignment."

"Positive Alignment" means that the motivated party changes their actions in ways the incentive creator likes. "Negative Alignment" means the opposite.

Whenever there are incentives offered to certain people/agents, there are likely to be cases of both Positive Alignment and Negative Alignment. The net effect will likely be either positive or negative. 

"Goodharting" is fairly vague and typically just refers to just the "Negative Alignment" portion. 

I'd expect this to make some discussion clearer.
"Will this new incentive be goodharted?" -> "Will this incentive lead to Net-Negative Alignment?" 

Other Name Options

Claude 3.7 recommended other naming ideas like:

• Intentional vs Perverse Responses
• Convergent vs Divergent Optimization
• True-Goal vs Proxy-Goal Alignment
• Productive vs Counterproductive Compliance

Refusing to say ‘please’ and ‘thank you’ would fall squarely in the category that people would have a reason to feel negative about.

It's not like wearing a suit in a swimming pool. Never saying "thank you" doesn't physically damage things. It just makes people upset because of the social inappropriateness, like the inappropriate suit.

Seeing you know the exact numbers, I wonder if you could connect with those other families? It's harder to get the best outcome if players do not cooperate and do not even know the wishes of others. Adding that this would be a valuable socializing opportunity would be somewhat hypocritical from me but it's still so.

I think the question—which goals will AGI agents have—is key to ask, but strikes me as interesting to consider only at the outset. Over longer-periods of time, is there any way that the answer is not just survival?

I have a hard time imagining that, ultimately, AGI agents which survive the most will not be those that are fundamentally trying to.