LessWrong 2.0 Reader

I would really love if some "let's make asi" people put some effort into making bad outcomes less bad. Like, it would really suck if we are going to be trapped in endless corporate punk hell, with superintentelligent nannies with correct (tm) opinions. Or infinite wedding parties or whatever. Just make sure that if you fuck up we all just get eaten by nanobots please. Permanent entrapment in misery would be a lot worse.

Training processes with varying (apparent) situational awareness

• 1:2.5 The AI seemingly isn't aware it is an AI except for a small fraction of training which isn't where much of the capabilities are coming from. For instance, the system is pretrained on next token prediction, our evidence strongly indicates that the system doesn't know it is an AI when doing next token prediction (which likely requires being confident that it isn't internally doing a substantial amount of general-purpose thinking about what to think about), and there is only a small RL process which isn't where much of the capabilities are coming from.

Abilities/intelligence come almost entirely from pretraining, so all the situation awareness and scheming capability that current (and future similar) frontier models possess is thus also mostly present in the base model.  The fact that you need to prompt them to summon out a situationally aware scheming agent doesn't seem like much of a barrier, and indeed strong frontier base models are so obviously misaligned/jail-breakable/dangerous that releasing them to the public is PR-harmful enough to motivate RLHF post training purely for selfish profit-motives.

> This implies that restricting when AIs become (saliently) aware that they are an AI could be a promising intervention, to the extent this is possible without greatly reducing competitiveness.

Who cares if it greatly reduces competitiveness in experimental training runs?

We need to figure out how to align superhuman models - models trained with > 1e25 efficient flops on the current internet/knowledge, which requires experimental iteration.  We probably won't get multiple iteration attempts for aligning SI 'in prod', so we need to iterate in simulation (what you now call 'model organisms').

We need to find alignment training methods that work even when the agent has superhuman intelligence/inference.  But 'superhuman' hear is relative - measured against our capabilities.  The straightforward easy way to accomplish this is training agents in simulations with much earlier knowledge cutoff dates, which isn't theoretically hard - just requires constructing augmented historical training datasets.  So you could train on a 10T+ token dataset of human writings/thoughts with cutoff 2010, or 1950, or 1700, etc.  These base models wouldn't be capable of simulating/summoning realistic situationally aware agents, their RL derived agents wouldn't be situationally sim-aware either, etc.

 

So in essence, even if reward truly isn't the optimization target at the outer level, that doesn't imply that all policies trained do not maximize the reward, right?

Interesting, I can see why that would be a feature. I don't mind the taste at all actually. Before, I had some of their smaller citrus flavored kind, and they dissolved super quick and made me a little nauseous. I can see these ones being better in that respect. 

I suspect that debating altruism is an unusually good opportunity for people to "generalize from one example".

People who enjoy helping others will go like: "Of course, everyone wants to help others, deep inside. It's just that when people are in need or in pain, their self-preservation instinct temporarily reduces this feeling, to make them focus on saving themselves. But as soon as we help them satisfy their physical or emotional needs, you will find that even the seemingly horrible people are actually good, when given the opportunity."

And this is kinda unfalsifiable, because if someone remains a horrible person no matter how much you give them (things, support, forgiveness), you can insist that there must be some need that wasn't sufficiently satisfied yet. And that person would obviously encourage this perspective, because it means that they will get even more things. And there will always be something missing, because the world is not perfect.

On the other hand, people who don't enjoy helping others, can rationalize away almost any observed goodness: "Yeah, they are just showing off (i.e. trading a little effort or money for higher social status). And they definitely expect to get something in return; if not today, then maybe tomorrow. They probably got something in return when you weren't watching. Okay, they never got anything in return, but they thought they would, they just miscalculated; that's not goodness, that's stupidity. Why the fuck would anyone do anything, if they don't expect to benefit from it somehow?"

And this too is kinda unfalsifiable, because almost always there is something, no matter how indirect or disproportionately small. And the very fact that you know about a good deed already makes it suspicious that the person wanted you to know, to get something in return, at least some status in your eyes. (And if you don't know about a good deed, then it cannot contradict your perspective, can it?)

This way, everyone can stay convinced that their general theory of altruism is correct.

So maybe the truth is that (1) people are different, and (2) even the same person can do different good deeds for different reasons, and (3) even one deed can have multiple reasons. For example:

• I may expect to maybe get something in return, but the probability of such thing happening multiplied by the average reward may be so small that this simply doesn't make sense economically; there are more profitable things I could be doing instead, if I only cared about my profit
• some people may help the poor to signal how rich they are, but that alone does not explain why they chose to signal their wealth this way, instead of e.g. buying a really expensive car, which is what some other rich people do
• some people are more likely to help others after their own needs are satisfied, but that may be just a subset of all people; other people respond to having their needs satisfied by simply having more needs, without any altruism manifesting as a side effect
• similarly, some people are more likely to help others after seeing a role model, but other people just laugh at the role model, or invent a hypothesis why the role model (1) actually secretly benefits from their seemingly selfless actions, and (2) is actually making the world a worse place

How people feel about receiving charity (e.g. whether they feel degraded by it) may also depend on their model of altruism, which probably is a result of what they would do in the reversed situations, and what motives they have observed (or hypothesized) at others. For example, the kind of person who would only help others to feel smugly superior to them, will probably fight hard against being a recipient of help. The kind of person who gives gladly will be more likely to also receive gladly. (Although, a scammer will also receive gladly; they won't feel humiliated by having successfully exploited others.)

Some smaller points I haven't see in the article:

When considering altruism as reciprocity, i.e. helping others while expecting to get something in return, it probably makes sense to distinguish a few different meanings:

• helping someone, because I expect that specific person to later pay back what they owe me
• helping people, and expecting that some of them will later somehow reciprocate and many probably won't, but I am okay with such outcome, because so far I am profitable on average -- helping others costs me little, and once in a while someone reciprocates in a way that feels like winning a lottery
• helping people in order to establish a general culture of "people in need should be helped" as an insurance in case I would later need some help myself, although I hope not to need this insurance

All of these could be summed up as "helping others, expecting to get something back", but they lead to different behaviors. In the first case, I would only help the people who seem most likely to pay it back later; I wouldn't help the homeless, or strangers. The second case... is actually how it works for me (I think it is not the true reason why I do it, but the fact is that it works). The third case, I think the difference is in the mood: if you only help others because you expect to be poor in future, it feels sad, and you will probably only try doing the minimum necessary.

I noticed a seeming paradox: When I help a person, I don't expect them to do something for me in return. And yet, I would feel better to learn that "this is the kind of person who helps others, when they can". At first I thought, okay, maybe this exposes some kind of hypocrisy or inconsistency on my end: I do not consciously expect to be paid back, but unconsciously I do, so I feel better when I learn that I have helped a person who is likely to reciprocate. But then I noticed that there is also another possible explanation: there are many people in need, and my possibilities to do good are limited; if I help a selfish person, it stops there; but if I help an altruistic person, they may later help someone else, and thus I may have started a chain reaction of good.

Similarly, helping a person who seems to be on a way to improvement feels better, to me. (I think this is not universal. At least I have heard that there are people who help others, but feel betrayed (?) when those people start working on themselves to be less likely to need help in the future.) One possible explanation is that people who get stronger are more likely to reciprocate. But another possible explanation is that when I help people, my hope is to make their situation better; and a person who works on themselves along with receiving my help is acting like a multiplier to that help. You know, "when you give a fish, you feed someone for a day; when you teach them fishing, you feed them for a lifetime", except people actually also need to eat when they are learning, so when you give someone a fish and then they learn fishing (sometimes they don't need you to teach), it's like you have fed them for a life using a single fish, which is an effective act of altruism.

I continue to be a fan of people trying to accomplish something in the world and reporting back on what happened. This is a good example of the genre, and on a subject near and dear to (part of) LessWrong's collective heart.

I confidently expect somebody will read a bunch of things on LessWrong, get excited about AI, and try to get the American government to Do Something. By default this attempt will not be particularly well aimed or effective, and every piece of information we can give on the obstacles will be useful. There have been updates since 2023 on government awareness and response to AI, though I suspect the core information in this post about how to get in contact with people. It might even be cause area agnostic; if I wanted to talk to congress people about education or biosecurity, my guess is having draft proposals ready would be useful.

As novel is as the Dialogue feature was, I'd be interested in a tightened up version of this that cut to the key points and takeaways. I'd also be interested in hearing from people who've done policy work whether this seems accurate and whether it leaves anything important out- better yet from people who tried using this as a guide! Overall, yeah, I weakly think this is worth including in a Best Of LessWrong collection.

On my model of what's going on, you probably want the lozenges to spend a while dissolving, so that you have fairly continuous exposure of throat and nasal tissue to the zinc ions. I find that they taste bad and astringent if I actively suck on them but are pretty unobtrusive if they just gradually dissolve over an hour or two (sounds like you had a similar experience). I sometimes cut the lozenges in half and let each half dissolve so that they fit into my mouth more easily, you might want to give that a try?

Yes - if anyone reading knows more about manufacturing and could comment on how easy it would be to convert, that would be very helpful.

I also agree it would be interesting to try to do more analysis of how much ASI and robotics could speed up construction of robot factories, by looking at different bottlenecks and how much they could help.

I'm not sure a robot workforce would have a huge effect initially, since there's already a large pool of human workers (though maybe you get some boost by making everything run 24/7). However, at later stages it might become hard to hire enough human workers, while with robots you could keep scaling.

Update: I'm pretty sure that making and running this site would be illegal in many jurisdictions.

To be fair, I expect a lot of the cases of identical copies modulo stochasticity to exist in the future, and indeed you could argue has already happened for AI, but I expect it to be more and more relevant by default, so FDT working in the identical copies case is still a really valuable niche.