My least favorite thing

One of the problems here is that, as of right now, there isn't much of a middle path between "Stay at Google and do nothing" and "quit your job to do alignment work full-time". Then there's the issue of status-seeking vs. altruism as a case of revealed vs. stated preferences. If there was a way [LW · GW] to make $750k a year and save the world, people would be all over that. I, personally, would be all over that.

But there isn't. If we use johnswentworth as an optimistic case, those who would go into independent AI alignment work full-time would make about $90k per year [LW · GW]. Of course, anyone that complains about the prospect of 'only' making 90k would be derided as a snot-nosed asshole, because we live in a world where hundreds of millions of people subsist upon less than two dollars a day. However, people's internal calculus chugs on just the same, and Charlie decides to stay at Google.

Do you tell every high school quarterback to drop everything and focus on getting in the NFL? Encouraging people to aim for low-probability events with huge failure downsides is almost cruel. You know what is likely to happen to Bob if he drops out? He's likely to waste all his day on video-games because he's addicted to them, and only the social pressures of school managed to break through his addiction to make him actually do stuff. All his friends are still in school and find his decision baffling, they still talk to him, but less and less over time. You know what becomes really hard once you drop out? Finding a girlfriend. Suddenly a year goes by with only a barely functional iphone app to show for it, and while Bob hasn't quite ruined his life, he is cursing his stupidity for dropping out of an ivy league school.

There are safer ways to experiment: ask for a semester off, try to build something during summer break instead of going at a fancy internship, maybe focus only the bare minimum on your classes while you do something else on the side, or take graduate level classes while you're first-year undergrad, or sacrifice your social life and sleep to work on something else concurrently with classes. Failing at any of these leaves you in a recoverable position.

(Edited to add: note that this is both informed by and potentially warped by personal experience and resultant subconscious biases. See my grandchild-comment: https://www.lesswrong.com/posts/JimXsFyEbPXrwzydN/my-least-favorite-thing?commentId=FZKMJctWjWmzjKb8N [LW(p) · GW(p)])

This is partially a story of distantly spaced equilibria though. (In this comment, I treat the educational-track “hamster wheel” from the OP both as given and as having the harmful side effects that the OP implies, to keep the topic connection. However, this is really a comment about equilibria more generally, and in reality, I am uncertain in the moment what I think of the specific case in the OP.)

In my model, you don't get off the hamster wheel because you lose a lot of opportunity if you do, partly (but not entirely!) because everyone else has common knowledge not to follow you and regarding expected relative traits of people on the wheel. If you try to give one-shot true advice, you will say to stay on the hamster wheel and you will be (probabilistically) right. But if everyone receives and follows similar advice, that has the side effect of perpetuating the hamster wheel.

Assuming the coordination benefits from the hamster wheel are a necessity (to avoid being outcompeted or otherwise), making an alternative viable requires creating common knowledge through iterated coordination updates outside the one-shot advice model. Giving suboptimal one-shot advice can be a route to this, but if you do it in the most straightforward way, you accept the problems with lying in the immediate sense as part of this. If it doesn't work, you weren't trustworthy, and that may become evident. If it does work, and the message was enough about the future, you might turn out to have told the truth via making it true, but still not in a straightforwardly truthful way, and that may still become evident. What effect that has has some cultural dependencies—there's chunks of American business culture, for instance, in which running a successful risky campaign of that type is considered admirable, and then there's the chaotic environments of politics…

Note that the suboptimality can also accrue from unintentional (or ‘unintentional’) value skew rather than lying, in which case the distrust also accrues from value skew, but the effect is similar.

By comparison, giving conventional one-shot advice but with an advertisement-like rider of “but if you have some risk tolerance, try this other thing” seems good to me. Attempting in other ways to get out of one-shot advice being a dominant force also seems promising.

If you want to get out of the “concrete convergence points” structure even more deeply rather than just shifting the loci of power, then that's a whole other ball game, I think.

Note that, in my current belief, posting this very comment has a likely side effect of reducing the credibility of forced attempts to get people out of the hamster wheel, but I accept this along the way, mainly because I think active coordination on dewheeling is better in expectation compared to the risked trust loss from poorly-calibrated attempts to forcibly dewheel in isolation, the latter being much worse for long-term dewheeling prospects. I can think of some counterarguments to this which I won't try to get into now both for lack of capacity and because that might have its own hazardous properties (I would be more amenable to describing them privately). There is a separate way in which I wonder whether analyzing the generational/cohort dynamics of this type of phenomenon might be useful, which I also don't have the capacity to go into right now and which I imagine may already be much better covered by e.g. sociological studies (but I haven't looked).

Strongly agree. Rationalist culture is instrumentally irrational here. It's very well known how important self-belief & a growth mindset is for success, and rationalists obsession with natural intelligence quite bad imo, to the point where I want to limit my interaction with the community so I don't pick up bad patterns.

I do wonder if you're strawmanning the advice a little, in my friend circles dropping out is seen as reasonable, though this could just be because a lot of my high-school friends already have some legible accomplishments and skills.

Aren't you kind of preaching to the choir? Who involved in AI alignment is actually giving advice like this?

Wouldn't the median respondent tell A and B something like "go start participating at the state of the art by reading and publishing on the Alignment Forum and by reading, reproducing, and publishing AI/ML papers, and maybe go apply for jobs at alignment research labs?"

I’m getting some new comments in the tone of, “it would be risky and irrational for Bob and Alice to not at least somewhat stay on the wheel.”

This is better addressed in a new article, which I’ll write soon.

My advice:

Alice:  go to college anyway.  If you can get into a better school do that; if not, that's ok too.  Take the minimum class load you can.  Take things that are fun, that you're interested in, that are relevant to alignment.  Have a ton of side projects.  Soak in the environment, cultivate ideas, learn, build.  Shoot for a b+ average gpa.  You're basically guaranteed employment no matter what you do here, and the ideas matter.

Bob:  focus on alignment where you can, but understand that your best bet may very well be to get the highest paying job you can and use that to fund research.  Think hard about that; high end salaries can be on the order of a million dollars a year.  Precommit to actually part with the cash if you go this route, because it's harder than you think.

Charlie:  raise the flag internally and keep it in everyone's mind.  Go for promo so that you both have more money to donate, and so you have more influence over projects which may make things worse.  Donate a quarter of your gross to alignment work; you can afford it.