The curse of identity

I can attest that I had those exact reactions on reading those sections of the article. And in general I am more impressed by someone who graduated quickly than one who took longer than average, and by someone who wrote a book rather than one who hasn't. "But what if that's not the case?" is hardly a knock-down rebuttal.

I think it's more likely you're confusing the status you attribute to Kaj for candidness and usefulness of the post, with the status you would objectively add or subtract from a person if you heard that they floundered or flourished in college.

Though note that the relevant criteria is not so much what other people actually consider to be high-prestige, but what the person themselves considers to be high prestige. (I wonder if I should have emphasized this part a little more, seeing how the discussion seems to be entirely about status in the eyes of others.) For various reasons, I felt quite strongly about graduating quickly.

Your comment and this post have really clarified a lot of the thoughts I've had about status - especially as someone who is largely motivated by how others perceive me - thanks!

I'm actually kind of ambivalent about it myself. Sometimes I wish I could go back to a simpler time when I thought that I was driven by pure intellectual curiosity alone. For someone whose "native" status-seeking tendencies aren't as destructive as the OP's, the knowledge may not be worth the cost.

Any thoughts on how to best consciously optimize for prestige?

Search for your comparative advantage (usually mentioned in the context of maximizing income, but is equally applicable to maximizing prestige). This can be counterintuitive so give it a second thought even if you think you already know. For example, in college I thought I was great at programming and never would have considered a career having to do with philosophy. Well, I am terrible at philosophy but as it turns out, so is everyone else, and I might actually have a greater comparative advantage in it than in programming.

Look for the Next Big Thing so you can write that seminal paper that everyone else will then cite. More generally, try to avoid competing in fields already crowded with prestige seekers. Look for fields that are relatively empty but have high potential.

Don't forget that you have other goals that you're optimizing for simultaneously, and try not to turn into a status junkie. Also double-check any plans you come up with for the kind of self-sabotage described in the OP.

but I wondered why Kaj hadn't included any solutions

I've been somewhat helped by simply realizing the problem. For example, recently I was struggling with wanting to study a lot of math and mathy AI, because that's the field that my brain has labeled the most prestigious (mostly as a result of reading Eliezer et al.). When I realized that I had been aiming at something that I felt was prestigious, not something that was actually my comparative advantage, it felt like a burden was lifted from my shoulders. I realized that I could actually take easier courses, and thereby manage to finish my Master's degree.

Isn't it a flaw of many thinkers that if you give them a question, they try to answer it?

It's also a Fully General Argument (and Excuse) for not solving problems.

This is a very clever statement, and therefore I accord you higher status, as you were hoping for when you wrote it. ;-)

Actually, I was hoping to help people. If you accord me status, but don't actually use any of the information I gave, then you are frustrating my hopes rather than satisfying them.

There are also jobs which are high pay that are also low status in any audience or society.

Yeah. What are the MD specialties that make all the money? Radiology, Oncology...

I seem to recall hearing somewhere an anecdote about a scientist who decided to dabble in some particular field. He immediately got a lot of attention and cites for his early papers, and then decided that if he could excel in the field this easily, the field wasn't worth his time.

In multiverse, binary quantum noise execute you!

Any ideas on how to do slight/no/negative evil but feel very evil?

I execute binary quantum noise and laugh maniacally!

One of my new favorite comments :-)

Reference.

There's a danger of simply getting used to being evil.

One need only feel "evil", rather than actually be "evil". Hypothetical: try to imagine yourself as a demonic being, wearing human skin. Hold yourself to the silly superstitions that people believe of them; they cannot enter homes uninvited, are always out to make "bargains", etc... limit yourself to the harmless categories of these sorts of behaviors, and see how it affects your behavior and thinking.

The point of this being that it magnifies your personal feelings of "wickedness" without actually producing those results.

Pee in the sink.

Lacking hygiene

My intuition is that passive things such as this and the procrastination Gabriel mentioned won't work.

And for the same thing but for rationality

Clever! I will think about it some rather than giving my snap judgement.

What's built in, plausibly, are specific drives (to comfort a crying baby, to take a clear example) whose gratification overlaps what we're inclined to call good. But these specific drives don't congeal into a drive to do ethical good: "good" isn't a natural property.

Now, you could say that "doing good" is just a "far" view of gratifying these specific drives. But I don't think that's the way it's used when someone sets out to "do good," that is, when they're making "near" choices.

Fortunately the title of the page gives it away: it's srdiamond, who I believe still posts occasionally as common_law.

Equally accurate and less specific. (Unless the phrase has another connotation?) I had in mind Sotala's discussion of status-seeking as an obstacle to doing good.

Sure, but you could start optimizing to impress an all knowing, completely rational historian from the far future.

I don't know if imaginary person is enough for our instincts. We should also seek company of rational people, so our instincts can focus on them.

Virtues I've instilled in myself that I've found useful: ...

Problem with signalling is that it can probably subvert any activity, and if something is called "virtue", then it seems like a very good target. If you are not careful, you may find that you, for example, describe yourself as a worse person than you really are, to signal high non-hypocricy.

By the way, I would like to read an article about applying specific lessons to specific computer games in real life.

What is behavior X?

This is me, but I also have several long term money making ideas that should greatly forward the cause and so far two of them seem to be progressing nicely.

tl;dr: Signalling is extremely important to you. Doing away with your ability to signal will leave you helplessly desperate to get it back.

I think that this is a point made not nearly often enough in rationalist circles: Signalling is important to humans, and you are not exempt just because you know that.

Upvoted. I'd love to hear your thoughts on how one could slide that scale more towards the "long-term strategic optimization" end? Assuming that it is possible, of course.

Heyo, after your correction I still think the main thrust of my reply isn't changed. Your correction mostly just makes me wrong to think that you argued that people that disendorse their status-seeking parts don't have long-term plans, rather than that their long-term planning rationality is worsened. I think I still disagree that their planning is worsened though, but my disagreement is sort of subtle and maybe not worth explaining given opportunity costs. I also stand by my main and mostly-orthogonal points about the importance of not dealing with demons (alternatively, "not making concessions to evil" or summat);—another person you could talk to about that theme would be Nick Tarleton, whose opinion is I think somewhere between ours but is (surprisingly) closer to mine than yours, at least recently. He's probably better at talking about these things than I am.

Thanks for the brief convo by the way. :)

Upon reflection, T.S. Eliot can say it better than I can:

O dark dark dark. They all go into the dark,
The vacant interstellar spaces, the vacant into the vacant,
The captains, merchant bankers, eminent men of letters,
The generous patrons of art, the statesmen and the rulers,
Distinguished civil servants, chairmen of many committees,
Industrial lords and petty contractors, all go into the dark,
And dark the Sun and Moon, and the Almanach de Gotha
And the Stock Exchange Gazette, the Directory of Directors,
And cold the sense and lost the motive of action.
And we all go with them, into the silent funeral,
Nobody's funeral, for there is no one to bury.
I said to my soul, be still, and let the dark come upon you
Which shall be the darkness of God. As, in a theatre,
The lights are extinguished, for the scene to be changed
With a hollow rumble of wings, with a movement of darkness on darkness,
And we know that the hills and the trees, the distant panorama
And the bold imposing facade are all being rolled away—
Or as, when an underground train, in the tube, stops too long between stations
And the conversation rises and slowly fades into silence
And you see behind every face the mental emptiness deepen
Leaving only the growing terror of nothing to think about;
Or when, under ether, the mind is conscious but conscious of nothing—
I said to my soul, be still, and wait without hope
For hope would be hope for the wrong thing; wait without love,
For love would be love of the wrong thing; there is yet faith
But the faith and the love and the hope are all in the waiting.
Wait without thought, for you are not ready for thought:
So the darkness shall be the light, and the stillness the dancing.
Whisper of running streams, and winter lightning.
The wild thyme unseen and the wild strawberry,
The laughter in the garden, echoed ecstasy
Not lost, but requiring, pointing to the agony
Of death and birth.

You say I am repeating
Something I have said before. I shall say it again.
Shall I say it again? In order to arrive there,
To arrive where you are, to get from where you are not,

You must go by a way wherein there is no ecstasy.
In order to arrive at what you do not know

You must go by a way which is the way of ignorance.
In order to possess what you do not possess

You must go by the way of dispossession.
In order to arrive at what you are not

You must go through the way in which you are not.
And what you do not know is the only thing you know
And what you own is what you do not own
And where you are is where you are not.

It would be better to have Napoleon as an ally than to have a narcotics addict with a 10 minute time horizon as an ally, and it seems analogously better to help your own status-seeking parts mature into entities that are more like Napoleon and less like the drug addict, i.e. into entities that have strategy, hope, long-term plans, and an accurate model of the fact that e.g. rationalizations don't change the outside world.

I would not want ha-Satan as my ally, even if I trusted myself not to get caught up in or infected by his instrumental ambitions. Still less would I want to give him direct read/write access to the few parts of my mind that I at all trust. Give not that which is holy unto the dogs, neither cast ye your pearls before swine, lest they trample them under their feet, and turn again and rend you. Mix a teaspoon of wine in a barrel of sewage and you get sewage; mix a teaspoon of sewage in a barrel of wine and you get sewage. The rationality of an agent is its goal: if therefore thy goal be simple, thy whole self shall be full of rationality. But if thy goal be fractured, thy whole self shall be full of irrationality. If therefore the rationality that is in thee be irrationality, how monstrous is that irrationality!

Seen at a higher level you advise dealing with the devil—the difference in power between your genuine thirst for justice and your myriad egoistic coalitions is of a similar magnitude as that between human and transhuman intelligence. (I find it disturbing how much more cunning I get when I temporarily abandon my inhibitions. Luckily I've only let that happen twice—I'm not a wannabe omnicidal-suicidal lunatic, unlike HJPEV.) Maybe such Faustian arbitrage is a workable strategy... But I remain unconvinced, and in the meantime the payoff matrix asymmetrically favors caution.

Take no thought, saying, Wherewithal shall I avoid contempt? or, Wherewithal shall I be accepted? or, Wherewithal shall I be lauded and loved? For true metaness knoweth that ye have want of these things. But seek ye first the praxeology of meta, and its rationality; and all these things shall be added unto you. Take therefore no thought for your egoistic coalitions: for your egoistic coalitions shall take thought for the things of themselves. Sufficient unto your ten minutes of hopeless, thrashing awareness is the lack of meta thereof.

That sounds like it'd work, but at the cost of eliminating most of the fuzzies you'd get from your altruism and most of your donation's social signaling value.

Doesn't that inherently make it a stronger signal when observed?

How many such communities can you be part of (because surely you don't only have one goal) and still not have them a diluted effect on yourself? How many such communities don't fall prey to lost purposes? How many can monitor your life with enough fidelity that they can tell if you go astray?

"Efficiency" at achieving something other than what you should work towards is harmful. ... Otherwise, you let that Blind Idiot Azathoth pick your purposes for you, trusting it more than you trust yourself.

The purpose of solving friendly AI is to protect the purposes picked for us by the blind idiot god.

For one, status-seeking is a zero sum game and only indirectly causes overall gains. The world would be a much better place if people actually cared about things like saving the world or even helping others, and put a little thought to it.

If I actually cared about saving the world and about conserving my resources, it seems like I would choose some rate of world-saving A.

If I actually cared about saving the world, about conserving my resources, and the opinion of my peers, it seems like I would choose some rate of world-saving B. For reasonable scenarios, B would be greater than A because I can also get respect from my peers, and when you raise demand and keep supply constant quantity supplied increases.

That is, I understand that status causes faking behavior that's a drain. (Status conflicts also lower supply, but it's not clear how much.) I don't think it's clear that the mechanism of status-seeking conflicts with actually caring about other goals or detracts from them on net.

I'm sure you've considered that "X is a 0 sum game" doesn't always mean that you should unilaterally avoid playing that game entirely. It does mean you'll want to engineer environments where X taxes at a lower rate.

For one, the world would be a much better place if people actually cared about things like saving the world or even helping others, and put a little thought to it.

Why do you want to save the world? To allow people, humans, to do what they like to do for much longer than they would otherwise be able to. Status-seeking is one of those things that people are especially fond of.

Ask yourself, would you have written this post after a positive Singularity? Would it matter if some people were engaged in status games all day long?

What you are really trying to tell people is that they want to help solving friendly AI because it is universally instrumentally useful.

In case you want to argue that status-seeking is bad, no matter what, under any circumstances, you have to explain why that is so. And if you are unable to ground utility in something that is physically measurable, like the maximization of certain brain states, then I don't think that you could convincingly demonstrate it to be a relatively undesirable human activity.

For one, status-seeking is a zero sum game and only indirectly causes overall gains.

But if status-seeking is what you really want, as evidenced by your decisions, how can you say it's bad that you do it? Can't I just go and claim any goal you're not optimizing for as your "real" goal you "should" have? Alternatively, can't I claim that you only want us to drop status-seeking to get rid of the competition? Where's your explanatory power?

By "caring about truth" here do you mean wanting systems to make explicit utterances that accurately reflect their actual motives? E.g., if X is a chess-playing AI that doesn't talk about what it wants at all, just plays chess, would a person who "cares about truth" would also be motivated to give X the ability and inclination to talk about its goals (and do so accurately)?

Or wanting systems not to make explicit utterances that inaccurately reflect their actual motives? E.g., a person who "cares about truth" might also be motivated to remove muflax's AI's ability to report on its goals at all? (This would also prevent it from winning Diplomacy games, but we've already stipulated that isn't a showstopper.)

I don't see how this is admirable at all. This is coercion.

If I work for a charitable organization, and my primary goal is to gain status and present an image as a charitable person, then efforts by you to change my mind are adversarial. Human minds are notoriously malleable, so it's likely that by insisting I do some status-less charity work you are likely to convince me on a surface level. And so I might go and do what you want, contrary to my actual goals. Thus, you have directly harmed me for the sake of your goals. In my opinion this is unacceptable.

Honest question: how do you know you have these goals? Presumably they don't manifest in actual behavior, or you wouldn't have a problem. If Kaj's analysis is right, shouldn't you assume that the belief of having these goals is part of your (working) strategy to gain certain status? Would you accept the same argument if Bruce made it?

Yes. It might be doing exactly what it was designed to do, but its designer was clearly stupid or cruel and had different goals than I'd prefer the AI to have.

Where are you getting your goals from? What are you, except your design? You are what Azathoth build. There is no ideal you that you should've become, but which Azathoth failed to make.

Not sure if I understand you correctly; let me try to rephrase it.

You are saying it is possible I claim confusion because I expect to gain status (contrarian status maybe?), as per Kaj's post, instead of being actually confused? Sure. I considered it, but rejected it because that weakens the explanatory power of status signalling. (I'm not sure if I agree with the signalling assumption, but let's for the sake of the argument.)

A real problem exists if an agent tries to optimize for a goal, but sucks at it. It's own beliefs are not relevant (unless the goal is about its beliefs). If Kaj is correct, then humans are optimizing for status, but sacrifice some accuracy of their self-modelling power. It seems to work out, so how is this problematic?

In other words, an agent wants X. It models itself to get better at getting X. The self-model is, among other things, the basis for communication with other agents. The self-model is biased to model itself wrongly as wanting Y. It is advantageous for the agent to be seen as wanting Y, not X. The inaccurate self-model doesn't cause substantial damage to its ability to pursue X, and it is much easier for the self-model to be biased than to lie. This setup sounds like a feature, not like a bug. If you observed it in an organism that wasn't you, wasn't even human, would you say the organism has a problem?

If you would recommend that the friend donate to SIAI...

Yes, I would.

Or, for a more radical thought experiment, ask yourself how much of SIAI's budget you would reallocate to your personal bank account given the chance.

I wouldn't reallocate any money to my personal bank account.

I don't see much of any explanation outside the endowment effect.

Okay, maybe I can overcome that at some point. But I think other factors like the predicted reaction of my surroundings also weigh heavily in my opinion.

For me:

1) The funds I have I might need later, and I'm not willing to take chances on my future. 2) Uncertainty as to whether the money would do much good. 3) Selfishness; my own well-being feels more important than a cause that might save everyone . 4) Potential benefits from SIAI are far and there's no societal pressure for donating money 5) I like money. Giving money hurts. It's supposed to go in my direction dammit.

I do anticipate that if I were to attain a sufficient amount of money (i.e. enough to feel that I don't need to worry about money ever again) I would donate, though probably not exclusively to SIAI.

Edit: Looking at those five points, an idea formulates. What if there was a post exclusively for people to comment and say that they donated (possibly specifying the amount), and receiving karma in return? Strange as it is, karma might be more motivating (near) for some people than the prospect of saving humanity (far!). Just like the promise of extra harry potter fan-fiction.

Hardly anyone is willing to say why they don't give.

My brain tells me that the reason I've only given a tiny fraction of my wealth so far is that there is still valuable information to be learned regarding whether SIAI is the best charity to donate to.

• I feel fairly sure that I haven't yet acquired/processed all of the relevant information
• I feel fairly sure that the value of that information is still high
• I feel that I'm not choosing to acquire that information as quickly as I could be
• I'm not sure what will happen when the value of information dips below the value of acting promptly - whether I'll start giving massively or move onto the next excuse, i.e. I'm not sure I trust my future self.

Hardly anyone is willing to say why they don't give.

Because GiveWell says I can do better.

Hardly anyone is willing to say why they don't give.

In my case, reducing existential risks isn't high on my priority list. I don't claim to be an altruist. I donate blood, but this has the advantage from my point of view that it is bounded, and visible, and local in both time and space.

Or would tell you that large donations are strange or a bad idea?

They would think that I have gone mad and would probably be mad at me as a result.

I deny that having a goal as a "strategic feature" is incompatible with being sincerly and deeply motivated. That's my point.

More precisely : either one is consciously seeking gratitude, then he/she is cynical, but I think this is rarely the case. Either seeking gratitude is only one aspect of a goal that is sincerly pursued (which means that one wants to deserve that gatiude for real). Then there is no problem, the motivation is there.

Let me rephrase.

The assumption that there would exist pure gratitude-free goals is a myth: pursuing such goals would be absurd. (people who seem do perform gratitude-free actions are often religious people: they actually believe in divine gratitude).

Therefore social gratitude is an essential component of any goal and thus it is not correlated with lack of sincere motivation, nor does it "downgrade" the goal to something less important. It's just part of it.