The danger of living a story - Singularity Tropes

I can see how for your audience, the story-like qualities would be a minus. On the other hand, I think the story bias has to do with how people cognitively process information and arguments. If you can't tell your mission & strategy as a story, it's a lot harder to get across your ideas, whatever your audience.

The battle was meant to be metaphorical - the battle to ensure that AI is Friendly rather than Unfriendly. And I didn't say anything about hostile humans - the problem is indifferent humans not giving you resources.

Also, I'm not arguing against SIAI, I just find it amusing how well the futurist sector maps onto a story outline - various protagonists passionate about fighting some great evil that others don't see and trying to build alliances and grow resources before time runs out. You can squiggle, but that's who you are. Instrumental rationality means figuring out how to make best positive use of it and avoid it biasing you.

Actually, I will comment (for the purposes of authenticity and from the belief that being more transparent about my motivations will increase mutual truth-finding) that while I'm not arguing "against" SIAI, this post is to some degree emerging from me exploring the question of SIAI's organizational instrumental rationality. I have the impression from a variety of angles/sources that it's pretty bad. Since I care about SIAI's success, it's one of the things I think about in the background - why, and how you could be more effective.

It's actually a disaster story, not a battle story, which is something I'm surprised Patrissimo missed in the opening paragraph.*

*(Possibly because disaster movie protagonists tend to be older, so it wouldn't fit that bit quite as well)

Your "enemies" are those too shortsighted to realise the true dangers, and your aim is to reveal the dangers to them, and save the world.

But as others have stated, with sufficient attention any story can be mapped to a combination of tropes (inverted, subverted and played straight).

There will be no robot war [...]

Maybe not - but if there are wars to come, they will probably involve many robots.

Quite possibly they will be fighting on both sides.

More:

• Overestimating the agency of everyone and everything
• Role-playing instead of trying to achieve goals
• Expecting too clean a distinction between protagonists and antagonists
• Underestimating the number, and overestimating the cohesiveness, of protagonists
• Overly anticipating unlikely but dramatic events

(Most of these are more accurately described as "errors almost everyone makes all the time" than "dangers of thinking you're in a story", but thinking of them that way seems pretty useful for identifying them.)

If you thought that you were living in a realistic/naturalistic story, you'd be underestimating your chance of making a significant difference. I have no idea what thinking you were living in a tragedy would do to your presuppositions.

I don't know if you have peasant ancestry, especially peasants living in a non- or partially-democratic society; I do, and when I talked to my grandparents and granduncles, as a young boy raised on hero stories and supported by his family, I was struck by how tremendously fatalistic their outlook on life was. While they worked hard and smart, and aimed at steadily improving their lot in life (and, eventually, succeeded - my father was able to get an engineering degree), the idea that they were and would always be part of the background scenery of history was very fixed in their mind; no matter how proud they were of their debt-free house and comfortable retirement money, they never even considered that it might be possible for them to be concerned about matters beyond their family's immediate needs - that was "gentlemen's stuff", while they were but "poor people".

To cut short, my point is that that type of literature is called "Realism" or "Naturalism" for a reason - because to a significant degree, and especially when compared with Romantic literature, that was actually how nineteenth-century peasants thought and lived (and not just peasants, I believe - Joseph Roth's Radetzkymarsch deals with the middle class but it's one of the most depressing novels I've ever read). The answer to your question is in those books themselves.

Five months ago, someone asked the question "What kind of evidence would convince you that you were in a story?". Will Newsome answered:

If something totally crazy seemed like it was about to happen and the world was at stake, like a technological singularity was about to occur or something, and I was called to work for the team of great minds that were trying their hardest to stop the destruction of the entire universe, dropping out of high school in the process, and meeting a beautiful girl who had been living literally a few houses down from me for the last 4 years without my knowing about it, who just so happened to be doing an essay on transhumanism for her English class and would just love to interview someone who was doing work for the Singularity Institute.

Oh wait...

Anything can be mapped to tropes, but not all tropes are the same. It matters what tropes your life, mission, or organization are mapped to! To skillfully navigate the world (I guess the LW term is "to win") you must know what tropes are being mapped to you, and what tropes your brain sees your identity as fitting into. That way you can manipulate others' perception of you (what stories are they telling about you? How are they telling those stories? Do they gain you status and resources), as well as making sure you aren't fooling yourself.

While I was deeply amused the first time you wrote that, I'm not sure that that is evidence of everything being mappable to tropes so much as how much common tropes are informed and shaped by actual history.

First, I'm not claiming a connection between truth and tropism, but this idea that everything is equally tropish seems wrong. Not everyone has the role of a protagonist fighting for humanity against a great inhuman evil that only they foresee, and struggling to gather allies and resources before time runs out. Yet Eliezer has that role.

Second, even though tropes apply to everyone's lives to some degree, it matters which tropes they are. For example, someone who sees themselves as a fundamentally misunderstood genius who deserves much more than society has given them is also living a trope - but it's a very different trope with very different results. Identifying the tropes you are living is useful - it helps in your personal branding, can teach you lessons about strategies for achieving your goal, and may show you pitfalls.

For example, I live a very similar trope set to Eliezer, which is why I notice it, and it poses many challenges in being effective, because it's tempting to (as Nick alluded to above) play the role rather than doing the work.

I'm sure I'm not the only one tempted here to make up some top-level post about how Eliezer chooses what he had for breakfast in a completely rational manner, and carefully avoids biases such as how his mother raised him on boiled eggs or how delicious chocolate chips taste, but thinks things out well enough that he does not have to jump through the same logical hoops every single morning...

Have you read Terry Pratchett's Witches Abroad?

And, of course, Granny Weatherwax.

That's basically the Reluctant Hero archetype. Of course, from the writer's viewpoint, the outstanding feature of that kind of hero is that you have to make sure his attempt to keep a non-story life fails or, well, you won't have a good story.

It would be nice to have stories about heroes who are aware of and resist the temptation to make their lives into stories.

Somehow I don't think those would make very good stories.

Mieville's Un Lun Dun has somewhat in that direction.

Znva punenpgre tbrf qverpgyl gb fbyivat ceboyrz engure guna pbyyrpgvat cybg pbhcbaf. Nyfb, cebcurpvrf ghea bhg abg gb jbex.

I'd prefer to be the comic relief, but I worry I'm a redshirt!

"So part of winning is being able to deal with human susceptibility to think in stories."

Exactly! It is especially relevant if you are trying to grow a following around an idea, which SIAI is. Winning requires wearing your Slytherin hat sometimes, and an effective Slytherin will manipulate the stories that they tell and the stories that are told about them.

SIAI's story by Frank Miller would be less insightful, but probably much more colorful.