LessWrong 2.0 Reader

I'm currently doing the Rethink Wellbeing IFS Course, and it's given me so much understanding of myself so quickly with no diminishing returns in sight yet, that it felt like the perfect time to apply More Dakka.

Therefore, I've used this list to generate ideas of how to apply More Dakka to my internal exploration, and found 30 strategies that sound super helpful :) 

A poem in honor of Stanislav Petrov

This is meant to be put to the Christmas carol "In the Bleak Midwinter" by Rossetti and Holst. Hopefully this can be occasionally sung like "For The Longest Term" is in EA spaces.

I tried to get Suno to sing this but can't yet get the lyrics, tune, and style all correct; this is the best attempt. I also will probably continue editing the lyrics because parts seem a bit rough, but I just wanted to write this up before everyone forgets about Petrov Day.

In the bleak midwinter
Petrov did forestall,
Smoke would block our sunlight,
Though it be mid-fall.
New York in desolation,
Moscow too,
In the bleak midwinter
We so nearly knew.

Computers blinked a warning:
Missiles on their way,
But Petrov chose to question
What the screens did say.
Had he sounded the alarm,
War would soon unfold,
Cities turned to ashes;
Ev'ry night so cold.

Toxic clouds loom o'er us,
Ash would fill the air,
Fields would yield no harvest,
Famine everywhere.
Radiation poisoning,
Sickness spreading wide,
Children weeping, starving,
With no place to hide.

But due to Petrov's wisdom
Spring will yet appear;
Petrov defied orders,
And reason conquered fear.
So we sing his story,
His deed we keep in mind;
From the bleak midwinter
He saved humankind.

Thanks, the game was very fun! I was playing as General Carter.

Some reflections:

• I looked at the citizens' comments, and while some of them were notable (@Jesse Hoogland [LW · GW] calling for the other side to nuke us <3), I didn't find anything important after the game started- I considered the overall change in their karma if one or two sides get nuked, but comments from the citizens were not relevant to decision-making (including threats around reputation or post downvotes).
• It was great to see the other side sharing my post [LW · GW] internally to calculate the probability of retaliation if we nuke them
• It was a good idea to ask whether looking at the source code is ok and then share it, which made it clear Petrovs won't necessarily have much information on whether the missiles they see are real.
• The incentives (+350..1000 LW karma) weren’t strong enough to make the generals try to win by making moves instead of winning by not playing, but I’m pretty happy with the outcome.
• It’s awesome to be able to have transparent and legible decision-making processes and trust each other’s commitments.
• One of the Petrovs preferred defeat to mutual destruction- I'm curious whether they'd report nukes if they were sure the nukes were real.

I'd claim that we kinda won the soft power competition:

• we proposed commitments to not first-strike;

  

• we bribed everyone (and then the whole website went down, but funnily enough, that didn’t affect our war room and diplomatic channel- deep in our bunkers, we were somehow protected from the LW downtime);

  

• we proposed commitments to report through the diplomatic channel if someone on our side made a launch, which disincentivized individual generals from unilaterally launching the nukes, allowed Petrovs to ignore scary incoming missiles, and possibly was necessary to win the game;

  

• finally, after a general on their side said they’ll triumph economically and culturally, General Brooks wrote a poem, and I generated a cultural gift, which made generals on the other side feel inspired. That was very wholesome and was highlighted in Ben Paces’s comment [LW(p) · GW(p)] after the game ended. I think our side triumphed here!

  

Thanks to everyone who participated for the experience!

I happened to log on at that time and thought someone had launched a nuke

Is anyone aware of any counterarguments that have been written up against this post's thesis?

As @faul_sname [LW · GW] notes, the $8T number (or $9.5T from on source cited in that wikipedia article) isn't plausible. At least, not without some very generous definitions of "cybercrime," "is," "costing," and "trillions."

By which I mean: if you squint really hard, and count all the money and time everyone everywhere is spending on all (digital and non-digital) cybercrime prevention and countermeasures, and try to estimate all the extra things people could do to generate value if they didn't have to worry about cybercrime, then sure, maybe you could get numbers up to a few trillion. 

But that's a bit like saying the cost of other crime includes all spending on the criminal and civil justice system, all spending on private security and surveillance by individuals and businesses, the entire salary of every cashier (since they wouldn't be needed if people would just count up their own purchases and leave payment), and every time someone doesn't do something because they don't want to go out wandering by themselves at 3am. Not actually a useful metric for deciding where it's worthwhile to increase or decrease resource allocations or to make regulatory decisions.

…But interestingly, if I then immediately ask you what you were experiencing just now, you won’t describe it as above. Instead you’ll say that you were hearing “sm-” at t=0 and “-mi” at t=0.2 and “-ile” at t=0.4. In other words, you’ll recall it in terms of the time-course of the generative model that ultimately turned out to be the best explanation.

In my review of Dennett's book [? · GW], I argued that this doesn't disprove the "there's a well-defined stream of consciousness" hypothesis since it could be the case that memory is overwritten (i.e., you first hear "sm" not realizing what you're hearing, but then when you hear "smile", your brain deletes that part from memory).

Since then I've gotten more cynical and would now argue that there's nothing to explain because there are no proper examples of revisionist memory.^[1] Because here's the thing -- I agree that if you ask someone what they experience, they're probably going to respond as you say in the quote. Because they're not going to think much about it, and this is just the most natural thing to reply. But do you actually remember understanding "sm" at the point when you first heard it? Because I don't. If I think about what happened after the fact, I have a subtle sensation of understand the word, and I can vaguely recall that I've heard a sound at the beginning of the word, but I don't remember being able to place what it is at the time.

I've just tried to introspect on this listening to an audio conversation, and yeah, I don't have any such memories. I also tried it with slowed audio. I guess reply here if anyone thinks they genuinely misremember this if they pay attention.

I found Section 6 particularly interesting! Here's how I understand it:

1. Most of our worries about AI stem from catastrophic scenarios, like AI killing everyone.
2. It seems that to prevent these outcomes, we don’t need to do extremely complex things, such as pointing AI towards the extrapolated values of humanity.
3. Therefore, we don’t need to focus on instilling a perfect copy of human values into AI systems.

From my understanding, this context relates to the "be careful what you wish for" problem with AI, where AI could optimize in dangerous or unexpected ways. There's a race here: can we control AI well enough to still gain its benefits?

However, I don't think you've provided enough evidence that this level of control is actually possible. Additionally, there’s the issue of deceptive alignment—I’m not convinced we could manage this "race" without receiving some kind of feedback from AI systems.

Finally, the description of the oracle AI in this section seems quite similar to the idea of corrigible AI.

I propose "token surprise" (as in type-token distinction). You expected this general type of thing but not that Ivanka would be one of the tokens instantiating it.

There might be a lesson here: If you play along the edge of threatening to destroy the world, you might actually destroy it even without making a decision to destroy it.