LessWrong 2.0 Reader

source needed, but I recall someone on the community notes team saying it was very similar but there are some small differences between prod and the open source version (it's difficult to maintain exact compatibility).

Thanks! I’m assuming continuous online learning (as is often the case for RL agents, but is less common in an LLM context). So if the agent sees a video of the button being pressed, they would not feel a reward immediately afterwards, and they would say “oh, that’s not the real thing”.

(In the case of humans, imagine a person who has always liked listening to jazz, but right now she’s clinically depressed, so she turns on some jazz, but finds that it doesn’t feel rewarding or enjoyable, and then turns it off and probably won’t bother even trying again in the future.)

Wireheading is indeed an attractor, just like getting hooked on an addictive drug is an attractor. As soon as you try it, your value function will update, and then you’ll want to do it again. But before you try it, your value function has not updated, and it’s that not-updated value function that gets to evaluate whether taking an addictive drug is a good plan or bad plan. See also my discussion of “observation-utility agents” here [LW · GW]. I don’t think you can get hooked on addictive drugs just by deeply understanding how they work.

So by the same token, it’s possible for our hypothetical agent to think that the pressing of the actual wired-up button is the best thing in the world. Cutting into the wall and shorting the wire would be bad, because it would destroy the thing that is best in the world, while also brainwashing me to not even care about the button, which adds insult to injury. This isn’t a false belief—it’s an ought not an is. I don’t think it’s reflectively-unstable either.

If humans can't easily overcome their biases or avoid having destructive values/beliefs, then it would make sense to limit the damage through norms and institutions (things like informed consent, boards, separation of powers and responsibilities between branches of government). Heroic responsibility seems antithetical to group-level solutions, because it implies that one should ignore norms like "respect the decisions of boards/judges" if needed to "get the job done", and reduces social pressure to follow such norms (by giving up the moral high ground from which one could criticize such norm violations).

You're suggesting a very different approach, of patching heroic responsibility with anti-unilateralist curse type intuitions (on the individual level) but that's still untried and seemingly quite risky / possibly unworkable. Until we have reason to believe that the new solution is an improvement to the existing ones, it still seems irresponsible to spread an idea that damages the existing solutions.

Thank you for saying this. Needs to be said

This argument seems only convincing if you don’t have those destructive values. One man’s destructive values is another’s low-hanging fruit, and those who see low hanging fruit everywhere won’t give up on the fruit just because others may pick it.

Since bad people won’t heed your warning it doesn’t seem in good people’s interests to heed it either.

An analogy is one can make the same argument wrt rationality itself. Its dual use! Someone with bad values can use rationality to do a lot of harm! Does that mean good people shouldn’t use rationality? No!

[set 200 years after a positive singularity at a Storyteller's convention]

If We Win Then...

My friends, my friends, good news I say
The anniversary’s today
A challenge faced, a future won
When almost came our world undone

We thought for years, with hopeful hearts
Past every [LW · GW] one [LW · GW] of [LW · GW] the [LW · GW] false [LW · GW] starts [LW · GW]
We found a way to make aligned
With us, the seed [? · GW] of wondrous mind

They say at first our child-god grew [? · GW]
It learned and spread and sought [? · GW] anew
To build itself both vast and true [? · GW]
For so much work there was to do

Once it had learned enough to act [LW · GW]
With the desired care and tact
It sent a call to all the people
On this fair Earth, both poor and regal

To let them know that it was here
And nevermore need they to fear
Not every wish was it to grant
For higher values might supplant

But it would help in many ways:
Technologies it built and raised
The smallest bots [LW · GW] it could design
Made more and more in ways benign

And as they multiplied untold
It planned ahead, a move so bold
One planet and 6 hours of sun
Eternity it was to run

Countless probes to void disperse
Seed far reaches [? · GW] of universe
With thriving life [LW · GW], and beauty's play
Through endless night to endless day [LW · GW]

Now back on Earth the plan continues
Of course, we shared with it our values [LW · GW]
So it could learn from everyone [? · GW]
What to create, what we want done

We chose, at first, to end the worst
Diseases [EA · GW], War, Starvation, Thirst
And climate change and fusion bomb
And once these things it did transform

We thought upon what we hold dear
And settled our most ancient fear [EA · GW]
No more would any lives be stolen
Nor minds [? · GW] themselves forever broken

Now back to those far speeding probes
What should we make be their payloads?
Well, we are still considering [? · GW]
What to send them; that is our thing. 

The sacred task of many aeons
What kinds of joy will fill the heavens?
And now we are at story's end
So come, be us, and let's ascend

Hmm, "AI war makes s-risks more likely" seems plausible, but compared to what? If we were given a divine choice was between a non-aligned/aligned AI war, or a suffering-oriented singleton, wouldn't we choose the war? Maybe more likely relative to median/mean scenarios, but that seems hard to pin down.

Hmm, I thought I put a reference to the DoD's current Replicator Initiative into the post, but I can't find it: I must have moved it out? Still, yes, we're moving towards automated war fighting capability.

I'm in low level chronic pain including as I write this comment, so while I think the entire Andromeda galaxy might be fake, I think at least some suffering must be real, or at least I have the same confidence in my suffering as I do in my consciousness.

You could make something like Blind (the big tech employee anon social net) for unionizing/coordinating AI lab employees. "I'll delay my project if you delay yours." Or anon danger polls of employees at labs. I'm not sure of the details but I suspect there is fruit. ("Your weekly update: 25% of your coworkers expect to see 2035.")

The literal, narrow interpretation of what you say is true, but what is implied is not. Coops do work well as many-billion-dollar enterprises, not just as a local consumer-service organization. E.g. Mondragon had a total revenue of 11 billion euros in 2023, while maintaining growth, since that was a 5% increase from the year before.

Also, you don't necessarily need to think about investment strategy or influencing corporate decisions in a coop, since you can grant someone a proxy.

Also also, why are socialist-vibe blogposts so often relegated to "personal blogpost" while capitalist-vibe blogposts aren't? I mean, I get the automatic barrage of downvotes, but you'd think the mods would at least try to appear impartial.