LessWrong 2.0 Reader

I am sort of confused at what you got from this post. You say "But also, it doesn't do the core intellectual work of replacing a pointer with its substance.", but, I think he explicitly does that here?

What does "bad faith" mean, though? It doesn't mean "with ill intent." Following Wikipedia, bad faith is "a sustained form of deception which consists of entertaining or pretending to entertain one set of feelings while acting as if influenced by another." The great encyclopedia goes on to provide examples: the solider who waves a flag of surrender but then fires when the enemy comes out of their trenches, the attorney who prosecutes a case she knows to be false, the representative of a company facing a labor dispute who comes to the negotiating table with no intent of compromising.

That is, bad faith is when someone's apparent reasons for doing something aren't the same as the real reasons. This is distinct from malign intent. The uniformed solider who shoots you without pretending to surrender is acting in good faith, because what you see is what you get: the man whose clothes indicate that his job is to try to kill you is, in fact, trying to kill you.

It feels like you wanted some entirely different post, about how to navigate when someone accuses someone of bad faith, for a variety of possible definitions of bad faith? (As opposed to this post, which is mostly saying "Avoid accusing people of bad faith. Instead, do some kind of more specific and useful thing." Which honestly seems like good advice to me even for most people who are using the phrase to mean something else. "I disapprove of what you're doing here and am leaving now" seems totally fine)

I wish that both the "tip top" and the "medium top" of educated people had consensus that both "property is downstream of power" and "prices are downstream of supply and demand".

I really don’t like that quote. I can see multiple interpretations for "property is downstream of power", many of them subject to debate, and none of the firm grounding (and non-ambiguity) of Econ 101 enjoyed by "price are downstream of supply and demand".

It looks like to me a false equivalence, trying to say "see ? both sides clearly have a point", where one point is actually clear, and the other isn’t so clear.

Would you mind clarifying what the consensus should be exactly on this point ? Maybe I’m just not interpreting "property is downstream of power" correctly.

First, RE the role of "solving alignment" in this discussion, I just want to note that:
1) I disagree that alignment solves gradual disempowerment problems.
2) Even if it would that does not imply that gradual disempowerment problems aren't important (since we can't assume alignment will be solved).
3) I'm not sure what you mean by "alignment is solved"; I'm taking it to mean "AI systems can be trivially intent aligned".  Such a system may still say things like "Well, I can build you a successor that I think has only a 90% chance of being aligned, but will make you win (e.g. survive) if it is aligned.  Is that what you want?" and people can respond with "yes" -- this is the sort of thing that probably still happens IMO.
4) Alternatively, you might say we're in the "alignment basin" -- I'm not sure what that means, precisely, but I would operationalize it as something like "the AI system is playing a roughly optimal CIRL game".  It's unclear how good of performance that can yield in practice (e.g. it can't actually be optimal due to compute limitations), but I suspect it still leaves significant room for fuck-ups.
5) I'm more interested in the case where alignment is not "perfectly" "solved", and so there are simply clear and obvious opportunities to trade-off safety and performance; I think this is much more realistic to consider.
6) I expect such trade-off opportunities to persist when it comes to assurance (even if alignment is solved), since I expect high-quality assurance to be extremely costly.  And it is irresponsible (because it's subjectively risky) to trust a perfectly aligned AI system absent strong assurances.  But of course, people who are willing to YOLO it and just say "seems aligned, let's ship" will win.  This is also part of the problem...


My main response, at a high level:
Consider a simple model:

• We have 2 human/AI teams in competition with each other, A and B.
• A and B both start out with the humans in charge, and then decide whether the humans should stay in charge for the next week.
• Whichever group has more power at the end of the week survives.
• The humans in A ask their AI to make A as powerful as possible at the end of the week.
• The humans in B ask their AI to make B as powerful as possible at the end of the week, subject to the constraint that the humans in B are sure to stay in charge.

I predict that group A survives, but the humans are no longer in power.  I think this illustrates the basic dynamic.

 

Responding to some particular points below:
 

Sure, but these things don't result in non-human entities obtaining power right? 

Yes, they do; they result in beaurocracies and automated decision-making systems obtaining power.  People were already having to implement and interact with stupid automated decision-making systems before AI came along.

Like usually these are somewhat negative sum, but mostly just involve inefficient transfer of power. I don't see why these mechanisms would on net transfer power from human control of resources to some other control of resources in the long run. To consider the most extreme case, why would these mechanisms result in humans or human appointed successors not having control of what compute is doing in the long run?

My main claim was not that these are mechanisms of human disempowerment (although I think they are), but rather that they are indicators of the overall low level of functionality of the world.  




 

But your P(doom) still only 0.6? Or are you considering disempowerment from AI separately?

I do think that convincing the government to pause AI in a way which sacrifices $3000 billion economic value, is relatively easier than directly spending $3000 billion on AI safety.

Maybe spending $1 is similarly hard to sacrificing $10-$100 of future economic value via preemptive regulation.

But $0.1 billion AI safety spending is so ridiculously little (1000 times less than capabilities spending), increasing it may still be the "easiest" thing to do. Of course we should still push for regulation at the same time (it doesn't hurt).

PS: what do you think of my open letter idea [LW · GW] for convincing the government to increase funding?

The "Taboo bad faith" title doesn't fit this post. I had hoped from the opening section that it was going in that direction, but it did not.

Most obviously, the post kept relying heavily on the terms "bad faith" and "good faith" and that conceptual distinction, rather than tabooing them.

But also, it doesn't do the core intellectual work of replacing a pointer with its substance. In the opening scenario where someone accuses their conversation partner of bad faith, conveying something along the lines of 'I disapprove of how you're approaching this conversation so I'm leaving', tabooing "bad faith" would mean articulating what pattern of behavior (they thought that) they saw and why disapproval & departure is an appropriate response. Zack doesn't try to do this, he just abandons this scenario to talk about other things involving his definition of "bad faith". (And similarly with "assume good faith".) I briefly hoped that the post would go in the "taboo your words" direction, describing what was happening in that sort of scenario with a clarity and precision that would make the label "bad faith" seem crude by comparison, but it did not. 

This post also doesn't manage to avoid the main pitfall that tabooing a word is meant to prevent, where people talk past each other because they're using the same word with different definitions. Even though he says at the start of the post that other people are using the term "bad/good faith" wrong according his understanding of the term, when he talks about the advice "assume good faith" he just plugs in his definition of "good faith" (and "assume") without noting that he's making an interpretation of what other people mean when they use the phrase and that they might mean something else. And similarly in other places like "being touchy about bad faith accusations seems counterproductive" and "the belief that persistent good faith disagreements are common would seem to be in bad faith". When someone says "you're acting in bad faith" are they claiming that you're showing the thing that Zack means by "bad faith"? Keeping that sort of thing straight is rationality 101 stuff that tabooing words helps with, and which this post repeatedly stumbles over.

I'm in! One suggestion. The URL for registering has de minimis info, https://www.havenbookings.space/events/Duncon

Ideally, that page should link to this one to help contextualize what's in the can

I edited the OP but wanted to add for people who missed it (bold part is new)

(Please generally be cautious on LessWrong talking about politics. I am interested in people commenting here who have read the LessWrong Political Prerequisites [? · GW] sequence. I'll be deleting or at least unhesitatingly strong downvoting comments that seem to be doing unreflective partisan dunking)

((But, that's not meant to mean "don't talk about political actions." If this is as big a deal as it sounds, I want to be able to talk about "what to do do?". But I want that talking-about-it to feel more like practically thinking through an action space, than blindly getting sucked into a political egregore))

I'm not sure if it means much, but I'd be very happy if AI safety could get another $50B from smart donors today.

I'd flag that [stopping AI development] would cost far more than $50B. I'd expect that we could easily lose $3T of economic value in the next few years if AI progress seriously stopped. 

I guess, it seems to me like duration is basically dramatically more expensive to get than funding, for amounts of funding people would likely want. 

From a software engineering perspective, misalignment is like a defect or a bug in software. Generally speaking, if a piece of software doesn't accept any user input is going to have fewer bugs than software that does. For a piece of software that doesn't accept any input or accepts some constrained user input, it's possible to formally prove that the software logic is correct. Think specialized software that controls nuclear power plants. To my knowledge, it's not possible to prove that software that accepts arbitrary unconstrained instructions from a user is defect free.

I claim that the Observer is the easiest ASI to align because it doesn't accept any instructions after it's been deployed and has a single very simple goal that avoids dealing with messy things like human happiness, human meaning, human intent, etc. I don't see how it could get simpler than that.