LessWrong 2.0 Reader

Two different meanings of “misuse”

The term "AI misuse" encompasses two fundamentally different threat models that deserve separate analysis and different mitigation strategies:

• Democratization of offense-dominant capabilities
  • This involves currently weak actors gaining access to capabilities that dramatically amplify their ability to cause harm. That amplification of ability to cause harm is only a huge problem if access to AI didn’t also dramatically amplify the ability of others to defend against harm, which is why I refer to “offense-dominant” capabilities; this is discussed in The Vulnerable World Hypothesis.
  • The canonical example is terrorists using AI to design bioweapons that would be beyond their current technical capacity (c.f.  Aum Shinrikyo, which failed to produce bioweapons despite making a serious effort)
• Power Concentration Risk
  • This involves AI systems giving already-powerful actors dramatically more power over others
  • Examples could include:
    • Government leaders using AI to stage a self-coup then install a permanent totalitarian regime, using AI to maintain a regime with currently impossible levels of surveillance.
    • AI company CEOs using advanced AI systems to become world dictator.
  • The key risk here is particular already-powerful people getting potentially unassailable advantages

These threats require different solutions:

• Misuse that involves offense-dominant capabilities can be addressed by preventing users of your AIs from doing catastrophically bad things, e.g. by training the models to robustly refuse requests that could lead to these catastrophic outcomes (which might require improvements in adversarial robustness), or by removing dangerous knowledge from the AI training data.
• Power concentration risks require different solutions. Technical measures to prevent users from using the AI for particular tasks don’t help against the threat of the lab CEO trying to use the AI for those harmful tasks, or the threat of the US government expropriating the AI system and using it for their own purposes. To resist against these threats, interventions include:
  • Computer security, to prevent powerful third parties from stealing model weights and using them in bad ways.
  • Security against insider threats, to make it harder for the AI to be misused internally. This is a classic insider threat problem; addressing it will require both technical interventions and workflow changes inside AI companies.

Many discussions of "AI misuse" focus primarily on interventions that only help with the first category, while using rhetoric that suggests they're addressing both. This creates a motte-and-bailey situation where:

• The "motte" (easily defensible position) is "we need to prevent terrorists from using AI for mass harm"
• The "bailey" (broader claim) is "our work on AI misuse prevention will solve the major misuse risks from AI"

This conflation is dangerous because it may lead us to overinvest in technical solutions that only address the less concerning risk, and underinvest in countermeasures for power concentration risks.

1. Butterfly ideas [LW · GW]?
2. By default I expect the author to have a pretty strong stance on the main idea of a post, also the content are usually already refined and complete, so the barrier of entry to having a comment that is valuable is higher.

Bob can choose whether to to hide this waste (at a cost of the utility loss by having $300 and worse listening experience, but a "benefit" of misleading Tim about his misplaced altruism)

True in my example. I acknowledge that my example is wrong and should have been more explicit about having an alternative. Quoting myself from the comment to Vladimir_Nesov:

Anyways, the unwritten thing is that Bob care about having a quality headphone and a good pair of shoes equally. So given that he already has an alright headphone, he would get more utility by buying a good pairs of shoes instead. It is essentially a choice between (a) getting a $300 headphone and (b) getting a $100 headphone and a $300 pair of shoes.

If the bad translation is good enough that the incremental value of a good translation doesn't justify doing it, then that is your answer.

I do accept this as the rational answer, doesn't mean it is not irritating. If A (skillful translator) cares about having a good translation of X slightly more than Y, and B (poor translator) cares about Y much more than X. If B can act first, he can work on X and "force" A (via expected utility) to work on Y. This is a failure of mine to not talk about difference in preference in my examples and expect people to extrapolate and infer it out.

I've noticed that there's a pretty big difference in the discussion that follows from me showing someone a draft of a post and asking for comments and the discussion in the comments section after I publish a post. The former is richer and more enjoyable whereas the latter doesn't usually result in much back and forth. And I get the sense that this is true for other authors as well.

I guess one important thing might be that with drafts, you're talking to people who you know. But I actually don't suspect that this plays much of a role, at least on LessWrong. As an anecdote, I've had some incredible conversations with the guy who reviews drafts of posts on LessWrong for free and I had never talked to him previously.

I wonder what it is about drafts. I wonder if it can or should be incorporated into regular posts.

That seems to be getting into Game Theory [? · GW] territory. One can model agents (players) with different strategies, even suboptimal ones. A lot of the insight from Game Theory isn't just about how to play a better strategy, but how changing the rules affects the game.

Again, seems like we are in agreement lol. I agree with what you said and I meant that, but tried to compress it into one sentence and failed to communicate.

This is the core reason why it is so difficult for ordinary people to pay their bills or raise families, despite earnings that would make them rich elsewhere or elsewhen. These productive actions are severely restricted, because if you are going to be productive then you have to do so ‘correctly’ and obey all sorts of rules and requirements.

There are plenty of good things that aren't restricted and bad things that are. But elites are human and aren't going to get it right every time, and you'll notice most the cases where they got it wrong.

Science hasn't increased our social skills nor our understanding of ourselves, modern wisdom and life advice is not better than it was 2000 years ago. 

Hard disagree, there's an entire field of psychology, decision theory and ethics using reflective equilibrium in light of science. 

Ancient wisdom can fail, but it's quite trivial for me to find examples in which common sense can go terribly wrong. It's hard to fool-proof anything, be it technology or wisdom.

Well some things go wrong more often than other things, wisdom goes wrong a lot of time, it isn't immune to memetic selection, there is not much mechanism to prevent you from falling for false memes. Technology after one point goes wrong wayyy less. A biology textbook is much more likely to be accurate and better on medical advice than a ayurvedic textbook. 

The whole "Be like water" thing is just flexibility/adaptability
 

Yes it's a metaphor for adaptiveness, but I don't understand where it may apply other than being a nice way to say "be more adaptive". It's like logical model like maths but for adaptiveness you import the idea of water-like adaptiveness into situations. 

As for that which is not connected to reality much (wisdom which doesn't seem to apply to reality), it's mostly just the axioms of human cognition/nature.

You know what might be an axiom of human cognition? Bayes rule and other axioms in statistics. I have found that I can bypass a lot of wisdom by using these axioms where others are stuck without a proper model in real life due to ancient wisdom. Eg; I stopped taking ayurvedic medication which contained carcinogens; when people spend hours thinking about certain principles in ethics or decision theory I know the laws to prevent such confusion etc

If you're in a good mood then the external world will seem better too. A related quote is "As you think, so you shall become" which is oddly simiar to the idea of cognitive behavioural therapy.

Honestly I agree with this part, I think this is the biggest weakness of rationalism. I think the failure to general purpose overcome akrasia is a failure of rationality. I find it hard to believe that there would be a person like david goggins but a rationalist. The obsession with accuracy doesn't play well with romanticism of motivation and self-overcoming, it's a battle you have to fight and figure out daily, and under the constraints of reality it becomes difficult.
 

Signal boost for the "username hiding" on homepage feature in settings - it seems cool, will see if it changes how I use LW.

I wonder also about a "hide karma by default". Though less sure if that will actually achieve the intended purpose, as karma can be a good filter when just skimming comments and not reading in detail. 

a belief is only really worthwhile if you could, in principle, be persuaded to believe otherwise

There's a point to be made here about why 'unconditional love' is unsatisfying to the extent the description as 'unconditional' is accurate.