LessWrong 2.0 Reader

From a Paul Christiano talk called "How Misalignment Could Lead to Takeover":

Assume we're in a world where AI systems are broadly deployed, and the world has become increasingly complex, where humans know less and less about how things work.

A viable strategy for AI takeover is to wait until there is certainty of success. If a 'bad AI' is smart, it will realize it won't be successful if it tries to take over, not a problem. 

So you lose when a takeover becomes possible, and some threshold of AIs behave badly. If all the smartest AIs you produce end up having the same problem, then takeover becomes more likely.

In other words, the smartest AIs will only take over if all the other AIs will also try to takeover.

Additionally, this is more likely to happen in an unstable world that could come from:

• War (or other exogenous shocks): making it harder for humans to detect and respond to problems.
• Cascading loss of control: some AIs try takeover and aren't fully successful, but make the world increasingly crazy, which cascades into other AIs (who previously thought it would be unsuccessful) now joining the takeover.
• Models interacting with each other: once some AIs realize could be successful, they may want to share that information with other models to see if other models may want to join in.

There's this tempting feeling of training your AI against these takeover attempts ("hey, look at this bad AI who tried to takeover the data center, don't do that!"), but you may just be training your model to learn that it needs to go much bigger if it wants to actually succeed at the takeover attempt.

Paul believes that if this kind of phenomenon is real, then he expects we can get compelling demonstrations in a lab (that would require some imagination to bridge the examples in the lab vs the wild). We'll still get demonstrations in the wild, but unclear if they will be big enough to make humanity reconsider things.

They likely use them in places where no air defence is present and still at some disatnce using JDAM. 

I think that I missed the main thing about stealth - they are stealth for radar on the distances like 100 km, but visible for radar on the distances like 10 km (arbitrary numbers). But optical observation on distances of 100 km is impossible (need large telescopes, but you need to know where to look). Also optical density of atmosphere starts playing role as well a spherical size of earth.

I think this is a great idea, except that on easy mode "a good specification of values and ethics to follow" means a few pages of text (or even just the prompt "do good things"), while other times "a good specification of values" is a learning procedure that takes input from a broad sample of humanity, and has carefully-designed mechanisms that influence its generalization behavior in futuristic situations (probably trained on more datasets that had to be painstakingly collected), and has been engineered to work smoothly with the reasoning process and not encourage perverse behavior.

Some issues have legible, material stakes.

Scrolling down... the table of how important are individual topics for young people; "student debt" is at its very bottom.

(Also, inflation on the very top? But isn't inflation a good thing if all you have is an enormous debt?)

But then no need for stealth at all?

I wasn't referring to the A10, but the use of e.g. f-35s in ground support roles - as heavily practised by the IDF for example.

This, and also most people on ACX respect Scott and his opinions, so if he demonstrates that he has put a lot of thought into this, and then he makes a conclusion, it will sound convincing to most.

Basically, we need to consider not just how many people believe some idea, but also how strongly. The typical situation with a conspiracy theory is that we have a small group that believes X very strongly, and a large group that believes non-X with various degrees of strength, from strongly to almost zero. What happens then is that people with a strong belief typically don't change their mind, while people with zero belief (who until now just took one side by default, because they never heard about the other) will flip a coin. Therefore the typical outcome is that the conspiracy theory becomes better known.

Or maybe the zero belief is not literally "never heard about theory" but "never met an actual person who also believes the theory" and as the debate starts, they find each other, and thus the conspiracy theory becomes socially acceptable (even being in a minority feels very different from being alone).

When the conspiracy theory is wildly known, and everyone already knows a few believers, most damage was already done.

Rationalist-adjacent community is often the opposite of the wider society, in that the mainstream beliefs are low-status, and we need to be reminded that they sometimes actually exist for a good reason. There is always this suspicion that people who have mainstream beliefs are simply too stupid to think independently. Therefore a debate will improve the case of the mainstream belief.

First of all, it‘s certainly important to distinguish between a probability model and a strategy. The job of a probability model is simply to suggest the probability of certain events and to describe how probabilities are affected by the realization of other events. A strategy on the other hand is to guide decision making to arrive at certain predefined goals.

Of course. As soon as we are talking about goals and strategies we are not talking about just probabilities anymore. We are also talking about utilities and expected utilities. However, probabilities do not suddenly change because of it. Probabilistic model is the same, there are simply additional considerations as well. 

My point is, that the probabilities a model suggests you to have based on the currently available evidence do NOT neccessarily have to match the probabilities that are relevant to your strategy and decisions.

Whether or not your probability model leads to optimal descision making is the test allowing to falsify it. There are no separate "theoretical probabilities" and "decision making probabilities". Only the ones that guide your behaviour can be correct. What's the point of a theory that is not applicable to practice, anyway?

If your model claims that the probability based on your evidence is 1/3 but the optimal decision making happens when you act as if it's 1/2, then your model is wrong and you switch to a model that claims that the probability is 1/2. That's the whole reason why betting arguments are popular.

If Beauty is awake and doesn‘t know if it is the day her bet counts, it is in fact a rational strategy to behave and decide as if her bet counts today.

Questions of what "counts" or "matters" are not the realm of probability. However, the Beauty is free to adjust her utilities based on the specifics of the betting scheme.

All your model suggests are probabilities conditional on the realization of certain events.

The model says that 

P(Heads|Red) = 1/3 

P(Heads|Blue) = 1/3

but

P(Heads|Red or Blue) = 1/2

Which obviosly translates in a betting scheme: someone who bets on Tails only when the room is Red wins 2/3 of times and someone who bets on Tails only when the room is Blue wins 2/3 of times, while someone who always bet on Tails wins only 1/2 of time.

This leads to a conclusion that observing event "Red" instead of "Red or Blue" is possible only for someone who has been expecting to observe event "Red" in particular. Likewise, observing HTHHTTHT is possible for a person who was expecting this particular sequence of coin tosses, instead of any combination with length 8.  See Another Non-Anthropic Paradox: The Unsurprising Rareness of Rare Events [LW · GW]

I am honored to be part of enabling more people from around the world to contribute to the safe and responsible development of AI. 

Clouds are a thing.