LessWrong 2.0 Reader

This is a narrow objection to the IMO hyperbolic focus on government assault risks.

Whether or not you face government assault risks depends on what you do. Most people don't face government assault risks. Some people engage in work or activism that results in them having government assault risks.

The Chinese government has strategic goals and most people are unimportant to those. Some people however work on topics like AI policy in which the Chinese government has an interest. 

Note you don't even need high resolution in all directions, just high enough to see whether it's worth zooming in/switching to a better camera.

Why would you need large telescopes?

Naked eye has angular resolution of 30m at 100km, you need something slightly better. A small lense should do it. Cameras + zoom lens are well understood mass produced components. And this is a highly parallelizable task.

From a Paul Christiano talk called "How Misalignment Could Lead to Takeover" (from February 2023):

Assume we're in a world where AI systems are broadly deployed, and the world has become increasingly complex, where humans know less and less about how things work.

A viable strategy for AI takeover is to wait until there is certainty of success. If a 'bad AI' is smart, it will realize it won't be successful if it tries to take over, not a problem. 

So you lose when a takeover becomes possible, and some threshold of AIs behave badly. If all the smartest AIs you produce end up having the same problem, then takeover becomes more likely.

In other words, the smartest AIs will only take over if all the other AIs will also try to takeover.

Additionally, this is more likely to happen in an unstable world that could come from:

• War (or other exogenous shocks): making it harder for humans to detect and respond to problems.
• Cascading loss of control: some AIs try takeover and aren't fully successful, but make the world increasingly crazy, which cascades into other AIs (who previously thought it would be unsuccessful) now joining the takeover.
• Models interacting with each other: once some AIs realize could be successful, they may want to share that information with other models to see if other models may want to join in.

There's this tempting feeling of training your AI against these takeover attempts ("hey, look at this bad AI who tried to takeover the data center, don't do that!"), but you may just be training your model to learn that it needs to go much bigger if it wants to actually succeed at the takeover attempt.

Paul believes that if this kind of phenomenon is real, then he expects we can get compelling demonstrations in a lab (that would require some imagination to bridge the examples in the lab vs the wild). We'll still get demonstrations in the wild, but unclear if they will be big enough to make humanity reconsider things.

They likely use them in places where no air defence is present and still at some disatnce using JDAM. 

I think that I missed the main thing about stealth - they are stealth for radar on the distances like 100 km, but visible for radar on the distances like 10 km (arbitrary numbers). But optical observation on distances of 100 km is impossible (need large telescopes, but you need to know where to look). Also optical density of atmosphere starts playing role as well a spherical size of earth.

I think this is a great idea, except that on easy mode "a good specification of values and ethics to follow" means a few pages of text (or even just the prompt "do good things"), while other times "a good specification of values" is a learning procedure that takes input from a broad sample of humanity, and has carefully-designed mechanisms that influence its generalization behavior in futuristic situations (probably trained on more datasets that had to be painstakingly collected), and has been engineered to work smoothly with the reasoning process and not encourage perverse behavior.

Some issues have legible, material stakes.

Scrolling down... the table of how important are individual topics for young people; "student debt" is at its very bottom.

(Also, inflation on the very top? But isn't inflation a good thing if all you have is an enormous debt?)

But then no need for stealth at all?

I wasn't referring to the A10, but the use of e.g. f-35s in ground support roles - as heavily practised by the IDF for example.

This, and also most people on ACX respect Scott and his opinions, so if he demonstrates that he has put a lot of thought into this, and then he makes a conclusion, it will sound convincing to most.

Basically, we need to consider not just how many people believe some idea, but also how strongly. The typical situation with a conspiracy theory is that we have a small group that believes X very strongly, and a large group that believes non-X with various degrees of strength, from strongly to almost zero. What happens then is that people with a strong belief typically don't change their mind, while people with zero belief (who until now just took one side by default, because they never heard about the other) will flip a coin. Therefore the typical outcome is that the conspiracy theory becomes better known.

Or maybe the zero belief is not literally "never heard about theory" but "never met an actual person who also believes the theory" and as the debate starts, they find each other, and thus the conspiracy theory becomes socially acceptable (even being in a minority feels very different from being alone).

When the conspiracy theory is wildly known, and everyone already knows a few believers, most damage was already done.

Rationalist-adjacent community is often the opposite of the wider society, in that the mainstream beliefs are low-status, and we need to be reminded that they sometimes actually exist for a good reason. There is always this suspicion that people who have mainstream beliefs are simply too stupid to think independently. Therefore a debate will improve the case of the mainstream belief.