LessWrong 2.0 Reader

Please consider this comment [LW(p) · GW(p)].

Please consider this comment [LW(p) · GW(p)]

This is useful part of code:

curl --header "Content-Type: application/json" --request POST --data '{"searchString": "How to improve the communication skills in Japanese?", "N": 5}' "http://[2600:1f18:17c:2d43:338d:2669:3fa5:82f8]:3000/search"

No execution of script. Returns libgen MD5 and epub CFI

Fwiw the thing that scares me is the combination of three things:

1. Producing an accurate estimate requires "understanding" the input
2. Our standard for the quality of estimates will be very high
3. The approach is purely algorithmic (rather than e.g. using the flexible intelligence of LLMs / humans)

You presumably agree with (1) and (3). For (2), it seems like there's a lot of correlations you need to take into account for this to produce estimates that aren't wildly off:

1. Underestimates of risk: These are cases where the presumption of independence neglects correlations between things that happen for catastrophe-causing events. For example, maybe a model is planning to defect when RSA-2048 is factored, and it has N different consistency checks it runs to make sure we aren't fooling it about whether it has been factored. The method has to see that there's a common cause for all the consistency checks to return 1, otherwise it will underestimate risk by a factor exponential in N. (And tbc I think even separately from more exotic examples like RSA-2048, it will be relatively common for this pattern to arise with networks, e.g. it happens any time that in reality there is a common cause that produces many pieces of evidence, and then the network aggregates those pieces of evidence to infer the common cause.)
2. Overestimates of risk: These are cases where the presumption of independence neglects correlations that rule out catastrophe, and so lead to overestimates. For example, the model might be willing to make up fake instructions for creating bioweapons to help with scifi writing, but never provide real instructions. So the method needs to capture the "understanding" of which instructions are fake vs real.

I agree this isn't a proof of impossibility, since a purely algorithmic approach (SGD) produced the "understanding" in the first place, so in theory a purely algorithmic approach could still capture all that understanding to produce accurate estimates. But it does seem heuristically like you should assign a fairly low probability that this pans out.

Please gpg sign the code and distribute it yourself if you trust it. It is less than 50 lines.

Could you send error? Works for me

Please gpg sign the code if you trust it. It is less than 50 lines.

And this is where the fundamental AGI-doom arguments – all these coherence theorems, utility-maximization frameworks, et cetera – come in. At their core, they're claims that any "artificial generally intelligent system capable of autonomously optimizing the world the way humans can" would necessarily be well-approximated as a game-theoretic agent.

These "coherence theorems" rest on very shaky ground. See this post. [LW · GW] I don't think it is appropriate to merely gesture at the existence of such theorems, as if they are widely accepted as unproblematic, without pointing to a specific one. There are other, informal, ways to make a similar point, but acting as if there are well-established, substantial proofs of the matter is not justified.

(this is completely sideways, but recently I found myself thinking that other exams should not be testing intelligence instead of their stated purpose. When an English test requires you to hold in your head facts, ..., it starts to be less about English and more about "something else".)

So, for starters, my own estimates of the likelihood of AI doom are nowhere near the 90%+ range, but seemingly much higher than Paul's.

My main concern with Paul's arguments about kindness, though, has nothing to do with AI specifically. I think the arguments greatly overestimate the prevalence and strength of kindness among humans. Humans produced St. Francis preaching to animals and Spinoza arguing it is impossible-in-principle for animals to suffer or for their apparent suffering to matter (and yes, many modern humans still believe that about many nonhuman animals, even if they wouldn't personally torture a puppy). We produced both Gandhi rejecting violence against enemies and Hitler freely committing genocide of other human groups. In both cases many millions of other nearby humans with an average distribution of starting views went along with these extremes to huge real-world effect. Human kindness and its boundaries depend quite a lot on how individual humans define their in-groups, and that kindness can very easily be (and often has been) zero for animals or other humans not within that circle.

In that context, I find "AI will inherit a sliver of kindness from humans," to be an extremely disconcerting claim that's far too weak to say anything about whether the AI will care about humans-in-general or any specific group of humans. There's a wide spectrum from Buddha to Stalin, and any point on that spectrum is compatible with Paul's claims. If AI is trained to emulate arbitrary human minds the way LLMs are, then it will be able to embody any of those points in response to the right conditions.

To be less abstract about it: let's suppose Paul is right and AI will inherit shards of all the values humans have. How many of your own children would need to be threatened with starvation for you to be willing to kill a dog for meat, or to just not have to feed it? In a world where your food potentially consists of all available free energy and you could, if you want, convert all matter into your descendants about whom you care that much, how confident are you that all humans with similar power levels would land on "pet" (or "live free in protected wilderness") rather than "meat"? Conversely, how confident are you that you could train another human to not care that much about themselves or their descendants, before letting them make that choice for you?

Humans do seem to have strong preferences over immediate actions.

I know, sometimes they do. But if they always did then they would be pretty useless. Habits are another example. Robust-to-new-obstacles behavior tends to be is driven by the future goals.

I expect that in multi-agent environments, there is significant pressure towards legibly having these kinds of strong preferences over immediate actions. As such, I expect that that structure of thing will show up in future intelligent agents, rather than being a human-specific anomaly.

Yeah same. Although legible commitments or decision theory can serve the same purpose better, it's probably harder to evolve because it depends on higher intelligence to be useful. The level of transparency of agents to each other and to us seems to be an an important factor. Also there's some equilibrium, e.g. in an overly honest society it pays to be a bit more dishonest, etc.

It does unfortunately seem easy and useful to learn rules like honest-to-tribe or honest-to-people-who-can-tell or honest-unless-it's-really-important or honest-unless-I-can-definitely-get-away-with-it. 

attempting to achieve their long-term goal at any cost

I think if you remove "at any cost", it's a more reasonable translation of "going hard". It's just attempting to achieve a long-term goal that is hard to achieve. I'm not sure what "at any cost" adds to it, but I keep on seeing people add it, or add monomaniacally, or ruthlessly. I think all of these are importing an intuition that shouldn't be there. "Going hard" doesn't mean throwing out your morality, or sacrificing things you don't want to sacrifice. It doesn't mean being selfish or unprincipled such that people don't cooperate with you. That would defeat the whole point.

It's not a binary choice between "care about process" and "care about outcomes" - it is possible and common to care about outcomes, and also to care about the process used to achieve those outcomes.

Yes!

It does seem to me that "we have a lot of control over the approaches the agent tends to take" is true and becoming more true over time.

No!

I doubt that systems trained with ML techniques have these properties. But I don't think e.g. humans or organizations built out of humans + scaffolding have these properties either

Yeah mostly true probably.

and I have a sneaking suspicion that the properties in question are incompatible with competitiveness.

I'm talking about stability properties like "doesn't accidentally radically change the definition of its goals when updating its world-model by making observations". I agree properties like this don't seem to be on the fastest path to build AGI.