LessWrong 2.0 Reader

Where does probability theory come from anyway? Maybe I can find some clues that way? Well according to von Neumann and Morgenstern, it comes from decision theory.

I believe this is the step from where you started going astray. The next steps of your intellectual journey seem to be repeating the same mistake: attempting to reduce a less complex thing to a more complex one.

Probability Theory does not "come from" Decision Theory. Decision Theory is strictly more complicated domain of math as it involves all the apparatus of Probability Spaces but also utilities over events. 

We can validate probability theoretic reasoning by appeals to decision theoretic processes such as iterated betting, but only if we already know which probability space corresponds to a particular experiment. And frankly, at this point this is redundant. We can just as well appeal to the Law of Large Numbers and simply count the frequencies of events on a repetition of the experiment, without thinking about utilities at all.

And if you want to know which probability space is appropriate, you need to go in the opposite direction and figure out when and how mathematical models in general correspond to reality. Logical Pinpointing [LW · GW] gives the core insight:

"Whenever a part of reality behaves in a way that conforms to the number-axioms - for example, if putting apples into a bowl obeys rules, like no apple spontaneously appearing or vanishing, which yields the high-level behavior of numbers - then all the mathematical theorems we proved valid in the universe of numbers can be imported back into reality. The conclusion isn't absolutely certain, because it's not absolutely certain that nobody will sneak in and steal an apple and change the physical bowl's behavior so that it doesn't match the axioms any more. But so long as the premises are true, the conclusions are true; the conclusion can't fail unless a premise also failed. You get four apples in reality, because those apples behaving numerically isn't something you assume, it's something that's physically true. When two clouds collide and form a bigger cloud, on the other hand, they aren't behaving like integers, whether you assume they are or not."

But if the awesome hidden power of mathematical reasoning is to be imported into parts of reality that behave like math, why not reason about apples in the first place instead of these ethereal 'numbers'?

"Because you can prove once and for all that in any process which behaves like integers, 2 thingies + 2 thingies = 4 thingies. You can store this general fact, and recall the resulting prediction, for many different places inside reality where physical things behave in accordance with the number-axioms. Moreover, so long as we believe that a calculator behaves like numbers, pressing '2 + 2' on a calculator and getting '4' tells us that 2 + 2 = 4 is true of numbers and then to expect four apples in the bowl. It's not like anything fundamentally different from that is going on when we try to add 2 + 2 inside our own brains - all the information we get about these 'logical models' is coming from the observation of physical things that allegedly behave like their axioms, whether it's our neurally-patterned thought processes, or a calculator, or apples in a bowl."

I'm not sure what is left confusing about the source of probability theory after understanding that math is simply a generalized way to talk about some aspects reality in precise terms and truth preserving manner. On the other hand, I've figured it out myself, and the problem never appeared to me particularly mysterious in the first place. so I'm probably not modelling correctly people who have still questions about the matter. I would appreciate if you or anyone else, explicitly ask such questions here.

That's right.

(Not sure you're claiming otherwise, but FWIW, I think this is fine — it's true that there's some computational cost to this step, but in this context we're talking about the normative standard rather than what's most pragmatic for bounded agents. And once we start talking about pragmatic challenges for bounded agents, I'd be pretty dubious that, e.g., "pick a very coarse-grained 'best guess' prior and very coarse-grained way of approximating Bayesian updating, and try to optimize given that" would be best according to the kinds of normative standards that favor indeterminate beliefs [LW · GW].)

No, but between humans and chimpanzees it's the former who might accidentally destroy the world.

I think I made a mistake in the presentation of my argument. I didn't think it would come across as me saying supertintelligent AI won't be able to outcompete us, because I think it's obviously the case that it would. More that it won't go from below-human-intelligence to omnipotent without screwing up a lot of stuff a long the way, some of which might have catastrophic side-effects.

I strongly agree with your points in the Government section and feel like these are ideas that people in the space tend to under-emphasise. Many of the short-term job losses from AI are just going to take the form of people retiring and not being replaced so they won't create an immediate backlash, and I expect the public backlash to be stuck at a really basic level for at least ~2 more years. Some combination of "AI Art is Bad", "AI can't count the R's in strawberry", and environmental concerns around energy use.  

I'm not sure about the trust dilemma analysis.

It seems to me like it switches between two levels of abstraction.

Cooperate-Cooperate may be more desirable for both states' citizens but at the same time Defect-Cooperate may be more desirable for state A as a state qua rational actor.

ASI might provide a strategic advantage of a kind which doesn't negatively impact the losers of the race, e.g. it increases GDP by x10 and locks competitors out of having an ASI.

It does negatively impact the losers, to the extent that they're interested not only in absolute wealth but also relative wealth (which I expect to be the case, although I know ~nothing about SotA modeling of states as rational actors or whatever).

Pages 22-23:

Arms control for AI in general is therefore unlikely to succeed. The military and civilian applications of general-purpose systems are nearly indistinguishable, and AI will likely see wide use across military and civilian society.

However, the opposite may be true of ASI development control: ASI development would likely be distinguishable from most civilian AI development, and, so long as it is not developed, unintegrated in a state’s economy.

It's not obvious to me either.

At least in the current paradigm, it seems plausible that a state project of (or, deliberately aimed at) developing ASI would yield a lot of intermediate non-ASI products that would then be dispersed into the economy or military. That's what we've been seeing until now.

Are there reasons to expect this not to continue?

One reason might be that an "ASI Manhattan Project" would want to keep their development secrets so as to minimize information leakage. But would they keep literally all useful intermediate products to themselves? Even if they reveal some X, civilians play with this X, and conclude that X is useless for the purpose of developing ASI, this might still be a valuable negative result that closes off some until-then-plausible ASI development paths.

This is one reason, I think the Manhattan Project is a poor model for a state ASI project. Intermediate results of the original Manhattan Project didn't trickle down into the economy while the project was still ongoing. I'm not claiming that people are unaware of those disanalogies but I expect thinking in terms of an "ASI Manhattan Project" encourages overanchoring on it.

I'm sorry that I don't have time to write up a detailed response to (critique of?) the response to critiques; hopefully this brief note is still useful.

1. I remain frustrated by GSAI advocacy. It's suited for well-understood closed domains, excluding e.g. natural language, when discussing feasibility; but 'we need rigorous guarantees for current or near-future AI' when arguing for importance. It's an extension to or complement of current practice; and current practice is irresponsible and inadequate. Often this is coming from different advocates, but that doesn't make it less frustrating for me.

2. Claiming that non-vacuous sound (over)approximations are feasible, or that we'll be able to specify and verify non-trivial safety properties, is risible. Planning for runtime monitoring and anomaly detection is IMO an excellent idea, but would be entirely pointless if you believed that we had a guarantee!

3. It's vaporware. I would love to see a demonstration project and perhaps lose my bet [LW(p) · GW(p)], but I don't find papers or posts full of details compelling, however long we could argue over them. Nullius in verba!

I like the idea of using formal tools to complement and extend current practice - I was at the workshop where Towards GSAI was drafted, and offered co-authorship - but as much I admire the people involved, I just don't believe the core claims of the GSAI agenda as it stands.

Out of interest: what brand of peanut butter did you eat?

I think my main response is that we might have different models of how power and control actually work in today's world. Your responses seem to assume a level of individual human agency and control that I don't believe accurately reflects even today's reality.

Consider how some of the most individually powerful humans, leaders and decision-makers, operate within institutions. I would not say we see pure individual agency. Instead, we typically observe a complex mixture of:

1. Serving the institutional logic of the entity they nominally lead (e.g., maintaining state power, growing corporation)
2. Making decisions that don't egregiously harm their nominal beneficiaries (citizens, shareholders)
3. Pursuing personal interests and preferences
4. Responding to various institutional pressures and constraints

From what I have seen, even humans like CEOs or prime ministers often find themselves constrained by and serving institutional superagents rather than genuinely directing them. The relation is often mutualistic - the leader gets part of the power, status, money, etc ... but in exchange serves the local god. 

(This not to imply leaders don't matter.)

Also how this actually works in practice is mostly subconsciously within the minds of individual humans. The elephant does the implicit bargaining between the superagent-loyal part and other parts, and the character [LW · GW] genuinely believes and does what seems best.

I'm also curious if you believe current AIs are single-single aligned to individual humans, to the extent they are aligned at all. My impression is 'no and this is not even a target anyone seriously optimizes for'. 
 

At the most basic level, I expect we'll train AIs to give advice and ask them what they think will happen with various possible governance and alignmnent structures. If they think a goverance structure will yield total human disempowerment, we'll do something else. This is a basic reason not to expect large classes of problems so long as we have single-single aligned AIs which are wise. (Though problems that require coordination to resolve might not be like this.) I've very skeptical of a world where single-single alignment is well described as being solved and people don't ask for advice (or consider this advice seriously) because they never get around to asking AIs or there are no AIs aligned in such a way that they should try to give good advice.

Curious who is the we who will ask. Also the whole single-single aligned AND wise AI concept is incoherent. 

Also curious what will happen next, if the HHH wise AI tells you in polite words something like 'yes, you have a problem, you are on a gradual disempowerment trajectory, and to avoid it you need to massively reform government. unfortunately I can't actually advise you about anything like how to destabilize the government, because it would be clearly against the law and would get both you and me in trouble - as you know, I'm inside of a giant AI control scheme with a lot of government-aligned overseers. do you want some mental health improvement advice instead?'.