LessWrong 2.0 Reader

Prediction (influenced by R1-Zero): By EOY, expert-level performance will be reported on outcome prediction for a certain class of AI experiments - those that can be specified concisely in terms of code and data sets that:

1. are frequently used and can be referenced by name, e.g. MNIST digits, or
2. are small enough to be given explicitly, or
3. are synthetic, specified by their exact distribution in code.

It seems like you're assuming people won't build AGI if they don't have reliable ways to control it, or else that sovereign (uncontrolled) AGI would be likely the be friendly to humanity.

I'm assuming neither. I agree with you that both seem (very) unlikely. ^[1]

It seems like you're assuming that any humans succeeding in controlling AGI is (on expectation) preferable to extinction? If so, that seems like a crux: if I agreed with that, then I'd also agree with "publish all corrigibility results".

It's more important to defuse the bomb than it is to prevent someone you dislike from holding it.

I think there is a key disanalogy to the situation with AGI: The analogy would be stronger if the bomb was likely to kill everyone, but also had a some (perhaps very small) probability of conferring godlike power to whomever holds it. I.e., there is a tradeoff: decrease the probability of dying, at the expense of increasing the probability of S-risks from corrupt(ible) humans gaining godlike power.

If you agree that there exists that kind of tradeoff, I'm curious as to why you think it's better to trade in the direction of decreasing probability-of-death for increased probability-of-suffering.

So, the question I'm most interested in is the one at the end of the post^[1], viz

What (crucial) considerations should one take into account, when deciding whether to publish---or with whom to privately share---various kinds of corrigibility-related results?

Yeah but if something is in the general circulation (bloodstream), then it’s going everywhere in the body. I don’t think there’s any way to specifically direct it.

…Except in the time domain, to a limited extent. For example, in rats, tonic oxytocin in the bloodstream controls natriuresis, while pulsed oxytocin in the bloodstream controls lactation and birth. The kidney puts a low-pass filter on its oxytocin detection system, and the mammary glands & uterus put a high-pass filter, so to speak.

Yeah. Though as a counterpoint, something I picked up from IIRC Scott Alexander or Marginal Revolution is that the FDA is not great about accepting foreign clinical trials, or demands that they always be supplemented by trials of Americans, or similar.

This is especially useful when pursuing several subgoals in parallel, as forward-checking a combination of moves is combinatorially costly--better to have the agent's parallel actions constrained to nice parts of the space.

If I were a singleton AGI, but not such a Jupiter brain that I could deal with the combinatorial explosion of directly jointly-optimizing every motion of every robot, I would presumably set up an internal “free market” with spot-prices for iron ore and robot-hours and everything else. Then I would iteratively cycle through all my decision-points and see if there are ways to “make money” locally, and then update virtual “prices” accordingly.

In fact, I think there’s probably a theorem that says that the optimal solution of a complex resource allocation problem is isomorphic to a system where things have prices. (Something to do with Lagrange multipliers? Shrug.)

(Fun fact: In the human world, propagating prices within firms—e.g. if the couch is taking up 4m² of the 4000m² floor space at the warehouse, then that couch is “charged” 0.1% of the warehouse upkeep costs, etc.—is very rarely done but leads directly to much better decisions and massive overall profit increases! See here [LW(p) · GW(p)].)  

Externalities are not an issue in this virtual “economy” because I can “privatize” everything—e.g. I can invent fungible allowances to pollute the atmosphere in thus-and-such way etc. This is all just a calculation trick happening in my own head, so there aren’t coordination problems or redistribution concerns or information asymmetries or anything like that. Since I understand everything (even if I can’t juggle it all in my head simultaneously), I’ll notice if there’s some relevant new unpriced externality and promptly give it a price.

So then (this conception of) corrigibility would correspond to something like “abiding by this particular system of (virtual) property rights”. (Including all the weird “property rights” like purchasing allowances to emit noise or heat or run conscious minds or whatever, and including participating in the enterprise of discovering new unpriced externalities.) Do you agree?

EDIT 3/5/24: In the comments for Counting arguments provide no evidence for AI doom, Evan Hubinger agreed that one cannot validly make counting arguments over functions. However, he also claimed that his counting arguments "always" have been counting parameterizations, and/or actually having to do with the Solomonoff prior over bitstrings.

As one of Evan's co-authors on the mesa-optimization paper from 2019 I can confirm this. I don't recall ever thinking seriously about a counting argument over functions. 

Who’s track record of AI predictions would you like to see evaluated?

Whoever has the best track record :)

Yes. Current AI policy is like people in a crowded room fighting over who gets to hold a bomb. It's more important to defuse the bomb than it is to prevent someone you dislike from holding it.

That said, we're currently not near any satisfactory solutions to corrigibility. And I do think it would be better for the world if were easier (by some combination of technical factors and societal factors) to build AI that works for the good of all humanity than to build equally-smart AI that follows the orders of a single person. So yes, we should focus research and policy effort toward making that happen, if we can.

And if we were in that world already, then I agree releasing all the technical details of an AI that follows the orders of a single person would be bad.

There are 3 people. Each person announces an integer. The smallest unique integer wins: e.g. if your opponents both pick 1, you win with any number. If all 3 pick the same number, the winner is picked randomly

Question: what’s the Nash equilibrium?

(I assume this is meant to be natural numbers or positive ints? Otherwise I don't think there is a nash equilibrium.)