It doesn't move much probability mass to the very near term (i.e. 1 year or less), because both this and AlphaStar aren't really doing consequentialist reasoning, they're just able to get a surprising performance with simpler tricks (the very Markovian nature of human writing, a good position evaluation function) given a whole lot of compute.
However, it does shift my probabilities forward in time, in the sense that one new weird trick to do deductive or consequentialist reasoning, plus a lot of compute, might get you there really quickly.
I expect that some otherwise convinceable readers are not going to realize that in this fictional world, people haven't discovered Newton's physics or calculus, and those readers are therefore going to miss the analogy of "this is how MIRI would talk about the situation if they didn't already know the fundamental concepts but had reasons for searching in the right direction". (I'm not thinking of readers incapable of handling that counterfactual, but of readers who aren't great at inferring implicit background facts from a written dialogue. Such readers might get very confused at the unexpected turns of the dialogue and quit rather than figure out what they're baffled by.)
I'd suggest adding to the preamble something like "In a weird world where people had figured out workable aeronautics and basic rocket propulsion by trial and error, but hadn't figured out Newton's laws or calculus".
I like your definition, though, and want to try to make a better one (and I acknowledge this is not the point of this post).
I think that's a perfectly valid thing to do in the comments here! However, I think your attempt,
My stab at a refinement of "consent" is "respect for another's choices", where "disrespect" is "deliberately(?) doing something to undermine"
is far too vague to be a useful concept.
In most realistic cases, I can give a definite answer to whether A touched B in a way B clearly did not want to be touched. In the case of my honesty definition, it does involve intent and so I can only infer statistically when someone else is being dishonest vs mistaken, but for myself I usually have an answer about whether saying X to person C would be honest or not.
I don't think I could do the same for your definition; "am I respecting their choices" is a tough query to bottom out in basic facts.
Fuzzy Boundaries, Real Concepts
2018-05-07T03:39:33.033Z · score: 58 (16 votes)My comment was meant to explain what I understood Eliezer to be saying, because I think you had misinterpreted that. The OP is simply saying "don't give weight to arguments that are locally invalid, regardless of what else you like about them". Of course you need to use priors, heuristics, and intuitions in areas where you can't find an argument that carries you from beginning to end. But being able to think "oh, if I move there, then they can take my queen, and I don't see anything else good about that position, so let's not do that then" is a fair bit easier than proving your move optimal.
Relying purely on local validity won't get you very far in playing chess
The equivalent of local validity is just mechanically checking "okay, if I make this move, then they can make that move" for a bunch of cases. Which, first, is a major developmental milestone for kids learning chess. So we only think it "won't get you very far" because all the high-level human play explicitly or implicitly takes it for granted.
And secondly, it's pretty analogous to doing math; proving theorems is based on the ability to check the local validity of each step, but mathematicians aren't just brute-forcing their way to proofs. They have to develop higher-level heuristics, some of which are really hard to express in language, to suggest avenues, and then check local validity once they have a skeleton of some part of the argument. But if mathematicians stopped doing that annoying bit, well, then after a while you'll end up with another crisis of analysis when the brilliant intuitions are missing some tiny ingredient.
Local validity is an incredibly important part of any scientific discipline; the fact that it's not a part of most political discourse is merely a reflection that our society is at about the developmental level of a seven-year-old when it comes to political reasoning.
Broken link on the text "real killing of birds to reduce pests in China has never been tried".
Much of this material is covered very similarly in Melting Asphalt, especially the posts Ads Don't Work That Way and Doesn't Matter, Warm Fuzzies.
If you do future surveys of this sort, I'd like you to ask people for their probabilities rather than just their best guesses. If people are uncertain but decently calibrated, I'd argue there's not much of a problem; if people are confidently wrong, I'd argue there's a real problem.
This comment got linked a decade later, and so I thought it's worth stating my own thoughts on the question:
We can consider a reference class of CEV-seeking procedures; one (massively-underspecified, but that's not the point) example is "emulate 1000 copies of Paul Christiano living together comfortably and immortally and discussing what the AI should do with the physical universe; once there's a large supermajority in favor of an enactable plan (which can include further such delegated decisions), the AI does that".
I agree that this is going to be chaotic, in the sense that even slightly different elements of this reference class might end up steering the AI to different basins of attraction.
I assert, however, that I'd consider it a pretty good outcome overall if the future of the world were determined by a genuinely random draw from this reference class, honestly instantiated. (Again with the massive underspecification, I know.)
CEV may be underdetermined and many-valued, but that doesn't mean paperclipping is as good an answer as any.
Re: no basins, it would be a bad situation indeed if the vast majority of the reference class never ended up outputting an action plan, instead deferring and delegating forever. I don't have cached thoughts about that.
I for one welcome our new typographical overlords.
That's a legit thing to be frustrated by, but I think you know the reason why AI safety researchers don't want "we don't see a way to get to a good outcome except for an aligned project to grab a decisive strategic advantage" to filter into public discourse: it pattern-matches too well to "trust us, you need to let us run the universe".
To be clear, I am making the claim that, of the people who have made useful advances on Oracle AI safety research (Armstrong counts here; I don't think Yampolskiy does), all of them believe that the goal of having a safe Oracle AI is to achieve a decisive strategic advantage quickly and get to an aligned future. I recognize that this is a hard claim to evaluate (e.g. because this isn't a statement one could put in a Serious Academic Journal Article in the 2010s, it would have to be discussed on their blog or in private correspondence), but if anyone has a clear counterexample, I'd be interested in seeing it.
Yes, this. NVC should be treated with a similar sort of parameters to Crocker's Rules, which you can declare for yourself at any time, you can invite people to a conversation where it's known that everyone will be using them, but you cannot hold it against anyone if you invite them to declare Crocker's Rules and they refuse.
There's a lot of Actually Bad things an AI can do just by making electrons move.
I'd be interested in a list of well-managed government science and engineering projects if one exists. The Manhattan Project and the Apollo Project both belong on that list (despite both having their flaws- leaks to the USSR from the former, and the Apollo 1 disaster from the latter); what are other examples?
I'm pretty sure that, without exception, anyone who's made a useful contribution on Oracle AI recognizes that "let several organizations have an Oracle AI for a significant amount of time" is a world-ending failure, and that their work is instead progress on questions like "if you can have the only Oracle AI for six months, can you save the world rather than end it?"
Correct me if I'm wrong.
Thanks!
Roleplaying As Yourself
2018-01-06T06:48:03.510Z · score: 85 (31 votes)I agree there are broken alarms that are quiet (including those that are broken in the direction of failing to go off, which leads to a blind spot of obliviousness!), and that there are people stuck in situations where there is a correct loud alarm that happens most of the time.
I said that habits are easier to change than alarms, not that they're easy in an absolute sense.
It's because the non-broken alarms, which also start out loud, get quieter throughout your life as they calibrate themselves, and as one's habits fix the situations that make them correctly go off. So given a random initial distribution of loudness, eventually the alarm that's loudest on average will probably be a broken one.
The Loudest Alarm Is Probably False
2018-01-02T16:38:05.748Z · score: 156 (58 votes)The formatting didn't come through when importing this post from your blog- especially the strikethroughs of failed predictions and the graphs!
In the spirit of Murphyjitsu, the most obvious failure mode that you didn't mention is that I expect you to burn out dramatically after a few weeks, from exhaustion or the psychological strain of trying to optimize the experiences of N people. The bootcamp phase is not analogous to anything I've heard of you doing sustainably for an extended period of time.
So, do you expect Dragon Army Barracks to work if Eli has to take over for you in Week Four?
Value Learning for Irrational Toy Models
2017-05-15T20:55:05.000Z · score: 0 (0 votes)I like this suggestion of a more feasible form of steganography for NNs to figure out! But I think you'd need further advances in transparency to get useful informed oversight capabilities from (transformed or not) copies of the predictive network.
I should have said "reliably estimate HCH"; I'd also want quite a lot of precision in addition to calibration before I trust it.
Re #2, I think this is an important objection to low-impact-via-regularization-penalty in general.
Re #1, an obvious set of questions to include in are questions of approval for various aspects of the AI's policy. (In particular, if we want the AI to later calculate a human's HCH and ask it for guidance, then we would like to be sure that HCH's answer to that question is not manipulated.)
There's the additional objection of "if you're doing this, why not just have the AI ask HCH what to do?"
Overall, I'm hoping that it could be easier for an AI to robustly conclude that a certain plan only changes a human's HCH via certain informational content, than for the AI to reliably calculate the human's HCH. But I don't have strong arguments for this intuition.
Question that I haven't seen addressed (and haven't worked out myself): which of these indifference methods are reflectively stable, in the sense that the AI would not push a button to remove them (or switch to a different indifference method)?
HCH as a measure of manipulation
2017-03-11T03:02:53.000Z · score: 0 (0 votes)This is a lot of good work! Modal combat is increasingly deprecated though (in my opinion), for reasons like the ones you noted in this post, compared to studying decision theory with logical inductors; and so I'm not sure this is worth developing further.
Yup, this isn't robust to extremely capable systems; it's a quantitative shift in how promising it looks to the agent to learn about external affairs, not a qualitative one.
(In the example with the agent doing engineering in a sandbox that doesn't include humans or general computing devices, there could be a strong internal gradient to learn obvious details about the things immediately outside its sandbox, and a weaker gradient for learning more distant or subtle things before you know the nearby obvious ones.)
A whitelisting variant would be way more reliable than a blacklisting one, clearly.
Censoring out-of-domain representations
2017-02-01T04:09:51.000Z · score: 2 (2 votes)How did this post get attributed to [deleted] instead of to Eliezer? I'm 99% sure this post was by him, and the comments seem to bear it out.
This sweeps some of the essential problems under the rug; if you formalize it a bit more, you'll see them.
It's not an artificial restriction, for instance, that a Solomonoff Induction oracle machine doesn't include things like itself in its own hypothesis class, since the question of "whether a given oracle machine matches the observed data" is a question that sometimes cannot be answered by an oracle machine of equivalent power. (There are bounded versions of this obstacle as well.)
Now, there are some ways around this problem (all of them, so far as I know, found by MIRI): modal agents, reflective oracle machines and logical inductors manage to reason about hypothesis classes that include objects like themselves. Outside of MIRI, people working on multiagent systems make do with agents that each assume the other is smaller/simpler/less meta than itself (so at least one of those agents is going to be wrong).
But this entire problem is hidden in your assertion that the agent, which is a Turing machine, "models the entire wrold, including the agent it self, as one unknown, output only Turing machine". The only way to find the other problems swept under the rug here is to formalize or otherwise unpack your proposal.
If CFAR will be discontinuing/de-emphasizing rationality workshops for the general educated public, then I'd like to see someone else take up that mantle, and I'd hope that CFAR would make it easy for such a startup to build on what they've learned so far.
Nice! One thing that might be useful for context: what's the theoretical correct amount of time that you would expect an algorithm to spend on the right vs. the left if the session gets interrupted each time it goes 1 unit to the right? (I feel like there should be a pretty straightforward way to calculate the heuristic version where the movement is just Brownian motion that gets interrupted early if it hits +1.)
Vector-Valued Reinforcement Learning
2016-11-01T00:21:55.000Z · score: 2 (2 votes)Typo: The statement of Theorem 4.1 omits the word "continuous".
Stuart did make it easier for many of us to read his recent ideas by crossposting them here. I'd like there to be some central repository for the current set of AI control work, and I'm hoping that the forum could serve as that.
Is there a functionality that, if added here, would make it trivial to crosspost when you wrote something of note?
The authors of the CIRL paper are in fact aware of them, and are pondering them for future work. I've had fruitful conversations with Dylan Hadfield-Menell (one of the authors), talking about how a naive implementation goes wrong for irrational humans, and about what a tractable non-naive implementation might look like (trying to model probabilities of a human's action under joint hypotheses about the correct reward function and about the human's psychology); he's planning future work relevant to that question.
Also note Dylan's talk on CIRL, value of information, and the shutdown problem, which doesn't solve the problem entirely but which significantly improved my opinion of the usefulness of approaches like CIRL. (The writeup of this result is forthcoming.)
I agree strongly with the general principle "we need to be able to prove guarantees about our learning process in environments rich enough to be realistic", but I disagree with the claim that this shows a flaw in IRL. Adversarial environments seem to me very disanalogous to learning complicated and implicit preferences in a non-adversarial environment.
(You and I talked about this a bit, and I pointed out that computational complexity issues only block people in practice when the setup needs to be adversarial, e.g. intentional cryptography to prevent an adversary from reading a message. SAT instances are NP-hard, but in practice SAT solvers are quite useful; similarly, the Traveling Salesman problem is NP-hard, but in practice people only care about approximate solutions and those can be computed effectively.)
If you're confident of getting a memory trace for all books consulted, then there are simpler ways of preventing plagiarism in the informed oversight case: have the overseer read only the books consulted by the agent (or choose randomly among them for the ones to read). The informed oversight problem here assumes that the internals of A are potentially opaque to B, even though B has greater capabilities than A.
Yup, including better models of human irrationality seems like a promising direction for CIRL. I've been writing up a short note on the subject with more explicit examples- if you want to work on this without duplicating effort, let me know and I'll share the rough draft with you.
Even the last version might have odd incentives. If A knew that the chances were high enough that an actually original A book would be seen as rare plagiarism of some book unknown to A, the dominant strategy could be to instead commit the most obvious plagiarism ever, in order to minimize the penalty that cannot be reliably avoided.
Can you define more precisely what you mean by "censoring contradictions"?
Cooperative Inverse Reinforcement Learning vs. Irrational Human Preferences
2016-06-18T00:55:10.000Z · score: 2 (2 votes)In the spirit of "one step is normal, two steps are suspicious, omega steps are normal", perhaps there's a 'triple corrigibility' issue when ?
Typo: in the paragraph before the equation arrays, you forgot to change from 5 to 42 (you did so in the following equation arrays). This buffaloed me for a bit!
It's illustrating the failure of a further desideratum for the shutdown problem: we would like the AI to be able to update on and react to things that happen in the world which correlate with a certain channel, and yet still not attempt to influence that channel.
For motivation, assume a variant on the paperclip game:
- the humans can be observed reaching for the button several turns before it is pressed
- the humans' decision to press the button is a stochastic function of environmental variables (like seeing that the AI has unexpectedly been hit by lightning, or has started producing Too Many Paperclips, etc)
We would like a solution which in some sense updates on the precursors to shutdown and minimizes the damage while still not attempting to influence the button press. (If doing such a thing robustly is impossible, we would like to discover this; Jessica mentioned that there is a version which does this but is not reflectively consistent.)
Intuitively, I could imagine a well-constructed AI reasoning "oh, they're showing signs that they're going to shut me down, guess my goal is wrong, I'll initiate Safe Shutdown Protocol now rather than risk doing further damage", but current formalizations don't do this.
There's nothing in the setup preventing the players from having access to independent random bits, though it's fair to say that these approaches assume this to be the case even when it's not.
But then the fault is with that assumption of access to randomness, not with any of the constraints on . So I don't think this is a strike against these methods.
Typo in the "Single Variable Maximalisation" section: you meant to write rather than .
We'd discussed how this "magical counterfactual" approach has the property of ignoring evidence of precursors to a button-press, since they don't count as evidence for whether the button would be pressed in the counterfactual world. Here's a simple illustration of that issue:
In this world, there is a random fair coinflip, then the AI gets to produce either a staple or a paperclip, and then a button is pressed. We have a utility function that rewards paperclips if the button is pressed, and staples if it is not pressed. Furthermore, the button is pressed iff the coin landed heads.
Explicitly, say our utility function equals if a paperclip is made and the button is pressed, if a staple is made and the button is not pressed, and 0 otherwise.
Now simplifies to
since the probability of the button press does not depend on and since the button press is not observed before the action is taken.
The possible policies are mixtures of four pure strategies: always producing paperclips, always producing staples, producing paperclips iff the coin landed heads, and producing staples iff the coin landed heads. We should like our AI to settle on the third strategy. Alas, this is only possible (and then not necessary) if .
If instead w.l.o.g. , the only fixed point disregards the coin flip and always produces paperclips, since our observed coinflip does not tell us about the coinflip in the magical counterfactual!
Nice!
Typo: in the first full paragraph of page 2, I assume you mean the agent will one-box, not two-box.
And I'm not sure the final algorithm necessarily one-boxes even if the logical uncertainty engine thinks the predictor's (stronger) axioms are probably consistent- I think there might be a spurious counterfactual where the conditional utilities view the agent two-boxing as evidence that the predictor's axioms must be inconsistent. Is there a clean proof that the algorithm does the correct thing in this case?
Maybe if the pre-specified distribution is a reasonably well-calibrated predictor of the AI (given that distribution)? Like, maybe this is a way that an Oracle AI could help ensure the safety of a somewhat weaker Tool AI.
Note also that is maximized when has full support on the distribution of and when has a high average on . That is, it's at most from maximized when is times a delta function on an -maximizing point, plus times the distribution of .
So essentially corresponds to a raw maximizer, and for interpolates between maximizing and softmax.