Posts
Comments
Quick approximate summary:
- John von Neumann first says, essentially, that the Industrial Revolution has made the world smaller, and that in earlier centuries the problems caused by it were contained to nations, but that now they extend to the entire globe.
- For a first example, he talks about cheap energy and industrial synthesis, though he predicts that nuclear fusion and transmutation would be much more practically feasible than they have turned out to be.
- He briefly mentions expected major improvement in automation, communication, and transportation.
- He then talks about anthropogenic climate change and the broad possibilities of geoengineering the climate.
- All of these technologies can vastly improve human life, or destroy it.
- He immediately rules out the "solution" of [preventing advances in technology] as both undesirable (because it blocks the positive uses) and impossible (because it would require total coordination and a total change in human values).
- He next considers the possibility of permanently avoiding war through diplomacy etc, and does not think that the 1950s drive for world peace will last long; furthermore, such an initiative would need to adapt to ever-more-powerful technologies as fast as they are introduced.
- He frames the upcoming decades as a dangerous but useful evolution, where we will either succeed or fail catastrophically, and doesn't sound especially optimistic. Our best hope is to innovate new political forms that are capable of handling major threats with patience, flexibility, and intelligence.
Alas, double-cruxing is a two-person game; you can't make another person do it, you can only do it with someone who genuinely wants to. For everyone else, the best related trick I know is nonviolent communication.
Meta: I approve of the practice of arguing against your own post in a comment.
How much less do you expect this to happen under the current system?
both historically and now, criticism is often met with counterarguments based on "style" rather than engaging with the technical meat of the criticism
Is there any group of people who reliably don't do this? Is there any indication that AI researchers do this more often than others?
Eliezer's real answer to this question is discussed in Timeless Control. Basically, choice is still meaningful in many-worlds or any other physically deterministic universe. There are incredibly few Everett branches starting from here where tomorrow I go burn down an orphanage, and this is genuinely caused by the fact that I robustly do not want to do that sort of thing.
If you have altruistic motivation, then the Everett branches starting from here are in fact better (in expectation) than the branches starting from a similar universe with a version of you that has no altruistic motivation. By working to do good, you are in a meaningful sense causing the multiverse to contain a higher proportion of good worlds than it otherwise would.
It really does all add up to normality, even if it feels counterintuitive.
Well, this post aged interestingly for those of us who know the author (who ended up working for a high-profile EA organization for some time).
Typo:
it is not the done the thing
it is not the done thing, perhaps?
Maybe you can get the best of both worlds by imagining you're writing a children's book, but that your editor is in fact an expert on the subject and you don't want to embarrass yourself in front of them.
And Robin Hanson was surprised that no big corporation wanted to implement a real prediction market?
This strongly resembles the argument given by Subhan in EY's post Is Morality Preference?, with a side order of Fake Selfishness. You might enjoy reading those posts along with others in their respective sequences. ("Is Morality Preference?" was part of the original metaethics sequence but didn't make the cut for Rationality: AI to Zombies.)
More to the point, the biggest mistake I see here is the one addressed in The Domain of Your Utility Function: yes, my moral preferences are a part of my map rather than the territory, but there's still a damn meaningful difference between egoism (preferences that point only to the part of my map labeled "my future experiences") and my actual moral preferences, which point to many other parts of the map as well.
I am struggling to understand the goal of the post.
The title was helpful to me in that regard. Each of these examples shows an agent who could run an honest process to get evidence on a question, but which prefers one answer so much that they try to stack the deck in that direction, and thereby loses the hoped-for benefits of that process.
Getting an honest Yes requires running the risk of getting a No instead.
Formatting request: can the footnote numbers be augmented with links that jump to the footnote text? (I presume this worked in Arbital but broke when it was moved here.)
I had a notion here that I could stochastically introduce a new goal that would minimize total suffering over an agent's life-history. I tried this, and the most stable solution turned out to be thus: introduce an overwhelmingly aversive goal that causes the agent to run far away from all of its other goals screaming.
did you mean: anhedonia
(No, seriously, your paragraph is an apt description of a long bout I had of depression-induced anhedonia; I felt so averse to every action that I ceased to feel wants, and I consistently marked my mood as neutral rather than negative despite being objectively more severely depressed than I was at other times when I put negative numbers in my mood tracker.)
The ideal thing is to judge Bob as if he were making the same prediction every day until he makes a new one, and log-score all of them when the event is revealed. (That is, if Bob says 75% on January 1st and 60% on February 1st, and then on March 1st the event is revealed to have happened, Bob's score equals 31*log(.25) + 28*log(.4). Then Bob's best strategy is to update his prediction to his actual current estimate as often as possible; past predictions are sunk costs.
The real-world version is remembering to dock people's bad predictions more, the longer they persisted in them. But of course this is hard.
538 did do this with their self-evaluation, which is a good way to try and establish a norm in the domain of model-driven reporting.
Let's note differences of degree here. Political systems differ massively in how easily decisionmakers can claim large spoils for themselves, and these differences seem to correlate with how pro-social the decisions tend to be. In particular, the dollar amounts of graft being alleged for politicians in liberal democracies are usually small compared to what despots regularly claim without consequence. (Which is not to say that it would be wise to ignore corruption in liberal democracies!)
I'm not convinced that Europe had more intellectual freedom on average than China, but because of the patchwork of principalities, it certainly had more variation in intellectual freedom than did a China that was at any given time either mostly unified or mostly at war; and all that you need for an intellectual revolution is the existence of a bastion of intellectual freedom somewhere.
It doesn't move much probability mass to the very near term (i.e. 1 year or less), because both this and AlphaStar aren't really doing consequentialist reasoning, they're just able to get a surprising performance with simpler tricks (the very Markovian nature of human writing, a good position evaluation function) given a whole lot of compute.
However, it does shift my probabilities forward in time, in the sense that one new weird trick to do deductive or consequentialist reasoning, plus a lot of compute, might get you there really quickly.
I expect that some otherwise convinceable readers are not going to realize that in this fictional world, people haven't discovered Newton's physics or calculus, and those readers are therefore going to miss the analogy of "this is how MIRI would talk about the situation if they didn't already know the fundamental concepts but had reasons for searching in the right direction". (I'm not thinking of readers incapable of handling that counterfactual, but of readers who aren't great at inferring implicit background facts from a written dialogue. Such readers might get very confused at the unexpected turns of the dialogue and quit rather than figure out what they're baffled by.)
I'd suggest adding to the preamble something like "In a weird world where people had figured out workable aeronautics and basic rocket propulsion by trial and error, but hadn't figured out Newton's laws or calculus".
I like your definition, though, and want to try to make a better one (and I acknowledge this is not the point of this post).
I think that's a perfectly valid thing to do in the comments here! However, I think your attempt,
My stab at a refinement of "consent" is "respect for another's choices", where "disrespect" is "deliberately(?) doing something to undermine"
is far too vague to be a useful concept.
In most realistic cases, I can give a definite answer to whether A touched B in a way B clearly did not want to be touched. In the case of my honesty definition, it does involve intent and so I can only infer statistically when someone else is being dishonest vs mistaken, but for myself I usually have an answer about whether saying X to person C would be honest or not.
I don't think I could do the same for your definition; "am I respecting their choices" is a tough query to bottom out in basic facts.
My comment was meant to explain what I understood Eliezer to be saying, because I think you had misinterpreted that. The OP is simply saying "don't give weight to arguments that are locally invalid, regardless of what else you like about them". Of course you need to use priors, heuristics, and intuitions in areas where you can't find an argument that carries you from beginning to end. But being able to think "oh, if I move there, then they can take my queen, and I don't see anything else good about that position, so let's not do that then" is a fair bit easier than proving your move optimal.
Relying purely on local validity won't get you very far in playing chess
The equivalent of local validity is just mechanically checking "okay, if I make this move, then they can make that move" for a bunch of cases. Which, first, is a major developmental milestone for kids learning chess. So we only think it "won't get you very far" because all the high-level human play explicitly or implicitly takes it for granted.
And secondly, it's pretty analogous to doing math; proving theorems is based on the ability to check the local validity of each step, but mathematicians aren't just brute-forcing their way to proofs. They have to develop higher-level heuristics, some of which are really hard to express in language, to suggest avenues, and then check local validity once they have a skeleton of some part of the argument. But if mathematicians stopped doing that annoying bit, well, then after a while you'll end up with another crisis of analysis when the brilliant intuitions are missing some tiny ingredient.
Local validity is an incredibly important part of any scientific discipline; the fact that it's not a part of most political discourse is merely a reflection that our society is at about the developmental level of a seven-year-old when it comes to political reasoning.
Broken link on the text "real killing of birds to reduce pests in China has never been tried".
Much of this material is covered very similarly in Melting Asphalt, especially the posts Ads Don't Work That Way and Doesn't Matter, Warm Fuzzies.
If you do future surveys of this sort, I'd like you to ask people for their probabilities rather than just their best guesses. If people are uncertain but decently calibrated, I'd argue there's not much of a problem; if people are confidently wrong, I'd argue there's a real problem.
This comment got linked a decade later, and so I thought it's worth stating my own thoughts on the question:
We can consider a reference class of CEV-seeking procedures; one (massively-underspecified, but that's not the point) example is "emulate 1000 copies of Paul Christiano living together comfortably and immortally and discussing what the AI should do with the physical universe; once there's a large supermajority in favor of an enactable plan (which can include further such delegated decisions), the AI does that".
I agree that this is going to be chaotic, in the sense that even slightly different elements of this reference class might end up steering the AI to different basins of attraction.
I assert, however, that I'd consider it a pretty good outcome overall if the future of the world were determined by a genuinely random draw from this reference class, honestly instantiated. (Again with the massive underspecification, I know.)
CEV may be underdetermined and many-valued, but that doesn't mean paperclipping is as good an answer as any.
Re: no basins, it would be a bad situation indeed if the vast majority of the reference class never ended up outputting an action plan, instead deferring and delegating forever. I don't have cached thoughts about that.
I for one welcome our new typographical overlords.
That's a legit thing to be frustrated by, but I think you know the reason why AI safety researchers don't want "we don't see a way to get to a good outcome except for an aligned project to grab a decisive strategic advantage" to filter into public discourse: it pattern-matches too well to "trust us, you need to let us run the universe".
To be clear, I am making the claim that, of the people who have made useful advances on Oracle AI safety research (Armstrong counts here; I don't think Yampolskiy does), all of them believe that the goal of having a safe Oracle AI is to achieve a decisive strategic advantage quickly and get to an aligned future. I recognize that this is a hard claim to evaluate (e.g. because this isn't a statement one could put in a Serious Academic Journal Article in the 2010s, it would have to be discussed on their blog or in private correspondence), but if anyone has a clear counterexample, I'd be interested in seeing it.
Yes, this. NVC should be treated with a similar sort of parameters to Crocker's Rules, which you can declare for yourself at any time, you can invite people to a conversation where it's known that everyone will be using them, but you cannot hold it against anyone if you invite them to declare Crocker's Rules and they refuse.
There's a lot of Actually Bad things an AI can do just by making electrons move.
I'd be interested in a list of well-managed government science and engineering projects if one exists. The Manhattan Project and the Apollo Project both belong on that list (despite both having their flaws- leaks to the USSR from the former, and the Apollo 1 disaster from the latter); what are other examples?
I'm pretty sure that, without exception, anyone who's made a useful contribution on Oracle AI recognizes that "let several organizations have an Oracle AI for a significant amount of time" is a world-ending failure, and that their work is instead progress on questions like "if you can have the only Oracle AI for six months, can you save the world rather than end it?"
Correct me if I'm wrong.
Thanks!
I agree there are broken alarms that are quiet (including those that are broken in the direction of failing to go off, which leads to a blind spot of obliviousness!), and that there are people stuck in situations where there is a correct loud alarm that happens most of the time.
I said that habits are easier to change than alarms, not that they're easy in an absolute sense.
It's because the non-broken alarms, which also start out loud, get quieter throughout your life as they calibrate themselves, and as one's habits fix the situations that make them correctly go off. So given a random initial distribution of loudness, eventually the alarm that's loudest on average will probably be a broken one.
The formatting didn't come through when importing this post from your blog- especially the strikethroughs of failed predictions and the graphs!
In the spirit of Murphyjitsu, the most obvious failure mode that you didn't mention is that I expect you to burn out dramatically after a few weeks, from exhaustion or the psychological strain of trying to optimize the experiences of N people. The bootcamp phase is not analogous to anything I've heard of you doing sustainably for an extended period of time.
So, do you expect Dragon Army Barracks to work if Eli has to take over for you in Week Four?
I like this suggestion of a more feasible form of steganography for NNs to figure out! But I think you'd need further advances in transparency to get useful informed oversight capabilities from (transformed or not) copies of the predictive network.
I should have said "reliably estimate HCH"; I'd also want quite a lot of precision in addition to calibration before I trust it.
Re #2, I think this is an important objection to low-impact-via-regularization-penalty in general.
Re #1, an obvious set of questions to include in are questions of approval for various aspects of the AI's policy. (In particular, if we want the AI to later calculate a human's HCH and ask it for guidance, then we would like to be sure that HCH's answer to that question is not manipulated.)
There's the additional objection of "if you're doing this, why not just have the AI ask HCH what to do?"
Overall, I'm hoping that it could be easier for an AI to robustly conclude that a certain plan only changes a human's HCH via certain informational content, than for the AI to reliably calculate the human's HCH. But I don't have strong arguments for this intuition.
Question that I haven't seen addressed (and haven't worked out myself): which of these indifference methods are reflectively stable, in the sense that the AI would not push a button to remove them (or switch to a different indifference method)?
This is a lot of good work! Modal combat is increasingly deprecated though (in my opinion), for reasons like the ones you noted in this post, compared to studying decision theory with logical inductors; and so I'm not sure this is worth developing further.
Yup, this isn't robust to extremely capable systems; it's a quantitative shift in how promising it looks to the agent to learn about external affairs, not a qualitative one.
(In the example with the agent doing engineering in a sandbox that doesn't include humans or general computing devices, there could be a strong internal gradient to learn obvious details about the things immediately outside its sandbox, and a weaker gradient for learning more distant or subtle things before you know the nearby obvious ones.)
A whitelisting variant would be way more reliable than a blacklisting one, clearly.
How did this post get attributed to [deleted] instead of to Eliezer? I'm 99% sure this post was by him, and the comments seem to bear it out.
This sweeps some of the essential problems under the rug; if you formalize it a bit more, you'll see them.
It's not an artificial restriction, for instance, that a Solomonoff Induction oracle machine doesn't include things like itself in its own hypothesis class, since the question of "whether a given oracle machine matches the observed data" is a question that sometimes cannot be answered by an oracle machine of equivalent power. (There are bounded versions of this obstacle as well.)
Now, there are some ways around this problem (all of them, so far as I know, found by MIRI): modal agents, reflective oracle machines and logical inductors manage to reason about hypothesis classes that include objects like themselves. Outside of MIRI, people working on multiagent systems make do with agents that each assume the other is smaller/simpler/less meta than itself (so at least one of those agents is going to be wrong).
But this entire problem is hidden in your assertion that the agent, which is a Turing machine, "models the entire wrold, including the agent it self, as one unknown, output only Turing machine". The only way to find the other problems swept under the rug here is to formalize or otherwise unpack your proposal.
If CFAR will be discontinuing/de-emphasizing rationality workshops for the general educated public, then I'd like to see someone else take up that mantle, and I'd hope that CFAR would make it easy for such a startup to build on what they've learned so far.
Nice! One thing that might be useful for context: what's the theoretical correct amount of time that you would expect an algorithm to spend on the right vs. the left if the session gets interrupted each time it goes 1 unit to the right? (I feel like there should be a pretty straightforward way to calculate the heuristic version where the movement is just Brownian motion that gets interrupted early if it hits +1.)