Posts
Comments
The other explanation I've heard bandied about is a polygenic version of sickle-cell anemia (where being heterozygous for the allele protects you from malaria but being homozygous gives you an awful disease).
In this model, there are a bunch of alleles that all push the phenotype in roughly the same direction, and having some of them is good, but past some threshold fitness starts rapidly declining.
(Further speculation: the optimum threshold is higher in the environment of civilization than in the ancestral environment, so these genes are experiencing massive positive selection over the last 10,000 years.)
This isn't a causal explanation, but it differs from the tower-foundation model in claiming that there's not a separate weakness to go looking for, just an optimum that's being surpassed- especially when two people near the optimum have children.
The NTSB report was released last week, showing that Uber's engineering was doing some things very wrong (with specifics that had not been reported before). Self-driving programs shouldn't go on public roads with that kind of system, even with a driver ready to take over.
Exactly. It seems like you need something beyond present imitation learning and deep reinforcement learning to efficiently learn strategies whose individual components don't benefit you, but which have a major effect if assembled perfectly together.
(I mean, don't underestimate gradient descent with huge numbers of trials - the genetic version did evolve a complicated eye in such a way that every step was a fitness improvement; but the final model has a literal blind spot that could have been avoided if it were engineered in another way.)
I just finished Story of Your Life, and I disagree with your attempt to retcon it as a physically possible story; I think the evidence is clear that Chiang intended it to be understood the way that most people understand it, and that he in fact was misunderstanding* the interaction between the variational principle and causality.
[Spoilers ahead.]
Firstly, it is a really plausible misunderstanding. If you don't deeply grok how decoherence works, then the principle of least action can really seem like it requires information to behave in nonsequential ways. (But in fact the Schrodinger equation exhibits causality, and variational principles are just the result of amplitudes nearly cancelling out for all paths whose action is not a local extremum.) Chiang is a gifted amateur in physics, so this misconception is not that unlikely.
Secondly, if your interpretation were his intended one, he could have done any number of things to suggest it! Louise directly tells the reader that this story is being told from the night of her daughter's conception, and that she now sees both future and past but does not act to change it. This is shown in the story to be the point of Heptapod B, and Chiang works at length to try and justify how understanding this language lets one understand (but not alter) the future. There's nothing else in the story to hint that Louise is an unreliable narrator.
If your interpretation were Chiang's, he would have to be intentionally misdirecting the audience to a degree you only see from authors like Nabokov, and not leaving any clue except for flawed science, which is common enough for dramatic license in science fiction that it really can't count as a clue. I doubt Chiang is doing that.
And finally, the whole thematic import of the story is ruined by your interpretation; the poignance comes from Louise's existentialist fatalism, giving life to the lines of the tragic play she is cast in, which is only meaningful insofar as she's telling the truth about perceiving forwards and backwards. (It's a variant on the Trafalmadorians, though interestingly Chiang hadn't yet read Slaughterhouse Five when he wrote this.) It's just a more boring story the way you see it!
*Possibly intentionally misunderstanding; I can imagine a science fiction author pestering their physics friend with "I've got a great story idea using this concept", the friend saying "It doesn't work that way", and the author asking "But if I pretend it does work that way, exactly how much physics knowledge would it take someone to be annoyed by it?"
Decision-theoretic blackmail is when X gets Y to choose A over B, not via acting to make the consequences of A more appealing to Y, but by making the consequences of B less appealing to Y.
The exceptions to this definition are pretty massive, though, and I don't know a principled emendation that excludes them.
1. There's a contract / social contract / decision-theoretic equilibrium, and within that, B will be punished. (This may not be a true counterexample, because the true choice is whether to join the contract... though this is less clear for the social contract than for the other two.)
2. Precommitting not to give in to blackmail is not itself blackmail. Of course, in an ultimatum game both players can imagine themselves as doing this.
Can anyone think of more exceptions, or a redefinition that clearly excludes these?
Removing things entirely seems extreme.
Dropout is a thing, though.
Shapley Values [thanks Zack for reminding me of the name] are akin to credit assignment: you have a bunch of agents coordinating to achieve something, and then you want to assign payouts fairly based on how much each contribution mattered to the final outcome.
And the way you do this is, for each agent you look at how good the outcome would have been if everybody except that agent had coordinated, and then you credit each agent proportionally to how much the overall performance would have fallen off without them.
So what about doing the same here- send rewards to each contributor proportional to how much they improved the actual group decision (assessed by rerunning it without them and seeing how performance declines)?
Customer service human interactions don't feel especially valuable to me, compared to intentional human interactions or even seeing other people walking down the street.
Good comment. I disagree with this bit:
I would, for instance, predict that if Superintelligence were published during the era of GOFAI, all else equal it would've made a bigger splash because AI researchers then were more receptive to abstract theorizing.
And then it would probably have been seen as outmoded and thrown away completely when AI capabilities research progressed into realms that vastly surpassed GOFAI. I don't know that there's an easy way to get capabilities researchers to think seriously about safety concerns that haven't manifested on a sufficient scale yet.
To copy myself in another thread, AlphaZero did some (pruned) game tree exploration in a hardcoded way that allowed the NN to focus on the evaluation of how good a given position was; this allowed it to kind of be a "best of both worlds" between previous algorithms like Stockfish and a pure deep reinforcement learner.
Re: your middle paragraph, I agree that you're correct about an RL agent doing metalearning, though we're also agreed that with current architectures it would take a prohibitive amount of computation to get anything like a competent general causal reasoner that way.
I'm not going to go up against your intuitions on imitation learning etc; I'm just surprised if you don't expect there's a necessary architectural advance needed to make anything like general causal reasoning emerge in practice from some combination of imitation learning and RL.
AlphaZero did some (pruned) game tree exploration in a hardcoded way that allowed the NN to focus on the evaluation of how good a given position was; this allowed it to kind of be a "best of both worlds" between previous algorithms like Stockfish and a pure deep reinforcement learner.
This is impossible for a game with an action space as large as StarCraft II, though; but in order to modify a game like Go, it would have to become completely different.
I'm not 100% sure about the example you raise, but it seems to me it's either going to have a decently prune-able game tree, or that humans won't be capable of playing the game at a very sophisticated level, so I'd expect AlphaZero-esque things to get superhuman at it. StarCraft is easier for humans relative to AIs because we naturally chunk concepts together (visually and strategically) that are tricky for the AI to learn.
DeepMind says it hopes the techniques used to develop AlphaStar will ultimately help it "advance our research in real-world domains".
But Prof Silver said the lab "may rest at this point", rather than try to get AlphaStar to the level of the very elite players.
Thanks, will do.
Virtue ethics seems less easily applicable to the domain of "what governmental policies to support" than to the domain of personal behavior, so I had a hard time thinking of examples. Can you?
Right, for them "alignment" could mean their desired concept, "safe for everyone except our targets".
DeepMind released their AlphaStar paper a few days ago, having reached Grandmaster level at the partial-information real-time strategy game StarCraft II over the summer.
This is very impressive, and yet less impressive than it sounds. I used to watch a lot of StarCraft II (I stopped interacting with Blizzard recently because of how they rolled over for China), and over the summer there were many breakdowns of AlphaStar games once players figured out how to identify the accounts.
The impressive part is getting reinforcement learning to work at all in such a vast state space- that took breakthroughs beyond what was necessary to solve Go and beat Atari games. AlphaStar had to have a rich enough set of potential concepts (in the sense that e.g. a convolutional net ends up having concepts of different textures) that it could learn a concept like "construct building P" or "attack unit Q" or "stay out of the range of unit R" rather than just "select spot S and enter key T". This is new and worth celebrating.
The overhyped part is that AlphaStar doesn't really do the "strategy" part of real-time strategy. Each race has a few solid builds that it executes at GM level, and the unit control is fantastic, but the replays don't look creative or even especially reactive to opponent strategies.
That's because there's no representation of causal thinking - "if I did X then they could do Y, so I'd better do X' instead". Instead there are many agents evolving together, and if there's an agent evolving to try Y then the agents doing X will be replaced with agents that do X'.
(This lack of causal reasoning especially shows up in building placement, where the consequences of locating any one building here or there are minor, but the consequences of your overall SimCity are major for how your units and your opponents' units would fare if they attacked you. In one comical case, AlphaStar had surrounded the units it was building with its own factories so that they couldn't get out to reach the rest of the map. Rather than lifting the buildings to let the units out, which is possible for Terran, it destroyed one building and then immediately began rebuilding it before it could move the units out!)
This means that, first, AlphaStar just doesn't have a decent response to strategies that it didn't evolve, and secondly, it doesn't do much in the way of a reactive decision tree of strategies (if I scout this, I do that). That kind of play is unfortunately very necessary for playing Zerg at a high level, so the internal meta has just collapsed into one where its Zerg agents predictably rush out early attacks that are easy to defend if expected. This has the flow-through effect that its Terran and Protoss are weaker against human Zerg than against other races, because they've never practiced against a solid Zerg that plays for the late game.
The end result cleaned up against weak players, performed well against good players, but practically never took a game against the top few players. I think that DeepMind realized they'd need another breakthrough to do what they did to Go, and decided to throw in the towel while making it look like they were claiming victory.
Finally, RL practitioners have known that genuine causal reasoning could never be achieved via known RL architectures- you'd only ever get something that could execute the same policy as an agent that had reasoned that way, via a very expensive process of evolving away from dominated strategies at each step down the tree of move and countermove. It's the biggest known unknown on the way to AGI.
[Cross-posted from Medium, written for a pretty general audience]
There are many words that could describe my political positions. But there's one fundamental label for me: I am a consequentialist.
Consequentialism is a term from ethics; there, it means the position that consequences are what truly make an action right or wrong, rather than rules or virtues. What that means is that for me, the most essential questions about policy aren't things like "what is fair" or "what rights do people have", although these are good questions. For me, it all boils down to "how do we make people's lives better?"
(There are some bits of nuance to the previous paragraph, which I've kept as a long endnote.)
"Make people's lives better" isn't a platitude- there's a real difference here! To explain, I want to point out that there are both consequentialists and non-consequentialists within different political camps. Let's consider socialists first and then libertarians second.
Many socialists believe both that (A) the world is headed for plutocratic disaster unless capitalism is overthrown, and that (B) labor markets and massive wealth disparities would be crimes even if they did not doom others to suffering. The difference is that some are more motivated by beliefs like (A), and could thus change their positions if convinced that e.g. the Nordic model was much better for future growth than a marketless society; while others are more motivated by beliefs like (B), and would continue to support pure socialism even if they were convinced it would mean catastrophe.
And many libertarians believe both that (A') the only engine that can consistently create prosperity for all is a free market with no interference, and that (B') taxation is a monstrous act of aggression and theft. The difference is that some are more motivated by beliefs like (A'), and thus could change their position if convinced that e.g. progressive taxation and redistribution would not destroy the incentives behind economic growth; while others are more motivated by beliefs like (B'), and would continue to support pure libertarianism even if they were convinced it would mean catastrophe.
I find it fruitful to talk with the first kind of socialist and the first kind of libertarian, but not the second kind of either. The second type just isn’t fundamentally interested in thinking about the consequences (except insofar as they can convince others by arguing for certain consequences). But among the first type, it’s possible to figure out the truth together by arguing about historical cases, studying natural experiments in policy, and articulating different theories.
I hope it's been helpful to draw out this distinction; I'd encourage you to first find fellow consequentialists among your natural allies, and expand from there when and if you feel comfortable. There's a lot that can be done to make the world a better place, and those of us who care most about making the world better can achieve more once we find each other!
P.S. The above focuses on the sort of political questions where most people's influence is limited to voting and convincing others to vote with them. But there's more ways to have an effect than that; I'd like to take one last moment to recommend the effective altruism movement, which investigates the best ways for people to have a big positive impact on the world.
---
Nuance section:
the position that consequences are what truly make an action right or wrong
There's a naive version of this, which is that you should seize any good immediate outcome you can, even by doing horrific things. That's... not a healthy version of consequentialism. The way to be less naive is to care about long-term consequences, and also to expect that you can't get away with keeping your behavior secret from others in general. Here's a good account of what non-naive consequentialism can look like.
the most essential questions about policy aren't things like "what is fair" or "what rights do people have", although these are good questions
In particular, fairness and rights are vital to making people's lives better! We want more than just physical comforts; we want autonomy and achievement and meaning, we want to have trustworthy promises about what the world will ask of us tomorrow, and we want past injustices to be rectified. But these can be traded off, in extreme situations, against the other things that are important for people. In a massive emergency, I'd rather save lives in an unfair way and try to patch up the unfairness later, than let people die to preserve fairness.
how do we make people's lives better?
This gets complicated and weird when you apply it to things like our distant descendants, but there are some aspects in the world today that seem fairly straightforward. Our world has built an engine of prosperity that makes food and goods available to many, beyond what was dreamt of in the past. But many people in the world are still living short and painful lives filled with disease and starvation. Another dollar of goods will do much more for one of them than for one of us. If we can improve their lives without destroying that engine, it is imperative to do that. (What consequentialists mostly disagree on is how the engine really works, how it could be destroyed, and how it could be improved!)
Well, damn. I started out reading this really skeptically, but the process fit my own past experiences of changing core beliefs frighteningly well. (The only thing that's ever noticeably worked for me is something like showing that one of my IFS parts is a hypocrite, which results in it more or less self-destructing and allowing change to happen.) I just emailed a Coherence Therapy practitioner, since I'm looking for a new therapist; this seems worth a try.
Quick approximate summary:
- John von Neumann first says, essentially, that the Industrial Revolution has made the world smaller, and that in earlier centuries the problems caused by it were contained to nations, but that now they extend to the entire globe.
- For a first example, he talks about cheap energy and industrial synthesis, though he predicts that nuclear fusion and transmutation would be much more practically feasible than they have turned out to be.
- He briefly mentions expected major improvement in automation, communication, and transportation.
- He then talks about anthropogenic climate change and the broad possibilities of geoengineering the climate.
- All of these technologies can vastly improve human life, or destroy it.
- He immediately rules out the "solution" of [preventing advances in technology] as both undesirable (because it blocks the positive uses) and impossible (because it would require total coordination and a total change in human values).
- He next considers the possibility of permanently avoiding war through diplomacy etc, and does not think that the 1950s drive for world peace will last long; furthermore, such an initiative would need to adapt to ever-more-powerful technologies as fast as they are introduced.
- He frames the upcoming decades as a dangerous but useful evolution, where we will either succeed or fail catastrophically, and doesn't sound especially optimistic. Our best hope is to innovate new political forms that are capable of handling major threats with patience, flexibility, and intelligence.
Alas, double-cruxing is a two-person game; you can't make another person do it, you can only do it with someone who genuinely wants to. For everyone else, the best related trick I know is nonviolent communication.
Meta: I approve of the practice of arguing against your own post in a comment.
How much less do you expect this to happen under the current system?
both historically and now, criticism is often met with counterarguments based on "style" rather than engaging with the technical meat of the criticism
Is there any group of people who reliably don't do this? Is there any indication that AI researchers do this more often than others?
Eliezer's real answer to this question is discussed in Timeless Control. Basically, choice is still meaningful in many-worlds or any other physically deterministic universe. There are incredibly few Everett branches starting from here where tomorrow I go burn down an orphanage, and this is genuinely caused by the fact that I robustly do not want to do that sort of thing.
If you have altruistic motivation, then the Everett branches starting from here are in fact better (in expectation) than the branches starting from a similar universe with a version of you that has no altruistic motivation. By working to do good, you are in a meaningful sense causing the multiverse to contain a higher proportion of good worlds than it otherwise would.
It really does all add up to normality, even if it feels counterintuitive.
Well, this post aged interestingly for those of us who know the author (who ended up working for a high-profile EA organization for some time).
Typo:
it is not the done the thing
it is not the done thing, perhaps?
Maybe you can get the best of both worlds by imagining you're writing a children's book, but that your editor is in fact an expert on the subject and you don't want to embarrass yourself in front of them.
And Robin Hanson was surprised that no big corporation wanted to implement a real prediction market?
This strongly resembles the argument given by Subhan in EY's post Is Morality Preference?, with a side order of Fake Selfishness. You might enjoy reading those posts along with others in their respective sequences. ("Is Morality Preference?" was part of the original metaethics sequence but didn't make the cut for Rationality: AI to Zombies.)
More to the point, the biggest mistake I see here is the one addressed in The Domain of Your Utility Function: yes, my moral preferences are a part of my map rather than the territory, but there's still a damn meaningful difference between egoism (preferences that point only to the part of my map labeled "my future experiences") and my actual moral preferences, which point to many other parts of the map as well.
I am struggling to understand the goal of the post.
The title was helpful to me in that regard. Each of these examples shows an agent who could run an honest process to get evidence on a question, but which prefers one answer so much that they try to stack the deck in that direction, and thereby loses the hoped-for benefits of that process.
Getting an honest Yes requires running the risk of getting a No instead.
Formatting request: can the footnote numbers be augmented with links that jump to the footnote text? (I presume this worked in Arbital but broke when it was moved here.)
I had a notion here that I could stochastically introduce a new goal that would minimize total suffering over an agent's life-history. I tried this, and the most stable solution turned out to be thus: introduce an overwhelmingly aversive goal that causes the agent to run far away from all of its other goals screaming.
did you mean: anhedonia
(No, seriously, your paragraph is an apt description of a long bout I had of depression-induced anhedonia; I felt so averse to every action that I ceased to feel wants, and I consistently marked my mood as neutral rather than negative despite being objectively more severely depressed than I was at other times when I put negative numbers in my mood tracker.)
The ideal thing is to judge Bob as if he were making the same prediction every day until he makes a new one, and log-score all of them when the event is revealed. (That is, if Bob says 75% on January 1st and 60% on February 1st, and then on March 1st the event is revealed to have happened, Bob's score equals 31*log(.25) + 28*log(.4). Then Bob's best strategy is to update his prediction to his actual current estimate as often as possible; past predictions are sunk costs.
The real-world version is remembering to dock people's bad predictions more, the longer they persisted in them. But of course this is hard.
538 did do this with their self-evaluation, which is a good way to try and establish a norm in the domain of model-driven reporting.
Let's note differences of degree here. Political systems differ massively in how easily decisionmakers can claim large spoils for themselves, and these differences seem to correlate with how pro-social the decisions tend to be. In particular, the dollar amounts of graft being alleged for politicians in liberal democracies are usually small compared to what despots regularly claim without consequence. (Which is not to say that it would be wise to ignore corruption in liberal democracies!)
I'm not convinced that Europe had more intellectual freedom on average than China, but because of the patchwork of principalities, it certainly had more variation in intellectual freedom than did a China that was at any given time either mostly unified or mostly at war; and all that you need for an intellectual revolution is the existence of a bastion of intellectual freedom somewhere.
It doesn't move much probability mass to the very near term (i.e. 1 year or less), because both this and AlphaStar aren't really doing consequentialist reasoning, they're just able to get a surprising performance with simpler tricks (the very Markovian nature of human writing, a good position evaluation function) given a whole lot of compute.
However, it does shift my probabilities forward in time, in the sense that one new weird trick to do deductive or consequentialist reasoning, plus a lot of compute, might get you there really quickly.
I expect that some otherwise convinceable readers are not going to realize that in this fictional world, people haven't discovered Newton's physics or calculus, and those readers are therefore going to miss the analogy of "this is how MIRI would talk about the situation if they didn't already know the fundamental concepts but had reasons for searching in the right direction". (I'm not thinking of readers incapable of handling that counterfactual, but of readers who aren't great at inferring implicit background facts from a written dialogue. Such readers might get very confused at the unexpected turns of the dialogue and quit rather than figure out what they're baffled by.)
I'd suggest adding to the preamble something like "In a weird world where people had figured out workable aeronautics and basic rocket propulsion by trial and error, but hadn't figured out Newton's laws or calculus".
I like your definition, though, and want to try to make a better one (and I acknowledge this is not the point of this post).
I think that's a perfectly valid thing to do in the comments here! However, I think your attempt,
My stab at a refinement of "consent" is "respect for another's choices", where "disrespect" is "deliberately(?) doing something to undermine"
is far too vague to be a useful concept.
In most realistic cases, I can give a definite answer to whether A touched B in a way B clearly did not want to be touched. In the case of my honesty definition, it does involve intent and so I can only infer statistically when someone else is being dishonest vs mistaken, but for myself I usually have an answer about whether saying X to person C would be honest or not.
I don't think I could do the same for your definition; "am I respecting their choices" is a tough query to bottom out in basic facts.
My comment was meant to explain what I understood Eliezer to be saying, because I think you had misinterpreted that. The OP is simply saying "don't give weight to arguments that are locally invalid, regardless of what else you like about them". Of course you need to use priors, heuristics, and intuitions in areas where you can't find an argument that carries you from beginning to end. But being able to think "oh, if I move there, then they can take my queen, and I don't see anything else good about that position, so let's not do that then" is a fair bit easier than proving your move optimal.
Relying purely on local validity won't get you very far in playing chess
The equivalent of local validity is just mechanically checking "okay, if I make this move, then they can make that move" for a bunch of cases. Which, first, is a major developmental milestone for kids learning chess. So we only think it "won't get you very far" because all the high-level human play explicitly or implicitly takes it for granted.
And secondly, it's pretty analogous to doing math; proving theorems is based on the ability to check the local validity of each step, but mathematicians aren't just brute-forcing their way to proofs. They have to develop higher-level heuristics, some of which are really hard to express in language, to suggest avenues, and then check local validity once they have a skeleton of some part of the argument. But if mathematicians stopped doing that annoying bit, well, then after a while you'll end up with another crisis of analysis when the brilliant intuitions are missing some tiny ingredient.
Local validity is an incredibly important part of any scientific discipline; the fact that it's not a part of most political discourse is merely a reflection that our society is at about the developmental level of a seven-year-old when it comes to political reasoning.
Broken link on the text "real killing of birds to reduce pests in China has never been tried".
Much of this material is covered very similarly in Melting Asphalt, especially the posts Ads Don't Work That Way and Doesn't Matter, Warm Fuzzies.
If you do future surveys of this sort, I'd like you to ask people for their probabilities rather than just their best guesses. If people are uncertain but decently calibrated, I'd argue there's not much of a problem; if people are confidently wrong, I'd argue there's a real problem.
This comment got linked a decade later, and so I thought it's worth stating my own thoughts on the question:
We can consider a reference class of CEV-seeking procedures; one (massively-underspecified, but that's not the point) example is "emulate 1000 copies of Paul Christiano living together comfortably and immortally and discussing what the AI should do with the physical universe; once there's a large supermajority in favor of an enactable plan (which can include further such delegated decisions), the AI does that".
I agree that this is going to be chaotic, in the sense that even slightly different elements of this reference class might end up steering the AI to different basins of attraction.
I assert, however, that I'd consider it a pretty good outcome overall if the future of the world were determined by a genuinely random draw from this reference class, honestly instantiated. (Again with the massive underspecification, I know.)
CEV may be underdetermined and many-valued, but that doesn't mean paperclipping is as good an answer as any.
Re: no basins, it would be a bad situation indeed if the vast majority of the reference class never ended up outputting an action plan, instead deferring and delegating forever. I don't have cached thoughts about that.
I for one welcome our new typographical overlords.
That's a legit thing to be frustrated by, but I think you know the reason why AI safety researchers don't want "we don't see a way to get to a good outcome except for an aligned project to grab a decisive strategic advantage" to filter into public discourse: it pattern-matches too well to "trust us, you need to let us run the universe".
To be clear, I am making the claim that, of the people who have made useful advances on Oracle AI safety research (Armstrong counts here; I don't think Yampolskiy does), all of them believe that the goal of having a safe Oracle AI is to achieve a decisive strategic advantage quickly and get to an aligned future. I recognize that this is a hard claim to evaluate (e.g. because this isn't a statement one could put in a Serious Academic Journal Article in the 2010s, it would have to be discussed on their blog or in private correspondence), but if anyone has a clear counterexample, I'd be interested in seeing it.
Yes, this. NVC should be treated with a similar sort of parameters to Crocker's Rules, which you can declare for yourself at any time, you can invite people to a conversation where it's known that everyone will be using them, but you cannot hold it against anyone if you invite them to declare Crocker's Rules and they refuse.
There's a lot of Actually Bad things an AI can do just by making electrons move.
I'd be interested in a list of well-managed government science and engineering projects if one exists. The Manhattan Project and the Apollo Project both belong on that list (despite both having their flaws- leaks to the USSR from the former, and the Apollo 1 disaster from the latter); what are other examples?