Previous LW discussions on taking over the world (last updated in 2013).
Comments of mine on "utopian hope versus reality" (dating from 2012).
Since that era, a few things have happened.
First change: LW is not quite the point of focus that it was. There was a rationalist diaspora into social media, and "Slate Star Codex" (and its associated subreddits?) became a more prominent locus of rationalist discussion. The most important "LW-ish" forums that I know about now, might be those which focus on quasi-technical discussion of AI issues like "alignment". I call them the most important because of...
Second change: The era of deep learning, and of commercialized AI in the guise of "machine learning", arrived. The fact that these algorithms are not limited to the resources of a single computer, but can in principle tap the resources of an entire data center or even the entire cloud of a major tech corporation, means that we have also arrived at the final stage of the race towards superintelligence.
In the past, taking over the world meant building or taking over the strongest superpower. Now it simply means being the first to create strongly superhuman intelligence; and saving the world means identifying a value system that will make an autonomous AI "friendly", and working to ensure that the winner of the mind race is guided by friendly rather than unfriendly values. Every other concern is temporary, and any good work done towards other causes, will potentially be undone by unfriendly AI, if unfriendly values win the AI race.
(I do not say with 100% certainty that this is the nature of the world, but this scenario has sufficient internal logic that, if it does not apply to reality, there must be some other factor which somehow overrides it.)
People like Schopenhauer and Benatar are just being realistic. Reality includes futility and horror on enormous scales. Perhaps the remaking of Earth by superhuman AI offers an imminent chance that even this can change, but it's just a chance.
So what is he saying? We never need to solve the problem of designing a human-friendly superintelligent agent?
This is the reverse of the usual argument that we should not believe we are going to have a googol descendants. Usually one says: to be living at the beginning of time means that you belong to a very special minority, therefore it would take more indexical information to single you out, compared to someone from the middle of history.
The thought experiment involves observers being in a coherent superposition. But I'm not now 100% sure that it involves actual quantum erasure, I was relying on other people's description. I'm hoping this will be cleared up without having to plough through the paper myself.
Anyway, LW may appreciate this analysis which actually quotes HPMOR.
It's a minor new quantum thought experiment which, as often happens, is being used to promote dumb sensational views about the meaning or implications of quantum mechanics. There's a kind of two-observer entangled system (as in "Hardy's paradox"), and then they say, let's also quantum-erase or recohere one of the observers so that there is no trace of their measurement ever having occurred, and then they get some kind of contradictory expectations with respect to the measurements of the two observers.
Undoing a quantum measurement in the way they propose is akin to squirting perfume from a bottle, then smelling it, and then having all the molecules in the air happening to knock all the perfume molecules back into the bottle, and fluctuations in your brain erasing the memory of the smell. Classically that's possible but utterly unlikely, and exactly the same may be said of undoing a macroscopic quantum measurement, which requires the decohered branches of the wavefunction (corresponding to different measurement outcomes) to then separately evolve so as to converge on the same state and recohere.
Without even analyzing anything in detail, it is hardly surprising that if an observer is subjected to such a highly artificial process, designed to undo a physical event in its totality, then the observer's inferences are going to be skewed somehow. So, you do all this and the observers differ in their quantum predictions somehow. In their first interpretation (2016), Frauchiger and Renner said that this proves many worlds. Now (2018), they say it proves that quantum mechanics can't describe itself. Maybe if they try a third time, they'll hit on the idea that one of the observers is just wrong.
Migraine is just an occasional problem. Living and working conditions are the truly chronic problem that have made me irrelevant.
Thanks for a response. I am actually most concerned about the things that I could be doing, that I don't see anyone else doing, and which aren't being done because I am operating at far below my potential. In my case, I think illness is very much just a symptom of the struggle to get on with things in an interfering environment.
The most ambitious thing that I can think of attempting, is to solve the AI value alignment problem in time for Earth's singularity. After this bout of sickness, and several days of dawdling while I waited to recover, I somehow have a new tactic for approaching the problem (it's more a personal tactic for engaging with the problem, than an idea for a solution). I hate the idea that this kind of experience is the price I pay for really pushing ahead, but it may be so.
One night, without sleep
2018-08-16T17:50:06.036Z · score: 16 (9 votes)Banning high-end GPUs so that only the government can have AI? They could do it, they might feel compelled to do something like it, but there would be serious resistance and moments of sheer pandemonium. They can say it's to protect humanity, but to millions of people it will look like the final step in the enslavement of humanity.
"Organization working on AI" vs "any other kind of organization" is not the important point. The important point is ALL. We are talking about a hypothetical organization capable of shutting down ALL artificial intelligence projects that it does not like, no matter where on earth they are. Alicorn kindly gives us an example of what she's talking about: "destroy all the GPUs on the planet and prevent the manufacture of new ones".
Just consider China, Russia, and America. China and America lead everyone else in machine learning; Russia has plenty of human capital and has carefully preserved its ability to not be pushed around by America. What do you envisage - the three of them agree to establish a single research entity, that shall be the only one in the world working on AI near a singularity threshold, and they agree not to have any domestic projects independent of this joint research group, and they agree to work to suppress rival groups throughout the world?
Despite your remarks about how the NSA could easily become the hub of a surveillance state tailored to this purpose, I greatly doubt the ability of NSA++ to successfully suppress all rival AI work even within America and throughout the American sphere of influence. They could try, they could have limited success - or they could run up against the limits of their power. Tech companies, rival agencies, coalitions of university researchers, other governments, they can all join forces to interfere.
In my opinion, the most constructive approach to the fact that there are necessarily multiple contenders in the race towards superhuman intelligence, is to seek intellectual consensus on important points. The technicians who maintain the world's nuclear arsenals agree on the basics of nuclear physics. The programmers who maintain the world's search engines agree on numerous aspects of the theory of algorithms. My objective here would be that the people who are working in proximity to the creation of superhuman intelligence, develop some shared technical understandings about the potential consequences of what they are doing, and about the initial conditions likely to produce a desirable rather than an undesirable outcome.
A great power can think about doing such things against an opponent. But I thought we were talking about a scenario in which some AI clique has halted *all* rival AI projects throughout the entire world, effectively functioning like a totalitarian world government, but without having actually crossed the threshold of superintelligence. That is what I am calling a fantasy.
The world has more than one great power, great powers are sovereign within their own territory, and you are not going to overcome that independence by force, short of a singularity. The rest of the world will never be made to stop, just so that one AI team can concentrate on solving the problems of alignment without having to look over its shoulder at the competition.
How are you going to stop a rival nuclear-armed state from doing whatever it wants on its own territory?
Can someone in China halt AI research at Google and Amazon? Can someone in America halt AI research at Tencent and Baidu? Could the NSA halt unapproved AI research just throughout America?
By a singularity I mean creation of superhuman intelligence that nothing in the world can resist.
My opinion: The capacity to forcibly halt all rival AI projects, is only to be expected in an AI project that has already produced a singularity. It is not a viable tactic if you are aiming to create a friendly singularity. In that case, there is no alternative to solving the problems of friendly values and value stability, and either reaching singularity first, or influencing those who will get there before you.
Doesn't this paper boil down to "Some factors in the Drake equation are highly uncertain, and we don't see any aliens, so those probabilities must be small after all?"
I guess it makes sense, given enough assumptions. There's a multiverse; in some fraction of universes there are intelligences which figure out the correct theory of the multiverse; some fraction of those intelligences come up with the idea of acausally coordinating with intelligences in other universes, via a shared model of the multiverse, and are motivated to do so; and then the various island populations of intelligences who are motivated to attempt such a thing, try to reason about each other's reasoning, and act accordingly.
I suppose it deserves its place in the spectrum of arcane possibilities that receive some attention. But I would still like to see someone model this at the "multiverse level". Using the language of programs: if we consider some set of programs that *hasn't* been selected precisely so that they will engage in acausal coordination - perhaps the set of *all* well-formed programs in some very simple programming language - what are the prospects for the existence of nontrivial acausal trade networks? They may be very rare, they may be vastly outnumbered by programs which made a modeling error and are "trading" with nonexistent partners, and so on.
Has anyone ever actually presented an argument for such propositions? Like describing an ensemble of toy possible worlds in which even attempting "acausal trade" is rational, let alone one in which these acausal coalitions of acausal traders exist?
It might makes some sense to identify with all your subjective duplicates throughout the (hypothetical) multiverse, on the grounds that some fraction of them will engage in the same decision process, so that how you decide here is actually how a whole sub-ensemble of "you"s will decide.
But acausal trade, as I understand it, involves simulating a hypothetical other entity, who by hypothesis is simulating *you* in their possible world, so as to artificially create a situation in which two ensemble-identified entities can interact with each other.
I mean... Do you, in this world, have to simulate not just the other entity, but also simulate its simulation of you?? So that there is now a simulation of you in *this* world? Or is that a detail you can leave out? Or do you, the original you, roleplay the simulation? Someone show me a version of this that actually makes sense.
A long time ago, something similar happened to me. I was an immortalist, I was living the struggle, I got the call that someone had died. I remember thinking, OK, now I have to bring back the dead as well.
Good luck preserving your intentions and your functionality. Hopefully there are people around you in real life who at least half understand and half sympathize with your response to the situation.
Actions can have unexpected consequences. When you started your 80-day sprint, I started my own. Good luck between now and the end of July.
The fact that I was basically serious, and in no way attempting to discourage @elriggs, and yet the comment is (after 12 hours) at -17, suggests that LW now has a problem with people who *do* want to do advanced things.
I do mean it rather seriously. There are theoretical frameworks already in play, directed at solving the two major subproblems that you identify, i.e. creating raw superintelligence, and (let's say) identifying a friendly value system. I actually find it conceivable that the required breakthroughs are not far away, in the same way that e.g. imminent solutions to math's Millennium Problems are conceivable - someone just needs to have the right insights.
An exercise for those who are a little more advanced: *actually* save the world within the next 80 days. In the context of AI safety at the singularity level, that would mean completely figuring out the theory required to have a friendly singularity, and then making it happen for real, by the end of July.
I'm thinking large numbers of synchronized reusable beacons - either recurrent novas or black holes - where a flash is produced by feeding the beacon with gas in a controlled way. For rapid reuse, you want local byproducts of the flash to get out of the way quickly, so the next batch of gas can be introduced. That could mean dwarf novas, or black hole processes in which the waste comes out in tightly focused jets.
There is a "remarkable recurrent nova" in the Andromeda Galaxy, which repeats on a timescale of months.
I only have an account on LW 1.0, not LW 2.0. Will my current account still exist after the migration?
Four hours of self-play and it's the strongest in the world. Soon the machines will be parenting us.
Knowledge of the terrain might be hard to get reliably
Knowing that the world is made of atoms should take an AI a long way.
If these people that develop [AGI] are friendly they might decide to distribute it to other people to make it harder for any one project to take off.
I hold to the classic definition of friendly AI as being AI with friendly values, which retains them (or even improves them) as it surpasses human intelligence and otherwise self-modifies. As far as I'm concerned, AlphaGo Zero demonstrates that raw problem-solving ability has crossed a dangerous threshold. We need to know what sort of "values" and "laws" should govern the choices of intelligent agents with such power.
And you're the tax collector? Answer the question.
Just answer the question.
Wake up! In three days, that AI evolved from knowing nothing, to comprehensively beating an earlier AI which had been trained on a distillation of the best human experience. Do you think there's a force in the world that can stand against that kind of strategic intelligence?
How much are you willing to lose?
A voice tells me that we're out of time. The future of the world will now be decided at Deep Mind, or by some other group at their level.
EurekAlert mentions the simulation argument, and the page implies that this was a press release from Oxford - even providing a media contact - though I have not found the document on Oxford's own website.
I am also skeptical of what the paper (arxiv) is actually saying, on a technical level. It reminds me of another paper a few months ago, which was hyped as exhibiting a "gravitational anomaly" in a condensed-matter system. From all that I could make out, there was no actual gravitational effect involved, only a formal analogy.
This paper seems to engage in exactly the same equivocation, now with the objective of proving something about computational complexity. But I'll have to study it in more detail to be sure.
It sounds like you want some second opinions and rational evaluation regarding your political conclusion - necessity of revolt. OK.
I can think of reasons for and reasons against such a conclusion, but probably you should spell out more of your reasoning first. For example, why will revolt help humanity survive?
I'm going to take a wild guess, and suggest that your attitude towards FAI research, and your experience of CFS, are actually related. I have no idea if this is a standard theory, but in some ways CFS sounds like depression minus the emotion - and that is a characteristic symptom in people who have a purpose they regard as supremely important, who find absolutely no support for their attempt to pursue it, but who continue to regard it as supremely important.
The point being that when something is that important, it's easy to devalue certain aspects of your own difficulties. Yes, running into a blank wall of collective incomprehension and indifference may have been personally shattering; you may be in agony over the way that what you have to do in order to stay alive, interferes with your ability to preserve even the most basic insights that motivate your position ... but it's an indulgence to think about these feelings, because there is an invisible crisis happening that's much more important.
So you just keep grinding away, or you keep crawling through the desert of your life, or you give up completely and are left only with a philosophical perspective that you can talk about but can't act on... I don't know all the permutations. And then at some point it affects your health. I don't want to say that this is solely about emotion, we are chemical beings affected by genetics, nutrition, and pathogens too. But the planes intersect, e.g. through autoimmune disorders or weakened disease resistance.
The core psychological and practical problem is, there's a difficult task - the great purpose, whatever it is - being made more difficult in ways that have no intrinsic connection to the problem, but are solely about lack of support, or even outright interference. And then on top of that, you may also have doubts and meta doubts to deal with - coming from others and from yourself (and some of those doubts may be justified!). Finally, health problems round out the picture.
The one positive in this situation, is that while all those negatives can reinforce each other, positive developments in one area can also carry across to another.
OK, so that's my attempt to reflect back to you, how you sound to me. As for practical matters, I have only one suggestion. You say
he travels to the USA for a few days every couple of months
so I suggest that you at least wait until his next visit, and use that extra time to understand better how all these aspects of your life intersect.
If ASI-provided immortal life were possible, you would already be living it.
... because if you're somewhere in an infinite sequence, you're more likely to be in the middle than at the beginning.
Suppose there's some idea, X, which you think might help to solve a problem, Y. And there's also a dumb version of X, X', which you know doesn't work, but which still has enthusiasts.
And then one day there's a headline: CAN IDEA X SOLVE PROBLEM Y? Only you find out that it's actually X', the dumb version of X, that is being presented to the world as X... and nothing is done to convey the difference between X' and the version of X that actually warrants attention.
That is, more or less, the situation I find myself in, with respect to this article. I wish there were some snappier way to convey the situation, without talking about X and X' and so on, but I haven't found a way to do it.
Problem Y is: explain why quantum mechanics works, without saying that things don't have properties until they are measured, and so on.
Idea X is, these days, usually called Bohmian mechanics. To the Schrodinger equation, which describes the time evolution of the wavefunction of quantum mechanics, it adds a classical equation of motion for the particles, fields, etc. The particles, fields, etc., evolve on a trajectory in state space which follows the probability current in state space, as defined by the Schrodinger equation.
The original version of this idea is due to de Broglie, who proposed that particles are guided by waves. This was called pilot-wave theory, because the wave "pilots" the particle.
Pilot-wave theory was proposed in the very early days of quantum theory, before the significance of entanglement was properly appreciated. The significance of entanglement is that you don't have one wavefunction per particle, you just have one big wavefunction which provides probabilities for joint configurations of particles.
A pilot-wave theory for many particles, in the form that de Broglie originally proposed - one wave per particle - contains no entanglement, and can't reproduce the multi-particle predictions of quantum mechanics, as Bell's theorem and many other theorems show. Bohmian mechanics can reproduce those predictions, because in Bohmian mechanics, the wavefunction that does the piloting is the single, entangled, multi-particle wave used in actual quantum mechanics.
All this is utterly basic knowledge for the people who work on Bohmian mechanics today. But meanwhile, apparently a group of people who work on fluid dynamics, have rediscovered de Broglie's original idea - "wave guiding a particle" - and are now promoting it as a possible explanation of quantum mechanics. They don't seem to care about the theorems proving that you can't get Bell-type correlations without using entangled waves.
So basically, this article describes the second-rate researchers in this field - in this case, people who are doing the equivalent of trying to force the square peg into the round hole - as if they are the intellectual leaders who define it!
Ironically, the man Yevtushenko is now dead too; but the world Yevtushenko, asteroid number 4234, lives on.
What about pederasty in ancient Greece, what about sex in all-male prisons... in both those cases, you have men who by current definitions are not gay, but rather bisexual. And in both cases you have recruitment into an existing sexual culture, whether through seduction or coercion.
Human sexuality can clearly assume an enormous variety of forms, and I don't have a unified theory of it. Obviously genes matter, starting with the basic facts of sex determination, and then in a more elusive way, having some effect on sexual dispositions in the mature organism.
And yes, natural selection will be at work. But, in this case it is heavily mediated by culture (which is itself a realm of replicator populations), and it is constrained by the evolvability of the human genome. I continue to think that the existence of nonreproductive sexuality is simply a side effect of our genomic and evolutionary "business model", of leaky sexual dimorphism combined with Turing-complete cognition.
There actually is a known replicator that assists the reproduction of gay phenotypes, but it's a behavior: gay sex! For a recent exposition, see the video that cost "Milo" his job.
I believe in evolution, I just don't believe in the gay germ.
But regardless of belief... I have some questions which I think are fair questions.
Are there any ideas about how and when the gay germ is acquired?
Are there any ideas about its mechanism of action?
If homosexuality has such a huge fitness penalty, why haven't we evolved immunity to the gay germ?
If someone hasn't experienced sex yet, but they already think they are gay, is that because of the gay germ?
Give it all to Fabian Tassano.
OK, let's talk about proximate causes and ultimate causes. The proximate causes are whatever leads to the formation of a particular human individual's sexuality. The ultimate causes are whatever it is that brought about the existence of a population of organisms in which a given sexuality is even possible.
My focus has been on proximate causes. I look at the role of fantasy, choice, and culture in shaping what a person seeks and what they can obtain, and the powerful conditioning effect of emotional and sexual reward once obtained, and I see no need at all to posit an extra category of cause, in order to explain the existence of homosexuality. It's just how some people end up satisfying their emotional and sexual drives.
What I had not grasped, is that the idea of the gay germ is being motivated by consideration of ultimate causes - all this talk of fitness penalties and failure to reproduce. I guess I thought Cochran was a science-jock who couldn't imagine being gay, and who therefore sought to explain it as resulting from an intruding perturbation of human nature.
I am frankly not sure how seriously I should take the argument that there has to be (in gwern's words) a "mechanism... to offset the huge fitness penalty". Humanity evolves through sexual selection, right? And there are lots of losers in every generation. Apparently that's part of our evolutionary "business model". Meanwhile I've argued that non-reproducing homosexuality is a human variation that arises quite easily, given our overall cognitive ensemble. So maybe natural selection has neither a clear incentive to eliminate it, nor a clear target to aim at anyway.
My guess is that it's somehow a spandrel of intelligence.
My theory and meta-theory: The gay germ theory is pretty silly. But the big myth to which it is a reaction, is the idea that people are simply "born that way". Cochran has a paranoid intuition that something else is happening, so he posits his gay germ. But what's really happening is sexual imprinting. A person's sexuality is greatly shaped by the first conditions under which they experience arousal, orgasm, and emotional bonding. Sexualities are "transmitted" in a way a little like languages. There's no "German germ" which makes people think and speak auf deutsch, instead there's some sort of deep learning based on early experience of a German-speaking environment. The acquisition of sexuality might be more like conditioning than learning, but it's still an acquired trait.
The big new problem in presidential decision theory is Coherent Extrapolated Trump.
The character from Molière learns a fancy name ("speaking in prose") for the way he already communicates. David Gross isn't saying that he is unfamiliar with the Bayesian view, he's saying that "Bayesian confirmation theory" is a fancy name for his existing epistemic practice.
An anonymous source supplies Gwern with juicy facts about this man Wright, they see print after a few weeks, and then within hours his home is raided by Australian federal police. I am reminded that the source of the Watergate leaks was in fact the deputy director of the FBI...
How does it support MWI?
Why doesn't your list of "things I've considered" include an option like, "Nutrition science is real science, and Campbell's work is correct"?
Sorry, I know this is the start of an ambitious sequence about economics and I shouldn't spoil that; and I know that you are defining "scarcity" according to some jargon definition that is part of academic economic discourse.
But do you understand that, by the ordinary understanding of the word, "scarcity is the presence of choice" is an utter inversion of reality? In the ordinary meaning, scarcity means shortage, it means lack, it means you don't have something that you need. And that means you don't have a choice! That you don't have the luxury of choice; that you will walk because you can't afford a bus, that you will borrow because you have run out of cash.
None of the replies to this post actually address the crucial part of the argument: the step from the existence of mathematical truths beyond the reach of a specific formalization of mathematical reasoning, to the existence of a reality beyond or outside a specific physical theory of everything, on the grounds that a mathematical theory of physics must be subject to Goedelian incompleteness.