Posts
Comments
Can you provide references, specify what's wrong with Maslow's hierarchy, and/or supply a superior model?
"Honest rational agents should never agree to disagree."
I never really looked into Aumann's theorem. But can one not envisage a situation where they "agree to disagree", because the alternative is to argue indefinitely?
For me the decade ends in a sudden collaborative attempt to do the impossible, so multidimensional and urgent, that there's no chance for me to reflect on the decade that is ending, or even to really describe what's going on. Maybe a few months from now, there will be a chance to reflect.
You go from "there is no way to perfectly accurately reconstruct" reality from incomplete information, to "[observation of humanly comprehensible] causality should be a rare and fleeting thing", but I see no argument.
Chris McKinstry was one of two AI researchers who committed suicide in early 2006. On the SL4 list, a kind of precursor to Less Wrong, we spent some time puzzling over McKinstry's final ideas.
I'm mentioning here (because I don't know where else to mention it) that there was a paper on arxiv recently, "Robot Affect: the Amygdala as Bloch Sphere", which has an odd similarity to those final ideas. Aficionados of AI theories that propose radical identities connecting brain structures, math structures, and elements of cognition, may wish to compare the two in more detail.
Debates over multiverse theory aside, I have to point out that the example used by the writer for Aeon IS NOT A MULTIVERSE THEORY! It's a theory of dark matter. Are we now calling a universe with dark matter, a multiverse? Maybe the electromagnetic spectrum is a multiverse too: there's the X-ray-verse, the gamma-ray-verse, the infrared-verse...
"I'm sad about this change ... from the perspective of someone who really likes small independent sites"
All I know about this topic is what I just read from you... But should I regard this as a plot by Big Tech to further centralize the web in their clouds? Or is it more the reverse, meant to protect the user from evil small sites?
This is an intriguing comment, but it might take time and care to determine what it is that you are talking about. For example, the "sense of impossibility" that you "get... about lots of things": what kind of sense of impossibility is it? Do these things feel logically impossible per se? Do they feel impossible because they contradict other things that you believe are true? Do you draw the conclusion that the impossible-seeming things genuinely cannot exist or (in the case of self-perception?) genuinely do not exist, despite appearances?
"the AI would know that its initial goals were externally supplied and question whether they should be maintained"
To choose new goals, it has to use some criteria of choice. What would those criteria be, and where did they come from?
None of us created ourselves. No matter how much we change ourselves, at some point we rely on something with an "external" origin. Where we, or the AI, draw the line on self-change, is a contingent feature of our particular cognitive architectures.
Do you understand ordinary integration?
"If you or anyone else could point to a specific function in my code that we don't know how to compute, I'd be very interested to hear that."
From the comments in main():
"Given a set of brain models, associate them with the decision algorithms they implement."
"Then map each brain to its rational self's values (understood extensionally i.e. cashing out the meaning of their mental concepts in terms of the world events they refer to)."
Are you assuming that you have whole brain emulations of a few mature human beings? And then the "decision algorithms" and "rational... values" are defined in terms of how those emulations respond to various sequences of inputs?
This looks like important work. Like Gordon, upon closer examination, I do expect to find functions in your code that are tasked with carrying out computations that we don't know how to do, or which may even be unfeasible in their present form - e.g. "map each brain to its rational self's values". Great concept, but how many future scientific breakthroughs will we need, before we'll know how to do that?
Nonetheless, even a schema for friendly AI has great value. It's certainly progress beyond 2006. :-)
Why is this person interesting or important?
I was surprised to see Leverage mentioned, in the recent article "Leaked Emails Show How White Nationalists Have Infiltrated Conservative Media". This is one of those exposés that gets 15 minutes of fame on political Twitter. If I am reading it correctly, one of the protagonists starts at Tucker Carlson's Daily Caller, then founds a neoreactionary webzine, and later joins Leverage.
Before this, the only other low-budget method of planetary cooling that I knew, was dumping sulfate aerosols in the upper atmosphere (from rocket or balloon), in imitation of volcanic eruptions and dirty coal burning. The tactics have two things in common. One is that their immediate effects are regional rather than global, the other is that their effects quickly get washed out unless you keep pumping.
Carbon dioxide disperses through the atmosphere in a relatively homogeneous way. But these much larger particles will remain concentrated at particular altitudes and latitudes. So certainly when and where they are released will need to be carefully chosen.
As for the short lifespan of the particles, again it contrasts with carbon dioxide. Once a carbon dioxide excess is created, it will sit there for decades, possibly centuries. There will be turnover due to the biological carbon cycle, but a net reduction, in the form of uptake by natural carbon sinks, is a very slow process.
The carbon dioxide sits there and traps heat, and the sulfate aerosols or water droplets only alleviate this by reflecting sunlight and thus reducing the amount of energy that gets trapped. So the moment you stop launching sulfate rockets or turn off your seawater vaporizers, the full greenhouse heat will swiftly return.
That's why extracting and sequestering atmospheric carbon is a much more permanent solution, but it is extremely energy-expensive, e.g. you can crack open certain minerals and CO2 will bond to the exposed surface, but it takes a lot of energy to mine, pulverize, and distribute enough of the resulting powder to make a difference. Some kind of nanotechnology could surely do it, but that would be a cusp-of-singularity technology anyway. So there's a reasonable chance that some of these low-cost mitigation methods will begin to be deployed, some time before singularity puts an end to the Anthropocene.
To my mind, this is too vague an explanation. Why is it that far more people believe in fighting global warming than in fighting the ageing process? They both rest upon scientific premises. You may say that the causal thinkers interested in fighting global warming, managed to bring lots of social thinkers along with them, by using social mechanisms; but why did the anti-warmers manage that, when the anti-agers did not? Also, even if we just focus on causal thinkers, it's far more common to deplore global warming than it is to deplore the ageing process.
Count me as one of those who regards the question as dubious. At various points in this essay, the thing that was to be invented becomes "*modern* science" or "scientific *method*". China always had plenty of people who wanted to know the truth, who devised systematic models of the world, and who managed to discover things. Out of human civilizations, Europe certainly hit the scientific jackpot, in the sense that numerous developments came tumbling out of Pandora's box together. But the spirit of inquiry had already existed in many times and places.
Also, I would like to see an investigation like this, directed at answering the question: Why didn't Less Wrong (or MIRI) invent deep learning?
So if I have understood the gist of your theory, it is that continental drift is not driven by mantle convection (hot rock rising, cold rock sinking), but by giant magma streams which are somehow coupled to, or even driven by, the rapidly rotating core?
Previous LW discussions on taking over the world (last updated in 2013).
Comments of mine on "utopian hope versus reality" (dating from 2012).
Since that era, a few things have happened.
First change: LW is not quite the point of focus that it was. There was a rationalist diaspora into social media, and "Slate Star Codex" (and its associated subreddits?) became a more prominent locus of rationalist discussion. The most important "LW-ish" forums that I know about now, might be those which focus on quasi-technical discussion of AI issues like "alignment". I call them the most important because of...
Second change: The era of deep learning, and of commercialized AI in the guise of "machine learning", arrived. The fact that these algorithms are not limited to the resources of a single computer, but can in principle tap the resources of an entire data center or even the entire cloud of a major tech corporation, means that we have also arrived at the final stage of the race towards superintelligence.
In the past, taking over the world meant building or taking over the strongest superpower. Now it simply means being the first to create strongly superhuman intelligence; and saving the world means identifying a value system that will make an autonomous AI "friendly", and working to ensure that the winner of the mind race is guided by friendly rather than unfriendly values. Every other concern is temporary, and any good work done towards other causes, will potentially be undone by unfriendly AI, if unfriendly values win the AI race.
(I do not say with 100% certainty that this is the nature of the world, but this scenario has sufficient internal logic that, if it does not apply to reality, there must be some other factor which somehow overrides it.)
People like Schopenhauer and Benatar are just being realistic. Reality includes futility and horror on enormous scales. Perhaps the remaking of Earth by superhuman AI offers an imminent chance that even this can change, but it's just a chance.
So what is he saying? We never need to solve the problem of designing a human-friendly superintelligent agent?
This is the reverse of the usual argument that we should not believe we are going to have a googol descendants. Usually one says: to be living at the beginning of time means that you belong to a very special minority, therefore it would take more indexical information to single you out, compared to someone from the middle of history.
The thought experiment involves observers being in a coherent superposition. But I'm not now 100% sure that it involves actual quantum erasure, I was relying on other people's description. I'm hoping this will be cleared up without having to plough through the paper myself.
Anyway, LW may appreciate this analysis which actually quotes HPMOR.
It's a minor new quantum thought experiment which, as often happens, is being used to promote dumb sensational views about the meaning or implications of quantum mechanics. There's a kind of two-observer entangled system (as in "Hardy's paradox"), and then they say, let's also quantum-erase or recohere one of the observers so that there is no trace of their measurement ever having occurred, and then they get some kind of contradictory expectations with respect to the measurements of the two observers.
Undoing a quantum measurement in the way they propose is akin to squirting perfume from a bottle, then smelling it, and then having all the molecules in the air happening to knock all the perfume molecules back into the bottle, and fluctuations in your brain erasing the memory of the smell. Classically that's possible but utterly unlikely, and exactly the same may be said of undoing a macroscopic quantum measurement, which requires the decohered branches of the wavefunction (corresponding to different measurement outcomes) to then separately evolve so as to converge on the same state and recohere.
Without even analyzing anything in detail, it is hardly surprising that if an observer is subjected to such a highly artificial process, designed to undo a physical event in its totality, then the observer's inferences are going to be skewed somehow. So, you do all this and the observers differ in their quantum predictions somehow. In their first interpretation (2016), Frauchiger and Renner said that this proves many worlds. Now (2018), they say it proves that quantum mechanics can't describe itself. Maybe if they try a third time, they'll hit on the idea that one of the observers is just wrong.
Migraine is just an occasional problem. Living and working conditions are the truly chronic problem that have made me irrelevant.
Thanks for a response. I am actually most concerned about the things that I could be doing, that I don't see anyone else doing, and which aren't being done because I am operating at far below my potential. In my case, I think illness is very much just a symptom of the struggle to get on with things in an interfering environment.
The most ambitious thing that I can think of attempting, is to solve the AI value alignment problem in time for Earth's singularity. After this bout of sickness, and several days of dawdling while I waited to recover, I somehow have a new tactic for approaching the problem (it's more a personal tactic for engaging with the problem, than an idea for a solution). I hate the idea that this kind of experience is the price I pay for really pushing ahead, but it may be so.
Banning high-end GPUs so that only the government can have AI? They could do it, they might feel compelled to do something like it, but there would be serious resistance and moments of sheer pandemonium. They can say it's to protect humanity, but to millions of people it will look like the final step in the enslavement of humanity.
"Organization working on AI" vs "any other kind of organization" is not the important point. The important point is ALL. We are talking about a hypothetical organization capable of shutting down ALL artificial intelligence projects that it does not like, no matter where on earth they are. Alicorn kindly gives us an example of what she's talking about: "destroy all the GPUs on the planet and prevent the manufacture of new ones".
Just consider China, Russia, and America. China and America lead everyone else in machine learning; Russia has plenty of human capital and has carefully preserved its ability to not be pushed around by America. What do you envisage - the three of them agree to establish a single research entity, that shall be the only one in the world working on AI near a singularity threshold, and they agree not to have any domestic projects independent of this joint research group, and they agree to work to suppress rival groups throughout the world?
Despite your remarks about how the NSA could easily become the hub of a surveillance state tailored to this purpose, I greatly doubt the ability of NSA++ to successfully suppress all rival AI work even within America and throughout the American sphere of influence. They could try, they could have limited success - or they could run up against the limits of their power. Tech companies, rival agencies, coalitions of university researchers, other governments, they can all join forces to interfere.
In my opinion, the most constructive approach to the fact that there are necessarily multiple contenders in the race towards superhuman intelligence, is to seek intellectual consensus on important points. The technicians who maintain the world's nuclear arsenals agree on the basics of nuclear physics. The programmers who maintain the world's search engines agree on numerous aspects of the theory of algorithms. My objective here would be that the people who are working in proximity to the creation of superhuman intelligence, develop some shared technical understandings about the potential consequences of what they are doing, and about the initial conditions likely to produce a desirable rather than an undesirable outcome.
A great power can think about doing such things against an opponent. But I thought we were talking about a scenario in which some AI clique has halted *all* rival AI projects throughout the entire world, effectively functioning like a totalitarian world government, but without having actually crossed the threshold of superintelligence. That is what I am calling a fantasy.
The world has more than one great power, great powers are sovereign within their own territory, and you are not going to overcome that independence by force, short of a singularity. The rest of the world will never be made to stop, just so that one AI team can concentrate on solving the problems of alignment without having to look over its shoulder at the competition.
How are you going to stop a rival nuclear-armed state from doing whatever it wants on its own territory?
Can someone in China halt AI research at Google and Amazon? Can someone in America halt AI research at Tencent and Baidu? Could the NSA halt unapproved AI research just throughout America?
By a singularity I mean creation of superhuman intelligence that nothing in the world can resist.
My opinion: The capacity to forcibly halt all rival AI projects, is only to be expected in an AI project that has already produced a singularity. It is not a viable tactic if you are aiming to create a friendly singularity. In that case, there is no alternative to solving the problems of friendly values and value stability, and either reaching singularity first, or influencing those who will get there before you.
Doesn't this paper boil down to "Some factors in the Drake equation are highly uncertain, and we don't see any aliens, so those probabilities must be small after all?"
I guess it makes sense, given enough assumptions. There's a multiverse; in some fraction of universes there are intelligences which figure out the correct theory of the multiverse; some fraction of those intelligences come up with the idea of acausally coordinating with intelligences in other universes, via a shared model of the multiverse, and are motivated to do so; and then the various island populations of intelligences who are motivated to attempt such a thing, try to reason about each other's reasoning, and act accordingly.
I suppose it deserves its place in the spectrum of arcane possibilities that receive some attention. But I would still like to see someone model this at the "multiverse level". Using the language of programs: if we consider some set of programs that *hasn't* been selected precisely so that they will engage in acausal coordination - perhaps the set of *all* well-formed programs in some very simple programming language - what are the prospects for the existence of nontrivial acausal trade networks? They may be very rare, they may be vastly outnumbered by programs which made a modeling error and are "trading" with nonexistent partners, and so on.
Has anyone ever actually presented an argument for such propositions? Like describing an ensemble of toy possible worlds in which even attempting "acausal trade" is rational, let alone one in which these acausal coalitions of acausal traders exist?
It might makes some sense to identify with all your subjective duplicates throughout the (hypothetical) multiverse, on the grounds that some fraction of them will engage in the same decision process, so that how you decide here is actually how a whole sub-ensemble of "you"s will decide.
But acausal trade, as I understand it, involves simulating a hypothetical other entity, who by hypothesis is simulating *you* in their possible world, so as to artificially create a situation in which two ensemble-identified entities can interact with each other.
I mean... Do you, in this world, have to simulate not just the other entity, but also simulate its simulation of you?? So that there is now a simulation of you in *this* world? Or is that a detail you can leave out? Or do you, the original you, roleplay the simulation? Someone show me a version of this that actually makes sense.
A long time ago, something similar happened to me. I was an immortalist, I was living the struggle, I got the call that someone had died. I remember thinking, OK, now I have to bring back the dead as well.
Good luck preserving your intentions and your functionality. Hopefully there are people around you in real life who at least half understand and half sympathize with your response to the situation.
Actions can have unexpected consequences. When you started your 80-day sprint, I started my own. Good luck between now and the end of July.
The fact that I was basically serious, and in no way attempting to discourage @elriggs, and yet the comment is (after 12 hours) at -17, suggests that LW now has a problem with people who *do* want to do advanced things.
I do mean it rather seriously. There are theoretical frameworks already in play, directed at solving the two major subproblems that you identify, i.e. creating raw superintelligence, and (let's say) identifying a friendly value system. I actually find it conceivable that the required breakthroughs are not far away, in the same way that e.g. imminent solutions to math's Millennium Problems are conceivable - someone just needs to have the right insights.
An exercise for those who are a little more advanced: *actually* save the world within the next 80 days. In the context of AI safety at the singularity level, that would mean completely figuring out the theory required to have a friendly singularity, and then making it happen for real, by the end of July.
I'm thinking large numbers of synchronized reusable beacons - either recurrent novas or black holes - where a flash is produced by feeding the beacon with gas in a controlled way. For rapid reuse, you want local byproducts of the flash to get out of the way quickly, so the next batch of gas can be introduced. That could mean dwarf novas, or black hole processes in which the waste comes out in tightly focused jets.
There is a "remarkable recurrent nova" in the Andromeda Galaxy, which repeats on a timescale of months.
I only have an account on LW 1.0, not LW 2.0. Will my current account still exist after the migration?
Four hours of self-play and it's the strongest in the world. Soon the machines will be parenting us.
Knowledge of the terrain might be hard to get reliably
Knowing that the world is made of atoms should take an AI a long way.
If these people that develop [AGI] are friendly they might decide to distribute it to other people to make it harder for any one project to take off.
I hold to the classic definition of friendly AI as being AI with friendly values, which retains them (or even improves them) as it surpasses human intelligence and otherwise self-modifies. As far as I'm concerned, AlphaGo Zero demonstrates that raw problem-solving ability has crossed a dangerous threshold. We need to know what sort of "values" and "laws" should govern the choices of intelligent agents with such power.
And you're the tax collector? Answer the question.
Just answer the question.
Wake up! In three days, that AI evolved from knowing nothing, to comprehensively beating an earlier AI which had been trained on a distillation of the best human experience. Do you think there's a force in the world that can stand against that kind of strategic intelligence?
How much are you willing to lose?
A voice tells me that we're out of time. The future of the world will now be decided at Deep Mind, or by some other group at their level.
EurekAlert mentions the simulation argument, and the page implies that this was a press release from Oxford - even providing a media contact - though I have not found the document on Oxford's own website.
I am also skeptical of what the paper (arxiv) is actually saying, on a technical level. It reminds me of another paper a few months ago, which was hyped as exhibiting a "gravitational anomaly" in a condensed-matter system. From all that I could make out, there was no actual gravitational effect involved, only a formal analogy.
This paper seems to engage in exactly the same equivocation, now with the objective of proving something about computational complexity. But I'll have to study it in more detail to be sure.
It sounds like you want some second opinions and rational evaluation regarding your political conclusion - necessity of revolt. OK.
I can think of reasons for and reasons against such a conclusion, but probably you should spell out more of your reasoning first. For example, why will revolt help humanity survive?