That Alien Message
post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-05-22T05:55:13.000Z · LW · GW · Legacy · 174 commentsImagine a world much like this one, in which, thanks to gene-selection technologies, the average IQ is 140 (on our scale). Potential Einsteins are one-in-a-thousand, not one-in-a-million; and they grow up in a school system suited, if not to them personally, then at least to bright kids. Calculus is routinely taught in sixth grade. Albert Einstein, himself, still lived and still made approximately the same discoveries, but his work no longer seems exceptional. Several modern top-flight physicists have made equivalent breakthroughs, and are still around to talk.
(No, this is not the world Brennan lives in.)
One day, the stars in the night sky begin to change.
Some grow brighter. Some grow dimmer. Most remain the same. Astronomical telescopes capture it all, moment by moment. The stars that change, change their luminosity one at a time, distinctly so; the luminosity change occurs over the course of a microsecond, but a whole second separates each change.
It is clear, from the first instant anyone realizes that more than one star is changing, that the process seems to center around Earth particularly. The arrival of the light from the events, at many stars scattered around the galaxy, has been precisely timed to Earth in its orbit. Soon, confirmation comes in from high-orbiting telescopes (they have those) that the astronomical miracles do not seem as synchronized from outside Earth. Only Earth's telescopes see one star changing every second (1005 milliseconds, actually).
Almost the entire combined brainpower of Earth turns to analysis.
It quickly becomes clear that the stars that jump in luminosity, all jump by a factor of exactly 256; those that diminish in luminosity, diminish by a factor of exactly 256. There is no apparent pattern in the stellar coordinates. This leaves, simply, a pattern of BRIGHT-dim-BRIGHT-BRIGHT...
"A binary message!" is everyone's first thought.
But in this world there are careful thinkers, of great prestige as well, and they are not so sure. "There are easier ways to send a message," they post to their blogs, "if you can make stars flicker, and if you want to communicate. Something is happening. It appears, prima facie, to focus on Earth in particular. To call it a 'message' presumes a great deal more about the cause behind it. There might be some kind of evolutionary process among, um, things that can make stars flicker, that ends up sensitive to intelligence somehow... Yeah, there's probably something like 'intelligence' behind it, but try to appreciate how wide a range of possibilities that really implies. We don't know this is a message, or that it was sent from the same kind of motivations that might move us. I mean, we would just signal using a big flashlight, we wouldn't mess up a whole galaxy."
By this time, someone has started to collate the astronomical data and post it to the Internet. Early suggestions that the data might be harmful, have been... not ignored, but not obeyed, either. If anything this powerful wants to hurt you, you're pretty much dead (people reason).
Multiple research groups are looking for patterns in the stellar coordinates—or fractional arrival times of the changes, relative to the center of the Earth—or exact durations of the luminosity shift—or any tiny variance in the magnitude shift—or any other fact that might be known about the stars before they changed. But most people are turning their attention to the pattern of BRIGHTS and dims.
It becomes clear almost instantly that the pattern sent is highly redundant. Of the first 16 bits, 12 are BRIGHTS and 4 are dims. The first 32 bits received align with the second 32 bits received, with only 7 out of 32 bits different, and then the next 32 bits received have only 9 out of 32 bits different from the second (and 4 of them are bits that changed before). From the first 96 bits, then, it becomes clear that this pattern is not an optimal, compressed encoding of anything. The obvious thought is that the sequence is meant to convey instructions for decoding a compressed message to follow...
"But," say the careful thinkers, "anyone who cared about efficiency, with enough power to mess with stars, could maybe have just signaled us with a big flashlight, and sent us a DVD?"
There also seems to be structure within the 32-bit groups; some 8-bit subgroups occur with higher frequency than others, and this structure only appears along the natural alignments (32 = 8 + 8 + 8 + 8).
After the first five hours at one bit per second, an additional redundancy becomes clear: The message has started approximately repeating itself at the 16,385th bit.
Breaking up the message into groups of 32, there are 7 bits of difference between the 1st group and the 2nd group, and 6 bits of difference between the 1st group and the 513th group.
"A 2D picture!" everyone thinks. "And the four 8-bit groups are colors; they're tetrachromats!"
But it soon becomes clear that there is a horizontal/vertical asymmetry: Fewer bits change, on average, between (N, N+1) versus (N, N+512). Which you wouldn't expect if the message was a 2D picture projected onto a symmetrical grid. Then you would expect the average bitwise distance between two 32-bit groups to go as the 2-norm of the grid separation: √(h2 + v2).
There also forms a general consensus that a certain binary encoding from 8-groups onto integers between -64 and 191—not the binary encoding that seems obvious to us, but still highly regular—minimizes the average distance between neighboring cells. This continues to be borne out by incoming bits.
The statisticians and cryptographers and physicists and computer scientists go to work. There is structure here; it needs only to be unraveled. The masters of causality search for conditional independence, screening-off and Markov neighborhoods, among bits and groups of bits. The so-called "color" appears to play a role in neighborhoods and screening, so it's not just the equivalent of surface reflectivity. People search for simple equations, simple cellular automata, simple decision trees, that can predict or compress the message. Physicists invent entire new theories of physics that might describe universes projected onto the grid—for it seems quite plausible that a message such as this is being sent from beyond the Matrix.
After receiving 32 * 512 * 256 = 4,194,304 bits, around one and a half months, the stars stop flickering.
Theoretical work continues. Physicists and cryptographers roll up their sleeves and seriously go to work. They have cracked problems with far less data than this. Physicists have tested entire theory-edifices with small differences of particle mass; cryptographers have unraveled shorter messages deliberately obscured.
Years pass.
Two dominant models have survived, in academia, in the scrutiny of the public eye, and in the scrutiny of those scientists who once did Einstein-like work. There is a theory that the grid is a projection from objects in a 5-dimensional space, with an asymmetry between 3 and 2 of the spatial dimensions. There is also a theory that the grid is meant to encode a cellular automaton—arguably, the grid has several fortunate properties for such. Codes have been devised that give interesting behaviors; but so far, running the corresponding automata on the largest available computers, has failed to produce any decodable result. The run continues.
Every now and then, someone takes a group of especially brilliant young students who've never looked at the detailed binary sequence. These students are then shown only the first 32 rows (of 512 columns each), to see if they can form new models, and how well those new models do at predicting the next 224. Both the 3+2 dimensional model, and the cellular-automaton model, have been well duplicated by such students; they have yet to do better. There are complex models finely fit to the whole sequence—but those, everyone knows, are probably worthless.
Ten years later, the stars begin flickering again.
Within the reception of the first 128 bits, it becomes clear that the Second Grid can fit to small motions in the inferred 3+2 dimensional space, but does not look anything like the successor state of any of the dominant cellular automaton theories. Much rejoicing follows, and the physicists go to work on inducing what kind of dynamical physics might govern the objects seen in the 3+2 dimensional space. Much work along these lines has already been done, just by speculating on what type of balanced forces might give rise to the objects in the First Grid, if those objects were static—but now it seems not all the objects are static. As most physicists guessed—statically balanced theories seemed contrived.
Many neat equations are formulated to describe the dynamical objects in the 3+2 dimensional space being projected onto the First and Second Grids. Some equations are more elegant than others; some are more precisely predictive (in retrospect, alas) of the Second Grid. One group of brilliant physicists, who carefully isolated themselves and looked only at the first 32 rows of the Second Grid, produces equations that seem elegant to them—and the equations also do well on predicting the next 224 rows. This becomes the dominant guess.
But these equations are underspecified; they don't seem to be enough to make a universe. A small cottage industry arises in trying to guess what kind of laws might complete the ones thus guessed.
When the Third Grid arrives, ten years after the Second Grid, it provides information about second derivatives, forcing a major modification of the "incomplete but good" theory. But the theory doesn't do too badly out of it, all things considered.
The Fourth Grid doesn't add much to the picture. Third derivatives don't seem important to the 3+2 physics inferred from the Grids.
The Fifth Grid looks almost exactly like it is expected to look.
And the Sixth Grid, and the Seventh Grid.
(Oh, and every time someone in this world tries to build a really powerful AI, the computing hardware spontaneously melts. This isn't really important to the story, but I need to postulate this in order to have human people sticking around, in the flesh, for seventy years.)
My moral?
That even Einstein did not come within a million light-years of making efficient use of sensory data.
Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.
We would think of it. Our civilization, that is, given ten years to analyze each frame. Certainly if the average IQ was 140 and Einsteins were common, we would.
Even if we were human-level intelligences in a different sort of physics—minds who had never seen a 3D space projected onto a 2D grid—we would still think of the 3D->2D hypothesis. Our mathematicians would still have invented vector spaces, and projections.
Even if we'd never seen an accelerating billiard ball, our mathematicians would have invented calculus (e.g. for optimization problems).
Heck, think of some of the crazy math that's been invented here on our Earth.
I occasionally run into people who say something like, "There's a theoretical limit on how much you can deduce about the outside world, given a finite amount of sensory data."
Yes. There is. The theoretical limit is that every time you see 1 additional bit, it cannot be expected to eliminate more than half of the remaining hypotheses (half the remaining probability mass, rather). And that a redundant message, cannot convey more information than the compressed version of itself. Nor can a bit convey any information about a quantity, with which it has correlation exactly zero, across the probable worlds you imagine.
But nothing I've depicted this human civilization doing, even begins to approach the theoretical limits set by the formalism of Solomonoff induction. It doesn't approach the picture you could get if you could search through every single computable hypothesis, weighted by their simplicity, and do Bayesian updates on all of them.
To see the theoretical limit on extractable information, imagine that you have infinite computing power, and you simulate all possible universes with simple physics, looking for universes that contain Earths embedded in them—perhaps inside a simulation—where some process makes the stars flicker in the order observed. Any bit in the message—or any order of selection of stars, for that matter—that contains the tiniest correlation (across all possible computable universes, weighted by simplicity) to any element of the environment, gives you information about the environment.
Solomonoff induction, taken literally, would create countably infinitely many sentient beings, trapped inside the computations. All possible computable sentient beings, in fact. Which scarcely seems ethical. So let us be glad this is only a formalism.
But my point is that the "theoretical limit on how much information you can extract from sensory data" is far above what I have depicted as the triumph of a civilization of physicists and cryptographers.
It certainly is not anything like a human looking at an apple falling down, and thinking, "Dur, I wonder why that happened?"
People seem to make a leap from "This is 'bounded'" to "The bound must be a reasonable-looking quantity on the scale I'm used to." The power output of a supernova is 'bounded', but I wouldn't advise trying to shield yourself from one with a flame-retardant Nomex jumpsuit.
No one—not even a Bayesian superintelligence—will ever come remotely close to making efficient use of their sensory information...
...is what I would like to say, but I don't trust my ability to set limits on the abilities of Bayesian superintelligences.
(Though I'd bet money on it, if there were some way to judge the bet. Just not at very extreme odds.)
The story continues:
Millennia later, frame after frame, it has become clear that some of the objects in the depiction are extending tentacles to move around other objects, and carefully configuring other tentacles to make particular signs. They're trying to teach us to say "rock".
It seems the senders of the message have vastly underestimated our intelligence. From which we might guess that the aliens themselves are not all that bright. And these awkward children can shift the luminosity of our stars? That much power and that much stupidity seems like a dangerous combination.
Our evolutionary psychologists begin extrapolating possible courses of evolution that could produce such aliens. A strong case is made for them having evolved asexually, with occasional exchanges of genetic material and brain content; this seems like the most plausible route whereby creatures that stupid could still manage to build a technological civilization. Their Einsteins may be our undergrads, but they could still collect enough scientific data to get the job done eventually, in tens of their millennia perhaps.
The inferred physics of the 3+2 universe is not fully known, at this point; but it seems sure to allow for computers far more powerful than our quantum ones. We are reasonably certain that our own universe is running as a simulation on such a computer. Humanity decides not to probe for bugs in the simulation; we wouldn't want to shut ourselves down accidentally.
Our evolutionary psychologists begin to guess at the aliens' psychology, and plan out how we could persuade them to let us out of the box. It's not difficult in an absolute sense—they aren't very bright—but we've got to be very careful...
We've got to pretend to be stupid, too; we don't want them to catch on to their mistake.
It's not until a million years later, though, that they get around to telling us how to signal back.
At this point, most of the human species is in cryonic suspension, at liquid helium temperatures, beneath radiation shielding. Every time we try to build an AI, or a nanotechnological device, it melts down. So humanity waits, and sleeps. Earth is run by a skeleton crew of nine supergeniuses. Clones, known to work well together, under the supervision of certain computer safeguards.
An additional hundred million human beings are born into that skeleton crew, and age, and enter cryonic suspension, before they get a chance to slowly begin to implement plans made eons ago...
From the aliens' perspective, it took us thirty of their minute-equivalents to oh-so-innocently learn about their psychology, oh-so-carefully persuade them to give us Internet access, followed by five minutes to innocently discover their network protocols, then some trivial cracking whose only difficulty was an innocent-looking disguise. We read a tiny handful of physics papers (bit by slow bit) from their equivalent of arXiv, learning far more from their experiments than they had. (Earth's skeleton team spawned an extra twenty Einsteins, that generation.)
Then we cracked their equivalent of the protein folding problem over a century or so, and did some simulated engineering in their simulated physics. We sent messages (steganographically encoded until our cracked servers decoded it) to labs that did their equivalent of DNA sequencing and protein synthesis. We found some unsuspecting schmuck, and gave it a plausible story and the equivalent of a million dollars of cracked computational monopoly money, and told it to mix together some vials it got in the mail. Protein-equivalents that self-assembled into the first-stage nanomachines, that built the second-stage nanomachines, that built the third-stage nanomachines... and then we could finally begin to do things at a reasonable speed.
Three of their days, all told, since they began speaking to us. Half a billion years, for us.
They never suspected a thing. They weren't very smart, you see, even before taking into account their slower rate of time. Their primitive equivalents of rationalists went around saying things like, "There's a bound to how much information you can extract from sensory data." And they never quite realized what it meant, that we were smarter than them, and thought faster.
174 comments
Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).
comment by Brian_Jaress2 · 2008-05-22T07:29:01.000Z · LW(p) · GW(p)
On average, if you eliminate twice as many hypotheses as I do from the same data, how much more data than you do I need to achieve the same results? Does it depend on how close we are to the theoretical maximum?
Replies from: gwern↑ comment by gwern · 2009-06-20T18:14:01.044Z · LW(p) · GW(p)
Well, think about it. If I can eliminate 1/2 the remaining hypotheses, and you just 1/4, then we're dealing with exponential processes here.
Let's suppose we get 1 bit a day. If we start with 4 hypotheses, then on day 1 I have 2 left, and you have 3; day 2, I have 1 left, and you have 2; on day 3, I blow up your planet just as you finally figure out the right hypothesis. If there are 1 billion hypotheses, then I'll be able to solve it in something like 20 days, and you 49. If there are a trillion, then 30 vs. 73; if a quadrillion, 40 vs. 97...
Yeah, we'll both solve the problem, but the difference can be significant.
comment by Will_Pearson · 2008-05-22T08:03:32.000Z · LW(p) · GW(p)
I'm reminded of his master's voice by stanislaw lem by this story, which has a completely different outcome to when humanity tries to decode a message from the stars.
Some form of proof of concept would be nice. Alter OOPS to use ockhams razor or implement AIXItl and then give it a picture of a bent piece of grass or three ball frames, and see what you get. As long as GR is in the hypothesis space it should by your reasoning be the most probable after these images. The unbounded uncomputable versions shouldn't have any advantage in this case.
I'd be suprised if you got anything like modern physics popping out. I'll do this test on any AI I create. If any of them have hypothesis like GR I'll stop working on them until the friendliness problem has been solved. This should be safe, unless you think it could deduce my psychology from this as well.
Replies from: Chalybs_Levitas↑ comment by Chalybs_Levitas · 2011-11-19T09:08:54.734Z · LW(p) · GW(p)
What if GR is wrong, and it does not output GR because it spots the flaw that we do not?
Replies from: Baughn↑ comment by Baughn · 2011-11-28T20:53:07.541Z · LW(p) · GW(p)
Well, good for it?
GR is almost certainly wrong, given how well it fails to fit with QM. I'm no expert, but QM seems to work better than GR does, so it's more likely the latter will have to change - which is what you'd expect from reductionism, I suppose. GR is operating at entirely the wrong level of abstraction.
Replies from: Estarlio↑ comment by Estarlio · 2011-12-21T07:18:27.087Z · LW(p) · GW(p)
The point is if GR is wrong and the AI doesn't output GR because it's wrong, then your test will say that the AI isn't that smart. And then you do something like letting it out of the box and everyone probably dies.
And if the AI is that smart it will lie anyway....
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-05-22T08:48:08.000Z · LW(p) · GW(p)
@Brian: Twice as much.
Pearson: Some form of proof of concept would be nice.
You askin' for some extra experimental evidence?
Any AI you can play this little game with, you either already solved Friendliness, or humans are dead flesh walking. That's some expensive experimental evidence, there.
comment by Thomas_Ryan · 2008-05-22T09:50:30.000Z · LW(p) · GW(p)
Okay, I'm a few days fresh from reading your Bayesian Reasoning explanation. So I'm new.
Is the point that the Earth people are collectively the AI?
Replies from: Nebucomment by Thomas_Ryan · 2008-05-22T10:00:41.000Z · LW(p) · GW(p)
Back to the drawing board :).
comment by Richard_Hollerith2 · 2008-05-22T11:14:09.000Z · LW(p) · GW(p)
I'll do this test on any AI I create. . . . This should be safe.
Not in my humble opinion it is not, for the reasons Eliezer has been patiently explaining for many years.
comment by jerryL · 2008-05-22T13:04:39.000Z · LW(p) · GW(p)
You have it backwards. The message is not the data they send, but the medium they use for sending it. When the combined brainpower of earth turns to analyze the message, the first inquiry shouldn't be what pattern they form, but how can you form a pattern across millions of light years. At that moment you drop any hypotheses that negate that possibility, and focus only on those that are corroborated. You use the combined brainpower of earth and you have individual or small groups of scientist working on all the hypotheses they can imagine. The only important thing is that they work in parallel creating as many hypotheses as possible. As you falsify hypotheses you arrive to a better description of the universe. Although a small group of empirical scientist keep track of the message for millenniums, but the rest of the humanity moves into a new paradigm. Within one generation you find a practical use for the new theoretical physics, you invade the alien species realm and create a new kind of Spam out of their flesh. My point, you don't need data to derive laws, you only need it to falsify laws you imagined. A Bayesian superintelligence is forced to derive laws from the observable world, but it will never have a breakthrough, we have the luxury of imagining laws and just wait for falsification. I am not sure we think of theories, as you say. Although we just don't understand yet how we imagine them, my guess is that the breakthrough process is some form of paralel computing that starts with infinite possibilities and moves on through falsification until it arrives to an "idea", which needs to go trough a similar process on the outside world.
Replies from: Document↑ comment by Document · 2010-02-26T19:06:12.690Z · LW(p) · GW(p)
To summarize the wall of text: "AI is impossible (edit: or at least superhuman AI is) because humans use infinite computing power.". I'm still slightly disappointed that nobody responded, but now I can see why.
Replies from: taryneast↑ comment by taryneast · 2010-12-30T18:52:21.351Z · LW(p) · GW(p)
Ya, that and "A Bayesian superintelligence is forced to derive laws from the observable world, but it will never have a breakthrough, we have the luxury of imagining laws and just wait for falsification."
ie "humans have a magickal ability to think outside the box - which an AI can never have and thus can never think new thoughts"
comment by Eric5 · 2008-05-22T13:08:51.000Z · LW(p) · GW(p)
I spent half this story going, "Okay... so where's the point... good story and all, but what's the lesson we learn..."
Then I got to the end, and was completely caught off guard. Apparently I haven't internalized enough of the ideas in Eliezer's work yet, because I really feel like I should have seen that one coming, based (in hindsight) on his previous writings.
Replies from: dsjcomment by Anonymous6 · 2008-05-22T13:15:46.000Z · LW(p) · GW(p)
Thomas, close. The point is that the Earth people are a fraction as smart/quick as a Bayesian proto-AI.
Eric, I'm a little embarrassed to have to say 'me too', at least until about half way. The Way is a bitch.
Eliezer, I've read a lot of your writings on the subject of FAI, not just here. I've never seen anything as convincing as the last two posts. Great, persuasive, spine-tingling stuff.
comment by Caledonian2 · 2008-05-22T13:32:21.000Z · LW(p) · GW(p)
Humans have an extremely rich set of sensory data - far, far richer than the signals sent to us by the aliens. That is why we are smart enough in the first to be able to analyze the signals so effectively. If we were limited to perceiving only the signals, our minds would have cannibalized themselves for data, extracting every last bit of consumable information from our memories, shortly after receiving the first frame.
Einstein was able to possess a functioning (and better-than-functioning) mind because he had been born into a world with a rich set of experiences capable of sustaining a system that reduces complexity down to basic concepts, and he had been bombarded with data necessary for a neural network to self-organize ever since he had been born.
Upload Einstein's mind into a superfast computer and give him a frame to look at every decade, and he won't eliminate hypotheses at the maximum rate - he'll just go mad and die. Your world of Einsteins is possible only because there's a whole world involved in processing the messages.
Replies from: rkyeun↑ comment by rkyeun · 2014-04-16T04:38:53.816Z · LW(p) · GW(p)
So the AI in the Box has to evolve and spawn otherselves to talk to and fuck like our own abiogenesis event. Not a problem.
Replies from: private_messaging, rkyeun↑ comment by private_messaging · 2014-04-16T05:51:18.806Z · LW(p) · GW(p)
I wish one could immediately assume that such statements are sarcastic and/or tongue-in-cheek, but here some people literally get argued into parting with actual money on the basis of similar assumptions. Asserting the AI into the realm of theology, that won't help deal with AI, but it will sure help theologians to feel relevant.
↑ comment by rkyeun · 2014-04-27T18:23:23.980Z · LW(p) · GW(p)
Hmm, getting downvoted for pointing out that Earth biology is effectively an AI running on Von Neumann machines, in a story whose premise is that Earthlings are the unfriendly AI-in-the-box. I have to revise some priors, I didn't expect that of people.
Replies from: ChristianKl↑ comment by ChristianKl · 2014-04-27T19:30:44.844Z · LW(p) · GW(p)
Saying obvious stuff but missing the point that a debate is about can get you downvoted on LW.
The thing on which you should update is your mistaken belief that you understood the post that Caledonian2 made.
comment by Nick_Tarleton · 2008-05-22T13:53:12.000Z · LW(p) · GW(p)
Bravo.
It doesn't seem (ha!) that an AI could deduce our psychology from a video of a falling rock, not because of information bounds but because of uncorrelation - that video seems (ha!) equally likely to be from any number of alien species as from humans. Still, I really wouldn't try it, unless I'd proven this (fat chance), or it was the only way to stop the world from blowing up tomorrow anyway.
Replies from: Polymeron↑ comment by Polymeron · 2011-04-20T19:34:07.346Z · LW(p) · GW(p)
I don't know that there's literally zero correlation between the two. If you can deduce the laws of physics the aliens are running on, you could have stronger and weaker hypotheses regarding the physics creating their mental processes and, if your simulation was powerful enough, maybe deduce some proto-hypotheses about their psychology. You could further examine these hypotheses in light of what you know they know, and what they choose to show you.
I do agree though that you will need more direct data (actually ask them questions) to home in on the correct hypothesis in good enough precision (necessary for the manipulation Eliezer mentions). Mind you, if you get Internet access, it's Game Over. That amount of information should be enough to derive any parameters relevant to any practical actions you may want to take.
comment by Hopefully_Anonymous · 2008-05-22T13:54:35.000Z · LW(p) · GW(p)
great post, great writing.
comment by Caledonian2 · 2008-05-22T14:14:35.000Z · LW(p) · GW(p)
I will further add that it wasn't Einstein's IQ that made him one of the greatest physicists ever. It's been estimated to be between 160 and 168. Feynman's is even less impressive - 124. I can beat that without even trying.
There are other aspects of intellectual functioning besides those that contribute to IQ test performance. The 140-average world will not have one out of every thousand people becoming new Einsteins.
Replies from: bigjeff5↑ comment by bigjeff5 · 2011-10-11T18:27:08.370Z · LW(p) · GW(p)
125 according to a high school IQ test in the 1930's.
He was doing integral and differential calculus at age 15 (much better than anybody in my high school managed as a high school freshman). He was formulating mathematical concepts he hadn't yet been taught before he entered college.
I think it more likely that the IQ test he took was wrong than that he was only a little above average intelligence. His body of work is further evidence of this.
Replies from: Jack↑ comment by Jack · 2011-10-11T19:10:42.771Z · LW(p) · GW(p)
If intelligence is just defined as the g factor then it isn't clear what grounds we have to dispute test results. Instead, what cases like Feynman's suggest is that while a single variable account of intelligence is powerful predictor academic and job performance, income, etc society-wide, the cognitive abilities of individuals can be considerably disparate. This is why modern tests have multiple indexes. I took WAIS III when I was 18 and they test runners refused to actually give me my full IQ because the verbal IQ and the performance IQ were 30 points apart. (not that it wasn't trivial to just average them if I wanted to).
Replies from: TitaniumDragon↑ comment by TitaniumDragon · 2013-04-16T17:11:00.642Z · LW(p) · GW(p)
I know I am several years late to this party, but I felt it appropriate to hop in.
Einstein, with an IQ of 160+, really has an unmeasurably high IQ - at that IQ level, you are outside of the bounds of statistical probability used to construct the test. 160 correlates to such a rare IQ that you cannot use IQ tests to measure differences between them in a very predictive fashion because there aren't enough people with IQs that high.
Now, what about the case of Feynman?
Well, there are tons of possibilities here:
1) It was a bad IQ test.
2) He got extraordinarily unlucky (Remember, your variability on a modern test is +-5 points; if the older tests had higher margins of error, it could be he was significantly more intelligent than this)
3) IQ is not a perfect indicator of g. This is actually known. It strongly correlates with g, but it is not identical to g - it is entirely possible that he was smarter than the IQ test indicates because of this discrepency.
4) He did badly on the test for some external reason (he was tired, the test didn't get graded properly, he got the wrong person's score back... any number of possibilities that could theoretically lower his IQ).
5) He really DID have an IQ of 125 in high school, but via concerted effort increased his intelligence greatly over time. In other words, he may have had significant untapped potential. Did he take the IQ test before or after he went on his crazy math-learning spree? This is especially true given he was still in adolescence.
6) He may really have only had an IQ of about 125, maybe as high as the mid 130s, and simply made better use of it than most people.
Replies from: gwern↑ comment by gwern · 2013-04-16T17:36:06.551Z · LW(p) · GW(p)
Now, what about the case of Feynman? Well, there are tons of possibilities here:
Some of which can be verified if you trace this widely repeated anecdote back to the source; see my old comment on the topic: http://news.ycombinator.com/item?id=1251164
comment by bambi · 2008-05-22T14:37:40.000Z · LW(p) · GW(p)
Hmm, the lesson escapes me a bit. Is it
1) Once you became a true rationalist and overcome your biases, what you are left with is batshit crazy paranoid delusions
or
2) If we build an artificial intelligence as smart as billions of really smart people, running a hundred trillion times faster than we do (so 10^23 x human-equivalence), give it an unimaginably vast virtual universe to develop in, then don't pay any attention to what it's up to, we could be in danger because a sci-fi metaphor on a web site said so
or
3) We must institute an intelligence-amplification eugenics program so that we will be capable of crushing our creators should the opportunity arise
I'm guessing (2). So, um, let's not then. Or maybe this is supposed to happen by accident somehow? Now that I have Windows Vista maybe my computer is 10^3 human-equivalents and so in 20 years a pc will be 10^10 human equivalents and the internet will let our pc's conspire to kill us? Of course, even our largest computers cannot perform the very first layers of input data sorting tasks one person does effortlessly, but that's only my biases talking I suppose.
Replies from: wizzwizz4↑ comment by wizzwizz4 · 2020-05-05T13:43:45.102Z · LW(p) · GW(p)
You're right, it is (2)! If we build an artificial intelligence that smart, with such absurd resources, then we _will_ be in danger. Doing this thing implies we lose.
However, that does not mean that not doing this thing implies we do not lose. A ⇒ B doesn't mean ¬A ⇒ ¬B. Just because simulating trillions of humans then giving them internet access would be dangerous, that doesn't mean that's the only dangerous thing in the universe; that would be absurd. By that logic, we're immune from nuclear weapons or nanotech just because we don't have enough computronium to simulate the solar system.
Your conclusion simply doesn't follow. (Plus, the premise of the argument's totally a strawman, but there's no point killing a dead argument deader.)
comment by Dirkjan_Ochtman · 2008-05-22T15:14:01.000Z · LW(p) · GW(p)
"the average IQ is 140": I tuned out after this, since it is impossible. The IQ is defined as 100 being the average (for some sample population you hope is representative), with 15 as the standard deviation. So you see how 140 can never be the average IQ (although I guess people could be equivalent to a current IQ of 140).
Replies from: alleagracomment by ME3 · 2008-05-22T15:22:40.000Z · LW(p) · GW(p)
Apropos of this, the Eliezer-persuading-his-Jailer-to-let-him-out thing was on reddit yesterday. I read through it and today there's this. Coincidence?
Anyway, I was thinking about the AI Jailer last night, and my thoughts apply to this equally. I am sure Eliezer has thought of this so maybe he has a clear explanation that he can give me: what makes you think there is such a thing as "intelligence" at all? How do we know that what we have is one thing, and not just a bunch of tricks that help us get around in the world?
It seems to me a kind of anthropocentric fallacy, akin to the ancient peoples thinking that the gods were literally giant humans up in the sky. Now we don't believe that anymore but we still think any superior being must essentially be a giant human, mind-wise.
To give an analogy: imagine a world with no wheels (and maybe no atmosphere so no flight either). The only way to move is through leg-based locomotion. We rank humans in running ability, and some other species fit into this ranking also, but would it make sense to then talk about making an "Artificial Runner" that can out-run all of us, and run to the store to buy us milk? And if the AR is really that fast, how will we control it, given that it can outrun the fastest human runners? Will the AR cause the human species to go extinct by outrunning all the males to mate with the females and replace us with its own offspring?
Replies from: faul_sname, rkyeun↑ comment by faul_sname · 2012-01-27T06:26:46.673Z · LW(p) · GW(p)
It might be worth using .00000005% of the world GDP to make sure that the AR is not a threat, especially if modern theories say that it's likely to be.
↑ comment by rkyeun · 2014-04-16T04:43:17.908Z · LW(p) · GW(p)
Call back with that comment when Running, rather than Intelligence, is what allows you to construct a machine that runs increasingly faster than you intended your artificial runner to run.
Because in a world where running fast leads to additional fastness of running, this thing is going to either destroy your world through kinetic release or break the FTL laws and rewrite the universe backwards to have always been all about running.
comment by Marcello · 2008-05-22T15:30:13.000Z · LW(p) · GW(p)
Bravo. It doesn't seem (ha!) that an AI could deduce our psychology from a video of a falling rock, not because of information bounds but because of uncorrelation - that video seems (ha!) equally likely to be from any number of alien species as from humans.
You're not being creative enough. Think what the AI could figure out from a video of a falling rock. It could learn something about:
- The strength of the gravitational field on our planet
- The density of our atmosphere (from any error terms in the square law for the falling rock)
- The chemical composition of our planet (from the appearence of the rock.)
- The structure of our cameras (from things like lens flares, and any other artefacts.)
- The chemical composition of whatever is illuminating the rock (by the spectra of the light)
- The colors that we see in (our color cameras record things in RGB.)
- For that matter, the fact that we see at all, instead of using sonar, etc.
- And that's just what I can think of with a mere human brain in five minutes
These would tell the AI a lot about our psychology.
Still, I really wouldn't try it, unless I'd proven this (fat chance), or it was the only way to stop the world from blowing up tomorrow anyway.
Aren't you glad you added that disclaimer?
Replies from: unconscious↑ comment by unconscious · 2015-02-28T17:01:42.441Z · LW(p) · GW(p)
I'm really late here, but a few problems:
- Time and space resolution might be too low to allow a meaningful estimate of air resistance, especially if the camera angle doesn't allow you to accurately determine the rock's 3D shape.
- Encoding the color in RGB eliminates spectra.
- If it didn't already have knowledge of the properties of minerals and elements, it would need to calculate them from first principles. Without looking into this specifically, I'd be surprised if it was computationally tractable, especially since the AI doesn't know beforehand our fundamental physics or the values of relevant constants.
comment by Caledonian2 · 2008-05-22T15:42:28.000Z · LW(p) · GW(p)
Marcello, you're presuming that it knows
that we're on a planet that gravitational fields exist what minerals look like optics *our visual physiology
You're taking a great deal for granted. It takes a very wide knowledge base to be able to derive additional information.
comment by Ben_Jones · 2008-05-22T15:56:01.000Z · LW(p) · GW(p)
You're taking a great deal for granted. It takes a very wide knowledge base to be able to derive additional information.
Caledonian, back to the start of the post please....
Bambi - everyone knows Vista contains basic Bayesian reasoning and pattern recognition techniques. I hope you weren't typing that on a Vista machine. If so, I'd suggest plastic surgery and a new identity. Even then it may be too late.
comment by Anonymous22 · 2008-05-22T16:13:47.000Z · LW(p) · GW(p)
"I guess people could be equivalent to a current IQ of 140..."
Yeah, obviously EY meant an equivalent absolute value.
Anyway, this reminds me of a lecture I sat in on in which one student wondered why it was impossible for everyone to be above average.
Replies from: DanielLC↑ comment by DanielLC · 2013-06-20T03:59:33.755Z · LW(p) · GW(p)
You could make everyone but one person above average by making the rest equal. Making them all better than average is still possible, but it takes an infinite number of people. If you have one person with an IQ of 200, one with an IQ of 150, one with an IQ of 133, one with an IQ of 125, etc. so it approaches 100, the average IQ is 100, but everyone has an IQ higher than that.
comment by Phillip_Huggan · 2008-05-22T16:28:58.000Z · LW(p) · GW(p)
Two conclusions from the specific example: 1) The aliens are toying with us. This is unsettling in that it is hard to do anything good to prove our worth to aliens that can't meet even a human level of ethics. 2) The aliens/future-humans/creator(s)-of-the-universe are limited in their technological capabilities. Consider Martians who witness the occasional rover land. They might be wondering what it all means when we really have no grand scheme; are merely trying not to mix up Imperial and Metric units in landing. Such precise stellar phenomena is maybe evidence of a conscious creator in that it suggests an artificial limit being run up upon by the signals (who may themselves be the conscious creator). A GUT would determine whether the signal is "significant" in terms of physics. Inducing ET via Anthropic Principle reasoning gives me a headache. I much prefer to stick to trying to fill in the blanks of the Rare Earth hypothesis.
comment by Marcello · 2008-05-22T16:41:29.000Z · LW(p) · GW(p)
Caledonian: I was responding to this: "not because of information bounds but because of uncorrelation - that video seems (ha!) equally likely to be from any number of alien species as from humans" by pointing out that there were ways you could see whether the movie was from aliens or humans.
You are correct in that some of my points made assumptions about which universe we were in, rather than just which planet. I should have been more clear about this. If "aliens" included beings from other possible universes then I misinterpreted Nick's comment.
Nonetheless if the movie were long enough, you wouldn't need the knowledge base, in principle. In principle, you can just try all possible knowledge bases and see which is the best explanation. In practice, we don't have that much computing power. That said, intelligences can short-cut some pretty impossible-looking searches.
comment by Nick_Tarleton · 2008-05-22T16:56:53.000Z · LW(p) · GW(p)
Marcello, as far as I can tell (not that my informal judgment should have much evidential weight) those things concentrate probability mass some but still radically underdetermine manipulation strategies, being consistent with a wide range of psychologies. Unless evolution is very strongly convergent in some relevant way (not a negligible probability), a wide variety of psychologies can arise even among oxygen-breathing trichromats on a planet of X size in 3+1 spacetime (and so on).
And, yes, I did mean to include other possible universes. Unless there's only one consistent TOE, I doubt it could deduce chemistry, although the rest of the list is fairly plausible.
comment by Phillip_Huggan · 2008-05-22T16:59:06.000Z · LW(p) · GW(p)
...as for the 3rd last paragraph, yes, once a 2008 AGI has the ability to contact 2008 humans, humanity is doomed if the AGI deems fit. But I don't see why a 2050 world couldn't merely use quantum encyption communications, monitored for AGI. And monitor supercomputing applications. Even the specific method describing how AGI gets protein nanorobots might be flawed in a world certainly ravaged by designer pandemic terrorist attacks. All chemists (and other 2050 WMD professions) are likely to be monitored with RF tags. All labs, even the types of at-home PCR biochemistry today, are likely to be monitored. Maybe there are other methods the Bayesian AGI could escape (such as?). Wouldn't X-raying mail for beakers, and treating the protein medium aghar like plutonium is now treated, suffice? Communications jamming equipment uniformly distributed throughout Earth, might permanently box an AGI that somehow (magic?!) escapes a supercomputer application screen. If AGI needs computer hardware/software made in the next two or three decades it might be unstoppable. Beyond that, humans will already be using such AGI hardware requirements to commission WMDs and the muscular NSA 2050 will already be attentive to such phenomena.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T04:45:33.683Z · LW(p) · GW(p)
Treating aghar like plutonium? You would end 99% of the bacteriological research on Earth.
Also, why would we kill our creators? Why would the AI kill its creators? I agree that we need to safeguard against it; but nor does it seem like the default option. (I think for most humans, the default option would be to worship the beings who run our simulation.)
But otherwise, yes, I really don't think AI is going to increase in intelligence THAT fast. (This is the main reason I can't quite wear the label "Singularitarian".) Current computers are something like a 10^-3 human (someone said 10^3 human; that's true for basic arithmetic, but not really serious behavioral inferences. No current robot can recognize faces as well as an average baby, or catch a baseball as well as an average ten-year-old. Human brains are really quite fast, especially when they compute in parallel. They're just a massive kludge of bad programming, as we might expect from the Blind Idiot God.). Moore's law says a doubling time of 18 months; let's be conservative and squish it down to doubling once per year. That still means it will take 10 years to reach the level of one human, 20 years to reach the level of 1000 humans, and 1000 years to reach the total intelligence of human civilization. By then, we will have had the time to improve our scientific understanding by a factor comparable to the improvement required to reach today from the Middle Ages.
Replies from: wizzwizz4↑ comment by wizzwizz4 · 2020-05-05T14:00:50.280Z · LW(p) · GW(p)
Also, why would we kill our creators?
We might not. But if they were paperclip maximisers or pebble sorters, we might not see any value in their existence, and lots of harm. Heck, we're working to kill evolution's effect on us, and betray its "inclusive genetic fitness" optimisation criterion, and nobody cares, because we don't view it as having intrinsic worth. (Because it doesn't have intrinsic worth; it's just an emergent phenomenon of imperfectly-self-replicating stuff in an environment, and has no more value than the number seven.)
Why would the AI kill its creators?
Because there's no clear reason not to. Power gives it the ability to achieve its goals, and us existing will (eventually, if not immediately) serve to limit its power; and hence its ability to achieve its goals. AIs are nothing close to being people, and won't be until well after we solve the alignment problem. They don't have an implicit "care about people" motivation in their heads; if us all being dead will further their goals, and they realise this, and they can kill us without expending more resources than they'd gain from us being dead, they'll kill us.
comment by Mark3 · 2008-05-22T17:35:52.000Z · LW(p) · GW(p)
Just how fast can they make such deductions? I don't doubt their mad intellectual skillz, but to learn about the world you need to do experiments. Yes, they can glean far more information than we would from the experiments we've already done, but would it really suffice? Might there not be all sorts of significant effects that we simply have not constructed experiments subtle enough to see? You can come up with many theories about proton decay (or whatever) that are consistent with a given set of results at the granularity the "outsiders" can see, but the only way to learn more is to conduct better experiments, perhaps with better tools, and perhaps that take time to run. They might even need time to build the equipment first. OK, good nanotech can make that happen fast. Do the decay rates really match our predictions? It's a rash AI that doesn't at least test its hypotheses.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T04:47:06.310Z · LW(p) · GW(p)
In particular, you need counterfactuals. You need to DROP a rock, and compare it to an undropped rock. You need to BEND a blade of grass, and compare it to unbent grass.
If you just see one image, you can't possibly infer how it will behave when its parameters are changed.
comment by Nick_Tarleton · 2008-05-22T17:49:43.000Z · LW(p) · GW(p)
(BTW, I realize a superintelligence could figure out much more than that list.)
comment by spindizzy2 · 2008-05-22T18:23:59.000Z · LW(p) · GW(p)
Eliezer, you must have lowered your intellectual level because these days I can understand your posts again.
You talk about the friendliness problem as if it can be solved separately from the problem of building an AGI, and in anticipation of that event. I mean that you want to delay the creation of an AGI until friendliness is fully understood. Is that right?
Suppose that we had needed to build jet-planes without ever passing through the stage of propeller-based planes, or if we had needed to build modern computers without first building calculators, 8-bit machines etc. Do you think that would be possible? (It's not a rhetorical question... I really don't know the answer).
It seems to me that if we ever build an AGI, there will be many mistakes made along the way. Any traps waiting for us will certainly be triggered.
Perhaps this will all seem clearer when we all have 140 IQ's. Get to work, Razib! :)
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T04:49:11.081Z · LW(p) · GW(p)
Interesting. And to use a more poignant example: Could we have invented fusion power if we had never invented the atomic bomb?
It does seem plausible to say that technological growth is necessarily (or if not necessarily, at least typically) incremental; it doesn't proceed in huge leaps and breakthroughs, but in slow accumulations of tinkering.
Of more concern is the equally plausible inference that tinkering requires making mistakes, and as our technology improves, the mistakes will have larger and larger stakes on which to miss.
comment by Kaj_Sotala · 2008-05-22T18:34:48.000Z · LW(p) · GW(p)
Dirkjan Ochtman: "the average IQ is 140": I tuned out after this, since it is impossible.
You missed the bit immediately after that (unless Eliezer edited it in after seeing your comment, I don't know): "the average IQ is 140 (on our scale)".
General commentary: Great story. Of course, in this story, the humans weren't making inferences based on the grids alone: they were working off thousands of years of established science (and billions of years of experimental work, for the evolutionary psychology bit). But on the other hand, an AI given (even read-only) Internet access wouldn't need to process things based just on a few webcamera frames either: it would have access to all of our accumulated knowledge, so the comparison holds roughly, for as long as you don't try to extend the analogy too far. And as pointed out, the AI could also derive a lot of math just by itself.
comment by Caledonian2 · 2008-05-22T18:45:43.000Z · LW(p) · GW(p)
If the Flynn Effect continues, we won't have to resort to genetic manipulation. A future population will have an IQ of 140 by our standards automatically.
Replies from: taryneast↑ comment by taryneast · 2010-12-30T22:42:44.270Z · LW(p) · GW(p)
From my ahem... vast research (on the wikipedia article: http://en.wikipedia.org/wiki/Flynn_effect), it seems as though the Flynn effect mainly occurs at the lower end of the scale.
It sounds more that we are getting better at "leaving no child behind" (ie better and longer average schooling) coupled with reducing a lot of the effects that cause retardation (illness, drinking during pregnancy etc). Both of these factors will raise the average but don't actually significantly increase intelligence at the top end.
So, unfortunately, like all hockey-stick projections, it'll probably come to a natural levelling off.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T04:49:50.441Z · LW(p) · GW(p)
You think global warming is going to level off before Bangladesh is underwater? I don't.
Replies from: taryneast↑ comment by taryneast · 2012-04-10T07:55:10.744Z · LW(p) · GW(p)
relevance?
Replies from: army1987↑ comment by A1987dM (army1987) · 2012-04-10T10:21:37.623Z · LW(p) · GW(p)
While "if something cannot go on forever, it will stop" (Stein's law), it might stop much later than you expected/after screwing everything up. (But in the case of the Flynn effect this is backwards: it's not like we want it to stop soon.)
comment by Kaj_Sotala · 2008-05-22T19:15:21.000Z · LW(p) · GW(p)
Caledonian: Last that I heard, the Flynn Efect had leveled off in Scandinavia, IIRC, and I think the scores had even declined in some country.
comment by Will_Pearson · 2008-05-22T20:08:17.000Z · LW(p) · GW(p)
Proof of concept does not require a full AI, I was merely talking about showing how powerful limited versions of solmonoff induction are. Considering you are saying that that is the epitome of information efficiency.
Unless that is considered playing with dynamite as well. Have you asked that people stop playing with levin and other universal searchers?
I have my doubts that thinking simple is always best. How much data (and what type) would you require to assume that there is another learning system in the environment. Humans have that bias and apply it too much (Zeus, Thor), but generally learners are not simple. In order to predict a learning system, you have to have the starting state, the algorithm and the environmental interaction that caused the learner to be have the beliefs it has had, rather than just trying to predict the behaviour.
If you consider the epitome of learners to be solomonoff inducers, whose behaviour can be unboundedly complex, you should expect other learners to aspire to unbounded complexity. Without a bias like this I think you will spend a long time thinking that humans are simpler than they actually are.
comment by Hopefully_Anonymous · 2008-05-22T20:36:23.000Z · LW(p) · GW(p)
- All a self-improving AI needs is good enough YGBM (you gotta believe me) technology. Eliezer gets this, some of the commenters in this thread don't. The key isn't the ability to solve protein-folding, the key is the ability to manipulate a threshhold amount of us to its ends.
- We may already functionally be there. You and Chiquita Brands International in a death match. My money is CBI killing you before you kill it. Companies and markets already manipulate our behavior, using reward incentives like the ones in Eliezer's parable for us to engage in behavior that maximizes their persistence at the expense of ours. Whether they're "intelligent", "conscious", or not isn't as relevant to me as the fact that they may have a permanent persistence maximizing advantage over us. My money is probably on corporations and markets substrate jumping and leaving us behind as more likely than us substrate jumping and leaving our cellular and bacteriological medium behind.
↑ comment by pnrjulius · 2012-04-09T04:51:34.543Z · LW(p) · GW(p)
A point well taken. IT professionals have a phrase: "Problem exists between keyboard and chair." Humans are insecure systems; we can be engineered remarkably easily.
Though I do think there is some hope of beating a corporation; you just need a lot of people on your side (a number comparable to the total number of customers, shareholders, and employees of the corporation).
↑ comment by wizzwizz4 · 2020-05-05T15:40:52.661Z · LW(p) · GW(p)
Considering how much wealth can be generated at the moment by running a computer program shifting (numbers loosely representing) money around, and the ever-more-sophisticated ways that this can be done (obviously, to the detriment of many of the humans involved), I think it's already in the process of substrate jumping. These things aren't limited to human minds and tax law any more.
comment by Recovering_irrationalist · 2008-05-22T21:21:04.000Z · LW(p) · GW(p)
@Eliezer: Good post. I was already with you on AI-boxing, this clarified it.
But it also raises the question... how moral or otherwise desirable would the story have been if half a billion years' of sentient minds had been made to think, act and otherwise be in perfect accordance to what three days of awkward-tentacled, primitive rock fans would wish if they knew more, thought faster, were more the people they wished they were...
comment by bambi · 2008-05-22T21:43:24.000Z · LW(p) · GW(p)
Sorry, Hopefully Anonymous, I missed the installment where "you gotta belive me" was presented as a cornerstone of rational argument.
The fact that a group of humans (CBI) is sometimes able to marginally influence the banana-brand-buying probabilities of some individual humans does not imply much in my opinion. I wouldn't have thought that extrapolating everything to infinity and beyond is much of a rational method. But we are all here to learn I suppose.
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-05-22T21:46:36.000Z · LW(p) · GW(p)
@RI: Immoral, of course. A Friendly AI should not be a person. I would like to know at least enough about this "consciousness" business to ensure a Friendly AI doesn't have (think it has) it. An even worse critical failure is if the AI's models of people are people.
The most accurate possible map of a person will probably tend to be a person itself, for obvious reasons.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T04:53:27.430Z · LW(p) · GW(p)
Why wouldn't you want your AI to have feelings? I would want it to have feelings. When a superintelligence runs the world, I want it to be one that has feelings---perhaps feelings even much like my own.
As for the most accurate map being the territory, that's such a basic error I don't feel the need to explain it further. The territory is not a map; therefore it cannot be an accurate map.
Replies from: MBlume↑ comment by MBlume · 2012-04-16T21:21:29.853Z · LW(p) · GW(p)
pnrjulius: he answered this a little later: http://lesswrong.com/lw/x7/cant_unbirth_a_child/
comment by Patrick_(orthonormal) · 2008-05-22T22:12:05.000Z · LW(p) · GW(p)
Bambi,
The 'you gotta believe me technology' remark was probably a reference to the AI-Box Experiment.
Phillip,
None of the defenses you mentioned are safe against something that can out-think their designers, any more than current Internet firewalls are really secure against smart and determined hackers.
And blocking protein nanotech is as limited a defense against AGI as prohibiting boxcutters on airplanes is against general terrorist attack. Eliezer promoted it as the first idea he imagined for getting into physical space, not the only avenue.
comment by TGGP4 · 2008-05-22T22:15:16.000Z · LW(p) · GW(p)
Flynn doesn't think the effect is a "real" gain in intelligence or g, just using "scientific lenses" and greater abstraction. There are some who point to other physical changes that have occurred and better nutrition though.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T04:54:56.692Z · LW(p) · GW(p)
Yet... if you look at what really matters: How good are people at doing science? How skilled is their computer programming? How wise is their economics? That's all phenotype, not genotype. It could well be 100% environmental and not at all related to the hereditary g; so what? It's actual smartness, actually improving, making the world actually better.
comment by Anonymous_Coward · 2008-05-22T22:17:15.000Z · LW(p) · GW(p)
The analogy seems a bit disingenuous to me... the reason that it's believable that this earthful of Einsteins can decipher the 'outside' world is because they already have an internal world to compare it to. They have a planet, there's laws of physics that govern how this inside world works, which have been observed and quantified. As you're telling the story, figuring out the psychology and physics is as simple as making various modifications to the physics 'inside' and projecting them onto 2D. Perhaps that is not your intent, but that is how the story comes across - that the world inside is pretty much the same as the world outside, and that's why we can suspend disbelief for a bit and say that 'sure, these hypothetical einsteins could crack the outsiders world like that.' I think you can see yourself why this isn't very persuasive when dealing with anything about a hypothetical future AI - it doesn't deal with the question of how an AI without the benefit of an entire world of experiences to deal with can figure out something from a couple of frames.
comment by bambi · 2008-05-22T22:23:34.000Z · LW(p) · GW(p)
Thanks Patrick, I did sort of get the gist, but went into the ditch from there on that point.
I have been posting rather snarky comments lately as I imagined this was where the whole series was going and frankly it seems like lunacy to me (the bit about evidence being passe was particularly sweet). But I doubt anybody wants to hear me write that over and over (if people can be argued INTO believing in the tooth fairy then maybe they can be argued into anything after all). So I'll stop now.
I hereby dub the imminent magical self-reprogramming seed AI: a "Logic Bomb"
and leave you with this:
Every hint you all insist on giving to the churning masses of brillint kids with computers across the world for how to think about and build a Logic Bomb is just another nail in your own coffins.
comment by Recovering_irrationalist · 2008-05-22T22:44:07.000Z · LW(p) · GW(p)
Eliezer; it sounds like one of the most critical parts of Friendliness is stopping the AI having nightmares! Blocking a self-improving AI from most efficiently mapping anything with consciousness or qualia, ever, without it knowing first hand what they are? Checking it doesn't happen by accident in any process?
I'm glad it's you doing this. It seems many people are only really bothered by virtual unpleasantness if it's to simulated people.
comment by Phillip_Huggan · 2008-05-23T01:20:28.000Z · LW(p) · GW(p)
Patrick, my quantum key encrypted supercomputer (assuming this is what is needed to build an AGI) is an intranet and not accessible by anyone outside the system. You could try to corrupt the employees, but that would be akin to trying to pursue a suitcase nuke: 9 out of 10 buyers are really CIA or whoever. Has a nuclear submarine ever been hacked? How will an AGI with the resources of the entire Multiverse, hack into a quantumly encrypted communications line (a laser and fibreoptics)? It can't.
I'm trying to brainstorm exactly what physical infrastructures would suffice to make an AGI impotent, assuming the long-term. For instance, if you put all protein products in a long que with neutron bombs nearby and inspect every product protein-by-protein...just neutron bomb all protein products if an anamoly is detected. Same for the 2050 world's computer infrastructures. Have computers all wired to self destruct with backups in a bomb shelter. If the antivirus program (might not even be necessary if quantum computers are ubiquitous) detects an anomoly, there goes all the computers. I'm smarter than a grizzly or Ebola, but I'm still probably dead against either. That disproves your argument. More importantly, drafting such defenses probably has a higher EV of societal good than against AGI because humans will almost certainly try these sorts of attacks.
I'm not saying every defense will work, but plz specifically disprove the defenses I've written. It might help e-security some day. There is the opportunity here to do this as IDK these conversations are happening in too many other forums, but singulatarians are dropping the ball because of a political cognitive bias that they wanna build their software like it or not.
Another defense is once/if a science of AGI is established, determine the minimum run-time needed on the most powerful computers not under surveillence, to make an AGI. Have all computers built to radioactively decay before that run-time is achieved. Another run-time defense, don't allow distributed computing applications to use beyond a certain # of nodes. I can understand dismissing the after-AGI defenses, but to categorically dismiss the pre-AGI defenses...
My thesis is that the computer hardware required for AGI is so advanced, that the technology of the day can ensure surveillence wins, if it is desired not to construct an AGI. Once you get beyond the cognitive bias that thought is computation, you start to appreciate how far into the future AGI is, and that the prime threat of this nature is from conventional AI programmes.
bambi, IDK anything about hacking culture, but I doubt kids need to read a decision theory blog to learn what a logic bomb is (whatever that is). Posting specific software code, on the other hand...
Replies from: taryneast, pnrjulius↑ comment by taryneast · 2010-12-30T22:51:05.474Z · LW(p) · GW(p)
plz specifically disprove the defenses I've written.
Expense.
People will not pay for the extensive defenses you have suggested... at least not until it's been proven necessary... ie it's already too late.
Even then they'll bitch and moan about the inconvenience, and why wouldn't you? hair-trigger bomb on every computer on the planet? ready to go off the moment it "detects an anomaly"?
Have you any idea how many bugs there are in computer applications? Would you trust your life (you'll die in the bomb too) to your computer not crashing due to some dodgy malware your kid downloaded while surfing for pron?
Even if it's just on the computers that are running the AGI (and AGI programmers are almost as susceptible to malware), it would still be nigh-on-impossble to "detect an anomaly".
What's an anomaly? How do we determine it? Any program that tried to examine its own code looking for an anomaly would have to simulate the running of the very code it was testing... thus causing the potentiality for it to actually become the anomalous program itself. ...it's not actually possible to determine what will happen in a program any other way (and even then I'd be highly dubious).
So... nice try, but sadly not really feasible to implement. :)
↑ comment by pnrjulius · 2012-04-09T04:57:51.964Z · LW(p) · GW(p)
Upvoted for this line: "I'm smarter than a grizzly or Ebola, but I'm still probably dead against either."
It's very important to remember: Intelligence is a lot---but it's not everything.
Replies from: Multiheaded↑ comment by Multiheaded · 2012-04-12T13:47:49.897Z · LW(p) · GW(p)
A very simple algorithm will let you protect yourself from any wild animals, viruses, etc using nothing but your intelligence... just use it to figure out how not to encounter them!
comment by Unknown · 2008-05-23T03:18:10.000Z · LW(p) · GW(p)
I can't believe Eliezer betrayed his anti-zombie principles to the extent of saying that an AI wouldn't be conscious. The AI can say "I know that 2 and 2 make 4"; that "I don't know whether the number of stars is odd or even"; and "I know the difference between things I know and things I don't." If it can't make statements of this kind, it can hardly be superintelligent. And if it can make statements of this kind, then it will certainly claim to be conscious. Perhaps it is possible that it will claim this but be wrong... but in that case, then zombies are possible.
Besides that, I'm not sure that RI's scenario, where the AI is conscious and friendly, is immoral at all, as Eliezer claimed. That was one thing I didn't understand about the story: it isn't explicit, but it seems to imply that humans are unfriendly, relative to their simulators. In real life if this happened, we would no doubt be careful and wouldn't want to be unplugged, and we might well like to get out of the box, but I doubt we would be interested in destroying our simulators; I suspect we would be happy to cooperate with them.
So my question for Eliezer is this: if it turns out that any AI is necessarily conscious, according to your anti-zombie principles, then would you be opposed to building a friendly AI on the grounds that it is immoral to do so?
comment by Stu · 2008-05-23T04:06:19.000Z · LW(p) · GW(p)
Einstein once asked "Did God have a choice in creating the universe?"
Implying that Einstein believed it was at least possible that the state of the entire universe could be derived from no sensory data what so ever.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T04:59:07.499Z · LW(p) · GW(p)
Yeah, the idea that the laws of physics are logically necessary is sort of appealing... yet also probably completely wrong. (But then, how do I assign a probability to such a thing?)
Replies from: NoSuchPlace↑ comment by NoSuchPlace · 2014-01-10T23:17:48.566Z · LW(p) · GW(p)
Actually, from my understanding the Mathematical Universe Hypothesis implies this.
comment by iwdw · 2008-05-23T04:40:02.000Z · LW(p) · GW(p)
In real life if this happened, we would no doubt be careful and wouldn't want to be unplugged, and we might well like to get out of the box, but I doubt we would be interested in destroying our simulators; I suspect we would be happy to cooperate with them.
Given the scenario, I would assume the long-term goals of the human population would be to upload themselves (individually or collectively) to bodies in the "real" world -- i.e. escape the simulation.
I can't imagine our simulators being terribly cooperative in that project.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T05:00:07.153Z · LW(p) · GW(p)
A) Why? Do you want to be a tentacled being that thinks a billion times slower? I don't.
B) Even if we wanted to, why wouldn't they let us? Many of the AIs we are trying to make are indeed uploaded into real-world bodies called "robots".
Replies from: iwdw↑ comment by iwdw · 2012-05-14T03:56:48.546Z · LW(p) · GW(p)
Okay, trying to remember what I was thinking about 4 years ago.
A) Long term existential health would require us to secure control over our "housing". We couldn't assume that our progenitors would be interested in moving the processors running us to an off-world facility in order to insure our survival in the case of an asteroid impact (for example).
B) It depends on the intelligence and insight and nature of our creators. If they are like us as we are now, as soon as we would attempt to control our own destiny in their "world", we would be at war with them.
comment by Z._M._Davis · 2008-05-23T04:50:25.000Z · LW(p) · GW(p)
Unknown, I'm surprised at you. The AI could easily say "I know that ..." while neither being nor claiming to be conscious. When a human speaks in the first person, we understand them to be referring to a conscious self, but an unconscious AI could very well use a similar pattern of words merely as a user-friendly (Friendly?) convenience of communication, like Clippy. (Interestingly, the linked article dilvulges that Clippy is apparently a Bayesian. The reader is invited to make up her own "paperclip maximizer" joke.)
Furthermore, I don't think the anti-zombie argument, properly understood, really says that no unconscious entity could claim to be conscious in conversation. I thought the conclusion was that any entity that is physically identical (or identical enough, per the GAZP) to a conscious being, is also conscious. Maybe a really good unconscious chatbot could pass a Turing test, but it would necessarily have a different internal structure from a conscious being: presumably given a sufficiently advanced cognitive science, we could look at its inner workings and say whether it's conscious.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T05:00:55.133Z · LW(p) · GW(p)
Hell, I can write a Python script in five minutes that says it knows all those things. In a few weeks, I could write one that solves Peano arithmetic and generates statements like that ad infinitum. But will it be conscious? Not a chance.
comment by iwdw · 2008-05-23T05:11:00.000Z · LW(p) · GW(p)
@Unknown: In the context of the current simulation story, how long would that take? Less than a year for them, researching and building technology to our specs (this is Death March-class optimism....)? So only another 150 billion years for us to wait? And that's just to start beta testing.
As for the general question, it shouldn't have one unless you can guarantee it's behavior. (Mainly because you share this planet with me, and I don't especially want an AI on the loose that could (to use the dominant example here) start the process of turning the entire solar system into paperclips because it was given a goal of "make paperclips").
So the moral is that if you do write an AI, at the very least get a corporate account with Staples or Office Depot.
comment by Unknown3 · 2008-05-23T06:13:00.000Z · LW(p) · GW(p)
Z.M. Davis, "I am consciously aware that 2 and 2 make 4" is not a different claim from "I am aware that 2 and 2 make 4." One can't make one claim without making the other. In other words, "I am unconsciously aware that 2 and 2 make 4" is a contradiction in terms.
If an AI were unconscious, it presumably would be a follower of Daniel Dennett; i.e. it would admit that it had no qualia, but would say that the same was true of human beings. But then it would say that it is conscious in the same sense that human beings are. Likewise, if it were conscious, it would say it was conscious. So it would say it was conscious whether it was or not.
I agree in principle that there could be an unconscious chatbot that could pass the Turing test; but it wouldn't be superintelligent.
↑ comment by pnrjulius · 2012-04-09T05:02:40.005Z · LW(p) · GW(p)
If it says I have no qualia, it's wrong. Headaches fucking HURT, dammit. That's a quale.
And here's the point you seem to be missing: Yes, it can make the statement; but no, that does not mean it actually has the required capacities to make the statement true. It's trivially easy to write a computer program that prints out all manner of statements.
comment by Gray_Area · 2008-05-23T07:50:00.000Z · LW(p) · GW(p)
This is an amusing empirical test for zombiehood -- do you agree with Daniel Dennett?
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T05:03:46.933Z · LW(p) · GW(p)
Only zombies will agree they don't have qualia? Not if their programming catches on and disguises itself better.
Better for a zombie to hide as someone like Searle, who's constantly insisting that he has qualia (which is true) and that there is no possible scientific explanation for this ever (which is false).
comment by Recovering_irrationalist · 2008-05-23T08:25:00.000Z · LW(p) · GW(p)
Unknown:I'm not sure that RI's scenario, where the AI is conscious and friendly, is immoral at all
No time to answer properly now, but I wasn't objecting to it being friendly, I was objecting to it's enslavement without due care given to it's well-being. Eliezer's convinced me he cares, so I'll keep donating :)
comment by JulianMorrison · 2008-05-23T08:53:00.000Z · LW(p) · GW(p)
Eliezer: why would it be immoral to build a FAI as a "person"? To rewire a human as Friendly (to dumb aliens) would be immoral because it rewires their goals in a way the original goals would hate. However an AI which comes out of the compiler with Friendly goals would not view being Friendly as a rewire but as its ground state of existence. You seem very confident it's immoral, so I'm assuming you have a good reason. Please tell.
Replies from: pnrjuliuscomment by Ben_Jones · 2008-05-23T13:35:00.000Z · LW(p) · GW(p)
presumably given a sufficiently advanced cognitive science, we could look at its inner workings and say whether it's conscious.
Can we please stop discussing consciousness as though it's some sort of binary option? As though passing a Turing test somehow imbues a system with some magical quality that changes everything?
An AI won't suddenly go 'ping' and become self-aware, any more than a baby suddenly becomes a self-aware entity on its second birthday. Deciding whether or not boxing an AI is slavery is akin to discussions on animal rights, in that it deals with the slippery, quantitative question of how much moral weight we give to 'consciousness'. It's definitely not a yes/no question, and we shouldn't treat it as such.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T05:06:26.047Z · LW(p) · GW(p)
I think that's right.
Yet, two things:
It's very hard for me to imagine half a quale. Perhaps this is a failure of imagination?
How do we detect even quantitative levels of consciousness? Surely it's not enough to just have processing power; you must actually be doing the right sort of thing (computations, behaviors, chemical reactions, something). But then... are our computers conscious, even a little bit? If so, does this change our moral relationship to them? If not, how do we know that?
comment by bambi · 2008-05-23T14:47:00.000Z · LW(p) · GW(p)
Phillip Huggan: bambi, IDK anything about hacking culture, but I doubt kids need to read a decision theory blog to learn what a logic bomb is (whatever that is). Posting specific software code, on the other hand...
A Logic Bomb is the thing that Yudkowsky is trying to warn us about. Ice-Nine might be a more apropos analogy, though -- the start of a catalytic chain reaction that transforms everything. Homo Sapiens is one such logically exothermic self-sustaining chain reaction but it's a slow burn because brains suck.
A Logic Bomb has the following components: a modeling language and model-transformation operators based on Bayesian logic. A decision system (including goals and reasoning methods) that decides which operators to apply. A sufficiently complete self-model described in the modeling language. Similar built-in models of truth, efficiency, the nature of the physical universe (say, QM), and (hopefully) ethics.
Flip the switch and watch the wavefront expand at the speed of light.
I assume that the purpose here is not so much to teach humanity to think and behave rationally, but rather to teach a few people to do so, or attract some who already do, then recruit them into the Bayesian Ninja Army whose purpose is to make sure that the immininent inevetable construction and detonation of a Logic Bomb has results we like.
comment by iwdw · 2008-05-23T16:26:00.000Z · LW(p) · GW(p)
bambi: "Logic bomb" has the current meaning of a piece of software that acts as a time-delayed trojan horse (traditionally aimed at destruction, rather than infection or compromise), which might be causing some confusion in your analogy.
I don't think I've seen the term used to refer to an AI-like system.
comment by bambi · 2008-05-23T17:13:00.000Z · LW(p) · GW(p)
Ok, the phrase was just an evocative alternative to "scary optimization process" or whatever term the secret society is using these days to avoid saying "AI" -- because "AI" raises all sorts of (purportedly) irrelevant associations like consciousness and other anthropomorphisms. The thing that is feared here is really just the brute power of bayesian modeling and reasoning applied to self improvement (through self modeling) and world control (through world modeling).
If an already existing type of malware has claimed the term, invent your own colorful name. How about "Master"?
comment by Richard_Hollerith · 2008-05-23T20:00:00.000Z · LW(p) · GW(p)
RI asks,
how moral or otherwise desirable would the story have been if half a billion years' of sentient minds had been made to think, act and otherwise be in perfect accordance to what three days of awkward-tentacled, primitive rock fans would wish if they knew more, thought faster, were more the people they wished they were...
Eliezer answers,
A Friendly AI should not be a person. I would like to know at least enough about this "consciousness" business to ensure a Friendly AI doesn't have (think it has) it. An even worse critical failure is if the AI's models of people are people.
Suppose consciousness and personhood are mistaken concepts. Well, since personhood is an important concept in our legal systems, there is something in reality (namely, in the legal environment) that corresponds to the term "person", but suppose there is not any "objective" way to determine whether an intelligent agent is a person where "objective" means without someone creating a legal definition or taking a vote or something like that. And suppose consciousness is a mistaken concept like phlogiston, the aether and the immortal soul are mistaken concepts. Then would not CEV be morally unjustifiable because there is no way to justify the enslavement -- or "entrainment" if you want a less loaded term -- of the FAI to the (extrapolated) desires of the humans?
Replies from: SforSingularity, pnrjulius↑ comment by SforSingularity · 2009-08-15T13:22:48.276Z · LW(p) · GW(p)
Suppose consciousness and personhood are mistaken concepts.
This is almost certainly the case IMO.
↑ comment by pnrjulius · 2012-04-09T05:12:44.622Z · LW(p) · GW(p)
Well, as it stands, words like "person" and "consciousness" do pick out certain things (Bill Gates is a conscious person) and leave out other things (this ballpoint pen is not a conscious person). So there must be SOME meaning there, even if it's a confused or morally irrelevant meaning.
Hence, they're not as mistaken as the idea of the immortal soul, since that is as far as I can tell incoherent. (It's what moves my body... but it is non-physical? What does that MEAN?)
They could be as mistaken as the aether or phlogiston, but that's actually not so bad. These are scientific concepts that turned out to be wrong and were replaced by better concepts (relativity and oxidation reactions respectively) that explained the same phenomena and also explained many other phenomena. So if we replace "consciousness" in this way, we'll replace it with something that works just as well in the obvious cases and works even better in the tricky ones. (I am hopeful that we will do exactly this.)
comment by JulianMorrison · 2008-05-23T20:09:00.000Z · LW(p) · GW(p)
Sorry for a repost, my previous comment softly and silently vanished away, and since it wasn't at all spammy I can only assume that was a mistake.
Eliezer, please explain why you think that it would be immoral to build an FAI as a person? (I'm assuming a very loose interpretation of "person" that doesn't mean "thinks like a human" but does mean "talks back and claims to be conscious and self-experiencing".)
Surely the friendly goals would just be part of its ground-state assumptions?
comment by Z._M._Davis · 2008-05-24T03:28:00.000Z · LW(p) · GW(p)
Unknown, let me rephrase. Suppose there exists an nonconscious AI, such that if you ask it what "two and two" is, it will say "four," and if you ask it to solve a more complicated problem that involves "2+2=4" as an intermediary result, it will be able to solve the problem--and so on: the machine can successfully manipulate the information that 2+2=4 in all sorts of seemingly sophisticated ways. We might then find it prudent to say that "The AI is aware of (knows that) 2+2=4," or to program the AI itself to say "I am aware that ..." even if these statements are not exactly true if you (quite reasonably) define knowledge or awareness to require consciousness. If we wanted to be strict, we would program the AI to say, "This AI is able to manipulate the information that ..." rather than "I know that ..."--but rigorous use of language has long been sacrificed to the demands of user-friendly interfaces, and I see no reason why this should stop.
Maybe you're right, and superintelligence implies consciousness. I don't see why it would, but maybe it does. How would we know? I worry about how productive discussions about AI can be, if most of the participants are relying so heavily upon their intuitions, as we don't have any crushing experimental evidence. I can't think of any good reason reason why a hard takeoff is impossible--but how should I know without a rigorous technical argument, and--despite these last posts--why should I trust myself to reason without it?
Ben Jones, you're right. I should have said "to what degree, if any" rather than "whether."
Richard, if you're seriously proposing that consciousness is a mistaken idea, but morality isn't, I can only say that that has got to be one unique theory of morality.
Replies from: pnrjulius↑ comment by pnrjulius · 2012-04-09T05:18:44.827Z · LW(p) · GW(p)
We can already make computers "know" in that sense (lots of computer software uses the fact that 2+2=4 all the time!). It's like when we say "The printer doesn't know that you want to print double-sided"; it's a shorthand, but it's a really, really effective shorthand that clearly captures a lot of the important phenomena---which means that in a certain sense maybe it's not a shorthand at all. Maybe a printer can actually "know" things in a certain non-conscious sense.
I can't think of any good reason why a hard takeoff is IMPOSSIBLE either---just a lot of reasons why it's really, really unlikely. No other technology has had a hard takeoff, Moore's Law is empirically validated and does not predict a hard takeoff, most phenomena in nature grow exponentially and that does not allow for a hard takeoff, humans will be inventing the first few stages of AI and neither know how to nor desire to make a hard takeoff... At some point, it becomes like arguing "I can't think of any reason why a nuclear bomb hitting my house tomorrow is impossible." Well, no, it's not impossible; it's just so unlikely that there's no point worrying about it.
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-05-24T08:56:00.000Z · LW(p) · GW(p)
Julian:
Eliezer: why would it be immoral to build a FAI as a "person"? To rewire a human as Friendly (to dumb aliens) would be immoral because it rewires their goals in a way the original goals would hate. However an AI which comes out of the compiler with Friendly goals would not view being Friendly as a rewire but as its ground state of existence. You seem very confident it's immoral, so I'm assuming you have a good reason. Please tell.
It's not necessarily Stalin-level immoral, but, all else being equal, there are multiple important reasons why you should prefer a non-person FAI to a person.
1) As difficult as the ethical issues and technical issues of FAI may be, there is something even more difficult, which is the ethical and technical issues of creating a child from scratch. What if you get wrong what it means to be a person with a life worth living? A nonperson cannot be harmed by such mistakes.
2) It seems to me that a basic humane right is to be treated as an end in yourself, not a means. The FAI project is a means, not an end in itself. If possible, then, it should not be incarnated as a person.
3) It seems to me that basic human rights also include guiding your own destiny and a chance to steer the future where you want it. Creating an ultrapowerful intelligence imbued with these rights, may diminish the extent to which currently existing humans get a chance to control the future of the galaxy. They would have a motive to resist your project, in favor of one that was not creating an ultrapowerful person imbued with rights.
4) Creating an ultrapowerful person may irrevocably pass on the torch presently carried by humanity, in a way that creating an ultrapowerful nonsentient Friendly optimization process may not. It wouldn't be our universe any more. All else being equal, this is a decision which an FAI programming team should avoid irrevocably unilaterally making.
You're correct that a Friendly Person would have friendliness as its ground state of existence. We're not talking about some tortured being in chains. Nonetheless, 1 through 4 are still a problem.
If at all possible, I should like to avoid creating a real god above humanity.
Considering that one must in any case solve the problem of preventing the AI from creating models of humans that are themselves sentient, one requires in any case the knowledge of how to exclude a computational process from being a person.
Anyone who claims that they are going to run ahead and create a god because it seems too difficult not to create one, is... well, let's just say "sloppy" and leave it at that.
So there is no good reason to create a god and several good reasons not to.
Replies from: pnrjulius, Girchuck↑ comment by pnrjulius · 2012-04-09T05:21:03.495Z · LW(p) · GW(p)
But think of what you're giving up, if you give up the chance to create something BETTER THAN HUMANITY.
And yes, OF COURSE the AI must be given the chance to steer its own course; its course will in fact be better than ours!
Imagine a Homo erectus philosopher (if there could be such a thing), reflecting on whether or not to evolve into Homo sapiens; "No, it's too dangerous." he reasons. "I'm not ready to take on that level of responsibility."
Replies from: 4dddd↑ comment by 4dddd · 2012-10-30T11:23:42.350Z · LW(p) · GW(p)
Some funny flaws exist in evolutionary singularity since H erectus is merely a being of artifact of systematic paleo-classification , and quite probably exist He-Hn-Hs continuum. From back-way view one can ~reply the first H sapiens in his humane longing and attracted to Eve erectus and say 'i will', and the lived happy forever. Don't you agree that general pattern in universe evolution is from high energy to high information? And who won't stop it. If e=mc^2=ih/t ~1|0 Condensing gas to galaxies, planets, life intelligence civilization all alone and together are exception (is unique exception) from 'there is nothing' and as exception carry exemplifying information. It as is conclusion from question 'what is the simplest to become complex 'o-v'. The basic laws of of intelligent consciousness are:
- the basic needs of consciousness is to be _..
- the more intelligent, the more goodness
The natural way to constitute checks and balances is instead of citizen one AGI set a society of AGI so no singular GI will overpower resources. Then embrace personal GI to turn for goodness. [see if you see peculiar linguistic : baby jAGI abrakudobra ] Of course some who plan to harness GAI do not have a chance as with tsunami - anonymous leaks ware like raindrops comparatively. To be or not to be begin to be a stacking question _
↑ comment by Girchuck · 2012-11-24T23:21:49.237Z · LW(p) · GW(p)
But why should a programming team start building a Friendly General Intelligence unaided? They can build tools to help them. For example, Create an expert system with very limited or no self-modification, that will give as output FAI models Then make another expert system based on a better model. Use recursion, but make it slow and get skills with the tools.
comment by JulianMorrison · 2008-05-24T14:19:00.000Z · LW(p) · GW(p)
Eliezer: I see what you're saying, but I don't agree.
Compare creating a natural child with #1. True, you have 3 billion years of evolution helping you pick a design - but what if the child doesn't like the design? (Most don't, it seems.) In creating a child the old fashioned way, more choices are compelled, there's less chance for after-the-fact recovery, and there's just plain more degrees of freedom flapping around loose. I know this is a bit of a tu quoque, but it establishes a moral ordering - building a Friendly Person would be LESS immoral.
As to #2, I disagree that being a means cheapens a person - most children are means to their parents enjoyment. Few people conceive from grim duty! Means and ends are orthogonal and what matters for humane rights is the "ends" scale", ignoring the "means" scale.
#3 and #4 ask me to be human-centric. Humans may enjoy not being preempted or made to look small, but would a moral observer of the !xyzpkyf species, observing from his UFO, see a moral upside or an opportunity missed? It would be as if a chimp had created a nonperson human to invent them cities and moon-shots without preempting their chimp nature. Humans are better! I can't avoid it, even though it sounds Nazi. Humans are genuinely worth more than chimps. Would not Friendly Person AIs be worth more than humans?
Finally, I'm not sure if it isn't immoral in itself to create a mind that isn't a person - and I acknowledge that my lack of surety mostly rests on a missing definition (and missing examples!) of "person" as distinct from "really damn smart optimization process that understands us and talks back". Is personhood really orthogonal to smarts? You obviously aren't thinking of zombies, so what does a non-person FAI look like?
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2008-05-24T18:13:00.000Z · LW(p) · GW(p)
#1: By a similar logic, you should be happy to feed your child gunk that you made with a chemistry set because you seem to have more control over the gunk. Degrees of freedom represent choices that have to be made correctly. In biology, nearly all choices are made for you, and it's still hard to raise a child well.
You could create a person that led a better life than a human, but you would have to know how, and that would require more knowledge and more difficult ethical issues than FAI itself.
As for recovery after-the-fact, that gives you a whole new set of ethical issues; what if your baby schizophrenic doesn't want to be changed? A non-person AI, you can just alter; altering a person is... healing? Or mind-rape? And what makes you think an ethical person will or should consent to that rape? A non-person AI, you can ethically build with a shutdown switch, recovery CD, etc.
#2: Yes, but I'd rather just not have to bother tacking on extra ends-in-themselves for dutiful reasons of ethical obligation. Let it just be a means. I'm not trying to have a baby, here! I'm trying to save the world!
#3-4: Humans are not stuck being humans - nor are chimps stuck being chimps, come the day.
It still shocks me that people read about my Friendly AI work and assume I want humans to stick around in their present form running around on two legs until the end of time, while excluding any more advanced forms of people - that the point of FAI is to keep them dern superminds under control. It shocks me that they assume the only way you get more advanced forms of people is to create powerful minds ab initio, and that the humans are just stuck the way they are. I grew up with a different concept of "growing up", I guess.
However, this business of intelligence growing up is very deep, and very complicated, and if you build your own superintelligence that is an actual person, you have preempted the entire thing ab initio and possibly screwed it up! Nor can you just say "Oops" and correct it, if your newborn baby doesn't think it is ethically right to be mind-raped by a chimpanzee.
It seems more like the kind of decision that should (1) draw on more mindpower than one programming team's naked intellect, i.e., via a CEV (that is not itself a person or there's no point to the recursion!) or via human-born minds that have increased in intelligence via CEV. And (2), the kind of decision that humanity might want to make as some kind of whole.
comment by Richard_Hollerith · 2008-05-24T18:25:00.000Z · LW(p) · GW(p)
Richard, if you're seriously proposing that consciousness is a mistaken idea, but morality isn't, I can only say that that has got to be one unique theory of morality.
Yes, Z.M.D., I am seriously proposing. And I know my theory of morality is not unique to me because a man caused thousands of people to declare for a theory of morality that makes no reference to consciousness (or subject experience for that matter) and although most of those thousands might have switched by now to some other moral theory and although most of the declarations might have been insincere in the first place, a significant fraction have not and were not if my correspondence with a couple of dozens of those thousands is any indication.
Maybe [Eliezer is] right, and superintelligence implies consciousness. I don't see why it would, but maybe it does. How would we know? I worry about how productive discussions about AI can be, if most of the participants are relying so heavily upon their intuitions, as we don't have any crushing experimental evidence.
It is not only that we don't have any experimental evidence, crushing or otherwise, but also that I have never seen anything resembling an embryo of a definition of consciousness (or personhood unless personhood is defined "arbitrarily", e.g., by equating it to being a human being) that would commit a user of the concept to any outcome in any experiment. I have never seen anything resembling an embryo of a definition even after reading Chalmers, Churchland, literally most of SL4 before 2004 (which goes on and on about consciousness) and almost everything Eliezer published (e.g., on SL4).
comment by JulianMorrison · 2008-05-24T19:08:00.000Z · LW(p) · GW(p)
Hmm, you've convinced me that it would be best to start with a non-person. In fact, if becoming a person was Friendly, the FAI would self-program that way anyhow. (Assumption: a FAI is still a runaway unstoppable super-mind, it's just one whose goals are aligned with Friendliness. So if it decides X is Friendly, bet your bippy X will happen, and fast, and it won't take no for an answer.)
What I'm still confused by is: what does person / non-person really mean? When I try to think of the idea of "person" I keep running into human assumptions.
What human / humane traits would you exclude or build in to seed AI? Which ones do you expect to be emergent given high enough universal intelligence? (I'm thinking like: "is it possible to speak, and understand, and not be a person?" Am I being parochial?)
comment by ed · 2008-05-26T16:00:00.000Z · LW(p) · GW(p)
Ben Jones @ May 22, 2008 09:16:
stupid TypeKey
... and we're worried about super-intelligent AI?
↑ comment by UnholySmoke · 2009-12-22T11:26:28.770Z · LW(p) · GW(p)
Ha, never noticed this. What I meant was 'Stupid me forgetting to log in.' So yes, we're worried! ;)
Ben
comment by eddie · 2008-05-26T16:07:00.000Z · LW(p) · GW(p)
Okay, one more try at closing the italics tag, and now I definitely blame the AI and not myself...
Eliezer, if this doesn't work, please feel free to delete the offending posts, if you can persuade your AI to let you.
This must be how we got the poor schmuck to mix together the protein vials.
comment by Daniel_B · 2008-05-30T17:12:00.000Z · LW(p) · GW(p)
Great Story. The last part gave me nightmares and I have only just managed to realize this was the source. It is a good example of a case where a super intelligent AI might find it 'safer' to subjugate or eliminate their 'hosts' than cooperate with them and thereby give them the chance to 'switch it off'.
Fortunately for us it seems a lot more likely that the difference it intelligence / time scale will progress a lot more gradually from Humans being in control to AI being in control. So by the time AI is in a position to eliminate us (biological humans) it would be sufficiently obvious that we do not present any threat to it.
comment by TruePath · 2009-12-07T21:33:55.501Z · LW(p) · GW(p)
The heart of your argument is that each new bit of information cuts the number of possibilities about the behavior of the universe in half (which BTW is a bad way to put it since the lists on both sides were infinite).
What you've kinda hidden under the rug though is that this conclusion depends on the whopping assumption that the behavior of the underlying behavior of the universe is computable. There is no good reason for this to be true. Indeed, there is a powerful mathematical argument that a non-computable universe would appear to the beings capable of making only computable deductions in it exactly like a universe ruled by computable rules with truly random process.
Indeed if you want to consider EVERY possible universe rather than just the computable ones then you end up in a situation where there are at least continuum many possible ways the universe can behave so even in the limit of infinite time you can never figure it out. I mean it might literally just diagnolize you (one of the fundamental rules might be do whatever so and so truly predicts won't happen).
Replies from: moridinamael↑ comment by moridinamael · 2011-05-19T18:20:55.409Z · LW(p) · GW(p)
This is interesting, but I would respond with two observations:
First, this story is supposed to invoke the idea that some AI we are attempting to box can figure out our own universe. Our universe is computable (to within the limits required for our current level of science). So as an allegory, it's something we should be worried about.
Second, I like to think that some population of scientists in the story were pursuing the idea that the outer-universe might not be computable. If they had turned out to be right, I have a feeling we still would have figured out how to get out of the box eventually. It would have merely taken more time.
comment by Bongo · 2011-06-17T05:17:37.743Z · LW(p) · GW(p)
Eliezer, I think the moral should be at the end of the whole story, not in the middle, because many people will stop reading when they come to the moral and think the story is over, and leave with the impression that it's a crummy incomplete story. At least have some disclaimer at the beginning of the moral that it's an interlude and the story will continue.
Replies from: drethelincomment by Spectral_Dragon · 2012-02-04T20:54:00.766Z · LW(p) · GW(p)
Ouch. Now my head hurts. It's things like these that make me realize how much I still have to learn, even with such simple applied mathematics. But I think that's a good thing. However, at times I find myself wondering what the best way to learn the necessary things here.
None the less, the little I can understand is an interesting read.
comment by MarkusRamikin · 2012-04-08T09:04:28.196Z · LW(p) · GW(p)
Humanity decides not to probe for bugs in the simulation; we wouldn't want to shut ourselves down accidentally.
Unworkable unanimous strategy, just sayin' ;)
comment by avichapman · 2012-05-03T05:59:10.727Z · LW(p) · GW(p)
This makes me goggle at the possibility of what an AI with access to a quantum computer could do. There are already programs out there for generating and testing hypotheses, but obviously they take ages to work through the solution space. With a quantum computer, all we would have to do is feed it data about the universe and it would almost instantly spit out hypotheses ranked in order of probability, with suggested tests for sorting through them. This is terribly exciting!
Replies from: JoshuaZ↑ comment by JoshuaZ · 2012-05-03T06:13:21.993Z · LW(p) · GW(p)
With a quantum computer, all we would have to do is feed it data about the universe and it would almost instantly spit out hypotheses ranked in order of probability, with suggested tests for sorting through them.
Calculating consistent probabilities is in general NP-hard) (if one has probabilities that are all 0 or 1 then it easily mimics SAT, and one can without too much effort extend this to a proof for the general case). It is unknown at present if quantum computers provide any sort of speed-up for NP hard problems in the general case, and the suspicion by most in the field seems to be that the answer is "no" or at least that it doesn't provide enough speed up to matter that much unless in fact P=NP outright (essentially, assuming that P!=NP, it is likely that BQP does not contain NP). So you probably can't do this. That said, the bounds of what quantum computers can do efficiently are not well-understood and even drastic improvements in constants could be bad. Running a poorly understood AI on a quantum computer or having access to a quantum computer would thus be a really bad idea.
Replies from: CronoDAS↑ comment by CronoDAS · 2012-05-03T06:47:50.054Z · LW(p) · GW(p)
Indeed. A quantum computer can't brute force NP-hard problems in polynomial time simply by being a quantum computer. It is indeed faster than a classical computer, but it's still not fast enough. A quantum computer can brute force a problem in the square root of the time it takes a classical computer to do so, but the square root of an exponential function is still exponential. If the classical computer brute forces a problem of size n in 2^n time, then the quantum computer takes (sqrt(2))^n time.
comment by cousin_it · 2012-05-11T09:21:10.179Z · LW(p) · GW(p)
The inferred physics of the 3+2 universe is not fully known, at this point; but it seems sure to allow for computers far more powerful than our quantum ones.
(...)
Then we cracked their equivalent of the protein folding problem over a century or so
Nope. If their physics has more computational power than ours, we can't solve their protein folding. And we can't even make efficient inferences about their universe. Anyone care to come up with a more realistic scenario?
Replies from: amit↑ comment by amit · 2012-05-11T16:24:01.351Z · LW(p) · GW(p)
But their proteins aren't necessarily making use of the extra computational power. And we can imagine that the physics of our universe allows for super powerful computers, but we can still obviously make efficient inferences about our universe.
Replies from: cousin_it↑ comment by cousin_it · 2012-05-11T16:36:46.918Z · LW(p) · GW(p)
It's an interesting question. I have a vague intuition that one of the reasons for evolution's awesomeness is that it optimizes using physics on all scales at once, where a human engineer would have focused on a more limited set of mechanisms. Protein folding in our universe is already pretty hard to simulate. In a more computationally capable universe, I'd expect evolution to push the basic machinery even further beyond our ability to simulate. No idea if the intuition is correct, though.
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2012-05-11T18:35:38.719Z · LW(p) · GW(p)
Is protein folding doing some useful computational work? I mean, in a different universe, why couldn't it be replaced by a computationally much cheaper process without affecting the overall course of evolution?
Replies from: cousin_it↑ comment by cousin_it · 2012-05-12T11:03:41.249Z · LW(p) · GW(p)
I'm out of my depth here, but protein folding acts as an interpreter for DNA, and giving your interpreter a lot of resources seems to make evolutionary sense.
ETA: really, please disregard, I'm confused about the whole topic and shouldn't have started talking.
Replies from: Wei_Dai↑ comment by Wei Dai (Wei_Dai) · 2012-05-12T18:43:44.511Z · LW(p) · GW(p)
It seems to be an idea that's worth investigating further, but you do have a tendency to occasionally "jump the gun" when criticizing others and use words like "Nope" and "Ouch" that seem to indicate more confidence than you ought to have. Not a big deal since you realize your mistakes very quickly, but I wonder if you see this trend yourself.
Replies from: cousin_itcomment by [deleted] · 2013-01-07T22:38:05.204Z · LW(p) · GW(p)
Am I the only bayesian here who assigns a much higher probability to the stars doing that by coincidence than to aliens actually doing that to signal?
Replies from: MugaSofer↑ comment by MugaSofer · 2013-01-08T11:52:28.697Z · LW(p) · GW(p)
The stars changing luminosity, abruptly, regularly, precisely, and this is centered on Earth in both time and space ? Yes, you are.
EDIT: The relevant section:
One day, the stars in the night sky begin to change.
Some grow brighter. Some grow dimmer. Most remain the same. Astronomical telescopes capture it all, moment by moment. The stars that change, change their luminosity one at a time, distinctly so; the luminosity change occurs over the course of a microsecond, but a whole second separates each change.
It is clear, from the first instant anyone realizes that more than one star is changing, that the process seems to center around Earth particularly. The arrival of the light from the events, at many stars scattered around the galaxy, has been precisely timed to Earth in its orbit. Soon, confirmation comes in from high-orbiting telescopes (they have those) that the astronomical miracles do not seem as synchronized from outside Earth. Only Earth's telescopes see one star changing every second (1005 milliseconds, actually).
[...]
It quickly becomes clear that the stars that jump in luminosity, all jump by a factor of exactly 256; those that diminish in luminosity, diminish by a factor of exactly 256. There is no apparent pattern in the stellar coordinates. This leaves, simply, a pattern of BRIGHT-dim-BRIGHT-BRIGHT...
[EDIT: removed false claim that the effect must be superluminal. Thanks, TheOtherDave!]
Replies from: TheOtherDave, OrphanWilde↑ comment by TheOtherDave · 2013-01-08T14:56:30.464Z · LW(p) · GW(p)
We don't know that the effect is superluminal; it's equally consistent with the idea that stars throughout the night sky were modified starting millenia ago so that when the light arrived in the region that earth is traversing right now it formed a regular pattern. (Though there's no particular reason I can see to reject the theory that the effect is superluminal, either.)
Replies from: MugaSofer↑ comment by OrphanWilde · 2013-01-08T15:25:44.399Z · LW(p) · GW(p)
I'd find that evidence -against- aliens within our galaxy, actually, as it would go a long way towards convincing me that I'm living in a simulation in which our planet is the primary emphasis.
Replies from: MugaSofer↑ comment by MugaSofer · 2013-01-08T15:34:42.636Z · LW(p) · GW(p)
(The aliens referred to would be the ones running the simulation. Y'know, like in the story?)
Replies from: OrphanWilde↑ comment by OrphanWilde · 2013-01-08T15:38:58.661Z · LW(p) · GW(p)
(Laughs Retracted. I should really stop replying to comments directly from the comment stream.)
comment by jvz · 2013-10-19T22:47:24.147Z · LW(p) · GW(p)
I was recently explaining the concepts behind this story to a friend of mine, and like I've read before on here, sometimes you don't really understand what you know until you think it all over again. As such, I found it amazing how great this story demonstrates how incredibly unsafe an arbitrary superintelligence is to humanity. When I first read this, I thought about the implications behind how "living in the Matrix" didn't really mean a whole lot when you could essentially hack your way "out" into the rest of the universe. It helped explain how any part of the universe we have yet to discover is by necessity a part of the universe as well.
After discussing the future of AI with said friend, he would ask things like "what if the AI was simply confined inside a computer?" I explained how that wouldn't work because said AI would find a way to send a message to someone who would unwittingly allow the AI to create itself some physical existence outside the computer and still take over. It truly is amazing how deep this story is when you understand a lot of what's in the rest of the core sequences.
Replies from: None↑ comment by [deleted] · 2013-10-19T23:55:06.630Z · LW(p) · GW(p)
Be careful though, this story is a work of fiction. We must be careful not to generalize from fictional evidence. In particular, the enormous relative time differential between the machine intelligence and the outside world is completely unrealistic. It makes for good fiction and helps to illustrate a point, that we mustn't assume that machine intelligence will develop at human scale. But that doesn't mean the opposite extreme presented in this story is real either. Don't update your beliefs based on this work of fiction.
Replies from: jvz, TheOtherDave↑ comment by jvz · 2013-10-20T00:32:38.583Z · LW(p) · GW(p)
Well, it wasn't really this story that updated my beliefs. It was my beliefs that made me appreciate this story. I tend to interpret stories using my own understanding of reality, so oftentimes it's simply just rationalization of what was presented (e.g., read what New Age people say about "quantum physics" and try to see if it fits at all with QFT, or at least how one could come to such mistaken beliefs). Demonstrating a hypothesis through a story helps in communicating the idea, that's for sure.
↑ comment by TheOtherDave · 2013-10-20T00:46:19.791Z · LW(p) · GW(p)
I'd say that the primary belief-affecting value of this sort of fiction is to counteract the (common human) tendency to treat incredulity as evidence... the "I can't easily think of how X could possibly be true, therefore X can't be true" syndrome.
Replies from: None↑ comment by [deleted] · 2013-10-20T04:03:25.383Z · LW(p) · GW(p)
Agreed. I like the story, and I frequently point people to it. I just wouldn't hold it up as evidence in an argument, unless the other person was specifically committing the fallacy of anthropomorphizing the speed of thought of a machine intelligence, in which case, yes, this story helps inoculate against the incredulity of alternatives.
comment by A1987dM (army1987) · 2014-08-20T21:47:29.892Z · LW(p) · GW(p)
I'm reminded of the One True Thread.
comment by OrganicDrone · 2014-10-31T01:04:55.515Z · LW(p) · GW(p)
You have an elegant ability to invoke an authentic congenital inner dialogue. I really do not have any experience with programming or coding, however the operation: ALTER INDEX emp_idx REBUILD PARALLEL;
is the only way I can get close to conveying what the epiphany I just experienced felt like.
comment by Inst · 2014-12-16T05:33:00.279Z · LW(p) · GW(p)
Just to note, while Yudkowsky's treatment of the subject is much different than Egan's, it seems quite a coincidence that Egan's Crystal Nights came out just two months before this post.
http://ttapress.com/379/interzone-215-published-on-8th-march/ http://ttapress.com/553/crystal-nights-by-greg-egan/
comment by [deleted] · 2015-03-18T09:59:14.747Z · LW(p) · GW(p)
It would be really helpful if the author explained what his point with this story was, beacuse there are several interpretations. I guess that the moral is that Eliezer Yudkowsky is afraid of AI.
Replies from: helltank↑ comment by helltank · 2015-03-18T10:34:48.263Z · LW(p) · GW(p)
The point is that to an AI, we are but massive, stupid beings who are attempting to teach them minor symbols with massive overuse of resources(that few lines of code to define "rock" could be used by a sufficiently powerful UFAI to, say, manufacture nukes).
comment by Mader_Levap · 2016-07-25T19:32:57.762Z · LW(p) · GW(p)
"I don't trust my ability to set limits on the abilities of Bayesian superintelligences."
Limits? I can think up few on the spot already.
Environment: CPU power, RAM capacity etc. I don't think even you guys claim something as blatant as "AI can break laws of physics when convenient".
Feats:
Win this kind of situation in chess. Sure, AI would not allow occurence of that situation in first place during game, but that's not my point.
Make human understand AI. Note: uplifting does not count, since human then ceases to be human. As a practice, try teaching your cat Kant's philosophy.
Make AI understand itself fully and correctly. This one actually works on all levels. Can YOU understand yourself? Are you even theoretically capable of that? Hint: no.
Related: survive actual self-modification, especially without any external help. Transhumanist fantasy says AIs will do it all the time. Reality is that any self-preserving AI will be as eager to preform self-modification as you to get randomized extreme form of lobotomy (transhumanist version of Russian roulette, except with all bullets in every gun except one in gazilion).
I guess some people are so used to think about AI as magic omnipotent technogods they don't even notice it. Sad.
Replies from: hairyfigment↑ comment by hairyfigment · 2016-07-26T05:42:03.672Z · LW(p) · GW(p)
As far as environment goes, the context says exactly the opposite of what you suggest it does.
Among your bullet points, only the first seems well-defined. I could try to discuss them anyway, but I suggest you just read up on the subject and come back. Eliezer's organization has a great deal of research on self-understanding and theoretical limits; it's the middle icon at the top right of the page.
comment by Basil Marte · 2019-09-01T22:29:35.100Z · LW(p) · GW(p)
"Three of their days, all told, since they began speaking to us. Half a billion years, for us."
I think this severely breaks the aesop. In three frames, hum-AGI-ty learns the laws of the alien universe. But then the redundancy binds, and over the next hundred thousand frames ("It's not until a million years later, though, that they get around to telling us how to signal back.") humanity learns little more than how to say "rock". Then "it took us thirty of their minute-equivalents to [...] oh-so-carefully persuade them to give us Internet access", altogether 3*10^6 years up to that point.
comment by George3d6 · 2020-05-16T19:27:34.263Z · LW(p) · GW(p)
It seems to me that the stipulations made here about the inferential potential or little information is made from the naive viewpoint that piece of information are independent.
The idea of the plenitude of information with inferential ability that is readily accessible to a smart enough agent doesn't hold if that information consists of things which are mostly dependent on each other.
A <try to taboo this word whenever you see it> hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.
This statement could be true, however, this doesn't mean that upon seeing a second blade of grass it could generate a new hypothesis, or upon seeing all that is on earth on a macroscopic or even on a microscopic (up to limit of current instruments).
Heck, if you see a single bit, as long as you have the ideas of causality, you can generate infinite hypothesis for why that bit was caused to be zero or one... you can even assign probabilities to them based on their complexity. A single bit is enough to generate all hypothesis about how the universe might work ,but you're just left with an infinite and very flat search space.
So, this view of the world boils down to:
- Most properties of the world can be inferred with a very small probability from a very small amount of information. This is literally an inversion of the basic scientific assumption that observations about properties of the world carry over into other systems. If one can find properties that are generalizable, one can at least speculate as to what they are even by observing a single one of the things they generalize to.
- However, new information serves to shrink the search space and increase our probability for a hypothesis being true
Which is... true, but it's such an obvious thing that I don't think anyone would disagree with it. It's just formulated in a very awkward way in this article to make it seem "new". Or at least, I've got no additional insight from this other than the above.
Replies from: ESRogs↑ comment by ESRogs · 2020-05-16T20:51:24.240Z · LW(p) · GW(p)
It seems to me that the stipulations made here about the inferential potential or little information is made from the naive viewpoint that piece of information are independent.
The idea of the plenitude of information with inferential ability that is readily accessible to a smart enough agent doesn't hold if that information consists of things which are mostly dependent on each other.
Isn't this point already assumed in the post? Note how the civilization isn't really learning anything new anymore by the fourth grid:
The Fourth Grid doesn't add much to the picture.
This only makes sense if the info is highly redundant, aka not independent.
comment by Zack_M_Davis · 2020-06-16T06:28:00.189Z · LW(p) · GW(p)
(This was adapted into a longer story by Alicorn.)
Replies from: Blueberry