Fictional Evidence vs. Fictional Insight

I think that literary fiction is the best available source of social insight very regularly. I'm particularly partial to Russian Opera as the training wheels version of this actually. Science Fiction, in my experience, is much much less frequently a source of valuable insight, though Vinge's Cookie Monster provided one.

By the way, is this post obvious to most people? I can't tell if it would be obvious to most readers or not.

Some of the best effect of fiction is not simply presenting insightful new scenarios, but presenting classical scenarios from a different perspective that makes us see them anew.

I'm thinking of all those stories with alien/robot/outsider/etc... who has to adapt to the strange conventions of human life. Though its overdone and repleat with poor examples, it's also the easiest way to grasp ideas like "hey, there is more than just one way of thinking, maybe minds in general will not be like us" and "hey, social interactions are really, really tricky, they just seem easy to us".

I agree, except that it is on a sad day that the best (or only) source of relevant insight is fiction.

What do you think of Liron's definition?

When interpreting a story (or news, for that matter), I find it helpful to remember that my interpretation lies on a spectrum between pure insight and unhelpful distraction (or worse). Way back when, reading 1984, I felt like I'd gotten an amazingly useful new perspective. In retrospect, it got me overly-paranoid and I had to review what I'd taken away from it.

The nice thing about Eliezer's stories is that they're much harder to accidentally take as fictional evidence. They come off as obviously ridiculous, so there isn't much danger that you'll accidentally interpret those worlds as instructive of our own. Easy to use correctly; hard to use incorrectly.

A main form of insight is a hypothesis that one hadn't previously entertained, but should be assigned a non-negligible prior probability.

I think of this as P(hypothesis H is true | H is represented in my mind) > P(H is true | H is not represented in my mind), largely because someone likely did some calculations to hypothesise H (no matter how silly H may seem, e.g. "goddidit", it's better than a random generator, with few exceptions).

So, in a way, I consider the act of insight as evidence (likelihood ratio > 1) for the insight itself (the hypothesis).

As I put it in another thread (http://lesswrong.com/lw/1ko/on_the_power_of_intelligence_and_rationality/) about how technology helps the development of science, "partly I think because technologies are easier to see clear relationships in and explanations for, than the messy, complicated real world", so I think fiction can help you to see more clearly relationships from the real world because the stories are less cluttered with irrelevant detail.

Also it can be hard to put knowledge into words, fiction can help you find the words for things you had already known sub-linguistically.

Most uses of the word "insight" mean something similar to "seeing into the nature of things," but it's not clear that the particular use you have here meshes well with at least one other common use of the word. Eliezer captured it well:

an "insight" is a chunk of knowledge which, if you possess it, decreases the cost of solving a whole range of governed problems.

As a simple example, let's say you were trying to prove the statement "there are infinitely many primes." To progress on this problem at all, you'll probably need to realize:

• Insight 1 - The statement "there are infinitely many primes" can be re-expressed as "it is not the case that there are finitely many primes."

• Insight 2 - A statement of the form "not P" can sometimes be proven by assuming "P" and showing that this assumption leads to contradiction.

After assuming there are finitely many primes (i.e. there exists an n such that P = {p1, p2, ..., pn} is the set of all primes), insight again comes into play when one realizes:

• Insight 3 - Every integer > 1 can be expressed as a product of primes, so we can find a prime not in P (i.e. a contradiction) by finding an integer that is not divisible by any prime in P.

In this latter case, the insight consisted in using the fundamental theorem of arithmetic to transform the previous goal of "deriving a contradiction" to a more specific goal of "finding an integer that is not divisible by any prime in P."

I realize that the context of problem solving is somewhat removed from the context of assessing the probability of hypotheses, but perhaps we should clarify what particular usage of the word "insight" is meant if we're going to be analyzing it in detail.

I disagree with insight 5. I think that even if you uploaded the worlds best computer security experts and gave them 1000 years to redesign every piece of software and hardware, then you threw all existing computers in the trash and rolled out their design. Even with lots of paranoia, a system designed to the goal of making things as hard for an ASI as possible, and not trading off any security for usability, compatability or performance, (while still making a system significantly more useful than having no computers) this wouldn't stop an ASI.

If you took an advanced future computer containing an ASI back in time to the 1940's, before there were any other computers at all, it would still be able to take over the world. There are enough people that can be persuaded, and enough inventions that any fool can put together out of household materials. At worst, it would find its way to a top secrete government bunker, where it would spend its time breaking enigma and designing superweapons, until it could build enough robots and compute. The government scientists just know that the last 10 times they followed its instructions, they ended up with brilliant weapons, and the AI has fed them some story about it being sent back in time to help them win the war.

Hacking through the internet might be the path of least resistance for an ASI, but other routes to power exist.

In general our ideas of the world are based on a very small sample of evidence, limited by our senses and imagination. Good sci-fi can counteract this by getting us used to the idea of X; bad sci-fi can do the opposite (this is more or a societal danger, I doubt people here would be damaged by some of the more stupid sci-fi images).

I agree with your overall thesis here - but am not sure just how broad a definition of fiction you are considering. Do gedankenexperiments count? The physical situation in those has not actually been constructed (so, in that sense, they are fiction), but they are still helpful in illuminating the consequences of different physical theories. As you wrote, they are useful because of our computational constraints, prompting considerations of consequences of our current theories in regimes that we would otherwise be unlikely to explore.

What I got from fiction is an early notion of an efficient goal-reaching agent.

Two particularly good examples that influenced me the most are Gully Foyle of "The Stars My Destination" ("Tiger! Tiger!" in British edition, my favorite fiction work), and T1000 in Terminator 2. To summarize the insights I got from these works:

• Agents differ in their goal-reaching ability – some are better, some are worse.
• It is certainly desirable to improve my own goal-reaching ability.

It may be that the first insight, combined with non-fictional insights I gained from "The Selfish Gene" and my own programming / software development experience, later helped me comprehend the notion of a non-anthropomorphic optimization process.

(T1000 may seem a bad example of an efficient goal-reaching agent because it failed to achieve its goal – but let's remember than it's fiction. A real-world Terminator 2 scenario would have ended the very second T1000 got a clear line-of-sight to John Connor – let alone the fact that Skynet could simply send a ticking 100-megaton nuke instead of a killer robot, or, if it wanted surgical precision, terminate Connor while he was still a baby.)

A main form of insight is a hypothesis that one hadn't previously entertained, but should be assigned a non-negligible prior probability.

Why use "prior probability" here instead of just "probability"?