Genies and Wishes in the context of computer science

Try to find a proof of theorem X, possibly using resources from the external world.

This could be an inspiration for a sci-fi movie: A group of scientists created a superhuman AI and asked it to prove the theorem X, using resources from the external world. The Unfriendly AI quickly took control over most of the planet, enslaved all humans and used them as cheap labor to build more and more microprocessors.

A group of rebels fights against the AI, but is gradually defeated. Just when the AI is trying to kill the protagonist and/or the people dearest to the protagonist, the protagonist finally understands the motivation of the AI. AI does not love humans, but neither it hates them... it is merely trying to get as much computing power as possible to solve the theorem X, which is the task it was programmed to do. So our young hero takes a pen and paper, solves the theorem X, and shows the solution to the AI... which upon seeing the solution prints the output and halts. Humans are saved!

(And if this does not seem like a successful movie scenario, maybe there is something wrong with our intuitions about superhuman AIs.)

Try to find a proof of theorem X, possibly using resources from the external world. Use this simple mathematical prior over possible external worlds.

A proof is not a known world state, though. The easy way would be to have a proof checker in the world model and the goal is to pound on the world in such a way so that the proof checker says "proof valid" (a known world state on the output of the proof checker), the obvious solution is to mess with the proof checker, while actual use of the resources runs into the problem of prediction of what this use will, exactly, produce, or how exactly it will happen - you don't know what's going to go into the proof checker if you act in the proof-finding-using-external-resources way. And if you don't represent all parts of the AI as embodied in the real world then the AI can not predict consequences of the damage to physical structures representing it.

The real killer though is that you got a really huge model, for which you need a lot of computational resources to begin with. Plus with a simple prior over possible worlds, you will be dealing with very super duper fundamental laws of physics (below quarks). A very huge number of technologies each of which is a lot more useful elsewhere.

I know the standard response to this, it's that if something doesn't work someone tries something different, but the different is very simple to picture: you restrict the model, a lot, which a: speeds up the AI by a mindbogglingly huge factor, and b: eliminates most of unwanted exploration (both are intrinsically related). You can't tell AI to "self improve", either, you have to define what improvement is, and a lot of improvement is about better culling of anything you can cull.

Congratulations! Please keep up this sort of work.

Thanks, I guess, but I do not view it as work. I am sick with cold, bored, and burnt out from doing actual work, and suffering from "someone wrong on the internet" syndrome, in combination with knowing that extremely rationalized wrongitude affects people like you.

I second this question. Who is arguing that a genie that "does what it's told" is easier to make than a genie that "does what is meant"? Eliezer didn't, at least not in this post:

The user interface doesn't take English inputs. The Outcome Pump isn't sentient, remember? But it does have 3D scanners for the near vicinity, and built-in utilities for pattern matching. So you hold up a photo of your mother's head and shoulders; match on the photo; use object contiguity to select your mother's whole body (not just her head and shoulders); and define the future function using your mother's distance from the building's center. The further she gets from the building's center, the less the time machine's reset probability.

You cry "Get my mother out of the building!", for luck, and press Enter.

struct domain{
.... any data
};
real Function(domain value){
    .... any code
}

Indeed. I would think (as someone who knows nothing of AI beyond following LW for a few years) that the likely AI risk is something that doesn't think like a human at all, rather than something that is so close to a human in its powers of understanding that it could understand a sentence well enough to misconstrue it in a manner that would be considered malicious in a human.

, I interpret it to mean a scenario where first a programmer comes up with a shoddy formalization of paperclip maximization that he thinks is safe but actually isn't, and then writes that formalization into the AI.

Well, you'd normally define a paperclip counter function that takes as it's input the state of some really shoddy simulator of Newtonian physics and material science, and then use some "AI" optimization software to find what sort of actions within this simulator produce simulated paperclips from the simulated spool of simulated steel wire with minimum use of electricity and expensive machinery. You also have some viewer for that simulator.

You need to define some context to define paperclip maximization in. Easy way to define a paperclip is as a piece of wire bent in a specific shape. Easy way to define a wire is just to have it as some abstract object with specific material properties, which you just have an endless supply of, coming out of a black box.

Thinking the opposite

I don't think I'm following -- what is the opposite of what?

Yep - humans can't even talk about misunderstandings of human language without postulating an agent that thinks like a human.

Some "simple" algorithms for "understanding" language - such as "All I need is a dictionary and a grammar parser" - lead to predictable categories of mistakes. For example, there's the story of the automatic translating software that, when asked to translate "The spirit is willing but the flesh is weak" into Russian and then back into English, returned "The vodka is good but the meat is rotten".

Well, one of my points is that there's no actual distinction. People make a distinction though, because firstly there's cognitive effort on both the saying and listening side to make communication clear, and there's a distinction between things that one side is responsible for and things that other side is responsible for. Secondarily, it is often selfish-optimal to misunderstand commands to some extent which can be attributed to an alternate understanding. Particularly prominent in lawyering.

And optimizers, regardless of how powerful they are, are useless for that.

The (powerful) optimizer needs to have a model of how its optimizations impact that which is to be optimized. A model it adapts. Hypotheses, in other words. "Maximize my life span" would need to deal with cancer.

They are of the "we have no good hypotheses and we need to generate/invent/create some" kind

Not really. Firstly, we do have chemistry figured out very well; it's just that cells are complicated and it is very difficult to find the consequences of our interventions, so we tend to throw fairly bad ideas at the wall and see what sticks. Secondarily, generating most plausible hypotheses that fit the data is also an optimization problem. And thirdly, observe that evolution - a rather messy optimization process - did decrease the per-cell cancer rate for a whale to the utterly minuscule fraction of that of a human. And of a human, to a fairly small fraction to that of a dog. (With an advanced optimization, whale's cellular biochemistry may be of use also - a whale has far more cells).