Friendly AI and the limits of computational epistemology

Upvoted for the accurate and concise summary of the big picture according to SI.

There are qualities which are part of reality - we know this because they are part of experience, and experience is part of reality - but which are not part of our physical description of reality.

This continues to strike me as a category error akin to thinking that our knowledge of integrated circuit design is incomplete because we can't use it to account for Java classes.

I have been publicly and repeatedly skeptical of any proposal to make an AI compute the answer to a philosophical question you don't know how to solve yourself, not because it's impossible in principle, but because it seems quite improbable and definitely very unreliable to claim that you know that computation X will output the correct answer to a philosophical problem and yet you've got no idea how to solve it yourself. Philosophical problems are not problems because they are well-specified and yet too computationally intensive for any one human mind. They're problems because we don't know what procedure will output the right answer, and if we had that procedure we would probably be able to compute the answer ourselves using relatively little computing power. Imagine someone telling you they'd written a program requiring a thousand CPU-years of computing time to solve the free will problem.

And once again, I expect that the hardest part of the FAI problem is not "winning the intelligence race" but winning it with an AI design restricted to the much narrower part of the cognitive space that integrates with the F part, i.e., all algorithms must be conducive to clean self-modification. That's the hard part of the work.

You invoke as granted the assumption that there's anything besides your immediately present self (including your remembered past selves) that has qualia, but then you deny that some anticipatable things will have qualia. Presumably there are some philosophically informed epistemic-ish rules that you have been using, and implicitly endorsing, for the determination of whether any given stimuli you encounter were generated by something with qualia, and there are some other meta-philosophical epistemology-like rules that you are implicitly using and endorsing for determining whether the first set of rules was correct. Can you highlight any suitable past discussion you have given of the epistemology of the problem of other minds?

eta: I guess the discussions here, or here, sort of count, in that they explain how you could think what you do... except they're about something more like priors than like likelihoods.

In retrospect, the rest of your position is like that too, based on sort of metaphysical arguments about what is even coherently postulable, though you treat the conclusions with a certainty I don't see how to justify (e.g. one of your underlying concepts might not be fundamental the way you imagine). So, now that I see that, I guess my question was mostly just a passive-aggressive way to object to your argument procedure. The objectionable feature made more explicit is that the constraint you propose on the priors requires such a gerrymandered-seeming intermediate event -- that consciousness-simulating processes which are not causally (and, therefore, in some sense physically) 'atomic' are not experienced, yet would still manage to generate the only kind of outward evidence about their experiencedness that anyone else could possibly experience without direct brain interactions or measurements -- in order to make the likelihood of the (hypothetical) observations (of the outward evidence of experiencedness, and of the absence of that outward evidence anywhere else) within the gerrymandered event come out favorably.

the problem with state-machine materialism is not that it models the world in terms of causal interactions between things-with-states; the problem is that it can't go any deeper than that, yet apparently we can.

I may have missed the part where you explained why qualia can't fit into a state machine-model of the universe. Where does the incompatibility come from? I'm aware that it looks like no human-designed mathematical objects have experienced qualia yet, which is some level of evidence for it being impossible, but not so strong that I think you're justified in saying a materialist/mathematical platonist view of reality can never account for conscious experiences.

Parts of this I think are brilliant, other parts I think are absolute nonsense. Not sure how I want to vote on this.

there is no way for an AI employing computational epistemology to bootstrap to a deeper ontology.

This strikes me as probably true but unproven.

My own investigations suggest that the tradition of thought which made the most progress in this direction was the philosophical school known as transcendental phenomenology.

You are anthropomorphizing the universe.

Your point sounds similar to Wei's point that solving FAI requires metaphilosophy.

Maybe I missed this, but did you ever write up the Monday/Tuesday game with your views on consciousness? On Monday, consciousness is an algorithm running on a brain, and when people say they have consciously experienced something, they are reporting the output of this algorithm. On Tuesday, the true ontology of mind resembles the ontology of transcendental phenomenology. What's different?

I'm also confused about why an algorithm couldn't represent a mass of entangled electrons.

That is all very interesting, but what difference does it practically make?

Suppose I were trying to build an AGI out of computation and physical sensors and actuators, and I had what appeared to me to be a wonderful new approach, and I was unconcerned with whether the device would "really" think or have qualia, just with whether it worked to do practical things. Maybe I'm concerned with fooming and Friendliness, but again, only in terms of the practical consequences, i.e. I don't want the world suddenly turned into paperclips. At what point, if any, would I need to ponder these epistemological issues?

Some brief attempted translation for the last part:

A "monad", in Mitchell Porter's usage, is supposed to be a somewhat isolatable quantum state machine, with states and dynamics factorizable somewhat as if it was a quantum analogue of a classical dynamic graphical model such as a dynamic Bayesian network (e.g., in the linked physics paper, a quantum cellular automaton). (I guess, unlike graphical models, it could also be supposed to not necessarily have a uniquely best natural decomposition of its Hilbert space for all purposes, like how with an atomic lattice you can analyze it either in terms of its nuclear positions or its phonons.) For a monad to be a conscious mind, the monad must also at least be complicated and [this is a mistaken guess] capable of certain kinds of evolution toward something like equilibria of tensor-product-related quantum operators having to do with reflective state representation[/mistaken guess]. His expectation that this will work out is based partly on intuitive parallels between some imaginable combinatorially composable structures in the kind of tensor algebra that shows up in quantum mechanics and the known composable grammar-like structures that tend to show up whenever we try to articulate concepts about representation (I guess mostly the operators of modal logic).

(Disclaimer: I know almost only just enough quantum physics to get into trouble.)

A monadic mind would be a state machine, but ontologically it would be different from the same state machine running on a network of a billion monads.

Not all your readers will understand that "network of a billion monads" is supposed to refer to things like classical computing machinery (or quantum computing machinery?).

Do you think that the outputs of human philosophers of mind, or physicists thinking about consciousness, can't be accurately modeled by computational processes, even with access to humans? If they can be predicted or heard, then they can be deferred to.

The simple solution is to demystify qualia: I don't understand the manner in which ionic transfer within my brain appears to create sensation, but I don't have to make the jump from that to 'sensation and experience are different from brain state'. All of my sense data comes through channels- typically as an ion discharge through a nerve or a chemical in my blood. Those ion discharges and chemicals interact with brain cells in a complicated manner, and "I" "experience" "sensation". The experience and sensation are no more mysterious than the identity.

I find Mitchell_Porter difficult to understand, but I've voted this up just for the well-written summary of the SI's strategy (can an insider tell me whether the summary is accurate?)

Just one thing though - I feel like this isn't the first time I've seen How An Algorithm Feels From Inside linked to as if it was talking about qualia - which it really isn't. It would be a good title for an essay about qualia, but the actual text is more about general dissolving-the-question stuff.

So what should I make of this argument if I happen to know you're actually an upload running on classical computing hardware?

Although the abstract concept of a computer program (the abstractly conceived state machine which it instantiates) does not contain qualia, people often treat programs as having mind-like qualities, especially by imbuing them with semantics - the states of the program are conceived to be "about" something, just like thoughts are.

So far as I can tell, I am also in the set of programs that are treated as having mind-like qualities by imbuing them with semantics. We go to a good deal of trouble to teach people to treat themselves and others as people; this seems to be a major focus of early childhood education, language learning, and so on. "Semantics" and "aboutness" have to do with language use, after all; we learn how to make words do things.

Consciousness may not be a one-place property ("I am conscious"); it is a two-place property ("I notice that I am conscious"; "I notice that you are conscious"; "You notice that I am conscious"). After all, most of the time we are not aware of our consciousness.

But the idea that a human being is a state machine running on a distributed neural computation is just a hypothesis, and I would argue that it is a hypothesis in contradiction with so much of the phenomenological data, that we really ought to look for a more sophisticated refinement of the idea.

I'm not aware of any "phenomenological data" that contradicts computationalism.

You have to factor in the idea that human brains have evolved to believe themselves to be mega-special and valuable. Once you have accounted for this, no phenomenological data contradicts computationalism.

You need to do the impossible one more time, and make your plans bearing in mind that the true ontology [...] something more than your current intellectual tools allow you to represent.

With the "is" removed and replaced by an implied "might be", this seems like a good sentiment...

...well, given scenarios in which there were some other process that could come to represent it, such that there'd be a point in using (necessarily-)current intellectual tools to figure out how to stay out of those processes' way...

...and depending on the relative payoffs, and the other processes' hypothetical robustness against interference.

(To the extent that decomposing the world into processes that separately come to do things, and can be "interfered with" or not, makes sense at all, of course.)

A more intelligible argument than the specific one you have been making is merely "we don't know whether there are any hidden philosophical, contextual, or further-future gotchas in whether or not a seemingly valuable future would actually be valuable". But in that case it seems like you need a general toolset to try to eventually catch the gotcha hypotheses you weren't by historical accident already disposed to turn up, the same way algorithmic probability is supposed to help you organize your efforts to be sure you've covered all the practical implications of hypotheses about non-weird situations. As a corollary: it would be helpful to propose a program of phenomenological investigation that could be expected to cover the same general sort of amount of ground where possible gotchas could be lurking as would designing an AI to approximate a universal computational hypothesis class.

If it matters, the only scenario I can think of specifically relating to quantum mechanics is that there are forms of human communication which somehow are able to transfer qubits, that these matter for something, and that a classical simulation wouldn't preserve them at the input (and/or the other boundaries).

In addition to being a great post overall, the first ~half of the post is a really excellent and compact summary of huge and complicated interlocking ideas. So, thanks for writing that, it's very useful to be able to see how all the ideas fit together at a glance, even if one already has a pretty good grasp of the ideas individually.

I've formed a tentative hypothesis that some human beings experience their own subjective consciousness much more strongly than others. An even more tentative explanation for why this might happen is that perhaps the brain regions responsible for reflective self awareness leading to that particular Strange Loop become activated or used intensely in some people to a greater degree and at an earlier age. Perhaps thinking a lot about "souls" at a young age causes you to strongly anchor every aspect of your awareness to a central observer-concept, including awareness of that observer-concept.

I don't know.

Anyway, this subpopulation of highly self-conscious people might feel strongly that their own qualia are, in fact, ontologically fundamental, and all their other sense data are illusionary. (I say "other" sense data because fundamentally your prefrontal cortex has no idea which objects of its input stream represent what we would naively classify as processed "sense data" versus which objects consist of fabrication and abstraction from other parts of the brain.)

The rest of the human population would be less likely to make statements about their subjective experience of their own awareness because that experience is less immediate for them.

I developed this hypothesis upon realizing that some people immediately know what you mean when you start talking about reflective self-awareness, and for these people the idea of an observer-spirit consciousness seems like a natural and descriptive explanation of their awarness. But then there are other people who look at you blankly when you make statements about your sense of yourself as a floating observer, no matter how you try to explain it, as if they really never reflected on their own awareness.

For myself, I think of "redness" as belonging to the same ontological class as an object in a C++ program - possessing both a physical reality in bits on a chip, and also complex representational properties within the symbolic system of C++. And I see no reason why my C++ program couldn't ultimately form a "awareness" object which then becomes aware of itself. And that could explain sentences the C++ program outputs about having an insistent sense of its own awareness. It actually sticks in my craw to say this. I, personally, have always had a strong sense of my own awareness and a powerful temptation to believe that there is some granular soul within me. I am not entirely satisfied with how this soul is treated by a reductionistic approach, but nor can I formulate any coherent objections, despite my best effots.

ed: Are people literally downvoting every reply that has anything good to say about the parent?

Upvoted for clarity.

I think, along with most LWers, that your concerns about qualia and the need for a new ontology are mistaken. But even granting that part of your argument, I don't see why it is problematic to approach the FAI problem through simulation of humans. Yes, you would only be simulating their physical/computational aspects, not the ineffable subjectiveness, but does that loss matter, for the purposes of seeing how the simulations react to different extrapolations and trying to determine CEV? Only if a) the qualia humans experience are related to their concrete biology and not to their computational properties, and b) the relation is two-ways, so the qualia are not epiphenomenal to behavior but affect it causally, and physics as we understand it is not causally closed. But in that case, you would not be able to make a good computational simulation of a human's behavior in the first place!

In conclusion, assuming that faithful computational simulations of human behavior are possible, I don't see how the qualia problem interferes with using them to determine CEV and/or help program FAI. There might be other problems with this line of research (I am not endorsing it) but the simulations not having an epiphenomenal inner aspect that true humans have does not interfere. (In fact, it is good--it means we can use simulations without ethical qualms!)

I think you're on the right track with your initial criticisms but qualia is the wrong response. Qualia is a product of making the same mistake as the platonists; it's reifying the qualities of objects into mental entities. But if you take the alternative (IMO correct) approach - leaving qualities in the world where they belong - you get a similar sort of critique because clearly reducing the world to just the aspects that scientists measure is a non-starter (note that it's not even the case that qualitative aspects can't be measured - i.e., you can identify colours using standardised samples - it's simply that these don't factor into the explanations reductionists want to privilege). This is where the whole qualia problem began: Descartes wanted to reduce the world to extended bodies and so he had to hide all the stuff that didn't fit in a vastly expanded concept of mind.

I've been trying to find a way to empathically emulate people who talk about quantum consciousness for a while, so far with only moderate success. Mitchell, I'm curious if you're aware of the work of Christof Koch and Giulio Tononi, and if so, could you speak to their approach?

For reference (if people aren't familiar with the work already) Koch's team is mostly doing experiments... and seems to be somewhat close to having mice that have genes knocked out so that they "logically would seem" to lack certain kinds of qualia that normal mice "logically would seem" to have. Tononi collaborates with him and has proposed a way to examine a thing that computes and calculates that thing's "amount of consciousness" using a framework he called Integrated Information Theory. I have not sat down and fully worked out the details of IIT such that I could explain it to a patient undergrad at a chalkboard, but the reputation of the people involved is positive (I've seen Koch's dog and pony show a few times and it has improved substantially over the years and he is pimping Tononi pretty effectively)... basically the content "smells promising" but I'm hoping I can hear someone else's well informed opinion to see if I should spend more time on it.

Also, it seems to be relevant to this philosophic discussion? Or not? That's what I'm wondering. Opinions appreciated :-)

To summarize (mostly for my sake so I know I haven't misunderstood the OP):

• 1.) Subjective conscious experience or qualia play a non-negligible role in how we behave and how we form our beliefs, especially of the mushy (technical term) variety that ethical reasoning is so bound up in.
• 2.) The current popular computational flavor of philosophy of mind has inadequately addressed qualia in your eyes because the universality of the extended church-turing thesis, though satisfactorily covering the mechanistic descriptions of matter in a way that provides for emulation of the physical dynamics, does not tell us anything about what things would have subjective conscious experiences.
• 3.) Features of quantum mechanics such as entanglement and topological structures in a relativistic quantum field provide a better ontological foundation for your speculative theories of consciousness which takes as inspiration phenomenology and a quantum mondadology.

EDIT: I guess the shortest synopsis of this whole argument is: we need to build qualia machines, not just intelligent machines, and we don't have any theories yet to help us do that (other than the normal, but delightful, 9 month process we currently use). I can very much agree with #1. Now, with #2, it is true that the explanatory gap of qualia does not yield to the computational descriptions of physical processes, but it is also true that the universe may just be constructed such that this computational description is the best we can get and we will just have to accept that qualia will be experienced by those computational systems that are organized in particular ways, the brain being one arrangement of such systems. And, for #3, without more information about your theory, I don't see how appealing to ontologically deeper physical processes would get you any further in explaining qualia, you need to give us more.

One question I had reading this is: What does it matter if our model of human consciousness is wrong? If we create FAI that has all of the outward functionality of consciousness I still would consider that a win. Not all eyes that have evolved are human eyes; the same could happen with consciousness. If we manufactured some mechanical "eye" that didn't model exactly the interior bits of a human eye but was still able to do what eyes do, shouldn't we still consider this an eye? It would seem nonsensical to me to question whether this mechanical eye "really" sees because the act of seeing is a subjective experience that can't be truly modeled or say that it's not "real" seeing because it's computational seeing.

which in its biophysical context would appear to be describing a mass of entangled electrons in that hypothetical sweet spot, somewhere in the brain, where there's a mechanism to protect against decoherence.

Which activity totally inaccessible to state machines do you think these electrons do?

I might be mistaken, but it seems like you're forwarding a theory of consciousness, as opposed to a theory of intelligence.

Two issues with that - first, that's not necessarily the goal of AI research. Second, you're evaluating consciousness, or possibly intelligence, from the inside, rather than the outside.

If everything comes together, then it will now be a straight line from here to the end.

To the end of what? The sequence? Or the humanity as we know it?

You need to find the true ontology, find the true morality, and win the intelligence race. For example, if your Friendly AI was to be an expected utility maximizer, it would need to model the world correctly ("true ontology"), value the world correctly ("true morality"), and it would need to outsmart its opponents ("win the intelligence race").

Is there one "true" ontology/morality? Most likely there are many, leading in different directions, not necessarily easy to rank based on our current morality.

Personally, I am not overly worried that SI will produce anything resembling a working AGI, let alone a self-improving one, on any kind of reasonable time frame, and there is little cause for concern at this point, and definitely no rush to fix any deep conceptual issues you think SI might have. The best outcome one can hope for is that the SI research will produce some interesting results and gets a certain amount of respect from the CS/AI crowd in general. Formulating and proving a theorem or two could be one of those results. Maybe an insight into machine learning.

I am not at all concerned that they would lock a "wrong" epistemology (MWI/Tegmark, whatever) into a self-improving AGI, partly because I think that algorithms for unnecessarily complicated models will be simplified, either externally by the programmers or internally by the machine, into something that better reflects the interaction with the external world.

A more likely outcome is that they hit an unexpected road block fairly quickly, one that makes them go back to the drawing board and reevaluate the basics.

Phenomenal experience does not give a non-theoretical access to existence claims. Qualia are theoretical tools of theories implemented as (parts of) minds. I do not (go as far as) posit "computational epistemology", just provide a constraint on ontology.

First, "ontology".

This makes me think you're going to next talk about your second objection, to "outsourcing the methodological and design problems of FAI research to uploaded researchers and/or a proto-FAI which is simulating or modeling human researchers", but you never mention that again. Was that intentional or did you forget? Do you have any new thoughts on that since the discussions in Extrapolating values without outsourcing?

OK.

In all seriousness, there's a lot you're saying that seems contradictory at first glance. A few snippets:

My message for friendly AI researchers is not that computational epistemology is invalid, or that it's wrong to think about the mind as a state machine, just that all that isn't the full story.

If computation epistemology is not the full story, if true epistemology for a conscious being is "something more", then you are saying that it is so incomplete as to be invalid. (Doesn't Searle hold similar beliefs, along the lines of "consciousness is something that brain matter does"? No uploading for you two!)

I do not think that SI intends to just program its AI with an apriori belief in the Everett multiverse

I'm not sure you appreciate the distance to go "just" in regards of provable friendliness theory, let alone a workable foundation of strong AI, a large scientific field in its own right.

The question of which "apriori beliefs" are supposed to be programmed or not programmed in the AI is so far off as to be irrelevant.

Also note that if it those beliefs turn out not be an invariant in respect to friendliness (and why should they?), they are going to be updated until they converge towards more accurate beliefs anyways.

For example, if your Friendly AI was to be an expected utility maximizer, it would need to model the world correctly ("true ontology"), value the world correctly ("true morality"), and it would need to outsmart its opponents ("win the intelligence race").

"Ontology + morals" corresponds to "model of the current state of the world + actions to change it", and the efficiency of those actions equals "intelligence". An agent's intelligence is thus an emergent property of being able to manipulate the world in accordance to your morals, i.e. it is not an additional property but is inherent in your so-called "true ontology".

Still upvoted.

Upvoted even though I only made it to the part about MWI dogmatism.

I hate to be that guy, but have you read the reductionism sequence? snerk

In all seriousness though, I think you're exactly right to question the underlying philosophical premises upon which SI is proceeding, especially regarding epistemology.

I just think the particular parts that you're questioning are mostly the wrong parts, and that, at best, you're only vaguely poking in the direction of where some actual issues may be.