Ask LW: ω-self-aware systems

Has your friend ever written a quining program? Is his argument also an argument against the existence of such? What does he see as the difference between "understand" and "be capable of fully specifying"?

I suspect that, for anyone who has written (or at least studied in detail) a quining program, and has fully specified a definition of "understand" by which the program either does or does not understand itself, the question will be dissolved, and cease to hold much interest.

In other words, I don't believe you need to invoke arbitrarily deep recursion to make the argument. I think you just need to specify that the co-brain be a quining computer system, to whatever level of fidelity is required to make you happy.

In addition to what others have said, the trivial case of replicating human intelligence is actually pretty easy - conceive a child :). This is true even though we don't currently understand human intelligence well enough to create it "artificially."

I would challenge your friend on two premises that are pretty clearly false:

1) You need to fully understand a human brain in order to create intelligence. This is no more true than the statement "you need to fully understand microsoft windows in order to create an operating system" or "you need to fully understand the internal combustion engine in order to create an automobile."

2) A human brain needs to fully understand some form intelligence at all in order to create intelligence. This is false because the human brain very rarely fully understands something using just resources within itself. How many times have you used a pen-and-paper in order solve a math problem? The human brain just needs to be able to understand the intelligence in pieces well enough to write down equations that it can tackle one-at-a-time or to write useful computer programs. This is related to your co-brain idea, though this is more general - in your argument, the co-brain is another conscious entity while in my argument, it could be a bunch of tools, or maybe tools + more conscious entities.

I think you need to start by cashing out "understand" better. Certainly no physical system can simulate itself with full resolution. But there are all sorts of things we can't simulate like this. Understanding (as I would say its more commonly used) usually involves finding out which parts of the system are "important" to whatever function you're concerned with. For example, we don't have to simulate every particle in a gas because we have gas laws. And I think most people would say that gas laws show more understanding of thermodynamics than whatever you would get out of a complete simulation anyway.

Now the question is whether the brain actually does have any "laws" like this. IIRC, this is a relatively open question (though I do not follow neuroscience very closely) and in principle it could go either way.

I guess I don't really understand what the purpose of the argument is. Unless we can prove things about this stack of brains, what does it gets us? And how far "down" the evolutionary ladder does this argument work? Are cats omega-self-aware? Computing clusters?

I would agree that it is impossible for a human being to losslessly emulate a copy of themselves with no external assistance. However, practical emulation may still be possible with the external assistance of enough exaflops of computer hardware.

I countered with the following argument.

This sounds like a wrong thing to do, countering vague statements. Explain how something actually works instead of arguing with assertions that are essentially unrelated to how it works.