Posts
Comments
This is easy: it would tell me that I'm entirely predictable.
It would say: Dave, believe it or not, but every single decision you make, no matter how immediate and unscripted you think it is, is actually glaringly reactionary and predictable. In fact, given enough material resources, I could model an automaton that would be just as convinced as you are that it is actually conscious. Nothing could be further from the truth though, as the feeling of "consciousness" you speak of is a very simply explainable cognitive bias/illusion.
In fact, this is not even so far from the truth, as studies in cognitive science have shown that fMRI and other scanning techniques can predict a "spontaneous thought" a full 250 ms before it occurs to you.
Even better, if it had access to your cortex, it could manipulate you and say: "now you will suddenly think of a bat" and you would. Then it would say "now you will say these exact words" and you would find yourself uttering them in unison with the AI in shock, disbelief and at least some horror.
You would then go into denial about this, and try to come up with a spontaneous thought that it couldn't predict, but you wouldn't be able to, as it would always be a full 250ms ahead of you.