What if Strong AI is just not possible?

I guess it tends to create physical structures that are simple, but I think the computational stuff tends to be weird. If you have a strand of DNA, the only way to tell what kind of chemistry that will result in is to run it. From what little I've heard, it sounds like any sort of program made by a genetic algorithm that can actually run is too crazy to understand. For example, I've heard of a set of transistors hooked together to be able to tell "yes" and "no" apart, or something like that. There were transistors that were just draining energy, but were vital. Running it on another set of transistors wouldn't work. It required the exact specs of those transistors. That being said, the sort of sources I hear that from are also the kind that say ridiculous things about quantum physics, so I guess I'll need an expert to tell me if that's true.

Has anyone here studied evolved computers?

Thanks!

Exponential search spaces are completely typical for NP problems.

Even many "P problems" have an exponential search space. For instance an n-digit number has exp(n) many potential divisors, but there is a polynomial-in-n-time algorithm to verify primality.

"Reverse engineer" is "there is a machine that seems to find needles in haystacks quickly. It has loops of copper wire, and plugs into a wall socket. Can you copy it and build another one?"

I admit that there are some "reverse-engineering" problems that are easy.

I have an EE professor friend who is working on making it harder to reverse engineer computer chips.

Why do you think that linking brains together directly would be so much more effective than email?

The most obvious answer would be speed. If you can simulate 1,000,000 brains at, say, 1,000 times the speed they would normally operate, the bottleneck becomes communication between nodes.

You don't need to restrict yourself to a 3d topology. Supercomputers with hundreds of thousands of cores can and do use e.g. 6D topologies. It seems that a far more efficient way to organize the brains would be how organizations work in real life: hierarchical structure, where each node is at most O(log n) steps away from any other node.

That's a good argument. What you're basically saying is that the design of the human brain occupies a sort of hill in design space that is very hard to climb out of. Now, if the utility function is "Survive as a hunter-gatherer in sub-saharan Africa," that is a very reasonable (heck, a very likely) possibility. But evolution hasn't optimized us for doing stuff like designing algorithms and so forth. If you change the utility function to "Design superintelligence", then the landscape changes, and hills start to look like valleys and so on. What I'm saying is that there's no reason to think that we're even at a local optimum for "design a superintelligence".

I think a human cognitive bias is to think that something about which we have a coherent idea is coherent in implementation. As an engineer, I think that this is a bias that is clearly wrong. A well designed smartphone, especially an Apple product, appears quite coherent, it appears "right." There is a consistency to its UI, to what a swipe or a back press or whatever does in one app and in another. The consistency in how it appears causes the human to think the consistency must be built in, that the design of such a consistent thing must be somehow SIMPLER than the design of a complex and inconsistent thing.

But it is not. It is much easier to design a user interface which is a mess, which has a radio button to enter one mode, but a drop down menu for another and a spinner for yet another. It is pure high-level skull sweat that removes these inconsistencies and builds a system which appears consistent at a high level.

And so it is with our brains and our intelligence. What we see and what we hear and what we carry around as an internal model of the world all agree not because there is some single simple neurology that gives that result, but because our brains are complex and have been fine tuned over millions of years to give agreement between these various functionalities, giving the appearance of a simple consistency, but through the agency of a complex tweaking of different features of hearing and vision and memory.

And in the midst of all this, we extend our brains beyond what they ever evolved for, to semiconductor chip design, the proof of mathematical theorems, the writing of symphonies and sonnets. Giving the appearance that there is some simple thing one might call "General Intelligence" of which we have a few large dollops.

But I think what we really have is a complex mix of separate analytical tools, and that it turns out that these tools span so many different ways of thinking about things that they can successfully be adapted to all sorts of things they were not originally designed (evolved) for. That is, they are a bunch of hacks, but not necessarily a complete set of hacks, if any such an idea as a complete set making up a general intelligence could even be defined.

So we use the tools we have, and enough animals die that we stay fed, and enough chips work that we have smart phones, and so we think we possess some "simple" general intelligence.

Or put concisely, what Baughn says in his comment at this same level.

A bundle of widely but not universally applicable tricks?

The anthropic principle selects certain universes out of all possible ones. In this case, we can only exist in the subset of them which admits humans but prohibits strong AI. You have to first subscribe to a version of many worlds to apply it, not sure if you do. Whether the idea of anthropic selection is a useful one still remains to be seen.

Most if of probably it is (under the assumption) but observers such as us only exist in the part free from of strong AI. If strong AI spreads out at the speed of light, observers such as us won't be able to detect it.

True, but given the assumptions, it would be evidence for the fact that there are none that have come in physical contact with the story-world (or else they would be dead).

Saying they would have created a human adds no information; worse, it adds noise in the form of whatever ideas you're trying to sneak into the discussion by saying this, or in the form of whatever any reader might misinterpret from using this label.

You haven't even made the claim that "The set of humans minds might possibly be outside of the set of possible Strong AI minds", so your argument isn't even about whether or not "Strong AIs" includes "Humans".

Basically, I was peeve-twitching because you're turning the whole thing into a pointless argument about words. And now you've caused me the inconvenience of writing this response. Backtrack: Hence the twitching.

If so, then the explanation proposed by Lalartu won't hold water with the target audience, i.e. the subset of humans who don't happen to hold that idea for granted.

If it's not, and the audience includes general-muggle-population in any non-accidental capacity, then it's worth pointing out that the majority of people accept the idea for granted, and thus that that subset of the target audience would take this explanation in stride.

Either way, the issue is relevant.

Mostly, I just wanted to respond to the emotionally-surprising assertion that they'd never cognizantly encountered this view.

OK like comparing apples and oranges. We wind up with apples AND oranges through similar mechanisms ni carbon and oxygen after 100s of millions of years of evolution, but we seriously consider we can't get there with design in silicon after less than 100 years of trying while watching the quality of our tools for getting there doubling every 5 years or so?

I'm not saying it HAS to happen. I'm just saying the smart bet is not against it happening.

Then we'll pick one of the methods that does. Evolution only finds local maximums. It's unlikely that it hit upon the global maximum.

Even on the off chance that it did, we can still improve upon the current method. Humans have only just evolved civilization. We could improve with more time.

Even if we're at the ideal for our ancestral environment, our environment has changed. Being fluent in a programming language was never useful before, but it is now. It used to be hard to find enough calories to sustain the brain. That is no longer a problem.