AI is not enough

There is an ambiguity in how people are using the world "algorithm"

Algorithm-1 is a technique that provably accomplishes some identifiable task X, perhaps with a known allowance for error. This is mostly what we talk about in computer science as "algorithms". Algorithm-2 is any process that can be described and that sometimes terminates. It might not come with any guarantees, it might not come with any clear objectives. A heuristic is an example of Algorithm-2.

Note that this distinction is observer-dependent. Unintelligible code is Algorithm-2, but it becomes Algorithm-1 when you learn what it does and why it works.

Human intelligence is an example of Algorithm-2 and not an example of Algorithm-1 for our purposes.

Machines can do both Algorithm-1 and Algorithm-2.

As near as I can tell, you are highlighting the fact that we don't have an Algorithm-1 for AI design. But that doesn't mean there isn't an Algorithm-2 that accomplishes it and doesn't mean we won't find that Algorithm-2.

Note that an Algorithm-2 can output an Algorithm-1 and a proof/explanation of it. There's nothing stopping us from building a giant-and-horrible combination of theorem prover, sandbox, code mutator, etc, that takes as input a spec, and might at some point output an algorithm, with a proof, that meets the spec.

I'm afraid just about everything here is wrong.

at some point we need something fundamentally non-algorithmic

No. Our brains are already implementing lots of algorithms. So far as we know, anything human beings come up with -- however creative -- is in some sense the product of algorithms. I suppose you could go further back -- evolution, biochemistry, fundamental physics -- but (1) it's hard to see how those could actually be relevant here and (2) as it happens, so far as we know those are all ultimately algorithmic too.

we will always have to rely on some non-algorithmical intelligence to find more intelligent solutions.

No (not even if you were right about ultimately needing something fundamentally non-algorithmic). Suppose you have some initial magic non-algorithmic step where the Finger of God implants intelligence into something (a computer, a human being, whatever). After that, that intelligent thing can design more intelligent things which design more intelligent things, etc. The alleged requirement to avoid an infinite regress is satisfied by that initial Finger-of-God step, even if everything after that is algorithmic. There's no reason to think that continued non-algorithmic stuff is called for.

we have no reason to suppose we can't find another more powerful one.

That might be true. It might even be true -- though I don't think you've given coherent reasons to think so -- that there'll always be a possible Next Big Thing that can't be found algorithmically. So what? A superintelligent AI isn't any less useful, or any less dangerous, merely because a magical new-AI-creating process might be able to create an even more superintelligent AI.

No algorithm can determine the simple axioms of the natural numbers from anything weaker.

It is not clear that this means anything. You certainly have given no reasons to believe it.

There is simply no way to derive the axioms from anything that doesn't already include it.

I think you are confusing derivations within some formal system such as Peano arithmetic (where, indeed, the only way to get the axioms is to begin with them, or with some other axioms that imply them) and (a quite different sort of) derivations outside that formal system, such as whatever Peano did to arrive at his axioms. I know of no reason to believe that the latter is fundamentally non-algorithmic, though for sure we don't know what algorithms would be best.

general intelligence necessarily has to transcend rules

I know of no reason to believe this, and it seems to me that if it seems true it's because what you imagine when you think about following rules is very simple rule-following, the sort of thing that might be done by a computer program at most a few pages in length running on a rather slow computer. In particular ...

since at the very least the rules can't be determined by rules

Whyever not? They have to be different rules, that's all.

Instead, we should expect a singularity that happens due to emergent intelligence.

"Emergence" is not magic.

not just one particular kind of intelligence like formal reasoning used by computers

Well, that might well be correct, in the sense that good paths to AI might well involve plenty of things that aren't best thought of as "formal reasoning". (Though, if they run on conventional computers, they will be equivalent in some sense to monstrously complicated systems of formal reasoning.)

You are assuming a concept of "algorithm" wide enough that everything an AI could do is considered an algorithm and simultaneously narrow enough that things humans do that don't seem obviously algorithmic don't count as algorithms, so you are simply begging the question. The concept of algorithm you use already implies that there are things a human can do and an AI can't right from the start.

I'm having trouble thinking of methods of problem solving that are non-algorithmic. Can anyone provide me with an example?

while(nothing else is a higher priority than this problem) {  
   stare at the problem and try to understand it  
   search my past memories for something related   // neural nets are good at this  
   for each relevant past memory,   
      try to apply a relevant technique that worked in the past  
      evaluate the result.   
      if it looks like progress  
         declare this to be the new version of the problem  

An interesting test of what appear to be general arguments against something

is whether you can use them to prove the opposite as well.

Clearly, if you want to use something non-algorithmic to solve a problem (say, if you're a human, which for the sake of argument we will pretend are "non-algorithmic"), you have to get that non-algorithm somehow. But then we have the problem to determine the non-algorithm to find that, and so on...
Obviously, at some point we have to actually find a non-algorithm to start with, so in any case at some point we need something fundamentally algorithmic to determine a solution to an problem that is solveable by a non-algorithm.

I think this post is filled with conceptual confusion, and I find it fascinating.

You never quite state the alternative to an algorithm. I propose that the only alternative is randomness. All processes in the universe are algorithms with elements of randomness.

I'm curious as to what you think the human brain does, if not an algorithm. I, like many on LW, believe the human brain can be simulated by a Turing machine (possible with a random generator). Concepts like "heuristics" or "intuition" or exploration" are algorithms with random elements. There is a lot of history on formalizing processes, and nothing known lies outside Turing machines (see the Church-Turing thesis).

In addition, I think the human brain is a lot more algorithmic than you think it is. A lot of Lukeprog's writings on neuroscience and psychology demonstrate ways in which our natural thoughts or intuitions are quite predictable.

at some point we have to actually find an algorithm to start with

The universe started with the laws of physics (which are known to be algorithms possibly with a random generator), and have run that single algorithm up to the present day.

What do you think about my proposed algorithm/random dichotomy?

Based on your comments, you are clearly an atheist, and therefore reject the argument of God existing because there has to be an uncaused cause.

Yet, your uncaused algorithm argument takes the exact same form. Isn't it the same counterargument?

My argument still holds in another form, though. Even if we assume the universe has a preexisting algorithm that just unfolds, we don't know which it is. So we can't determine the best seed AI from that either, effectively we still have to start from b). Unless we get the best seed AI by accident (which seems unlikely to me) there will be room for a better seed AI which can only determined if we start with a totally new algorithm (which the original seed AI is unable to, since then it would have to delete itself). We, having the benefit of not knowing our algorithm, can built a better seed AI which the old seed AI couldn't built because it already has a known algorithm it must necessarily build on.

Indeed, a good seed AI would at some point suggest us to try another seed AI, because it infers that its original code is unlikely to be the best possible at optimal self-modification. Or it would say "Delete this part of my source code and rewrite it, it doesn't seem optimal to me, but I can't rewrite it because I can't modify this part without destroying myself or basing the modification on the very part that I want to fundamentally rewrite".

I see in which case my argument fails:

If we assume a prexisting algorithm for the universe already (which most people here seem to do), then everything else could be derived from that, including all axioms of natural numbers, since we assume the algorithm to be more powerful then them at the start. Step b) is simply postulated to be already fulfilled, with the algorithm just being there (and "just being there" is not an algorithm), so that we already have an algorithm to start with (the laws of nature).

The "laws of nature" simply have to be taken as granted. There is nothing deeper than that. The problem is that then we face a universe which is essentially abitrary, since from what we know the laws could be anything abitrary else as well (these laws would just be the way they are, too). But this is obviously not true. The laws of nature are not abitrary, there is a deeper order in them which can't stem from any algorithm (since this would just be another abitrary algorithm). But if this is the case, I believe we have no reason to suppose that this order just stopped working and lets do one algorithm all the rest. We would rather expect it to be still continuously active. Then the argument works. The non-algorithmic order can always yield more powerful seed AIs (since there is no single most powerful algorithm, I think we can agree on that), so that AI is not sufficient for an ever increasing general intelligence.

So we face a problem of worldview here, which really is independent of the argument and this is maybe not the right place to argue it (if it is even useful to discuss it, I am not sure about that, either).

What I write here may be quite simple (and I am certainly not the first to write about it)

Feel free to post relevant links, then. Until then, downvoted.

Any bitstring to a certain length N, can be produced.

Some of them are algorithms. Some can be proved with a rigor. Some can be tested statistically.

I don't see your point as valid.

I like this post more for the responses than for the argument. It's obvious that a lot of people here mean something by "algorithm" that is so general as to be useless: they mean any kind of (deterministic) procedure or process at all. Algorithms are effective procedures for calculation. Equations, laws, theories, and so forth, are not algorithms (note that algorithms are procedures and not descriptions). The sort of argument you're making was common to early discussions of AI. The idea that everything a person can do can be explained algorithmically is really more a presupposition of AI rather than something it actively argues for. The idea is that the presupposition will be vindicated by the creation of AI programs that show the disputed properties.