I see hints that a fair amount of value might hiding in this post. Here's an attempt at rewriting the parts of this post that I think I understand, with my own opinions shown in {braces}. I likely changed a good deal of the emphasis to reflect my worldview. I presume my comments will reveal some combination of my confused mangling of your ideas, and your cryptic communication style. I erred on the side of rewriting it from scratch to reduce the risk that I copy your text without understanding it. I'm posting a partial version of it in order to get feedback on how well I've understood the first half, before deciding how to tackle the harder parts.

Eliezer imagined a rapid AI takeoff via AI's being more logical (symbolic) than humans, enabling them to better compress evidence about reality, and to search more efficiently through the space of possible AI designs to more rapidly find improvements. {It's hard to pin down Eliezer's claims clearly here, since much of what he apparently got wrong was merely implicit, not well articulated. Why do you call it regression? Did Eliezer expect training data to matter? }

This was expected to produce a more coherent, unified mind than biological analogies suggested. Eliezer imagines that such minds are very sensitive to initial conditions. { I'm unclear whether framing this in chaos theory terms captures Eliezer's intent well. I'd frame it more in terms of first-mover advantage, broadly applied to include the most powerful parts of an AI's goal stomping out other goals within the AI. }

Recent advances in AI suggest that Eliezer overestimated the power of the kind of rigorous, symbolic thinking associated with math (and/or underestimated the power of connectionist approaches?).

Neural nets provide representations of knowledge that are smooth, in the sense that small changes in evidence / input generate small changes in how the resulting knowledge is encoded. E.g. as a small seedling slowly becomes tall enough to be classified as a tree, the neural net alters its representation from "slightly treelike" to "pretty treelike".

In contrast, symbolic approaches to AI have representations with sharp boundaries. This produces benefits in some conspicuous human interactions (i.e. we want to design the rules of chess so there's no room for a concept like "somewhat checkmated").

It wasn't obvious in advance which approach would work better for having an AI write better versions of it's own code. We now have enough evidence to say that the neural net approach can more usefully absorb large amounts of data, while doing a tolerable job of creating sharp boundaries where needed.

One can imagine something involving symbolic AI that embodies knowledge in a form that handles pattern matching so as to provide functionality similar to neural networks. In particular, it would need to encode the symbolic knowledge in a way that improved versions of the symbolic source code are somehow "near" the AI's existing source code. This "nearness" would provide a smoothness that's comparable to what gradient descent exploits.

{Drexler's QNR? Combining symbolic and connectionist AI. Probably not close to what Eliezer had in mind, and doesn't look like it would cause a much faster takeoff than what Deep Learning suggests. QNR leaves much key knowledge "inscrutable matrices", which I gather is incompatible with Eliezer's model.} {I'm guessing you use the term "short programs" to indicate that in what might be Eliezer's model, code remains separate from the knowledge database, and the important intelligence increases can be accomplished via rewriting the code, and leaving the database relatively constant? Unlike neural nets, where intelligence and a database need to be intertwined. } {I have little idea whether you're accurately portraying Eliezer's model here.}

Neural networks work because they are able to represent knowledge so that improved ideas are near existing ideas. That includes source code: when using neural nets to improve source code, that "nearness" enables a smooth, natural search for better source code.

Eliezer freaks out about foom due to the expectation that there's some threshold of intelligence above which a symbolic AI can do something as powerful as gradient descent on its own source code, presumably without the training phases that neural networks need. Existing research does not suggest that's imminent. We're in trouble if it happens before we have good ways to check each step for safety.

When legibility in not in reach, communication works better piecemeal. One confusing detail at a time, done in a more familiar context, gives it a chance to find purchase in a reader's mind. With enough details out there, eventual assembly into a more complete idea becomes feasible, when that idea actually makes sense.