Giant (In)scrutable Matrices: (Maybe) the Best of All Possible Worlds

To the connectivists such as myself, your point 0 has seemed obvious for a while, so the EY/MIRI/LW anti-neural net groupthink was/is a strong sign of faulty beliefs [LW(p) · GW(p)]. And saying "oh but EY/etc didn't really think neural nets wouldn't work, they just thought other paradigms would be safer" doesn't really help much if no other paradigms ever had a chance. Underlying much of the rationalist groupthink on AI safety is a set of correlated incorrect anti-connectivist beliefs which undermines much of the standard conclusions [LW · GW].

Mm, I fear this argument is self-contradictory to a significant extent.

Interpretability is premised on the idea that it's possible to reduce a "connectionist" system to a more abstract, formalized representation.

• Consider the successful interpretation of a curve detector. Once we know what function a bunch of neurons implements, we can tear out these neurons, implement that function in a high-level programming language, then splice that high-level implementation into the NN in place of the initial bunch-of-neurons. If the interpretation is correct, the NN's behavior won't change.
• Scaling this trick up, the "full" interpretation of a NN should allow us to re-implement the entire network in high-level-programming manner; I think Neel Nanda even did something similar here [LW · GW].
• Redwood Research's outline here [LW · GW] agrees with this view. An "interpretation" of a NN is basically a transform of the initial weights-and-biases computational graph into a second, "simpler" higher-level computational graph.

So inasmuch as interpretability is possible, it implies the ability to transform a connectionist system into what's basically GOFAI. And if we grant that, it's entirely coherent to wish that we've followed the tech-development path where we're directly figuring out how to build advanced AI in a high-level manner. Instead, we're going about it in a round-about fashion: we're generating incomprehensible black-boxes whose internals we're then trying to translate into high-level representations.

The place where "connectionism" does outperform higher-level approaches is "blind development". If we don't know how the algorithm we want is supposed to work, only know what it should do, then such approaches may indeed be absolutely necessary. And in that view, it should be clear why evolution never stumbled on anything else: it has no idea what it's doing, so of course it'd favour algorithms that work even if you have no idea what you're doing.

(Though I don't think brains' uniformity is entirely downstream even of that. Computers can also be viewed as "a massive number of simple, uniform units connected to each other", the units being transistors. Which suggests that it's more of a requirement for general-purpose computational systems, imposed by the constraints of our reductionist universe. Not a constraint on the architecture of specifically intelligent systems.

Perhaps all good computational substrates have this architecture; but the software that's implemented on these substrates doesn't have to. And in the case of AI, the need for connectionism is already fulfilled by transistors, so AI should in principle be implementable via high-level programming that's understandable by humans. There's no absolute need for a second layer of connectionism in the form of NNs.)

I agree with your second point though, that complexity is largely the feature of problem domains, not agents navigating them. Agents' policy functions are likely very simple, compared to agents' world-models.

I feel like this is missing the point. To me, when I heard the description of neural nets as large hard-to-interpret matrices, this clicked with me. Why? Because I'd spent years studying the brain. It's not brain weights that make them more interpretable as systems. Heck no. All the points made about how biological networks are bad for interpretability are quite on point. The reason I think brains are more interpretable at a system level is for two reasons:

1. Modularity Look at the substantia nigra in the brain. That's a module. It performs a critical, specific, hardwired task. The cortex is somewhat rewirable, but not very. And these modules are mostly located in the same places and performing mostly the same operations between people. Even between different species!
2. Limited rewirability The patterns of function in a brain get much more locked in than in a neural network. If you were trying to hide your fear from a neuro scientist who had the ability to monitor your brain activity with implanted electrodes, could you do it? I think you couldn't, even with practice. You could train yourself to feel less fear, but you couldn't experience the same amount of fear but hide the activity in a different part of your brain. Even if you also got to monitor your own brain activity and practice. I just don't think that the brain is rewirable enough to manage that. Some stuff you could move or obfuscate, a bit, but it would be really hard and only partially successful.

So when people like Conjecture talk about building Cognitive Emulations that have highly verifiable functionality... I think that's an awesome idea! I think that will work! (If given enough time and resources, which I fear it won't be) And i think they should use big matrices of numbers to build them, but those matrices should be connected up to each other in specific hardcoded ways with deliberate information bottlenecks. I do not endorse spiking neural nets as inherently more interpretable. I endorse highly modular systems with deliberate restrictions on their ability to change their wiring.

The frequently accompanying, action-relevant claim -- that substantially easier-to-interpret alternatives exist -- is probably false and distracts people with fake options. That's my main thesis.

I agree with this claim (anything inherently interpretable in the conventional seems totally doomed). I do want to push back on an implicit vibe of "these models are hard to interpret because of the domain, not because of the structure" though - interpretability is really fucking hard! It's possible, but these models are weird and cursed and rife with bullshit like superposition, it's hard to figure out how to break them down into interpretable units, they're full of illusions, and confused and janky representations, etc.

I don't really have a coherent and rigorous argument here, I just want to push back on the vibe I got from your "interpretability has many wins" section - it's really hard!

Related: Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc [LW · GW] and some good discussion in its comments section.

First of all, I strongly agree that intelligence requires (or is exponentially easier to develop as) connectionist systems. However, I think that while big, inscrutable matrices may be unavoidable, there is plenty of room to make models more interpretable at an architectural level.

Well, I ask you -- do you think any other ML model, trained over the domain of all human text, with sufficient success to reach GPT-4 level perplexity, would turn out to be simpler?

I have long thought that Transformer models are actually too general purpose for their own good. By that I mean that the $O\left(n^2\right)$ operations they do, using all-to-all token comparisons for self-attention, is actually extreme overkill for what an LLM needs to do.

Sure, you can use this architecture for moving tokens around and building implicit parse trees and semantic maps and a bunch of other things, but all these functions are jumbled together in the same operations and are really hard to tease out. Recurrent models with well-partitioned internal states and disentangled token operations could probably do more with less. Sure, you can build a computer in Conway's Game of Life (which is Turing-complete), but using a von Neumann architecture would be much easier to work with.

Embedded within Transformer circuits, you can find implicit representations of world models [LW · GW], but you could do even better [LW · GW] from an interpretability standpoint by making such maps explicit. Give an AI a mental scratchpad that it depends on for reasoning (DALL-E, Stable Diffusion, etc. sort of do this already, except that the mental scratchpad is the output of the model [an image] rather than an internal map of conceptual/planning space), and you can probe that directly to see what the AI is thinking about.

Real brains tend to be highly modular, as Nathan Helm-Burger pointed out. The cortex maps out different spaces (visual, somatosensory, conceptual, etc.). The basal ganglia perform action selection and general information routing. The cerebellum fine-tunes top-down control signals. Various nuclei control global and local neuromodulation. And so on. I would argue that such modular constraints actually made it easier for evolution to explore the space of possible cognitive architectures.

I like this post overall. I'd guess that connectionism is the real-world way to do things, but I don't think I'm quite as confident that there aren't alternatives. I basically agree with the main points, though.

to mind control a [maze-solving agent] to pursue whatever you want with just a single activation

I want to flag that this isn't quite what we demonstrated. It's overselling our results a bit. Specifically, we can't get the maze-solving agent to go "wherever we want" via the demonstrated method. Rather, in many mazes, I'd estimate we can retarget the agent to end up in about half of the maze locations. Probably a slight edit would fix this, like changing "whatever you want" to "a huge range of goals."

• Model-based RL is more naturally interpretable than end-to-end-trained systems because there’s a data structure called “world-model” and a data structure called “value function”, and maybe each of those data structures is individually inscrutable, but that’s still a step in the right direction compared to having them mixed together into just one data structure. For example, it’s central to this proposal [LW · GW].
• More generally, we don’t really know much about what other kinds of hooks and modularity can be put into a realistic AI. You say “probably not possible” but I don’t think the “probably” is warranted. Evolution wasn’t going for human-interpretability, so we just don’t know either way. I would have said “We should be open to the possibility that giant inscrutable matrices are the least-bad of all possible worlds” or something.
• If Architecture A can represent the same capabilities as Architecture B with fewer unlabeled nodes (and maybe a richer space of relationships between nodes), then that’s a step in the right direction.
• I think you’re saying that “asynchronous” neural networks (like in biology) are more inscrutable than “synchronous” matrix multiplication, but I don’t think that claim is really based on anything, except for your intuitions, but your intuitions are biased by the fact that you’re never tried to interpret an “asynchronous” neural network, which in turn is closely tied to the fact that nobody knows how to program one that works. Actually, my hunch is that the asynchronicity is an implementation detail that could be easily abstracted away.
• If we think of interpretability as a UI into the trained model, then the problem is really to simultaneously co-design a learning algorithm & interpretability approach that work together and dynamically scale up to sufficient AI intelligence. I think you would describe success at that design problem as “Ha! The inscrutable matrix approach worked after all!”, and that Eliezer would describe success at that same design problem as “Ha! We figured out a way to build AI without giant inscrutable matrices!” (The matrices are giant but not inscrutable.)

On the downside, if matrix multiplication is the absolute simplest, most interpretable AI blueprint there is -- if it's surrounded on all sides by asynchronous non-digital biological-style designs, or other equally weird alien architectures -- that sounds like pretty bad news. Instead of hoping that we luck out and get a simpler, more interpretable architecture in future generations of more-powerful AI, we will be more likely to switch to something much more inscrutable, perhaps at just the wrong time. (ie, maybe matrix-based AIs help design more-powerful successor systems that are less interpretable and more alien to us.)

Yup agreed, all intelligence must be connectionist; but I think you can go further than empirical and claim that getting ahead of "vanilla" connectionism is possible only by other things that are, themselves, also connectionism. In other words, the interesting paradigms are the subsets of connectionism. After all, category theory is connectionist too.

1. Please enable this for promotion to the front page

2. Is this proof that intelligence must be connectionist? Of course not.

Well it seems to be strong evidence that intelligence is connectionist; "absence of evidence is evidence of absence [? · GW]".

Current NN matrices are dense and continuous weighted. A significant part of the difficulty of interpretability is that they have all to all connections; it is difficult to verify that one activation does or does not affect another activation.

However we can quantize the weights to 3 bit and then we can probably melt the whole thing into pure combinational logic. While I am not entirely confident that this form is strictly better from an interpretability perspective, it is differently difficult.

"Giant inscrutable matrices" are probably not the final form of current NNs, we can potentially turn them into different and nicer form.

Well, I ask you -- do you think any other ML model, trained over the domain of all human text, with sufficient success to reach GPT-4 level perplexity, would turn out to be simpler?

If we're literally considering a universal quantifier over the set of all possible ML models, then I'd think it extremely likely that there does exist a simpler model with perplexity no worse than GPT-4. I'm confused as to how you (seem to have) arrived at the opposite conclusion.

Imagine an astronomer in the year 1600, who frequently refers to the "giant inscrutable movements" of the stars. [...]

I think the analogy to {building intelligent systems} is unclear/weak. There seem to be many disanalogies:

In the astronomy case, we have

• A phenomenon (the stars and their motions) that we cannot affect

• That phenomenon is describable with simple laws.

• The "piles of numbers" are detailed measurements of that phenomenon.

• It is useful to take more measurements, doing so is helpful for finding the simple laws.

In the {building AIs} case, we have

• A phenomenon (intelligence) which we can choose to implement in different ways, and which we want to harness for some purpose.

• That phenomenon is probably not entirely implementable with simple programs. (As you point out yourself.)

• The "piles of numbers" are the means by which some people are implementing the thing; as opposed to measurements of the one and only way the thing is implemented.

• And so: implementing AIs as piles-of-numbers is not clearly helpful (and might be harmful) to finding better/simpler alternatives.

So, I predict with high confidence that any ML model that can reach the perplexity levels of Transformers will also present great initial interpretive difficulty.

I do agree that any realistic ML model that achieves GPT-4-level perplexity would probably have to have at least some parts that are hard to interpret. However, I believe it should (in principle) be possible to build ML systems that have highly interpretable policies (or analogues thereof), despite having hard-to-interpret models.

I think if our goal was to build understandable/controllable/safe AI, then it would make sense to factor the AI's mind into various "parts", such as e.g. a policy, a set of models, and a (set of sub-)goals.

In contrast, implementing AIs as giant Transformers precludes making architectural distinctions between any such "parts"; the whole AI is in a(n architectural) sense one big uniform soup. Giant Transformers don't even have the level of modularity of biological brains designed by evolution.

Consequently, I still think the "giant inscrutable tensors"-approach to building AIs is terrible from a safety perspective, not only in an absolute sense, but also in a relative sense (relative to saner approaches that I can see).

Tentative GPT4's summary. This is part of an experiment. 
Up/Downvote "Overall" if the summary is useful/harmful.
Up/Downvote "Agreement" if the summary is correct/wrong.
If so, please let me know why you think this is harmful. 
(OpenAI doesn't use customers' data anymore for training, and this API account previously opted out of data retention)

TLDR:
The article argues that deep learning models based on giant stochastic gradient descent (SGD)-trained matrices might be the most interpretable approach to general intelligence, given what we currently know. The author claims that seeking more easily interpretable alternatives could be misguided and distract us from practical efforts towards AI safety.

Arguments:
1. Generally intelligent systems might inherently require a connectionist approach.
2. Among known connectionist systems, synchronous matrix operations are the most interpretable.
3. The hard-to-interpret part of matrices comes from the domain they train on and not their structure.
4. Inscrutability is a feature of our minds and not the world, so talking about "giant inscrutable matrices" promotes unclear thought.

Takeaways:
1. Deep learning models' inscrutability may stem from their complex training domain, rather than their structure.
2. Synchronous matrix operations appear to be the easiest-to-understand, known approach for building generally intelligent systems.
3. We should not be seeking alternative, easier-to-interpret paradigms that might distract us from practical AI safety efforts.

Strengths:
1. The author provides convincing examples from the real world, such as the evolution of brain sizes in various species, to argue that connectionism is a plausible route to general intelligence.
2. The argument that synchronous matrix operations are more interpretable than their alternatives, such as biologically inspired approaches, is well-supported.
3. The discussion on inscrutability emphasizes that our understanding of a phenomenon should focus on its underlying mechanisms, rather than being misled by language and intuition.

Weaknesses:
1. Some arguments, such as the claim that ML models' inscrutability is due to their training domain and not their structure, are less certain and based on the assumption that the phenomenon will extend to other models.
2. The arguments presented are ultimately speculative and not based on proven theories.

Interactions:
1. The content of this article may interact with concepts in AI interpretability, such as feature importance and attribution, which mehtods aim to improve our understanding of AI models.

Factual mistakes:
I am not aware of factual mistakes in my summary.

Missing arguments:
1. The article does not address how any improvements in interpretability would affect AI alignment efforts or the risks associated with AGI.
2. The article does not explore other potential interpretability approaches that could complement or augment the synchronous matrix operations paradigm.

A couple points. Your bit about connectivism IMO ignores the fact that biological systems have their own constraints. If you looked at animals and guessed that the most energy efficient, simple to maintain way to move on most terrains is probably legs, you'd be wrong, it's wheels or threads. But legs are what organic life that needs to keep all its moving parts connected could come up with.

A similar problem might be at work with connectivism. We designed MLPs in the first place by drawing inspiration from neurons, so we were kinda biased. As a general rule, we know that, if made arbitrarily large, they're essentially universal functions. Any universal function ought to be potentially equivalent; you could make it a crazily high dimensional spline or what have you, the point is you have a domain and a codomain and a bunch of parameters you can tune to make that function fit your data as closely as possible. What determines the choice is then practicality of both computation and parameter fitting. Connected systems we use in ML happen to be very convenient for this, but I'd be fairly surprised if that was the same reason why biology uses them. It would mean our brain does gradient descent and backpropagation too. And for ML too there might be better choices that we just haven't discovered yet.

That said, I don't think connected systems are necessarily a bad paradigm. I agree with you that the complexity is likely due to the fact that we're fitting a really really complex function in a space of dimensionality so high the human mind can't even begin to visualize it, so no wonder the parameters are hard to figure out. What might be possible though is to design a connected system to have dedicated, distinct functional areas that make it more "modular" in its overall structure (either by prearranging those, or by rearranging blocks between stages of training). That could make the process more complex or the final result less efficient, the same way if you compile a program for debugging it isn't as performant as one optimized for speed. But the program compiled for debugging you can look inside, and that's kind of important and useful. The essence of the complaint here is IMO that as far as these trade-offs go, researchers seem all-in on the performance aspect and not spending nearly enough effort debugging their "code". It may be that the alternative to giant inscrutable matrices might just be a bunch of smaller, more scrutable ones, but hey. Would be progress!

We can discuss anything that exists, that might exist, that did exist, that could exist, and that could not exist. So no matter what form your predict-the-next-token language model takes, if it is trained over the entire corpus of the written word, the representations it forms will be pretty hard to understand, because the representations encode an entire understanding of the entire world.

Perhaps. 

Imagine a huge number of very skilled programmers tried to manually hard code a ChatGPT in python. 

Ask this pyGPT to play chess, and it will play chess. Look under the hood, and you see a chess engine programmed in. Ask it to solve algebra problems, a symbolic algebra package is in there. All in the best neat and well commented code.

Ask it to compose poetry, and you have some algorithm that checks if 2 words rhyme. Some syllable counter. Etc. 

Rot13 is done with a hardcoded rot13 algorithm. 

Somewhere in the algorithm is a giant list of facts, containing "Penguins  Live In Antarctica".  And if you change this fact to say "Penguins Live in Canada", then the AI will believe this. (Or spot it's inconsistency with other facts?) 

And with one simple change, the AI believes this consistently. Penguins appear when this AI is asked for poems about canada, and don't appear in poems about Antarctica. 

When asked about the native canadian diet, it will speculate that this likely included penguin, but say that it doesn't know of any documented examples of this. 

Can you build something with ChatGPT level performance entirely out of human comprehensible programmatic parts?

Obviously having humans program these parts directly would be slow. (We are still talking about a lot of code.) But if some algorithm could generate that code? 

But if the universal failure of nature and man to find non-connectionist forms of general intelligence does not move you

Firstly, AIXI exists, and we agree that it would be very smart if we had the compute to run it. 

Secondly I think there is some sort of slight of had here. 

ChatGPT isn't yet fully general. Neither is a 3-sat solver.  3-sat looks somewhat like what you might expect a non-connectionist approach to intelligence to look like. There are a huge range of maths problems that are all theoretically equivalent to 3 sat.

In the infinite limit, both types of intelligence can simulate the other at huge overhead, In practice, they can't. 

Also, non-connectionist forms of intelligence are hard to evolve, because evolution works in small changes. 

we can make matrices more intelligible if we make their dimensions meaning not random, but corresponding to some known in advance dimensions, like [levels of dog, level of cat, level of human, level of ape]