AI Is Not Software

An AI system is software in something like the same way a human being is chemistry: our bodies do operate by means of a vast network of cooperating chemical processes, but the effects of those processes are mostly best understood at higher levels of abstraction, and those don't behave in particularly chemistry-like ways.

Sometimes our bodies go wrong in chemistry-based ways and we can apply chemistry-based fixes that kinda work. We may tell ourselves stories about how they operate ("your brain doesn't have enough free serotonin, so we're giving you a drug that reduces its uptake") but they're commonly somewhere between "way oversimplified" and "completely wrong".

Sometimes an AI system goes wrong in code-level ways and we can apply code-based fixes that kinda work. We may tell ourselves stories about how they operate ("the training goes unstable because of exploding or vanishing gradients, so we're introducing batch normalization to make that not happen") but they're commonly somewhere between "way oversimplified" and "completely wrong".

But most of the time, if you want to understand why a person does certain things, thinking in terms of chemistry won't help you much; and most of the time, if you want to understand why an AI does certain things, thinking in terms of code won't help you much.

just name it after another phase maybe lol. gelware? liquidware. fluidware? gasware? plasmaware?? etherware?!

True to say that there's a distinction between software and data. Photo editor, word processor, video recorder: software. Photo, document, video: data.

I think similarly there's a distinction within parts of "the AI", where the weights of the model are data (big blob of stored numbers that the training software calculated). Seems inaccurate though, to say that AI "isn't software" when you do still need software running that uses those weights to do the inference.

I guess I take your point, that some of the intuitions people might have about software (that it has features deliberately designed and written by a developer, and that when it goes wrong we can go patch the faulty function) don't transfer. I would just probably frame that as "these intuitions aren't true for everything software does" rather than "this thing isn't software".

I am not sure this is correct. Actually I feel pretty confident it is not.

In the broader scope of systems engineering, especially for embedded control systems, every component has a probability less than 1.0 of working. A good system design factors in the likelihood of failure for any single component. Generally there are redundancies to improve the probability to a series probability. For example, the Tesla cybertruck uses 2 front steering motors, with 2 controllers, 2 network links, at least 2 steering angle sensors, and 2 power supplies. In order for the steering to fail at least one component in both systems has to fail during the same drive, or the fault has to be undetected in the A system at startup and the B system fails on the same drive.

Human engineers sit down and multiply probabilities for the above and add redundancy or more reliable components until the design satisfied a target pFail.

With ML models you know or can measure the following :

1. How confident is the model in its perception of the current situation. (Empirical, measured in training)
2. Is the current system input within the latent space of the training distribution (from a parallel model)
3. The neural sim that lockstep predicts n future environment states conditional on actions : how confident is it? (Empirical, measured in training on real data)
4. How costly are errors in the current situation?
5. In the current situation, what classes of error are likely? (For example an autonomous vehicle at 65mph draws from the "high speed collision" risk pool)

With this information it is possible to build an outer control algorithm that uses multiple ML models to accomplish a larger task. And this algorithm can use the above information to shut down the model or revert control to a safer policy (stop the robot, engage the brakes) when the risk is high. One obvious note: "confidence" and "in distribution" are empirical metrics measured during training, they are not a number the model can "choose" what to give in an agentic way.

This outer framework is software. It's how autonomous car stacks except comma.ai and donkey car work. The framework people put around their discord bots using AI is similar. The framework doesn't just let the model do anything it wants, it restricts model actions to discord messages but only when a keyword to trigger the bot is emitted.

For any use of AI, present or future that I can anticipate, human engineers will use software frameworks, with more restrictive frameworks and multiple parallel models for AI applications that have high risks.

Do you agree or disagree? If you disagree, do you acknowledge that this is how present engineers are doing this task? Why do you believe future designs will be less restricted?

I very strongly disagree. In my opinion, this argument appears fatally confused about the concept of "software." 

As others have pointed out, this post seems to be getting at a distinction between code and data, but many of the examples of software given by OP contain both code and data, as most software does. Perhaps the title should have been "AI is Not Code," but since it wasn't I think mine is a legitimate rebuttal. 

I'm not trying to make an argument by definition. My comment is about properties of software that I think we would likely agree on. I think OP both ignores some properties software can have while assuming all software shares other separate properties, to the detriment of the argument.

I  think the post is correct in pointing out that traditional software is not similar to AI in many ways, but that's where my agreement ends.

1: Software, I/O, and such

Most agree on the following basic definition: software is a set of both instructions and data, hosted on hardware, that governs how input data is transformed to some sort of output. As you point out, inputs and outputs are not software.

For example, photos of a wedding or a vacation aren’t software, even if they are created, edited, and stored using software.

Yes.

Second, when we run the model, it takes the input we give it and performs “inference” with the model. This is certainly run on the computer, but the program isn’t executing code that produces the output, it’s using the complicated probability model which grew, and was stored as a bunch of numbers. 

No! It is quite literally executing code to produce the output! Just because this specific code and the data it interacts with specifies a complicated probability model that does not mean it is not software. 

Every component of the model is software. Even the pseudorandomness of the model outputs is software (torch.randn(), often). There is no part of this inference process that generates outputs that is not software. To run inference is only to run software.

2: Stochasticity

The model responds to input by using the probability model to estimate the probability of difference responses, in order to output something akin to what the input data did - but it does so in often unexpected or unanticipated ways.

Software is often, but is not necessarily deterministic. Software can have stochastic or pseudorandom outputs. For example, software that generates pseudorandom numbers is still software. The fact that AI generates stochastic outputs humans don't expect does not make it not software.

Also, software is not necessarily interpretable and outputs are not necessarily expected or expectable.

3: Made on Earth by Humans

First, we can talk about how it is created. Developers choose a model structure and data, and then a mathematical algorithm uses that structure and the training data to “grow” a very complicated probability model of different responses... The AI model itself, the probability model which was grown, is generating output based on a huge set of numbers that no human has directly chosen, or even seen. It’s not instructions written by a human.

Neither a software's code nor its data is necessarily generated by humans.

4: I have bad news for you about software engineering

Does software work? Not always, but if not, it fails in ways that are entirely determined by the human’s instructions.

This is just not true, many bugs are caused by specific interactions between inputs and the code + data, some also caused by inputs, code, data, and hardware (buffer overflows being the canonical example). You could get an error due to cosmic bit flips, that has nothing to do with humans or instructions at all! Data corruption... I could go on and on.

For example, unit tests are written to verify that the software does what it is expected to do in different cases. The set of cases are specified in advance, based on what the programmer expected the software to do. 

... or the test is incorrect. Or both the test and the software are incorrect. Of course this assumes you wrote tests, which you probably didn't. Also, who said you can't write unit tests for AI? You can, and people do. All you have to do is fix the temperature parameter and random seed. One could argue benchmarks are just stochastic tests...

If it fails a single unit test, the software is incorrect, and should be fixed.

Oh dear. I wish the world worked like this. 

Badly written, buggy software is still software. Not all software works, and it isn't always software's fault. Not all software is fixable or easy to fix.

5: Implications

What we call AI in 2024 is not software. It's kind of natural to put it in the same category as other things that run on a computer, but thinking about LLMs, or image generation, or deepfakes as software is misleading, and confuses most of the ethical, political, and technological discussions.

In my experience, thinking of AI as software leads to higher quality conversations about the issues. Everyone understands at some level that software can break, be misused, or be otherwise in-optimal for any number of reasons. 

I have found that when people begin to think AI is not software, they often devolve into dorm room philosophy debates instead of dealing with its many concrete, logical, potentially fixable issues. 

Seems like an issue of code/data segmentation. Programs can contain compile time constants, and you could turn a neural network into a program that has compile time constants for the weights, perhaps "distilling" it to reduce the total size, perhaps even binarizing it.

Arguably, video games aren't entirely software by this standard, because they use image assets.

Formally segmenting "code" from "data" is famously hard because "code as data" is how compilers work and "data as code" is how interpreters work. Some AI techniques involve program synthesis.

I think the relevant issue is copyright more than the code/data distinction? Since code can be copyrighted too.

I think it would also be useful to explain that AI is trained. So, much like a dog, if if does something we don't want ti to, be may be able to train it out of doing that, but the process is much more like training.a dog than fixing a bug in software.

Sure, but this is not new. You start by saying "AI in 2024" but this is true of everything that has been called AI and a lot of things that maybe should have been called AI, such as the PageRank algorithm. Credit scores have made decisions based on based on statistical models since the 50s.