"Open Source AI" isn't Open Source

The vast majority of uses of software are via changing configuration and inputs, not modifying code and recompiling the software. (Though lots of Software as a Service doesn't even let you change configuration directly.) But software is not open in this sense unless you can recompile, because it's not actually giving you full access to what was used to build it.

The same is the case for what Facebook call open-source LLMs; it's not actually giving you full access to what was used to build it.

I don't think that's accurate. the data is the code, the model is just the binary format the code gets compiled to.

Yeah. I think it is fairly arguable that open data sets aren't required for open source; it's not the form you'd prefer to modify it in as a programmer, and they're not exactly code to start with, Shakespeare didn't write his plays as programming instructions for algorithms to generate Shakespeare-like plays. No one wants a trillion tokens that take ~$200k to 'compile' as their starting point to build from after someone else has already done that and made it available. (Hyperbolic; but the reasons someone wants that generally aren't the same reasons they'd want to compile from source code.) Open datasets are nice information to have, but lack a reasonable reproduction cost or much direct utility beyond explanation.

Llama's state as not open source is much more strongly made by the restrictions on usage, as noted in the prior discussion. There is a meaningful distinction between open dataset and closed dataset, but I'd put the jump between Mistral and Llama to be from 'open source with hidden supply chain' to 'open weights with restrictive licensing', where the jump between Mistral and RedPajama is more 'open source with hidden supply chain' to 'open source with revealed supply chain'.

Oh, I do, they're just generally not quite the best available/most popular for hobbyists. Some I can find quickly enough are Pythia and OpenLLaMA, and some of the RedPajama models Together.ai trained on their own RedPajama dataset (which is freely available and described). (Also the mentioned Falcon and MPT, as well as StableLM. You might have to get into the weeds to find out how much of the data processing step is replicable.)

(It's going to be expensive to replicate any big pretrained model though, and possibly not deterministic enough to do it perfectly; especially since datasets sometimes adjust due to removing unsafe data, the recipes for data processing included random selection and shuffling from the datasets, etc. Smaller examples where people have fine-tuned using the same recipe coincidentally or intentionally have gotten identical model weights though.)