Google "We Have No Moat, And Neither Does OpenAI"

My prior, not having looked too carefully at the post or the specific projects involved, is that probably any claims that an open source model is 90% as good as GPT4 or indistinguishable are hugely exaggerated or otherwise not a fair comparison. In general in ML, confirmation bias and overclaiming is very common and as a base rate the vast majority of papers that claim some kind of groundbreaking result end up just never having any real impact.

Also, I expect facets of capabilities progress most relevant to existential risk will be especially constrained strongly by base model quality. I would agree that open source is probably better at squeezing stuff out of small models, but my model is that wrt existential risk relevant capabilities progress this is less relevant (cf the bitter lesson).

I feel like the title of this post is slightly misleading. It's not like there was any kind of official Google statement saying they have no moat. It was some random Google Researchers whose private document leaked. 

A better title would be "Unknown Google researcher says Google has no moat, and neither does OpenAI". I think people seem to be taking this a lot more seriously because they don't know who wrote it, and are treating it as some kind of consensus within Google.

Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.

Widely, does this come as a surprise? I recall back to the GPT2 days where the 4chan and Twitter users of AIDungeon discovered various prompting techniques we use today. More access means more people trying more things, and this should already be our base case because of how open participation in open source has advanced and improved OSS projects.

I'm worried that up until now, this community has been too focused on the threat of big companies pushing capabilities ahead and not focused enough on the threat posed by open-source AI. I would love to see more discussions of regulations in order to mitigate this risk. I suspect it would be possible to significantly hamper these projects by making the developers of these projects potentially liable for any resulting misuse.

I have no idea how you think this would work.

First, any attempt at weakening liability waivers will cause immediate opposition by the entire software industry. (I don't even know under what legal theory of liability this would even operate.) Remember under American law, code is free speech. So...second, in the case you're somehow (somehow!) able to pass something (while there's a politicized and deadlocked legislature) where a coalition that includes the entire tech industry is lobbying against it and there isn't an immediate prior restraint to speech challenge...what do you think you're going to do? Go after the mostly anonymous model trainers? A lot of these people are random Joe Schmoes with no assets. Some of the SD model trainers which aren't anonymous already have shell corporations set up, both to shield their real identities and to preemptively tank liability in case of artist nuisance lawsuits.

The fact that they can squeeze more performance out of models with smaller datasets and less compute points to a current computing overhang [? · GW], which reduces the odds that Altman's strategy of short timelines/slow takeoff can be done safely.

Aren't a lot of the doomy scenarios predicated on a single monolithic AI though? (Or multipolar AI that all agree to work togrther for naughtiness for some reason)

A bunch of them being tinkered on by lots of people seems like an easier path to alignment and as a failsafe in terms of power distribution.

You have lots of smaller scale dangers introduced but they certaintly dont seem to me to rise to the level of x or s risk in the near term.

What have we had thus far? A bunch of think tanks using deductive reasoning with no access to good models and a few monoliths with all the access. Seems to me that having the capability to actually run experiments at a community level will neccesarily boost efforts on alignment and value loading more than it assists actual AGI being born.

This reminds me of stories (maybe exaggerated) about how Microsoft became internally terrified of GPLv3 and Linux in the late 90s and early 2000s.

You can think of it as “dangerous capabilities in everyone’s hands”, but I prefer to think of it as “everyone in the world can work on alignment in a hands-on way, and millions of people are exposed to the problem in a much more intuitive and real way than we ever foresaw”.

Ordinary people without PhDs are learning what capabilities and limitations LLMs have. They are learning what capabilities you can and cannot trust an LLM with. They are coming up with creative jailbreaks we never thought of. And they’re doing so with toy models that don’t have superhuman powers of reasoning, and don’t pose X-risks.

It was always hubris to think only a small sect of people in the SF bay area could be trusted with the reins to AI. I’ve never been one to bet against human ingenuity, and I’m not about to bet against them now that I’ve seen the open source community use LLaMa to blaze past every tech company.

I think this speaks of a move from focusing on building the largest models possible, to optimising smaller models for mundane utility. Since smaller models have a hard ceiling on capabilities, this is good news as it pushes off AGI.

I have a list of specific technical advances in algorithms/architecture that I suspect will push these open-source LLMs further in capabilities when integrated. Especially increasing effective agency & strategic planning. I think these advances, if like 80% of them are discovered and are as successful as I expect, are enough with like a 65 billion or so param LLM, to begin a successful RSI process. Slow at first, but able to accelerate since I still think there's a lot of algorithmic efficiency gain sitting on the table. What I don't know is what to do about that scenario. If this is the case, then the world is going to look pretty crazy a couple years from now.

The Substack post only mentions that a researcher leaked the document, not that any researcher authored it. The document could've been written up by one or more Google staffers who aren't directly doing the research themselves, like a project manager or a research assistant. 

Nothing in the document should necessarily be taken as representative of Google, or any particular department, though the value of any insights drawn from the document could vary based on what AI research project(s)/department the authors of the document work on/in. This document is scant evidence in any direction of how representative the statements made are of Google and its leadership, or any of the teams or leaders of any particular projects or departments at Google focused on the relevant approaches to AI research.

This gives me the title for season 1 episode 1 of Uncanny Valley, the spinoff of Silicon Valley focused on AI alignment: "I Have No Moat and I Must Scream". (It's a Harlan Ellison reference.)

I'd like to see open-sourced evaluation and safety tools. Seems like a good thing to push on.

I think this is absolutely accurate and here's why:

1. open-source will surely beat the big guys. Openai and google do NOT have a monopoly on smart people, there's far more of us that do not work for them. Any smart person who is not allowed on the winning team will simply make their own team win. Therefore openAI and google will be left in the dust in 2 years flat. And I don't see anything stopping researchers from leaving and then just giving their truly awesome ideas to open-source after they take home the big bucks.

2. They have no moat, at least not one that is going to be there for long. 
 - First mover advantage is not an advantage in every game. America has been king the world ever since they joined WW2 late. And the first movers got the reputation they deserved. As they say, sometimes it's better to wait and let the competition draw a bit of blood before you jump into the fray.

 - data moat? nope we got open data. network effects moat? nope, we got github, technology moat? apparently not a very thick one. Entrenched customers? yeah for openAI. But once you're paying for multiple Ai subscriptions and you have to get rid of one... maybe it's byebye openai. 

The only moat they have is that they have a lot of money to pay researchers. And they can pour millions into training models. So if they can win by spending more money then that's how they can do it. OpenAI can win by building on top of their user base. But openAI shows no signs of doing that, they are just becoming microsofts toy, and google can pour as much money as they want wherever they want it's not going to buy them something better than open source.

⚫ What is the cost of even Cat level A.I. (CLAI)?

The Information had a good scoop recently. They said recently:

⚫ OpenAI’s losses roughly doubled to around $540 million last year as it developed ChatGPT and hired key employees from Google, according to three people with knowledge of the startup’s financials.

The previously unreported figure reflects the steep costs of training its machine-learning models during the period before it started selling access to the chatbot. Furthermore reports online indicate that ChatGPT costs approximately $700,000 a day just to run, or more.

Even as revenue has picked up—reaching an annual pace of hundreds of millions of dollars just weeks after OpenAI launched a paid version of the chatbot in February—those costs are likely to keep rising as more customers use its artificial intelligence technology and the company trains future versions of the software.

⚫ Reflecting that capital drain, CEO Sam Altman has privately suggested OpenAI may try to raise as much as $100 billion in the coming years to achieve its aim of developing artificial general intelligence that is advanced enough to improve its own.

⚫ A leak out of Google claims that they have no moat against open-source LLMs that could one day overtake them. (Read semianalysis on Sustack for this scoop). simonwillison and LessWrong have blogged about this too.

What is clear is few of even the best Generative A.I. startups today, will even be around in 2030. They will have failed to make money. Which makes all the hype and hubris a bit of a head fake.

Microsoft's investment in OpenAI isn't likely to age well given the pace of innovation in LLMs. Very few companies or even Nation states can easily bear the costs of developing this stuff and they will get disrupted by new tech here rather easily. That won't be a problem for the Beijing Academy of Artificial Intelligence (BAAI).