Building AI Research Fleets

post by Ben Goldhaber (bgold), Jesse Hoogland (jhoogland) · 2025-01-12T18:23:09.682Z · LW · GW · 3 comments

Contents

  From AI scientist to AI research fleet
  Recommendations
    Individual practices
    Organizational changes[2]
    Community-level actions
  Further Reading
    On Automation in AI Safety
    On Research Automation
    On Automation Generally
      Algorithmic trading
      Industrial research, big pharma, biotech research, defense & national laboratory research
      Quality control in flexible manufacturing systems
      Medical & legal automation
None
3 comments

From AI scientist to AI research fleet

Research automation is here (1, 2, 3). We saw it coming and planned ahead, which puts us ahead of most (4, 5 [LW · GW], 6 [LW · GW]). But that foresight also comes with a set of outdated expectations that are holding us back. In particular, research automation is not just about “aligning the first AI scientist”, it’s also about the institution-building problem of coordinating the first AI research fleets.

Research automation is not about developing a plug-and-play “AI scientist”. Transformative technologies are rarely straightforward substitutes for what came before. The industrial revolution was not about creating mechanical craftsmen but about deconstructing craftsmen into assembly lines of specialized, repeatable tasks. Algorithmic trading was not just about creating faster digital traders but about reimagining traders as fleets of bots, quants, engineers, and other specialists. AI-augmented science will not just be about creating AI “scientists.”

Why? New technologies come with new capabilities and limitations. To fully take advantage of the benefits, we have to reshape our workflows around these new limitations. This means that even if AIs eventually surpass human abilities across the board, roles like “researcher” will likely transform dramatically during the transition period.

The bottleneck to automation is not just technological but also institutional. The problem of research automation is not just about training sufficiently capable and aligned models. We face an “institutional overhang” where AI capabilities are outpacing our ability to effectively organize around their weaknesses. Factories had to develop new management techniques, quality control systems, and worker training programs to make assembly lines effective. Trading firms had to build new risk management frameworks, compliance systems, and engineering cultures to succeed at algorithmic trading. So too, research institutions will need to reinvent themselves around AI or fall behind.  

The scaling labs have already moved beyond the traditional academic model. Consider the use of matrix management structures where research engineers work across multiple projects, standardized research workflows that enable fast iteration, and cross-cutting infrastructure teams that maintain the computational foundation for research. Labs employ specialized roles like research engineers, infrastructure specialists, and research managers that don't fit neatly into the academic hierarchy.

Deepmind’s recent Nobel prize is a hint of more to come.

A vision: the automated research fleet. Imagine tomorrow’s research lab: not individual AI models confined to chat windows but vast digital fleets of specialized AI agents working in concert. Each agent masters its own niche in the research pipeline: proving theorems, reviewing literature, generating hypotheses, running experiments, analyzing results, communicating outcomes, developing new techniques, conceptualizing entirely new paradigms…

Automation raises the level of abstraction so that everyone becomes a middle manager — every researcher the director of a research institution of their own. And it changes the basic patterns of human-AI interaction: the prompter will become the prompted — instead of crafting careful prompts in chat interfaces, human researchers receive updates and requests for guidance from their AI project leads, who independently pursue established research objectives.

This future may appear wasteful at first glance. Imagine thousands of AI instances running in parallel, testing slight variations of the same approach, with almost all attempts failing. Or hundreds of different AI instances in a shared chat that redundantly process the same tokens. But this apparent inefficiency is a feature, not a bug. Ford’s assembly lines overproduced standardized parts; McLean’s containers shipped half-empty; early cloud computing wasted countless unused FLOPs. Just as these “inefficiencies” enabled unprecedented flexibility and scale in their industries, the parallel processing power of AI research fleets will unlock new possibilities in scientific discovery. The ability to rapidly test hundreds of variations, explore multiple paths simultaneously, and fail fast will become a cornerstone of future research methodology.

Recommendations

The scaling labs already understand that research automation is here – they're building the infrastructure and organizational patterns for automated research at scale. For AI safety to stay relevant, we need to adapt and accelerate. Here are our recommendations for transitioning toward AI research fleet management:

Individual practices

Beware AI slop. We are not pollyanish AI enthusiasts — much of the content currently produced by AI is bad and possibly harmful [LW · GW]. Continue to whet your tastes on pre-2023 human-sourced content.

Organizational changes[2]

Beware AI slop. You shouldn’t use AI systems blindly for all of your coding and research. At the same time, you should tolerate early automation mistakes (from, e.g., AI code slop) as learning opportunities for your organization to develop better quality control processes.

Community-level actions

In general, we recommend working forwards from your existing workflows rather than working backwards from any idealistic vision of what automated AI safety research should look like. Too much theorizing is a real risk. Work iteratively with what you have.

We personally are starting today, and think you should too. The race for AI safety isn't one we chose, but it's one we have to win.

Thanks to Raemon and Daniel Murfet for feedback on a draft of this post.

Further Reading

On Automation in AI Safety

On Research Automation

On Automation Generally

Algorithmic trading

MacKenzie, D. (2021). "Trading at the Speed of Light: How Ultrafast Algorithms Are Transforming Financial Markets." Princeton University Press.  

MacKenzie, D. (2019). "How Algorithms Interact: Goffman's 'Interaction Order' in Automated Trading." Theory, Culture & Society 36(2): 39-59.

Zuckerman, G. (2019). “The Man who Solved the Market: How Jim Simons Launched the Quant Revolution”. New York, NY, Portfolio / Penguin.

Industrial research, big pharma, biotech research, defense & national laboratory research

Hounshell, D.A. and Smith, J.K. (1988). "Science and Corporate Strategy: Du Pont R&D, 1902-1980." Cambridge University Press.

Henderson, R. (1994). "Managing Innovation in the Information Age." Harvard Business Review 72(1): 100-105.

Quality control in flexible manufacturing systems

Hayes, R. H., & Jaikumar, R. (1988). "Manufacturing's crisis: New technologies, obsolete organizations." Harvard Business Review, 66(5), 77-85.

Goldratt, Eliyahu, (1984). “The Goal: a Process of Ongoing Improvement". Great Barrington, MA :North River Press

Medical & legal automation

Jha, S. and Topol, E. (2016). "Adapting to Artificial Intelligence: Radiologists and Pathologists as Information Specialists." JAMA 316(22): 2353-2354.

Remus, D. and Levy, F. (2017). "Can Robots Be Lawyers? Computers, Lawyers, and the Practice of Law." Georgetown Journal of Legal Ethics 30: 501-558.

  1. ^

     Consider actually reading the docs.

  2. ^

     In a sense we are all corporations now. All of these suggestions also apply to how you organize AIs in your personal life.

3 comments

Comments sorted by top scores.

comment by Daniel Murfet (dmurfet) · 2025-01-12T19:47:16.210Z · LW(p) · GW(p)

Thanks Jesse, Ben. I agree with the vision you've laid out here.

I've spoken with a few mathematicians about my experience using Claude Sonnet and o1, o1-Pro for doing research, and there's an anecdote I have shared a few times which gets across one of the modes of interaction that I find most useful. Since these experiences inform my view on the proper institutional form of research automation, I thought I might share the anecdote here.

Sometime in November 2024 I had a striking experience with Claude Sonnet 3.5. At the end of a workday I regularly paste in the LaTeX for the paper I’m working on and ask for its opinion, related work I was missing, and techniques it thinks I might find useful. I finish by asking it to speculate on how the research could be extended. Usually this produces enthusiastic and superficially interesting ideas, which are however useless.

On this particular occasion, however, the model proceeded to elaborate a fascinating and far-reaching vision of the future of theoretical computer science. In fact I recognised the vision, because it was the vision that led me to write the document. However, none of that was explicitly in the LaTeX file. What the model could see was some of the initial technical foundations for that vision, but the fancy ideas were only latent. In fact, I have several graduate students working with me on the project and I think none of them saw what the model saw (or at least not as clearly).

I was impressed, but not astounded, since I had already thought the thoughts. But one day soon, I will ask a model to speculate and it will come up with something that is both fantastic and new to me.

Note that Claude Sonnet 3.5/3.6 would, in my judgement, be incapable of delivering on that vision. o1-Pro is going to get a bit further. However, Sonnet in particular has a broad vision and "good taste" and has a remarkable knack of "surfing the vibes" around a set of ideas. A significant chunk of cutting edge research comes from just being familiar at a "bones deep" level with a large set of ideas and tools, and knowing what to use and where in the Right Way. Then there is technical mastery to actually execute when you've found the way; put the vibe surfing and technical mastery together and you have a researcher.

In my opinion the current systems have the vibe surfing, now we're just waiting for the execution to catch up.

comment by PeterMcCluskey · 2025-01-13T02:46:22.309Z · LW(p) · GW(p)

I was just thinking about writing a post that overlaps with this, inspired by a recent Drexler post. I'll turn it into a comment.

Leopold Aschenbrenner's framing of a drop-in remote worker anthropomorphizes AI in a way that risks causing AI labs to make AIs more agenty than is optimal.

Anthropomorphizing AI is often productive. I use that framing a fair amount to convince myself to treat AIs as more capable than I'd expect if I thought of them as mere tools. I collaborate better when I think of the AI as a semi-equal entity.

But it feels important to be able to switch back and forth between the tool framing and the worker framing. Both framings have advantages and disadvantages. The ideal framing is likely somewhere in between that seems harder to articulate.

I see some risk that AI labs turning AIs into agents, when if they were less focused on replacing humans they might lean more toward Drexler's (safer) services model.

Please, AI labs, don't anthropomorphize AIs without carefully considering when that's an appropriate framing.

comment by Jonas Hallgren · 2025-01-12T20:08:55.677Z · LW(p) · GW(p)

Well said. I think that research fleets will be a big thing going forward and you expressed why quite well. 

I think there's an extension that we also have to make with some of the safety work we have, especially for control and related agendas. It is to some extent about aligning research fleets and not individual agents.

I've been researching ways of going about aligning & setting up these sorts of systems for the last year but I find myself very bottlenecked by not being able to communicate the theories that exists in related fields that well. 

It is quite likely that RSI happens in lab automation and distributed labs before anything else. So the question then becomes how one can extend the existing techniques and theory that we currently have to distributed systems of research agents? 

There's a bunch of fun and very interesting decentralised coordination schemes and technologies one can use from fields such as digital democracy and collective intelligence. It is just really hard to prune what will work and to think about what the alignment proposals should be for these things. You usually have emergence which for Agent-Based Models which research systems are a sub-part of and often the best way to predict problems is to actually run the experiments in those systems. 

So how in the hell are we supposed to predict the problems without this? What are the experiments we need to run? What types of organisation & control systems should be recommended to governance people when it comes to research fleets?