Public Weights?

Something I haven't yet personally observed in threads on this broad topic is the difference in risk modeling from the perspective of the potential malefactor. You note that outside a hackathon context, one could “take a biology class, read textbooks, or pay experienced people to answer your questions”—but especially that last one has some big-feeling risks associated with it. What happens if the experienced person catches onto what you're trying to do, stops answering questions, and alerts someone? The biology class is more straightforward, but still involves the risky-feeling action of talking to people and committing in ways that leave a trail. The textbooks have the lowest risk of those options but also require you to do a lot more intellectual work to get from the base knowledge to the synthesized form.

This restraining effect comes only partly in the form of real social risks to doing things that look ‘hinky’, and much more immediately in the form of psychological barriers from imagined such risks. People who are of the mindset to attempt competent social engineering attacks often report them being surprisingly easy, but most people are not master criminals and shy away from doing things that feel suspicious by reflex.

When we move to the LLM-encoded knowledge side of things, we get a different risk profile. Using a centralized, interface-access-only LLM involves some social risk to a malefactor via the possibility of surveillance, especially if the surveillance itself involves powerful automatic classification systems. Content policy violation warnings in ChatGPT are a very visible example of this; many people have of course posted about how to ‘jailbreak’ such systems, but it's also possible that there are other hidden tripwires.

For an published-weights LLM being run on local, owned hardware through generic code that's unlikely to contain relevant hidden surveillance, the social risk to experimenting drops into negligible range, and someone who understands the technology well enough may also understand this instinctively. Getting a rejection response when you haven't de-safed the model enough isn't potentially making everyone around you more suspicious or adding to a hidden tripwire counter somewhere in a Microsoft server room. You get unlimited retries that are punishment-free from this psychological social risk modeling perspective, and they stay punishment-free pretty much up until the point where you start executing on a concrete plan for harm in other ways that are likely to leave suspicious ripples.

Structurally this feels similar to untracked proliferation of other mixed-use knowledge or knowledge-related technology, but it seems worth having the concrete form written out here for potential discussion.

This is the main driving force behind why my intuition agrees with you that the accessibility of danger goes up a lot with a published-weights LLM. Emotionally I also agree with you that it would be sad if this meant it were too dangerous to continue open distribution of such technology. I don't currently have a well-formed policy position based on any of that.

Could a virologist actually tell you how to start a pandemic? The paper you're discussing says they couldn't:

Fortunately, the scientific literature does not yet feature viruses that are particularly likely to cause a new pandemic if deliberately released (with the notable exception of smallpox, which is largely inaccessible to non-state actors due to its large genome and complex assembly requirements). Threats from historical pandemic viruses are mitigated by population immunity to modern-day descendants and by medical countermeasures, and while some research agencies actively support efforts to find or create new potential pandemic viruses and share their genome sequences in hopes of developing better defenses, their efforts have not yet succeeded in identifying credible examples.

The real risk would come from biological design tools (BDTs), or other AI systems capable of designing new pathogens that are more lethal and transmissible than existing ones. I'm not aware of any existing BDTs that would allow you to design more capable pathogens, but if they exist or emerge, we could place specific restrictions on those models. This would be far less costly than banning all open source LLMs. 

My guess is that the synthesis functionality of current LLMs is actually adding something here and a no-LLM group would do quite a bit worse, but 83% of people seem to disagree with me:

This is a random nit, but a market with a probability of 17% does not imply that 83% of people disagree with you. I don't know what fraction of people agree with you, I just know that by whatever mechanism Manifold traders are willing to trade, the current price is at 17%.

(Back in 2017 I asked for examples of risk from AI, and didn't like any of them all that much. Today, "someone asks an LLM how to kill everyone and it walks them through creating a pandemic" seems pretty plausible.)

My impression from the 2017 post is that concerns were framed as “superintelligence risk” at the time.  The intended meaning of that term wasn’t captured in the old post, but it’s not clear to me that an LLM answering questions about how to create a pandemic qualifies as superintelligence?

This contrast seems mostly aligned with my long-standing instinct that folks worried about catastrophic risk from AI have tended to spend too much time worrying about machines achieving agency and not enough time thinking about machines scaling up the agency of individual humans.

I expect the supposedly dangerous information (that the authors are careful not to actually tell you) is some combination of obvious (to a person of ordinary skill in the art), useless, and wrong, roughly analogous to the following steps for building a nuclear bomb:

1. Acquire 100 kilograms of highly enriched uranium.
2. Assemble into a gun-type fission weapon.
3. Earth-Shattering Kaboom!

This "draw the rest of the fucking owl" kind of advice is good for a laugh, and as fodder for fear-mongering about actually open AI (not to be confused with the duplicitously named OpenAI), but little else.