Biosecurity Culture, Computer Security Culture

The problem with biosecurity is that most people working in the field are aligned with universities that do dangerous research. As a result they are more like the computer security department of a big company than they are like independent hackers in the field of computer security.

It seems that the position of the biosafety EA people is that if someone would actually act like a computer security hacker, they would be completely shunned and not have the reputation to achieve anything in that field. 

There's a EA post that speaks about how a lot of research that's done is gain of function research and that anyone who speaks with biosafety people should not speak about gain of function and instead enhanced potential pandemic pathogens. That because most of the research in the field includes gain of function research.

This basically mean that everyone in the field knows that Fauci perjured himself when claiming that there's no gain-of-function research in Wuhan but everyone in the field is afraid to do so.

The NIH saying that the research in Wuhan somehow wasn't on enhanced potential pandemic pathogens because the coronavirus (and thus it was somehow justified to fund it while the moratorium was in place) on which they worked wasn't a human illness was pretty bullshit. 

It should be obvious to everyone that it's bullshit if you have a transparent conversation about it. A lot of people in power in the field have a lot to lose if there's an open debate and thus do everything not to have that debate. Unfortunately, it might be correct that a biosafety EA would burn to many bridges when speaking honestly and transparently.

One key difference I see is that tremendous amounts of fungible value is locked away behind (hopefully) secure computing infrastructure, so in a world with keep-quiet norms there would be a tremendous financial incentive to defect on those norms.

As far as I know, no corresponding financial incentive exists for biosecurity (unless you count stuff like "antibiotics are an exploit against the biology of bacteria that people will pay lots of money for").

I think it makes a huge difference that most cybersecurity desasters only cost money (or cause damage to a company's reputation and loss of confidential information of customers) while a biosecurity desaster can kill a lot of people. This post seems to ignore this?

Imagine that the Morris worm never happened, nor Blaster, nor Samy. A few people independently discovered SQL injection but kept it to themselves. [...]

That hypothetical world is almost impossible, because it's unstable. As soon as certain people noticed that they could get an advantage, or even a laugh, out of finding and exploiting bugs, they'd do it. They'd also start building on the art, and they'll even find ways to organize. And finding out that somebody had done it would inspire more people to do it.

You could probably have a world without the disclosure norm, but I don't see how you could have a world without the actual exploitation.

We have driverless cars, robosurgeons, and simple automated agents acting for us, all with the security of original Sendmail.

None of those things are exactly bulletproof as it is.

But having the whole world at the level you describe basically sounds like you've somehow managed to climb impossibly high up an incredibly rickety pile of junk, to the point where instead of getting bruised when you inevitably do fall, you're probably going to die.

Introducing the current norms into that would be painful, but not doing so would just let it get keeping worse, at least toward an asymptote.

and the level of caution I see in biorisk seems about right given these constraints.

If that's how you need to approach it, then shouldn't you shut down ALL biology research, and dismantle the infrastructure? Once you understand how something works, it's relatively easy to turn around and hack it, even if that's not how you originally got your understanding.

Of course there'd be defectors, but maybe only for relatively well understood and controlled purposes like military use, and the cost of entry could be pretty high. If you have generally available infrastructure, anybody can run amok.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Cybersecurity seems in a pretty bad state globally - it's not completely obvious to me that a historical norm of "people who discover things like SQL injection are pretty tight-lipped about them and share them only with governments / critical infrastructure folks / other cybersecurity researchers" would have led to a worse situation than the one we're in cybersecuritywise...