Yoshua Bengio: Reasoning through arguments against taking AI safety seriously

post by Judd Rosenblatt (judd) · 2024-07-11T23:53:17.187Z · LW · GW · 3 comments

This is a link post for https://yoshuabengio.org/2024/07/09/reasoning-through-arguments-against-taking-ai-safety-seriously/

Contents

3 comments

He starts by emphasizing

The issue is so hotly debated because the stakes are major: According to some estimates, quadrillions of dollars of net present value are up for grabs, not to mention political power great enough to significantly disrupt the current world order.

[...]

The most important thing to realize, through all the noise of discussions and debates, is a very simple and indisputable fact: while we are racing towards AGI or even ASI, nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans.

And goes on to do a pretty great job addressing "those who think

3 comments

Comments sorted by top scores.

comment by faul_sname · 2024-07-13T20:25:57.888Z · LW(p) · GW(p)

In the section "For those who think that open-source AGI code and weights are the solution"

If we had the DNA sequence of an extremely dangerous virus, would it be best to share it publicly or not? If the answer is obvious to you in this case, think twice about the case for AGI algorithms and parameters.

The National Institute of Health's answer is "yes". Here's variola major (smallpox) for example. So those arguing that it's a bad idea to share ML algorithms and artifacts should either make the case that the NIH is wrong to share the smallpox genome or make the case that sharing some subset of ML algorithms and artifacts is more dangerous than sharing the smallpox genome.

In fairness, some people have in fact made decent cracks at the argument that sharing some types of ML-related information is more dangerous than sharing the smallpox genome. Still, I think the people arguing that the spread of knowledge is the thing we want to target, rather than the spread of materials, could do a better job of making that argument. But the common-sense "you wouldn't share the genome of a dangerous virus" argument doesn't work because we would, in fact, share the genome of a dangerous virus (and I personally think that it's actively good that we share the genomes of dangerous viruses, because it allows for stuff like this).

Replies from: nathan-helm-burger, whestler
↑ comment by Nathan Helm-Burger (nathan-helm-burger) · 2024-07-22T18:07:08.211Z · LW(p) · GW(p)

Personally, my take is that this is a dangerous mistake on the part of NIH and the worldwide academic community generally, that so much dangerous information has been released in regards to virology and bioweapons. Unfortunately, I'm clearly currently in the minority among scientists who think this. I wish I had a good idea of how to change people's minds on this issue, and the connected one of publishing AI capabilities implementation details.

↑ comment by whestler · 2024-07-15T10:30:40.826Z · LW(p) · GW(p)

I initially thought there must be some simple reason that publishing the DNA sequence is not a dangerous thing to do, like "ok, but given that you would need a world class lab and maybe even some techniques which haven't even been invented yet to get it to work, it's not a dangerous thing to publish".

According to this article from 2002, synthesising smallpox would be tricky, but within the reach of a terrorist organisation. Other viruses may be easier. 

“Scientifically, the results are not surprising or astounding in any way,” says virologist Vincent Racaniello of Columbia University. “The point here, of course, is that the DNA can be synthesized from the [genetic] sequence, and this could be done by any third-rate terrorist.”

Apparently, large organisations like the NIH are foolhardy enough to publish dangerous data like this. I wonder if there's some other justification, like "the data was already public, in such a way that it could not be removed"