I think that post assumes an incorrect model of scientific progress.

First: It's not about people at all, it's about ideas. And it seems much more defensible to claim that the impact ideas have on scientific problems is dominated by outliers: so-called "paradigms". Quantum mechanics, general relativity, Shannon's information theory, or the idea of applying the SGD algorithm to train very deep neural networks – all of those fields have "very compact generators" in terms of ideas.

These ideas then need to be propagated and applied, and in some sense, that takes up a "bigger" chunk of the concept-space than the compact generators themselves. Re-interpreting old physical paradigms in terms of the new theory, deriving engineering solutions and experimental setups, figuring out specific architectures and tricks for training ML models, etc. The raw information content of all of this is much higher than that of the "paradigms"/"compact generators". But that doesn't mean it's not all downstream of said generators, in a very strong sense.

Second: And this is where the "Great Man Theory" can be re-introduced again, in a way more true to reality. It's true that the bulk of the work isn't done by lone geniuses. But as we've just established, the bulk of the work is relatively straightforward propagation and application of a paradigm's implications. Not necessarily trivial – you still need to be highly mathematically gifted to make progress in many cases, say – but straightforward. And also factorizable: once a paradigm is in place, it can be propagated/applied in many independent directions at once.

The generation of paradigms themselves, however, is something a relatively small group of people can accomplish, by figuring out, based on hard-to-specify post-rigorous intuitions, in which direction the theory needs to be built. And this is something that might require unusual genius/talent/skillset [LW(p) · GW(p)].

Tenuously related: On this model, I think the purported "decline of genius" – the observation that there are no Einsteins or von Neumanns today – is caused by a change in scientific-research pipelines. Previously, a lone genius trying to cause a paradigm shift needed to actually finalize the theory before it'd be accepted, and publish it all at once. Nowadays, they're wrapped up in a cocoon of collaborators from the get-go, the very first steps of a paradigm shift are published, and propagation/application steps likewise begin immediately. So there's less of a discontinuity, both in terms of how paradigm shifts happen, and to whom they're attributed.

I generally like the linked essay by Beren, but I don’t like this linkpost, especially the title, because I dispute that Beren’s essay is on the topic of “the problem with infohazards as a concept”. My very strong impression is that Beren (like me) thinks that “infohazards” is a perfectly valid and useful concept. In particular, Beren’s essay starts with:

Obligatory disclaimer: This post is meant to argue against overuse of infohazard norms in the AI safety community and demonstrate failure modes that I have personally observed. It is not an argument for never using infohazards anywhere or that true infohazards do not exist. None of this is meant to be an absolute statement. Caveat emptor. Use common sense. If you have actually stumbled across some completely new technique which speeds up training by 100x or whatever then, in general, you should not share it. Most of the concerns here are presented with a mistake theory frame – it is assumed that these pathologies arise not due to any specific bad intentions but rather due to natural social and epistemic dynamics.

My opinion is: Infohazards exist. But people have to figure out in each individual instance whether something is or isn’t an infohazard (i.e., whether the costs of keeping it secret outweigh the benefits, versus the other way around). And (from my perspective) figuring that out is generally very hard.

• For one thing, figuring out what is or isn’t an infohazard is inherently hard, because there are a bunch of considerations entering the decision, all of which involve a lot of uncertainty, partly because they may involve trying to guess things about future intellectual & tech progress (which is notoriously hard).
• For another thing, it has poor feedback mechanisms—so you can be really bad at deciding what is or isn’t an infohazard, for a very long time, without noticing and correcting.

In this background context, Beren is arguing that, in his experience, people in AI alignment are systematically going wrong by keeping things secret that, all things considered, would have been better discussed openly.

That’s (a priori) a plausible hypothesis, and it’s well worth discussing whether it’s true or false. But either way, I don’t see it as indicating a problem with infohazards as a concept.

Sorry if I’m misunderstanding the OP’s point or putting words in anyone’s mouth.

Even Einstein – the archetypal lone genius

Einstein wouldn't be my first choice, if you want an archetypal example of a "Great Man" scientist. A better first choice would be Shannon. In the case of information theory, I'd say the Great Man model is just obviously basically correct.

With that example in mind, the "Great Man theory is just false" claim goes straight out the window; fields presumably vary in the extent to which Great Man models apply. So the relevant question is: is alignment more like information theory, or <whatever field would be the archetypal example of lots and lots of people all making marginal contributions>.

Many times have I heard people talk about ideas they thought up that are ‘super infohazardous’ and ‘may substantially advance capabilities’ and then later when I have been made privy to the idea, realized that they had, in fact, reinvented an idea that had been publicly available in the ML literature for several years with very mixed evidence for its success – hence why it was not widely used and known to the person coming up with the idea.

I’d be very interested if anyone has specific examples of ideas like this they could share (that are by now widely known or obviously not hazardous). I’m sympathetic to the sorts of things the article says, but I don’t actually have any picture of the class of ideas it’s talking about.

The main problem I see that are relevant to infohazards are that it encourages a "Great Man Theory" of progress in science, which is basically false, and this still holds despite vast disparities in ability, since no one person or small group is able to single handedly solve scientific fields/problems by themselves, and the culture of AI safety already has a bit of a problem with using the "Great Man Theory" too liberally.

I found other parts of the post a lot more convincing than this part of the post, and almost didn't read it because you highlighted this part of the post. Thankfully I did!

Here are the headings:

• Infohazards prevent seeking feedback and critical evaluation of work
• Thinking your work is infohazardous leads to overrating novelty or power of your work
• Infohazards assume an incorrect model of scientific progress
• Infohazards prevent results from becoming common knowledge and impose significant frictions
• Infohazards imply a lack of trust, but any solution will require trust
• Infohazards amplify in-group and social status dynamics
• Infohazards can be abused as tools of power
• Infohazards fail the ‘skin in the game’ test

Alright, I think I've figured out what my disagreement with this post is.

A field of research pursues the general endeavor of finding out things there are to know about a topic. It consists of building an accurate map of the world, of how-things-work, in general.

A solution to alignment is less like a field of research and more like a single engineering project. A difficult one, for sure! But ultimately, still a single engineering project, for which it is not necessary to know all the facts about the field, but only the facts that are useful.

And small groups/individuals do put together single engineering projects all the time! Including very large engineering projects like compilers, games & game engines, etc.

And, yes, we need solving alignment to be an at least partially nonpublic affair, because some important insights about how to solve alignment will be dual use, and the whole point is to get the people trying to save the world to succeed before the people functionally trying to kill everyone, not to get the people trying to save the world to theoretically succeed if they as much time as they wanted.

(Also: I believe this post means "exfohazard", not "infohazard" [LW · GW])

From the link:

Specifically, I think it should only be acceptable to claim something is infohazardous when you have strong empirical evidence that 1.) it substantially advances capabilities (i.e. more than the median NeurIPS paper), 2.) It empirically works on actual ML systems at scale, 3.) it is already not reasonably known within the ML community, and 4.) when there is no reason to expect differential impact on safety vs capabilities i.e. when the idea has no safety implications and is pure capabilities.

If we’re proposing this as a general criterion, as opposed to “a starting-point heuristic that would have been an improvement in Beren’s microenvironment”, then I have some complaints.

First, before we even start, I think “claim something is infohazardous” is not where I’d necessarily be locating the decision point. I want to ask: What exactly are you deciding whether or not to say, and to whom, and in what context? For example, if there are paradigm shifts between us and TAI, then the biggest infohazards would presumably involve what those paradigm-shifts are. And realistically, whatever the answer is, someone somewhere has already proposed it, but it hasn’t caught on, because it hasn’t so far been demonstrated to work well, and/or seems a priori implausible to most people, and/or is confusing or antimemetic [? · GW] for some other reason, etc. So if a random person says “I think the paradigm shift is X”, that’s probably not an infohazard even if they’re right, because random people make claims like that all the time and nobody listens. By contrast, if they are not a random person but rather a prominent leader in the field giving a NeurIPS keynote, or unusually good at explaining things in a compelling way, or controlling an AI funding source, or publishing turnkey SOTA code, or whatever, that might be different.

Second, “it empirically works on actual ML systems at scale” shouldn’t be a necessary criterion in general, in my opinion. For example, if there are paradigm shifts between us and TAI (as I happen to believe), then maybe we can think of actual ML systems as airplanes, and TAI as a yet-to-be-invented rocket ship, in an imaginary world where modern aircraft exist but rocket ships haven’t been invented yet. Anyway, early rocket ships are going to be far behind the airplane SOTA; in fact, for quite a while, they just won’t work at all, because they need yet-to-be-invented components like rocket fuel. Developing those components is important progress, but would not “empirically” advance the (airplane) SOTA.

I think about this a lot in the context of neuroscience. IMO the most infohazardous AI work on earth right now involves theoretical neuroscience researchers toiling away in obscurity, building models that don’t do anything very impressive. But we know that fully understanding the human brain is a recipe for TAI. So figuring out a little piece of that puzzle is unarguably a step towards TAI. The key is that: one piece of the puzzle, in the absence of all the other pieces, is not necessarily going to do anything impressive.

Third, (4) is not really how I would frame weighing the costs and benefits. For example, I would be especially interested in the question: “If this is true, then do I expect capabilities researchers to figure this out and publish it sooner or later, before TAI?” If so, it might make sense to just sit on it and find something else to work on until it’s published by someone else, even if it does have nonzero safety/alignment implications. Especially if there are other important alignment things to work on in the meantime (which there probably are). This has the side benefit of being a great strategy in the worlds where the idea is wrong anyway. It depends on the details in lots of ways, of course. More discussion & nuance in my post here [LW · GW], and see also Charlie’s post on tech trees [LW · GW].

(None of this is to invalidate Beren’s personal experience, which I think is pretty different from mine in various ways, especially that he’s working closer to the capabilities research mainstream, and was working at Conjecture which actually wanted to be out ahead of everyone else in (some aspects of) AI capabilities IIUC, whereas I’m speaking from my own experience doing alignment research with has a path-to-impact which mostly doesn’t require that.)

I think the question you need to ask yourself is "Given that there are O(10-100x) more Capabilities researchers than Alignment researchers, if I sit on this (exciting but untested) Capabilities idea I just had (which I don't recall reading anywhere), will that delay Capabilities by more months than the infohazard procedures required to keep this quiet will slow down Alignment? And how sure am I that this idea (if in fact viable) is a pure Capabilities idea and doesn't have any significant Alignment uses?"

I don't think the concept of infohazard as applied to AI alignment/safety has anything to do with the Great Man Theory. If we bought the Great Man Theory, we would also have to believe that at any time a random genius could develop ASI using only their laptop and unleash it onto the world, in which case, any hope of control is moot. Most people who support AI governance don't believe things are quite that extreme, and think that strategies ranging from "controlling compute" to "making it socially disreputable to work on AI capabilities" may effectively delay the development of AGI by significantly hurting the big collaborative projects.

On the flip side, would alignment work proceed much faster and more successfully if thousands of researchers with billions of dollars of funding worked on it exclusively, building on each others' intuition? Of course it would. But we do not live in that world. We live in a world in which economic incentives have aligned things so that the power of numbers lies on the side of capabilities. In this world, if you share your genius interpretability insight (which you may have, because while science isn't made only of Great Men on average, it certainly can advance in random leaps and bounds depending on individual contributions, and here even a mere 10 or 20 years of difference in the time of a discovery can be crucial), well, it's much more likely that it will just be used to make an even more powerful AI before anyone manages to use it to align the ones we have. So, keeping info contained isn't the best thing to do in some absolute sense, but it may be the best thing we can do now, as inefficient as it is, because it's the only random edge that a small community competing with a much larger and more powerful one can hope to gain.