Competition: Amplify Rohin’s Prediction on AGI researchers & Safety Concerns

Some information:

I don't intend to engage

I won't be responding to this thread until after the competition ends. If you're not sure about something in the question, assume what (you think) I would assume. Feel free to argue for a specific interpretation in a comment if you think it's underspecified.

Why I chose this question

In any realistic plan that ends with "and then we deployed the AI knowing we had mitigated risk X", it seems to me like we need prestigious AI researchers to have some amount of consensus that X was actually a real risk. If you want a company to use a safer technique, the key decision-makers at the company will need to believe that it's necessary in order to "pay the alignment tax", and their opinions will be shaped by the higher-up AI researchers at the company. If you want a government to regulate AI, you need to convince the government that X is a real risk, and the government will probably defer to prestigious AI experts.

So it seems like an important variable is whether the AI community agrees that X is a real risk. (Also obviously whether in reality X is a real risk, but I'm assuming that it is for now.) It may be fine if it's only the AGI researchers -- if a company knows it can build AGI, it probably ignores the opinions of people who think we can't build AGI.

Hence, this question. While an answer to the explicit question would be interesting to me, I'm more excited to have better models of what influences the answer to the question.

Prior reasoning

Copied over from the Google Doc linked above. Written during the 30 minutes I had to come up with a prior.

Seems very unlikely that if I ran this survey now the fraction would be 0.5. Let's give that 0.1%, which I can effectively ignore.

My timelines for "roughly human-level reasoning" have a median of let's say 20 years; it seems likely we get significant warning shots a few years before human-level reasoning, that lead to increased interest in safety. It's not crazy that we get a warning shot in the next year from e.g. GPT-3, though I'd be pretty surprised, call it ~1%.

My model for how we get to consensus is that there's a warning shot, or some excellent paper, or a series of these sorts of things, that increases the prominence of safety concerns, especially among "influencers", then this percolates to the general AI community over the next few years. I do think it can percolate quite quickly, see e.g. the fairly large effects of Concrete Problems in AI safety or how quickly fairness + bias became mainstream. (That's fewer examples than I'd like...) So let's expect 1 - 10 years after the first warning shot. Given median timelines of 20 years + warning shot shortly before then + 1-10 years to reach consensus, probably median for this question should also be ~20 years? (Note even if we build human-level reasoning before a majority is reached, the question could still resolve positively after that, since human-level reasoning != AI researchers are out of a job)

There's also a decent chance that we don't get a significant warning shot before superintelligent AI (maybe 10% e.g. via fast takeoff or good safety techniques that don't scale to AGI), or that not enough people update on it / consensus building takes forever / the population I chose just doesn't pay attention to safety for some reason (maybe another 15%), so let's say 25% that it never happens. Also another ~10% for AGI / warning shots not happening before 2100, or the safety community disappearing before 2100, etc. So that's 35% on >2100 (which includes never).

So 25% for never leaves 75% for "at some point" of which I argued the median is 20 years, so I should have ~35% from now till 20 years, and 30% from 20 years till 2100 (since it's 10% on >2100 but not never).

Then I played around with Elicit until I got a distribution that fit these constraints and looked vaguely lognormal.

Rohin has created his posterior distribution! Key differences from his prior are at the bounds:

• He now assigns 3% rather than 0.1% to the majority of AGI researchers already agreeing with safety concerns.
• He now assigns 40% rather than 35% to the majority of AGI researchers agreeing with safety concerns after 2100 or never.

Overall, Rohin’s posterior is a bit more optimistic than his prior and more uncertain.

Ethan Perez [LW · GW]’s snapshot wins the prize for the most accurate prediction of Rohin's posterior. Ethan kept a similar distribution shape while decreasing the probability >2100 less than the other submissions.

The prize for a comment that updated Rohin’s thinking goes to Jacob Pfau [LW(p) · GW(p)]! This was determined by a draw with comments weighted proportionally to how much they updated Rohin’s thinking.

Thanks to everyone who participated and congratulations to the winners! Feel free to continue making comments and distributions, and sharing any feedback you have on this competition.

My old prediction for when the fraction be >= 0.5: elicited

My old prediction for Rohin's posterior: elicited

I went through the top 20 list of most cited AI researchers on google scholar (thanks to Amanda for linking), and estimated that roughly 9 of them may qualify under Rohin's criterion. Of those 9, my guess was that 7/9 would answer 'Yes' on Rohin's question 3.

My sampling process was certainly biased. For one, AI researchers are likely to be more safety conscious than industry experts. My estimate also involved considerable guesswork, so I down-weighted the estimated 7/9 to a 65% chance that the >=0.5 threshold will be met within the first couple years. Given the extreme difference between my distribution and the others posted, I guess there's a 1/3 chance that my estimate based on the top 20 sampling will carry significant weight in Rohin's posterior.

The justification for the rest of my distribution is similar to what others have said here and elsewhere about AI safety. My AGI timeline is roughly in line with the metaculus estimate here. Before the advent of AGI, a number of eventualities are possible: a warning shot occurs, perhaps theoretical consensus will emerge, perhaps industry researchers will be oblivious to safety concerns because of a principal-agent nature to the problem, perhaps AGI will be invented before safety is worked out, etc.

Edit: One could certainly do a better job of estimating where the sample population of researchers currently stands by finding a less biased population. Maybe people interviewed by Lex Fridman, that might be a decent proxy for AGI-research-fame?

I think 1% in the next year and a half is significantly too low.

Firstly, conditioning on AGI researchers makes a pretty big difference. It rules out most mainstream AI researchers, including many of the most prominent ones who get the most media coverage. So I suspect your gut feeling about what people would say isn't taking this sufficiently into account.

Secondly, I think attributing ignorance to the outgroup is a pretty common fallacy, so you should be careful of that. I think a clear majority of AGI researchers are probably familiar with the concept of reward gaming by now, and could talk coherently about AGIs reward gaming, or manipulating humans. Maybe they couldn't give very concrete disaster scenarios, but neither can many of us.

And thirdly, once you get agreement that there are problems, you basically get "we should fix the problems first" for free. I model most AGI researchers as thinking that AGI is far enough away that we can figure out practical ways to prevent these things, like better protocols for giving feedback. So they'll agree that we should do that first, because they think that it'll happen automatically anyway.

In my experience, a highly predictive feature of "agreeing with safety concerns" is how much someone thinks about/works on RL (or decision-making more broadly). For example, many scientists in RL-driven labs (DeepMind, OpenAI) agree with safety concerns, while there is almost no understanding of safety concerns (let alone agreement that they are valid) among researchers in NLP (mainly driven by supervised learning and unsupervised learning); it's easier to intuitively motivate safety concerns from the perspective of RL and demos of where it fails (esp. with concerns like reward gaming and instrumental convergence). Thus, a useful way to decompose Rohin's question is:

1. How many top researchers doing AGI-related work think about RL?
   1. How many of the above researchers agree with safety concerns?
2. How many top researchers doing AGI-related work don't think about RL?
   1. How many of the above researchers agree with safety concerns?

We can add the numbers from 1.1 and 2.1 and divide by the total number of top AGI researchers (here, 2000).

I'd argue that more researchers are in category 2 (non-RL researchers) than 1 (RL researchers). A lot of recent progress towards AGI has been driven by improvements in representation learning, supervised learning, and unsupervised learning. Work in these areas is AGI-related in the relevant sense to Rohin; if we develop AGI without RL (e.g., GPT-10), we'd need the non-RL researchers who develop these models to coordinate and agree with safety concerns about e.g. releasing the models. I think it will continue to be the case for the foreseeable future that >50% of AGI-related researchers aren't doing RL, as representation learning, supervised learning, unsupervised learning, etc. all seem quite important to developing AGI.

The upshot (downshot?) of the above is that we'll probably need a good chunk of non-RL but AGI-related researchers to agree with safety concerns. Folks in this group seem less receptive to safety concerns (probably because they don't obviously come up as often as in RL work), so I think it'll take a pretty intuitively compelling demonstration/warning-shot to convince people in the non-RL group, visible enough to reach across several subfields in ML; preferably, these demos should be of direct relevance to the work in people doing non-RL work (i.e., directly showing how NLP systems are Stuart-Russell-style dangerous to convince NLP people). I think we'll need pretty advanced systems to get these kinds of demos, roughly +/-5 years from when we get AGI (vs. Rohin's prior estimate of 1-10 years before AGI). So overall, I think Rohin's posterior should be shifted right ~5 years.

Here is my Elicit snapshot of what I think Rohin's posterior will be after updating on all comments here. It seems like all the other comments are more optimistic than Rohin prior, so I predict that Rohin's posterior will become more optimistic, even though I think the concerns I've outlined above outweigh/override some of the considerations in the other comments. In particular, I think you'll get an overestimate of "agreement on safety concerns" by looking only at the NeurIPS/ICML community which is pretty RL-heavy relative to e.g. the NLP and Computer Vision communities (which will still face AGI-related coordination problems). The same can be said about researchers who explicitly self-identify with "Artificial Intelligence" or "Artificial General Intelligence" (historically focused on decision-making and games). Looking at the 100 most cited NLP researchers on Google Scholar, I found one who I could recognize as probably sympathetic to safety concerns (Wojciech Zaremba) and similar for Computer Vision.

My snapshot. I put 2% more mass on the next 2 years and 7% more mass on 2023-2032. My reasoning:

1. 50% is a low bar.

2. They just need to understand and endorse AI Safety concerns. They don't need to act on them.

3. There will be lots of public discussion about AI Safety in the next 12 years.

4. Younger researchers seem more likely to have AI Safety concerns. AI is a young field. (OTOH, it's possible that lots of the top cited/paid researchers in 10 years time are people active today).

Notes

• Field of AGI research plausibly commenced on 1956 with Dartmouth conference. What happens if one uses Laplace's rule? Then a priori pretty implausible that it will happen, if it hasn't happened soon.

• How do information cascades work in this context? How many researchers would I expect to have read and recall a reward gaming list (1, 2, 3, 4)

• Here is A list of good heuristics that the case for AI x-risk fails [LW · GW]. I'd expect that these, being pretty good heuristics, will keep having an effect on AGI researchers that will continue keeping them away from considering x-risks.

• Rohin probably doesn't actually have enough information or enough forecasting firepower to predict that it hasn't happened at 0.1%, and be calibrated. He probably does have the expertise, though. I did some experiments a while ago, and "I'd be very surprised if I were wrong" translated for me to a 95%. YMMV.

• An argument would go: "The question looks pretty fuzzy to me, having moving parts. Long tails are good in that case, and other forecasters who have found some small piece of evidence are over-updating." Some quotes:

  There is strong experimental evidence, however, that such self-insight is usually faulty. The expert perceives his or her own judgmental process, including the number of different kinds of information taken into account, as being considerably more complex than is in fact the case. Experts overestimate the importance of factors that have only a minor impact on their judgment and underestimate the extent to which their decisions are based on a few major variables. In short, people's mental models are simpler than they think, and the analyst is typically unaware not only of which variables should have the greatest influence, but also which variables actually are having the greatest influence. (Source: Psychology of Intelligence Analysis , Chapter 5)

  Our judges in this study were eight individuals, carefully selected for their expertise as handicappers. Each judge was presented with a list of 88 variables culled from the past performance charts. He was asked to indicate which five variables out of the 88 he would wish to use when handicapping a race, if all he could have was five variables. He was then asked to indicate which 10, which 20, and which 40 he would use if 10, 20, or 40 were available to him.

  We see that accuracy was as good with five variables as it was with 10, 20, or 40. The flat curve is an average over eight subjects and is somewhat misleading. Three of the eight actually showed a decrease in accuracy with more information, two improved, and three stayed about the same. All of the handicappers became more confident in their judgments as information increased. (Source: Behavioral Problems of Adhering to a Decision Policy)

  • I'm not sure to what extent this is happening with forecasters here: finding a particularly interesting and unique nugget of information and then over-updating. I'm also not sure to what extent I actually believe that this question is fuzzy and so long tails are good.

Here is my first entry to the competition. Here is my second and last entry to the competition. My changes are that I've assigned some probability (5%; I'd personally assign 10%) that it has already happened.

Some notes about that distribution:

• Note that this is not my actual distribution, this is my guess as to how Rohin will update
• My guess doesn't manipulate Rohin's distribution much; I expect that Rohin will not in fact change his mind a lot.
• In fact, this is not exactly my guess as how Rohin will update. That is, I'm not maximizing expected accuracy, I'm ~maximizing the chance of getting first place (subject to spending little time on this)

Some quick comments at forecasters:

• I think that the distinction between the forecaster's beliefs and Rohin's is being neglected. Some of the snapshots predict huge updates, which really don't seem likely.

I think it's >1% likely that the one of the first few surveys Rohin conducted would result in a fraction of >0.5.

Evidence from When Will AI Exceed Human Performance?, in the form of median survey responses of researchers who published at ICML and NIPS in 2015:

• 5% chance given to Human Level Machine Intelligence (HLMI) having an extremely bad long run impact (e.g. human extinction)
• Does Stuart Russell's argument for why highly advanced AI might pose a risk point at an important problem? 39% say at least important, 70% at least moderately important.
• But on the other hand, only 8.4% said working on this problem is now is more valuable than other problems in the field. 28% said as valuable as other problems.
• 47% agreed that society should prioritize "AI Safety Research" more than it currently was.

These seem like fairly safe lower bounds compared to the population of researchers Rohin would evaluate, since concern regarding safety has increased since 2015 and the survey included all AI researchers rather than only those whose work is related to AGI.

These responses are more directly related to the answer to Question 3 ("Does X agree that there is at least one concern such that we have not yet solved it and we should not build superintelligent AGI until we do solve it?") than Question 2 ("Does X broadly understand the main concerns of the safety community?"). I feel very uncertain about the percentage that would pass Question 2, but think it is more likely to be the "bottleneck" than Question 3.

Given these considerations, I increased the probability before 2023 to 10%, with 8% below the lower bound. I moved the median | not never up to 2035 as a higher probability pretty soon also means a sooner median. I decreased the probability of “never” to 20%, since the “not enough people update on it / consensus building takes forever / the population I chose just doesn't pay attention to safety for some reason” condition seems not as likely.

I also added an extra bin to ensure that the probability continues to decrease on the right side of the distribution.

My snapshot

Note: I'm interning at Ought and thus am ineligible for prizes.

Seems like it could be helpful if people who've thought about this would also predict on the question of what the survey value would be today. (e.g. via elicit snapshots)

I thought about how I could most efficiently update my and Rohin’s views on this question.

My best ideas are:
1. Get information directly on this question. What can we learn from surveys of AI researchers or from public statements from AI researchers?

2. Get information on the question’s reference class. What can we learn about how researchers working on other emerging technologies that might have huge risks thought about those risks?

I did a bit of research/thinking on these, which provided a small update towards thinking that AGI researchers will evaluate AGI risks appropriately.

I think that there’s a bunch more research that would be helpful -- in particular, does anyone know of surveys of AI researchers on their views on safety?

I answered the following subquestion to help me answer the overall question: “How likely is it that the condition Rohin specifies will not be met by 2100?”

This could happen due to any of the following non-mutually exclusive reasons:

1. Global catastrophe before the condition is met that makes it so that people are no longer thinking about AI safety (e.g. human extinction or end of civilization): I think there's a 50% chance

2. Condition is met sometime after the timeframe (mostly, I'm imagining that AI progress is slower than I expect): 5%

3. AGI succeeds despite the condition not being met: 30%

4. There's some huge paradigm shift that makes AI safety concerns irrelevant -- maybe most people are convinced that we'll never build AGI, or our focus shifts from AGI to some other technology: 10%

5. Some other reason: 20%

I thought about this subquestion before reading the comments or looking at Rohin’s distribution. Based on that thinking, I thought that there was a 60% chance that the condition would not be met by 2100.

I answered the following subquestion to help me answer the overall question: “How likely is it that the condition Rohin specified would already be met (if he went out and talked to the researchers today)?”

Considerations that make it more likely:

1. The considerations identified in ricaz [LW(p) · GW(p)]’s and Owain [LW(p) · GW(p)]’s comments and their subcomments

2. The bar for understanding safety concerns (question 2 on the "survey") seems like it may be quite low. It seems to me that researchers entirely unfamiliar with safety could gain the required level of understanding in just 30 minutes of reading (depends on how Rohin would interpret his conversation with the researcher in deciding whether to mark “Yes” or “No”)

Considerations that make it less likely:

1. I’d guess that currently, most AI researchers have no idea what any of the concrete safety concerns are, i.e. they’d be “No”s on question 2

2. The bar for question 3 on the "survey" ("should we wait to build AGI") might be pretty high. If someone thinks that some safety concerns remain but that we should cautiously move forward on building things that look more and more like AGI, does that count as a "Yes" or a "No"?

3. I have the general impression that many AI researchers really dislike the idea that safety concerns are serious enough that we should in any way slow down AI research

I thought about this subquestion before reading the comments or looking at Rohin’s distribution. Based on that thinking, I thought that there was a 25% chance that the condition Rohin specified would already be met.

Note: I work at Ought, so I'm ineligible for the prizes

If there's an AGI within, say, 10 years and it mostly keeps the world recognizable so there are still researchers to have opinions, does that resolve as "never" or according to whether the AGI wants them to be convinced? Because if the latter, I expect that they will in hindsight be convinced that we should have paid more attention to safety. If the former, I submit that his prior doesn't change. If the latter, I submit that the entire prior is moved 10 years to the left (and the first 10 years cut off, and then renormalize along the y axis).

Forecast - 25 mins

• I thought it was more likely that in the short run there could be a preference cascade among top AGI researchers, and as others have mentioned due to the operationalization of top AGI researchers might be true already.
• If this doesn't become a majority concern by 2050, I expect it will be because of another AI Winter, and I tried to have my distribution reflect that (a little hamfistedly).

I think the following is underspecified:

Does X agree that there is at least one concern such that we have not yet solved it and we should not build superintelligent AGI until we do solve it?

What counts as building superintelligent AGI?

This could mean anything from working on foundational theory which could be used to facilitate building an AGI, to finishing the final phase of training on a fully functional AGI implementation.

In the former case you're going to get close to 0% agreement. In the latter, well over 50% already (I hope!).

I don't see any natural/clear in-the-spirit-of-the-question interpretation. E.g. if we set the bar as low as: "build superintelligent AGI" = "do non-safety work related to AGI", then you'll never get above 50% agreement, since anyone who fulfils (1) must deny (3) or admit they're already being irresponsible.

I don't think it's a useful question without clarification of this.

As things stand, it'll be pretty easy for a researcher to decide that they're not building superintelligent AGI, almost regardless of what they're doing. It's then easy to concede that safety problems should be solved first, since that can always mean later.

On the other hand, requiring that a researcher agrees with Rohin on interpretation of "build superintelligent AGI" in order to say that they "agree with safety concerns" seems a high bar.

My distribution

My biggest differences with Rohin's prior distribution are:

1. I think that it's much more likely than he does that AGI researchers already agree with safety concerns

2. I think it's considerably more likely than he does that the majority of AGI researchers will never agree with safety concerns

These differences are explained more on my distribution and in my other comments.

The next step that I think would help the most to make my distribution better would be to do more research [LW(p) · GW(p)].

If people don't have a strong sense of who these people are/would be, you can look through this google scholar citation list (this is just the top AI researchers, not AGI researchers).