[Crosspost] AI X-risk in the News: How Effective are Recent Media Items and How is Awareness Changing? Our New Survey Results.

post by otto.barten (otto-barten) · 2023-05-04T14:09:00.134Z · LW · GW · 0 comments

This is a link post for https://forum.effectivealtruism.org/posts/YweBjDwgdco669H72/ai-x-risk-in-the-news-how-effective-are-recent-media-items

Contents

  PART 1: Effectiveness per media item
    Media Items Examined:
    Results:
  PART 2: Awareness change over time and awareness as a function of length and trust level
    Media Items Examined:
    Results:
None
No comments

This is a summary of a follow-up study conducted by the Existential Risk Observatory, which delves into a greater number media items. To access our previous study, please follow this link [EA · GW]. The data collected will be presented in two separate posts. The first post, which is the current one, has two parts. The first part examines the key indicators used in the previous research, such as "Human Extinction Events" and "Human Extinction Percentage," along with a new key indicator called "Concern Level." The Concern Level indicator assesses participants' level of concern about AI existential risk on a scale of 0 to 10 before and after the intervention. The second part analyzes the changes in public awareness about AI existential risk over time. It also explores the connection between the effectiveness of different media formats, namely articles and videos, and their length in raising awareness. In addition, it investigates how trust levels are related to the effectiveness of media sources in increasing public awareness of AI existential risk. In the second post, the research covers a new aspect of this study: participants' opinions on an AI moratorium and their likelihood of voting for it.

PART 1: Effectiveness per media item

This research aimed to evaluate the effectiveness of AI existential risk communication in increasing awareness of the potential risks posed by AI to human extinction.

Research Objectives: The objective of the study was to determine the effectiveness of AI existential risk communication in raising public awareness. This was done by examining the changes in participants' views on the likelihood and ranking of AI as a potential cause of extinction before and after the intervention. Furthermore, the study evaluated the difference in the level of concern of participants before and after the intervention.

Measurements and Operationalization: Three primary measurements - "Human Extinction Events," "Human Extinction Percentage," and "Concern Level" - were utilized to examine alterations in participants' perceptions. The coding scheme that was previously used in our research was employed to assess participants' increased awareness of AI. The data was gathered through Prolific, a platform that locates survey respondents based on predefined criteria. The study involved 350 participants, with 50 participants in each survey, who were required to be at least 18 years old, residents of the United States, and fluent in English.

Data Collection and Analysis: Data was collected through surveys in April 2023. The data analysis comprised three main sections: (1) comparing changes in the key indicators before and after the intervention, (2) exploring participants' views on the possibility of an AI moratorium and their likelihood of voting for it, and (3) assessing the number of participants who were familiar with or had confidence in the media channel used in the intervention.

Media Items Examined:

  1. CNN: Stuart Russell on why A.I. experiments must be paused
  2. CNBC: Here's why A.I. needs a six-month pause: NYU Professor Gary Marcus
  3. The Economist: How to stop AI going rogue
  4. Time 1: Why Uncontrollable AI Looks More Likely Than Ever | Time
  5. Time 2: The Only Way to Deal With the Threat From AI? Shut It Down | Time
  6. FoxNews Article: Artificial intelligence 'godfather' on AI possibly wiping out humanity: ‘It's not inconceivable’ | Article
  7. FoxNews Video: White House responds to concerns about AI development | Video

Results: 

Human Extinction Events 

The graph below displays the percentage of increased awareness across various media sources. The Economist survey showed the highest increase in awareness at 52 percent, followed by CNBC, CNN, and Time (Eliezer Yudkowsky) surveys, all with a 48 percentage increase. FoxNews surveys, in both video and article formats, had the lowest increase in awareness at around 30 percent. The success of The Economist, CNBC, CNN, and Time may suggest that media items from reputable sources are more effective than those considered to have lower trustworthiness, such as FoxNews. To evaluate this hypothesis, participants were asked if they trusted the news channel from which they consumed the media item. The Economist, CNBC, CNN, and Time had more than 80 percent of participants indicating some level of trust in the newspaper, whereas the corresponding figures for the FoxNews surveys (Article and Video) were 55 and 62 percent, averaging around 58 for the media channel. In addition, media items from CNBC, CNN, and Time featured well-known individuals or university professors, which may have contributed to their higher effectiveness.

Figure 1: Percentage of participants who exhibited higher awareness per the Human Extinction Events indicator after the intervention across surveys.

Human Extinction Percentage

The graph below presents a picture of the distribution of the Human Extinction Percentage indicator before and after the intervention across various media sources. All surveys experienced an increase in both mean and median percentage from before to after the intervention. The Time (Yudkowsky) survey displayed the highest change in mean percentage with an increase of 11 percent, followed closely by CNBC and CNN with a 9 percent increase. Meanwhile, The Economist and the FoxNews video surveys had approximately an 8 point increase in mean percentage. The second Time (Barten and Yampolskiy) and FoxNews Article survey recorded the lowest increase in mean percentage, both around 4 percent.

The results further support the argument that media items from reputable sources have a greater impact on the public than those considered to be from less trustworthy sources. However, it is important to note that the Human Extinction Percentage indicator had some conflicting results, particularly with the FoxNews Video survey showing a relatively high output, comparable to that of The Economist, while the second Time (Barten and Yampolskiy) survey had the lowest performance.

Interestingly, the median change for CNN, both FoxNews surveys, and the second Time (Barten and Yampolskiy) survey recorded the lowest changes in median as shown in the graph below. This may suggest that the high mean percentage changes seen in the CNN and FoxNews surveys may be due to some of the data points being much higher in the range compared to the other surveys. This may indicate that for those who do trust a media channel, such as FoxNews, it can lead to significantly higher levels of change in percentage.

Figure 2: Pre-post summary of the distribution of percentage values for the Human Extinction Percentage indicator across surveys.

Concern Level

The graph below depicts the distribution of the Concern Level indicator across various media sources before and after the intervention. All the surveys exhibited an increase in both the mean and median values of concern levels from before to after the intervention. The Economist and CNN surveys demonstrated the most substantial increase in mean value, with an increase of 1.5 points. CNBC and both Time surveys showed a similar trend with a mean increase of 1.3 points. The FoxNews surveys reported the lowest mean increase in concern levels, with 0.9 for the article version and 0.7 for the video version. As in the previous section, we can argue that media items from reputable sources may be more effective in raising concern levels. Additionally, the Economist survey had an impressive increase in the median value of concern levels, averaging more than 2.5 points from before to after the intervention. This could suggest that media sources with a high level of trustworthiness can have a significant impact on changing people's level of concern regarding AI existential risk.

Figure 3: Pre-post summary of the distribution of values from a scale from 0 to 10 for the Concern Level indicator across surveys.

PART 2: Awareness change over time and awareness as a function of length and trust level

Research Questions: The objective of the study was to determine the effectiveness of AI existential risk communication in raising public awareness. To achieve this, the research analyzed the relationship between the length of media items and their impact on raising awareness, measured by the key indicator Human Extinction Events. In addition, the study investigated how the level of trust in media sources is related to the magnitude of awareness increase, as shown by the same key indicator.

Measurements, Operationalization and Data Collection/Analysis: Part 2 of the research adheres to the same methodology and data collection approach as Part 1, but with two notable adjustments. Firstly, it narrows its attention to the "Human Extinction Events" key indicator to scrutinize how the length of media items and trust levels influence the effectiveness of AI existential risk communication. Secondly, it incorporates data from the previous research iteration, conducted in December 2022, and the current one, conducted in April 2023, to provide a more comprehensive analysis.

Media Items Examined:

First Iteration (December 2022):

  1. Elon Musk (EM): "I Tried To Warn You" - Elon Musk LAST WARNING (2023)
  2. PewDiePie (PDP): WE ARE ALL GOING TO D1E. (AI)
  3. CNN (1CNN): Hawking: A.I. could be end of human race | CNN
  4. CNBC (1CNBC): Stephen Hawking says A.I. could be 'worst event in the history of our civilization'
  5. Salon: Human-level AI is a giant risk. Why are we entrusting its development to tech CEOs?
  6. Vox: The case that AI threatens humanity, explained in 500 words
  7. Washington Post (WP): What is the worst-case AI scenario? Human extinction.

Second Iteration (April 2023):

  1. CNN (2CNN): Stuart Russell on why A.I. experiments must be paused
  2. CNBC (2CNBC): Here's why A.I. needs a six-month pause: NYU Professor Gary Marcus
  3. The Economist (TE): How to stop AI going rogue
  4. Time 1 (T1): Why Uncontrollable AI Looks More Likely Than Ever | Time
  5. Time 2 (T2): The Only Way to Deal With the Threat From AI? Shut It Down | Time
  6. FoxNews Article (FNA): Artificial intelligence 'godfather' on AI possibly wiping out humanity: ‘It's not inconceivable’
  7. FoxNews Video (FNV): White House responds to concerns about AI development

Results:

Change in Public Awareness of AI Existential Risk Over Time

In both research iterations, participants were asked to rank three events that they believed could cause human extinction in the next century, before and after the intervention, which was used as the Human Extinction Events indicator. To assess the level of public awareness of AI existential risk, the study measured the percentage of participants who mentioned AI in the first instance the question was asked before the intervention. The figure below shows the change in awareness from the first round, conducted in December 2022, to the second round, conducted in April 2023. The percentage of participants who mentioned AI increased from approximately 6.9 percent to 12.3 percent pre-intervention, indicating a growth of 178 percent, which is almost double the amount of the first iteration. This increase in awareness of AI existential risk may be attributed to the surge in media coverage related to the release of ChatGPT. In the second iteration of the study, participants were also asked if they knew the AI chatbot ChatGPT, to which at least 80 percent responded positively in all surveys.

Figure 4: Change in awareness of AI existential risk among participants over time.

Article Length and Increase in Awareness

This section of the study examines the connection between the length of articles, measured in words, and the increase in awareness indicated by the Human Extinction Events measure. A positive correlation can be observed, with a higher percentage of participants achieving greater awareness as article length increases. However, there are some exceptions to this trend, such as the Washington Post article, which has a similar word count to the Salon article but a significantly lower increase in awareness percentage. While the trend is apparent, additional data collection is necessary to draw a more definitive conclusion.

Figure 5: The relationship between article length (in words) and increased awareness.

Video Length and Increase in Awareness

This section of the study examines the relationship between the duration of videos, measured in minutes, and the increase in awareness based on the Human Extinction Events indicator. A small positive correlation can be observed, with a greater percentage of participants achieving a higher level of awareness as video length increases. However, the CNN video from the first iteration stands out as an outlier, performing significantly better than its counterpart from the second iteration. This suggests that the trend may not be as clear as with articles. Nonetheless, to reach a more conclusive outcome, further data collection would be necessary.

 

Figure 6: The relationship between video length (in minutes) and increased awareness.

Trust levels and Increase in Awareness

This section of the study explores the relationship between participants' trust levels in media sources, as indicated by their response to a question about whether they trust the media source of the media item, and the increase in awareness indicated by the Human Extinction Events indicator. A generally positive correlation can be observed, with a higher percentage of participants achieving greater awareness as their trust levels in the media source increase. However, certain exceptions exist, such as the Washington Post and CNN articles which were also previously mentioned. The CNN and Washington Post articles had awareness increase values respectively far above and below the line of expected awareness increase based on their trust levels. Despite these exceptions, other articles seem to follow a consistent trend, suggesting that greater trust in a media source results in higher awareness increase. For instance, media sources like The Economist and CNN, which were more trusted, have a higher awareness increase, whereas FoxNews, with lower trust levels, has lower awareness increase. Nevertheless, more data is required to arrive at a more definitive conclusion.

 

 

Figure 7: The relationship between trust level (in percentage) and increased awareness.

Conclusion: Based on the results of this study, it can be suggested that AI existential risk communication from reputable media sources can be an effective means of raising public awareness and concern regarding the potential risks associated with AI. The data indicates that media items from sources with higher levels of trustworthiness, such as The Economist, CNBC, CNN, and Time, have a greater impact on the public than those sources considered to be less trustworthy, such as FoxNews.

The study also reveals that all key indicators increased significantly across all surveys after the intervention, suggesting that the AI existential risk communication was effective in changing participants' perceptions. Additionally, the median and mean changes in both indicators were generally higher for surveys from more reputable media sources.

In addition, there appears to be a correlation between media exposure and awareness of AI existential risk. The study found that participants who were exposed to longer articles or videos had higher levels of awareness of AI existential risk. Furthermore, participants who trusted the media source also had higher levels of awareness. However, there were some outliers to these trends, such as the Washington Post and CNN articles. Overall, the study suggests that media exposure and trust are important factors in increasing awareness of AI existential risk. However, more data is needed to make a more conclusive result.

Although the study offers valuable insights into how various media sources can increase public awareness of AI existential risk and influence perceptions, there are limitations that require further investigation. One of the limitations is that the study only focused on a particular demographic - residents of the United States who are fluent in English. Therefore, the generalizability of the findings to other populations with different cultural backgrounds or language preferences is limited. To address this limitation, future studies should include diverse populations to provide more comprehensive insights.

Additionally, the study only examined the short-term effects of media interventions, and there is a need to explore the long-term effects of different types of media sources on public perceptions and behaviors regarding AI existential risk. To achieve this, longitudinal studies could be conducted to track changes in awareness and trust levels over time as media coverage of AI-related issues evolves.

Future research should also explore the impact of various media types such as podcasts, social media posts, documentaries, and news articles of different lengths on awareness and trust levels related to AI existential risks. This will provide a better understanding of how different media types influence public perceptions and attitudes towards AI existential risk.

Furthermore, the study relied solely on self-reported data, which may be biased or inaccurate. Future research should consider using other sources of data, such as behavioral data, to corroborate the findings. This will improve the reliability of the results and provide a more comprehensive understanding of the relationship between media exposure and public perceptions of AI existential risk.

Lastly, while the study found that media sources with higher trustworthiness tend to have a greater impact on public perceptions, it did not investigate the reasons behind this phenomenon. Future research could explore the factors that contribute to the perceived trustworthiness of media sources. This could be achieved by analyzing the impact of cultural and societal factors on awareness and trust levels related to AI existential risks in different regions of the world.

0 comments

Comments sorted by top scores.