Posts
Comments
I found this article useful. Any plans to update this for 2024?
Wow, high praise for MATS! Thank you so much :) This list is also great for our Summer 2024 Program planning.
Another point: Despite our broad call for mentors, only ~2 individuals expressed interest in mentorship who we did not ultimately decide to support. It's possible our outreach could be improved and I'm happy to discuss in DMs.
I don't see this distribution of research projects as "Goodharting" or "overfocusing" on projects with clear feedback loops. As MATS is principally a program for prosaic AI alignment at the moment, most research conducted within the program should be within this paradigm. We believe projects that frequently "touch reality" often offer the highest expected value in terms of reducing AI catastrophic risk, and principally support non-prosaic, "speculative," and emerging research agendas for their “exploration value," which might aid potential paradigm shifts, as well as to round out our portfolio (i.e., "hedge our bets").
However, even with the focus on prosaic AI alignment research agendas, our Summer 2023 Program supported many emerging or neglected research agendas, including projects in agent foundations, simulator theory, cooperative/multipolar AI (including s-risks), the nascent "activation engineering" approach our program helped pioneer, and the emerging "cyborgism" research agenda.
Additionally, our mentor portfolio is somewhat conditioned on the preferences of our funders. While we largely endorse our funders' priorities, we are seeking additional funding diversification so that we can support further speculative "research bets". If you are aware of large funders willing to support our program, please let me know!
There seems to be a bit of pushback against "postmortem" and our team is ambivalent, so I changed to "retrospective."
Thank you!
Ok, added!
FYI, the Net Promoter score is 38%.
Ok, graph is updated!
Do you think "46% of scholar projects were rated 9/10 or higher" is better? What about "scholar projects were rated 8.1/10 on average" ?
We also asked mentors to rate scholars' "depth of technical ability," "breadth of AI safety knowledge," "research taste," and "value alignment." We ommitted these results from the report to prevent bloat, but your comment makes me think we should re-add them.
Yeah, I just realized the graph is wrong; it seems like the 10/10 scores were truncated. We'll upload a new graph shortly.
Cheers, Vaniver! As indicated in the figure legend for "Mentor ratings of scholar research", mentors were asked, “Taking the above [depth/breadth/taste ratings] into account, how strongly do you support the scholar's research continuing?” and prompted with:
- 10/10 = Very disappointed if [the research] didn't continue;
- 5/10 = On the fence, unsure what the right call is;
- 1/10 = Fine if research doesn't continue.
Mentors rated 18% of scholar research projects as 10/10 and 28% as 9/10.
Also, last year's program was 8-weeks and this year's program is 10-weeks.
Buck Shlegeris, Ethan Perez, Evan Hubinger, and Owain Evans are mentoring in both programs. The links show their MATS projects, "personal fit" for applicants, and (where applicable) applicant selection questions, designed to mimic the research experience.
Astra seems like an obviously better choice for applicants principally interested in:
- AI governance: MATS has no AI governance mentors in the Winter 2023-24 Program, whereas Astra has Daniel Kokotajlo, Richard Ngo, and associated staff at ARC Evals and Open Phil;
- Worldview investigations: Astra has Ajeya Cotra, Tom Davidson, and Lukas Finnvedan, whereas MATS has no Open Phil mentors;
- ARC Evals: While both programs feature mentors working on evals, only Astra is working with ARC Evals;
- AI ethics: Astra is working with Rob Long.
MATS has the following features that might be worth considering:
- Empowerment: Emphasis on empowering scholars to develop as future "research leads" (think accelerated PhD-style program rather than a traditional internship), including research strategy workshops, significant opportunities for scholar project ownership (though the extent of this varies between mentors), and a 4-month extension program;
- Diversity: Emphasis on a broad portfolio of AI safety research agendas and perspectives with a large, diverse cohort (50-60) and comprehensive seminar program;
- Support: Dedicated and experienced scholar support + research coach/manager staff and infrastructure;
- Network: Large and supportive alumni network that regularly sparks research collaborations and AI safety start-ups (e.g., Apollo, Leap Labs, Timaeus, Cadenza, CAIP);
- Experience: Have run successful research cohorts with 30, 58, 60 scholars, plus three extension programs with about half as many participants.
Our alumni were surveyed about what amount of funding they would consider sufficient to participate in MATS and the median (discounting the $0 responses) was quite low.
Yes
We agree, which is why we note, "We think that ~1 more median MATS scholar focused on AI safety is worth 5-10 more median capabilities researchers (because most do pointless stuff like image generation, and there is more low-hanging fruit in safety)."
MATS' goals:
- Find + accelerate high-impact research scholars:
- Pair scholars with research mentors via specialized mentor-generated selection questions (visible on our website);
- Provide a thriving academic community for research collaboration, peer feedback, and social networking;
- Develop scholars according to the “T-model of research” (breadth/depth/epistemology);
- Offer opt-in curriculum elements, including seminars, research strategy workshops, 1-1 researcher unblocking support, peer study groups, and networking events;
- Support high-impact research mentors:
- Scholars are often good research assistants and future hires;
- Scholars can offer substantive new critiques of alignment proposals;
- Our community, research coaching, and operations free up valuable mentor time and increase scholar output;
- Help parallelize high-impact AI alignment research:
- Find, develop, and refer scholars with strong research ability, value alignment, and epistemics;
- Use alumni for peer-mentoring in later cohorts;
- Update mentor list and curriculum as the alignment field’s needs change.
Types of organizations that conduct alignment research, differentiated by funding model and associated market forces:
- Academic research groups (e.g., Krueger's lab at Cambridge, UC Berkeley CHAI, NYU ARG, MIT AAG);
- Research nonprofits (e.g., ARC Theory, MIRI, FAR AI, Redwood Research);
- "Mixed funding model" organizations:
- "Alignment-as-a-service" organizations, where the product directly contributes to alignment (e.g., Apollo Research, Aligned AI, ARC Evals, Leap Labs);
- "Alignment-on-the-side" organizations, where product revenue helps funds alignment research (e.g., Conjecture);
- Scaling labs, where alignment research is mostly directed towards improving product (e.g., Anthropic, DeepMind, OpenAI).
The Summer 2023 Cohort has 460 applicants. Our last cohort included 57 scholars.
As an educational seminar and independent research program, MATS cannot offer J1 visas. We can support scholars' ESTA and B1/B2 visa applications, however.
John's scholars have historically only had to seek LTFF funding for the 4-month extension program subsequent to the in-person Scholars Program. They are otherwise treated like other scholars.
Hi Pulkit. Unfortunately, applications have closed for our Summer 2023 Cohort. Hopefully, we will launch applications for our Winter Cohort soon!
I'm somewhere in the middle of the cognitivist/enactivist spectrum. I think that e.g. relaxed adversarial training is motivated by trying to make an AI robust to arbitrary inputs it will receive in the world before it leaves the box. I'm sympathetic to the belief that this is computationally intractable; however, it feels more achievable than altering the world in the way I imagine would be necessary without it.
I'm not an idealist here: I think that some civilizational inadequacies should be addressed (e.g., better cooperation and commitment mechanisms) concurrent with in-the-box alignment strategies. My main hope is that we can build an in-the-box corrigible AGI that allows in-deployment modification.
I agree with you that AI is generally seen as "the big thing" now, and we are very unlikely to be counterfactual in encouraging AI hype. This was a large factor in our recent decision to advertise the Summer 2023 Cohort via a Twitter post and a shout-out on Rob Miles' YouTube and TikTok channels.
However, because we provide a relatively simple opportunity to gain access to mentorship from scientists at scaling labs, we believe that our program might seem attractive to aspiring AI researchers who are not fundamentally directed toward reducing x-risk. We believe that accepting such individuals as scholars is bad because:
- We might counterfactually accelerate their ability to contribute to AI capabilities;
- They might displace an x-risk-motivated scholar.
Therefore, while we intend to expand our advertising approach to capture more out-of-network applicants, we do not currently plan to reduce the selection pressures for x-risk-motivated scholars.
Another crux here is that I believe the field is in a nascent stage where new funders and the public might be swayed by fundamentally bad "AI safety" projects that make AI systems more commercialisable without reducing x-risk. Empowering founders of such projects is not a goal of MATS. After the field has grown a bit larger while maintaining its focus on reducing x-risk, there will hopefully be less "free energy" for naive AI safety projects, and we can afford to be less choosy with scholars.
Mentorship is critical to MATS. We generally haven't accepted mentorless scholars because we believe that mentors' accumulated knowledge is extremely useful for bootstrapping strong, original researchers.
Let me explain my chain of thought better:
- A first-order failure mode would be "no one downloads experts' models, and we grow a field of naive, overconfident takes." In this scenario, we have maximized exploration at the cost of accumulated knowledge transmission (and probably useful originality, as novices might make the same basic mistakes). We patch this by creating a mechanism by which scholars are selected for their ability to download mentors' models (and encouraged to do so).
- A second-order failure mode would be "everyone downloads and defers to mentors' models, and we grow a field of paradigm-locked, non-critical takes." In this scenario, we have maximized the exploitation of existing paradigms at the cost of epistemic diversity or critical analysis. We patch this by creating mechanisms for scholars to critically examine their assumptions and debate with peers.
I think we agree on a lot more than I realized! In particular, I don't disagree with your general claims about pathways to HRAD through Alignment MVPs (though I hold some credence that this might not work). Things I disagree with:
- I generally disagree with the claim "alignment approaches don't limit agentic capability." This was one subject of my independent research before I started directing MATS. Hopefully, I can publish some high-level summaries soon, time permitting! In short, I think "aligning models" generally trades off bits of optimization pressure with "making models performance-competitive," which makes building aligned models less training-competitive for a given degree of performance.
- I generally disagree with the claim "corrigibility is not a useful, coherent concept." I think there is a (narrow) attractor basin around "corrigibility" in cognition space. Happy to discuss more and possibly update.
- I generally disagree with the claim "provably-controllable highly reliable agent design is impossible in principle." I think it is possible to design recursively self-improving programs that are robust to adversarial inputs, even if this is vanishingly hard in practice (which informs my sense of alignment difficulty only insomuch as I hope we don't hit that attractor well before CEV value-loading is accomplished). Happy to discuss and possibly update.
- I generally disagree with the implicit claim "it's useful to try aligning AI systems via mechanism design on civilization." This feels like a vastly clumsier version of trying to shape AGIs via black-box gradient descent. I also don't think that realistic pre-AGI efficient markets we can build are aligned with human-CEV by default.
- We broadened our advertising approach for the Summer 2023 Cohort, including a Twitter post and a shout-out on Rob Miles' YouTube and TikTok channels. We expected some lowering of average applicant quality as a result but have yet to see a massive influx of applicants from these sources. We additionally focused more on targeted advertising to AI safety student groups, given their recent growth. We will publish updated applicant statistics after our applications close.
- In addition to applicant selection and curriculum elements, our Scholar Support staff, introduced in the Winter 2022-23 Cohort, supplement the mentorship experience by providing 1-1 research strategy and unblocking support for scholars. This program feature aims to:
- Supplement and augment mentorship with 1-1 debugging, planning, and unblocking;
- Allow air-gapping of evaluation and support, improving scholar outcomes by resolving issues they would not take to their mentor;
- Solve scholars’ problems, giving more time for research.
- Defining "good alignment research" is very complicated and merits a post of its own (or two, if you also include the theories of change that MATS endorses). We are currently developing scholar research ability through curriculum elements focused on breadth, depth, and epistemology (the "T-model of research"):
- Breadth-first search (literature reviews, building a "toolbox" of knowledge, noticing gaps);
- Depth-first search (forming testable hypotheses, project-specific skills, executing research, recursing appropriately, using checkpoints);
- Epistemology (identifying threat models, backchaining to local search, applying builder/breaker methodology, babble and prune, "infinite-compute/time" style problem decompositions, etc.).
- Our Alumni Spotlight includes an incomplete list of projects we highlight. Many more past scholar projects seem promising to us but have yet to meet our criteria for inclusion here. Watch this space.
- Since Summer 2022, MATS has explicitly been trying to parallelize the field of AI safety as much as is prudent, given the available mentorship and scholarly talent. In longer-timeline worlds, more careful serial research seems prudent, as growing the field rapidly is a risk for the reasons outlined in the above article. We believe that MATS' goals have grown more important from the perspective of timelines shortening (though MATS management has not updated on timelines much as they were already fairly short in our estimation).
- MATS would love to support senior research talent interested in transitioning into AI safety! Our scholars generally comprise 10% Postdocs, and we would like this number to rise. Currently, our advertising strategy is contingent on the AI safety community adequately targeting these populations (which seems false) and might change for future cohorts.
I don't disagree with Shimi as strongly as you do. I think there's some chance we need radically new paradigms of aligning AI than "build alignment MVPs via RLHF + adversarial training + scalable oversight + regularizers + transparency tools + mulligans."
While I do endorse some anthropocentric "value-loading"-based alignment strategies in my portfolio, such as Shard Theory and Steve Byrnes' research, I worry about overly investing in anthropocentric AGI alignment strategies. I don't necessarily think that RLHF shapes GPT-N in a manner similar to how natural selection and related processes shaped humans to be altruistic. I think it's quite likely that the kind of cognition that GPT-N learns to predict tokens is more akin to an "alien god" than it is to human cognition. I think that trying to value-load an alien god is pretty hard.
In general, I don't highly endorse the framing of alignment as "making AIs more human." I think this kind of approach fails in some worlds and might produce models that are not performance-competitive enough to outcompete the unaligned models others deploy. I'd rather produce corrigible models with superhuman cognition coupled with robust democratic institutions. Nevertheless, I endorse at least some research along this line, but this is not the majority of my portfolio.
Thank you for recommending your study guide; it looks quite interesting.
MATS does not endorse “maximizing originality” in our curriculum. We believe that good original research in AI safety comes from a combination of broad interdisciplinary knowledge, deep technical knowledge, and strong epistemological investigation, which is why we emphasize all three. I’m a bit confused by your reference to Adam’s post. I interpret his post as advocating for more originality, not less, in terms of diverse alignment research agendas.
I think that some of the examples you gave of "non-alignment" research areas are potentially useful subproblems for what I term "impact alignment." For example, strong anomaly detection (e.g., via mechanistic interpretability, OOD warning lights, or RAT-style acceptability verification) can help ensure "inner alignment" (e.g., through assessing corrigibility or myopia) and infosec/governance/meme-curation can help ensure "outer alignment," "paying the alignment tax," and mitigating "vulnerable world" situations with stolen/open sourced weaponizable models. I think this inner/outer distinction is a useful framing, though not the only way to carve reality at the joints, of course.
I think Metzinger or Clark could give interesting seminars, though I'm generally worried about encouraging the anthropomorphism of AGI. I like the "shoggoth" or "alien god" memetic framings of AGI, as these (while wrong) permit a superset of human-like behavior without restricting assumptions of model internals to (unlikely and optimistic, imo) human-like cognition. In this vein, I particularly like Steve Byrnes' research as I feel it doesn't overly anthropomorphize AGI and encourages the "competent psychopath with alien goals" memetic framing. I'm intrigued by this suggestion, however. How do you think Metzinger or Clark would specifically benefit our scholars?
(Note: I've tried to differentiate between MATS' organizational position and mine by using "I" or "we" when appropriate.)
MATS' framing is that we are supporting a "diverse portfolio" of research agendas that might "pay off" in different worlds (i.e., your "hedging bets" analogy is accurate). We also think the listed research agendas have some synergy you might have missed. For example, interpretability research might build into better AI-assisted white-box auditing, white/gray-box steering (e.g., via ELK), or safe architecture design (e.g., "retargeting the search").
The distinction between "evaluator" and "generator" seems fuzzier to me than you portray. For instance, two "generator" AIs might be able to red-team each other for the purposes of evaluating an alignment strategy.
MATS aims to find and accelerate alignment research talent, including:
- Developing scholar research ability through curriculum elements focused on breadth, depth, and originality (the "T-model of research");
- Assisting scholars in producing impactful research through research mentorship, a community of collaborative peers, dedicated 1-1 support, and educational seminars;
- Aiding the creation of impactful new alignment organizations (e.g., Jessica Rumbelow's Leap Labs and Marius Hobbhahn's Apollo Research);
- Preparing scholars for impactful alignment research roles in existing organizations.
Not all alumni will end up in existing alignment research organizations immediately; some return to academia, pursue independent research, or potentially skill-up in industry (to eventually aid alignment research efforts). We generally aim to find talent with existing research ability and empower it to work on alignment, not necessarily through existing initiatives (though we certainly endorse many).
In general, MATS gives our chosen mentors the final say on certain curriculum elements (e.g., research projects offered, mentoring style, mentor-specific training program elements) but offers a general curriculum (e.g., Alignment 201, seminars, research strategy/tools workshops) that scholars can opt into.
I think there should be much more seminars about philosophy of science, methodology of science, and strategy of research in the program. Perhaps, not at the expense of other seminar content, but via increasing the number of seminar hours. Talks about ethics, and in particular about ethical naturalism (such as Oliver Scott Curry’s “Morality as Cooperation”), or interaction of ethics and science more generally, also seem essential.
MATS is currently focused on developing scholars as per the "T-model of research" with three main levers:
- Mentorship (weekly meetings + mentor-specific curriculum);
- Curriculum (seminars + workshops + Alignment 201 + topic study groups);
- Scholar support (1-1s for unblocking + research strategy).
The "T-model of research" is:
- Breadth-first search (literature reviews, building a "toolbox" of knowledge, noticing gaps);
- Depth-first search (forming testable hypotheses, executing research, recursing appropriately, using checkpoints);
- Originality (identifying threat models, backchaining to local search, applying builder/breaker methodology, babble and prune, "infinite-compute/time" style problem decompositions, etc.).
In the Winter 2022-23 Cohort, we ran several research strategy workshops (focusing on problem decomposition and strategy identification) and had dedicated scholar support staff who offered regular, airgapped 1-1 support for research strategy and researcher unblocking. We will publish some findings from our Scholar Support Post-Mortem soon. We plan to run further research strategy and “originality” workshops in the Summer 2023 Cohort and are currently reviewing our curriculum from the ground up.
Currently, we are not focused on “ethical naturalism” or similar as a curriculum element as it does not seem broadly useful for our cohort compared to project-specific, scholar-driven self-education. We are potentially open to hosting a seminar on this topic.
Independent researchers sometimes seem to think that if they come up with a particularly brilliant idea about alignment, leading AGI labs will adopt it. Not realising that this won’t happen no matter what partially for social reasons
MATS generally thinks that separating the alignment problem into “lowering the alignment tax” via technical research and “convincing others to pay the alignment tax” via governance interventions is a useful framing. There are worlds in which the two are not so cleanly separable, of course, but we believe that making progress toward at least one of these goals is probably useful (particularly if more governance-focused initiatives exist). We also support several mentors whose research crosses this boundary (e.g., Dan Hendrycks, Jesse Clifton, Daniel Kokotajlo).
I see in the previous SERI MATS cohort, there was a debate about this, titled “How much should be invested in interpretability research?", but apparently there is no recording.
We did record this debate but have yet to publish the recording as the speakers have not yet given approval. MATS believe that interpretability research might have many uses.
People fail to realise that leading AGI labs, such as OpenAI and Conjecture (I bet DeepMind and Anthropic, too, even though they haven’t publicly stated this) do not plan to align LLMs “once and forever”, but rather use LLMs to produce novel alignment research, which will almost certainly go in package with novel AI architectures (or variations on existing proposals, such as LeCun’s H-JEPA).
Among many other theories of change, MATS currently supports:
- Evaluating models for dangerous capabilities, to aid safety standards, moratoriums, and alignment MVPs:
- Owain Evans' evaluations of situational awareness
- Ethan Perez’s and Evan Hubinger's demonstrations of deception and other undesirable traits
- Daniel Kokotajlo’s dangerous capabilities evaluations
- Dan Hendrycks’ deceptive capabilities benchmarking
- Building non-agentic AI systems that can be used to extract useful alignment research:
- Evan Hubinger’s conditioning predictive models
- Janus’ simulators and Nicholas Kees Dupois’ cyborgism
- Interpretability and model editing research that might build into better model editing, better auditing, better regularizers, better DL science, and many other things:
- Neel Nanda’s and Conjecture’s mechanistic interpretability research
- Alex Turner’s RL interpretability and model editing research
- Collin Burns’ “gray-box ELK”
- Research that aims to predict or steer architecture-agnostic or emergent systems:
- Vivek Hebbar’s research on “sharp left turns," etc.
- John Wentworth’s selection theorems research
- Alex Turner’s and Quintin Pope’s shard theory research
- Technical alignment-adjacent governance and strategy research:
- Jesse Clifton’s and Daniel Kokotajlo’s research on averting AI conflict and AI arms races
- Dan Hendrycks’ research on aligning complex systems
A talent recruitment and onboarding organization targeting cyber security researchers to benefit AI alignment,
This EA Infosec bookclub seems like a good start, but more could be done!
offers of funding for large compute projects that focus on alignment
This NSF grant addresses this!
An organization that does for the Open Philanthropy worldview investigations team what GCP did to supplement CEA's workshops and 80,000 Hours’ career advising calls.
Rethink Priorities are doing this!
A dedicated bi-yearly workshop for AI safety university group leaders that teaches them how to recognize talent, foster useful undergraduate research projects, and build a good talent development pipeline or “user journey” (including a model of alignment macrostrategy and where university groups fit in).
Several workshops have run since I posted this, but none quite fits my vision.
A combined research mentorship and seminar program that aims to do for AI governance research what MATS is trying to do for technical AI alignment research.
The GovAI Summer & Winter Fellowships Program is addressing this niche!
Yeah, we hope to hold another cohort starting in Nov. However, applying this time might be good practise and, if the mentor is willing, you could just defer to Winter!
An incomplete list of possibly useful AI safety research:
- Predicting/shaping emergent systems (“physics”)
- Learning theory (e.g., shard theory, causal incentives)
- Regularizers (e.g., speed priors)
- Embedded agency (e.g., infra-Bayesianism, finite factored sets)
- Decision theory (e.g., timeless decision theory, cooperative bargaining theory, acausal trade)
- Model evaluation (“biology”)
- Capabilities evaluation (e.g., survive-and-spread, hacking)
- Red-teaming alignment techniques
- Demonstrations of emergent properties/behavior (e.g., instrumental powerseeking)
- Interpretability (“neuroscience”)
- Mechanistic interpretability (e.g., superposition, toy models, automated circuit detection)
- Gray box ELK (e.g., Collin Burns’ research)
- Feature extraction/sparsity (including Wentworth/Bushnaq style “modularity” research)
- Model surgery (e.g., ROME)
- Alignment MVP (“psychology”)
- Sampling simulators safely (conditioning predictive models)
- Scalable oversight (e.g., RLHF, CAI, debate, RRM, model-assisted evaluations)
- Cyborgism
- Prompt engineering (e.g., jailbreaking)
- Strategy/governance (“sociology”)
- Compute governance (e.g., GPU logging/restrictions, treaties)
- Model safety standards (e.g., auditing policies)
- Infosecurity
- Multi-party authentication
- Airgapping
- AI-assisted infosecurity
Main takeaways from a recent AI safety conference:
- If your foundation model is one small amount of RL away from being dangerous and someone can steal your model weights, fancy alignment techniques don’t matter. Scaling labs cannot currently prevent state actors from hacking their systems and stealing their stuff. Infosecurity is important to alignment.
- Scaling labs might have some incentive to go along with the development of safety standards as it prevents smaller players from undercutting their business model and provides a credible defense against lawsuits regarding unexpected side effects of deployment (especially with how many tech restrictions the EU seems to pump out). Once the foot is in the door, more useful safety standards to prevent x-risk might be possible.
- Near-term commercial AI systems that can be jailbroken to elicit dangerous output might empower more bad actors to make bioweapons or cyberweapons. Preventing the misuse of near-term commercial AI systems or slowing down their deployment seems important.
- When a skill is hard to teach, like making accurate predictions over long time horizons in complicated situations or developing a “security mindset,” try treating humans like RL agents. For example, Ph.D. students might only get ~3 data points on how to evaluate a research proposal ex-ante, whereas Professors might have ~50. Novice Ph.D. students could be trained to predict good research decisions by predicting outcomes on a set of expert-annotated examples of research quandaries and then receiving “RL updates” based on what the expert did and what occurred.
These days have particular significance in my AGI timelines ranking and I think are a good default spread based on community opinion. However, there is no reason you shouldn't choose alternate years!
Seems right. Explore vs. exploit is another useful frame.
Reasons that scaling labs might be motivated to sign onto AI safety standards:
- Companies who are wary of being sued for unsafe deployment that causes harm might want to be able to prove that they credibly did their best to prevent harm.
- Big tech companies like Google might not want to risk premature deployment, but might feel forced to if smaller companies with less to lose undercut their "search" market. Standards that prevent unsafe deployment fix this.
However, AI companies that don’t believe in AGI x-risk might tolerate higher x-risk than ideal safety standards by the lights of this community. Also, I think insurance contracts are unlikely to appropriately account for x-risk, if the market is anything to go by.