Ryan Kidd's Shortform

I am a Manifund Regrantor. In addition to general grantmaking, I have requests for proposals in the following areas:

• Funding for AI safety PhDs [LW · GW] (e.g., with these supervisors), particularly in exploratory research connecting [LW · GW] AI safety theory with empirical ML research.
• An AI safety PhD [LW · GW] advisory service that helps prospective PhD students choose a supervisor and topic (similar to Effective Thesis, but specialized for AI safety).
• Initiatives to critically examine current AI safety macrostrategy (e.g., as articulated by Holden [? · GW] Karnofsky [? · GW]) like the Open Philanthropy AI Worldviews Contest and Future Fund Worldview Prize [EA · GW].
• Initiatives to identify and develop "Connectors [LW · GW]" outside of academia (e.g., a reboot of the Refine [LW · GW] program, well-scoped contests [LW · GW], long-term mentoring [LW · GW] and peer-support programs).
• Physical community spaces for AI safety in AI hubs outside of the SF Bay Area or London (e.g., Japan, France, Bangalore).
• Start-up incubators for projects, including evals/red-teaming/interp companies, that aim to benefit AI safety, like Catalyze Impact, Future of Life Foundation, and YCombinator's request for Explainable AI start-ups.
• Initiatives to develop and publish expert consensus on AI safety macrostrategy cruxes, such as the Existential Persuasion Tournament and 2023 Expert Survey on Progress in AI (e.g., via the Delphi method, interviews [LW · GW], surveys [LW · GW], etc.).
• Ethics/prioritization research into:
  • What values to instill in artificial superintelligence?
  • How should AI-generated wealth be distributed?
  • What should people do in a post-labor society?
  • What level of surveillance/restriction is justified by the Unilateralist's Curse?
  • What moral personhood will digital minds have?
  • How should nations share decision making power regarding transformative AI?
• New nonprofit startups that aim to benefit AI safety.

Hourly stipends for AI safety fellowship programs, plus some referents. The average AI safety program stipend is $26/h.

Edit: updated figure to include more programs.

↑ comment by metawrong · 2024-11-01T18:20:37.732Z · LW(p) · GW(p)

LASR (https://www.lasrlabs.org/) is giving a £11,000 stipend for a 13 week program, assuming 40h/week it works out to ~$27

Replies from: ryankidd44, ryankidd44

↑ comment by Ryan Kidd (ryankidd44) · 2024-11-01T19:58:53.495Z · LW(p) · GW(p)

Updated figure with LASR Labs and Pivotal Research Fellowship at current exchange rate of 1 GBP = 1.292 USD.

↑ comment by Ryan Kidd (ryankidd44) · 2024-11-01T19:46:50.831Z · LW(p) · GW(p)

That seems like a reasonable stipend for LASR. I don't think they cover housing, however.

I just left a comment on PIBBSS' Manifund grant proposal (which I funded $25k) that people might find interesting.

Main points in favor of this grant

1. My inside view is that PIBBSS mainly supports “blue sky” or “basic” research, some of which has a low chance of paying off, but might be critical in “worst case [AF · GW]” alignment scenarios (e.g., where “alignment MVPs” don’t work, or “sharp left turns [? · GW]” and “intelligence explosions [? · GW]” are more likely than I expect). In contrast, of the technical research MATS supports, about half is basic research (e.g., interpretability, evals, agent foundations) and half is applied research (e.g., oversight + control, value alignment). I think the MATS portfolio is a better holistic strategy for furthering AI alignment. However, if one takes into account the research conducted at AI labs and supported by MATS, PIBBSS’ strategy makes a lot of sense: they are supporting a wide portfolio of blue sky research that is particularly neglected by existing institutions and might be very impactful in a range of possible “worst-case” AGI scenarios. I think this is a valid strategy in the current ecosystem/market and I support PIBBSS!
2. In MATS’ recent post, “Talent Needs of Technical AI Safety Teams [LW · GW]”, we detail an AI safety talent archetype we name “Connector”. Connectors bridge exploratory theory and empirical science, and sometimes instantiate new research paradigms. As we discussed in the post, finding and developing Connectors is hard, often their development time is on the order of years, and there is little demand on the AI safety job market for this role. However, Connectors can have an outsized impact on shaping the AI safety field and the few that make it are “household names” in AI safety and usually build organizations, teams, or grant infrastructure around them. I think that MATS is far from the ideal training ground for Connectors (although some do pass through!) as our program is only 10 weeks long (with an optional 4 month extension) rather than the ideal 12-24 months, we select scholars to fit established mentors’ preferences rather than on the basis of their original research ideas, and our curriculum and milestones generally focus on building object-level scientific skills rather than research ideation and “gap-identifying”. It’s thus no surprise that most MATS scholars are “Iterator” archetypes. I think there is substantial value in a program like PIBBSS existing, to support the development of “Connectors” and pursue impact in a higher-variance way than MATS.
3. PIBBSS seems to have decent track record for recruiting experienced academics in non-CS fields and helping them repurpose their advanced scientific skills to develop novel approaches to AI safety. Highlights for me include Adam Shai’s “computational mechanics” approach to interpretability and model cognition, Martín Soto’s “logical updatelessness” approach to decision theory, and Gabriel Weil’s “tort law” approach to making AI labs liable for their potential harms on the long-term future.
4. I don’t know Lucas Teixeira (Research Director) very well, but I know and respect Dušan D. Nešić (Operations Director) a lot. I also highly endorsed Nora Ammann’s vision (albeit while endorsing a different vision for MATS). I see PIBBSS as a highly competent and EA-aligned organization, and I would be excited to see them grow!
5. I think PIBBSS would benefit from funding from diverse sources, as mainstream AI safety funders have pivoted more towards applied technical research (or more governance-relevant basic research like evals). I think Manifund regrantors are well-positioned to endorse more speculative basic research, but I don’t really know how to evalutate such research myself, so I’d rather defer to experts. PIBBSS seems well-positioned to provide this expertise! I know that Nora had quite deep models of this while Research Director and in talking with Dusan, I have had a similar impression. I hope to talk with Lucas soon!

Donor's main reservations

1. It seems that PIBBSS might be pivoting away from higher variance blue sky research to focus on more mainstream AI interpretability. While this might create more opportunities for funding, I think this would be a mistake. The AI safety ecosystem needs a home for “weird ideas” and PIBBSS seems the most reputable, competent, EA-aligned place for this! I encourage PIBBSS to “embrace the weird”, albeit while maintaining high academic standards for basic research, modelled off the best basic science institutions.
2. I haven’t examined PIBBSS’ applicant selection process and I’m not entirely confident it is the best version it can be, given how hard MATS has found applicant selection and my intuitions around the difficulty of choosing a blue sky research portfolio. I strongly encourage PIBBSS to publicly post and seek feedback on their applicant selection and research prioritization processes, so that the AI safety ecosystem can offer useful insight. I would also be open to discussing these more with PIBBSS, though I expect this would be less useful.
3. My donation is not very counterfactual here, given PIBBSS’ large budget and track record. However, there has been a trend in typical large AI safety funders away from agent foundations and interpretability, so I think my grant is still meaningful.

Process for deciding amount

I decided to donate the project’s minimum funding ($25k) so that other donors would have time to consider the project’s merits and potentially contribute. Given the large budget and track record of PIBBSS, I think my funds are less counterfactual here than for smaller, more speculative projects, so I only donated the minimum. I might donate significantly more to PIBBSS later if I can’t find better grants, or if PIBBSS is unsuccessful in fundraising.

Conflicts of interest

I don't believe there are any conflicts of interest to declare.

Why does the AI safety community need help founding projects?

1. AI safety should scale
   1. Labs need external auditors for the AI control plan [LW · GW] to work
   2. We should pursue many research bets [LW · GW] in case superalignment/control fails
   3. Talent leaves MATS/ARENA and sometimes struggles to find meaningful work for mundane reasons, not for lack of talent or ideas
   4. Some emerging research agendas don’t have a home
   5. There are diminishing returns at scale [LW · GW] for current AI safety teams; sometimes founding new projects is better than joining an existing team
   6. Scaling lab alignment teams are bottlenecked by management capacity, so their talent cut-off is above the level required to do “useful AIS work”
2. Research organizations (inc. nonprofits) are often more effective than independent researchers
   1. “Block funding model” is more efficient, as researchers can spend more time researching, rather than seeking grants, managing, or other traditional PI duties that can be outsourced
   2. Open source/collective projects often need a central rallying point (e.g., EleutherAI, dev interp at Timaeus, selection theorems and cyborgism agendas seem too delocalized, etc.)
3. There is (imminently) a market for for-profit AI safety companies [LW · GW] and value-aligned people should capture this free energy or let worse alternatives flourish
   1. If labs or API users are made legally liable for their products, they will seek out external red-teaming/auditing consultants to prove they “made a reasonable attempt” to mitigate harms
   2. If government regulations require labs to seek external auditing, there will be a market for many types of companies
   3. “Ethical AI” companies might seek out interpretability or bias/fairness consultants
4. New AI safety organizations struggle to get funding and co-founders despite having good ideas
   1. AIS researchers are usually not experienced entrepeneurs (e.g., don’t know how to write grant proposals for EA funders, pitch decks for VCs, manage/hire new team members, etc.)
   2. There are not many competent start-up founders in the EA/AIS community and when they join, they don’t know what is most impactful to help
   3. Creating a centralized resource for entrepeneurial education/consulting and co-founder pairing would solve these problems

Main takeaways from a recent AI safety conference:

• If your foundation model is one small amount of RL away from being dangerous and someone can steal your model weights, fancy alignment techniques don’t matter. Scaling labs cannot currently prevent state actors from hacking their systems and stealing their stuff. Infosecurity is important to alignment.
• Scaling labs might have some incentive to go along with the development of safety standards as it prevents smaller players from undercutting their business model and provides a credible defense against lawsuits regarding unexpected side effects of deployment (especially with how many tech restrictions the EU seems to pump out). Once the foot is in the door, more useful safety standards to prevent x-risk might be possible.
• Near-term commercial AI systems that can be jailbroken to elicit dangerous output might empower more bad actors to make bioweapons or cyberweapons. Preventing the misuse of near-term commercial AI systems or slowing down their deployment seems important.
• When a skill is hard to teach, like making accurate predictions over long time horizons in complicated situations or developing a “security mindset [? · GW],” try treating humans like RL agents. For example, Ph.D. students might only get ~3 data points on how to evaluate a research proposal ex-ante, whereas Professors might have ~50. Novice Ph.D. students could be trained to predict good research decisions by predicting outcomes on a set of expert-annotated examples of research quandaries and then receiving “RL updates” based on what the expert did and what occurred.

An incomplete list of possibly useful AI safety research:

• Predicting/shaping emergent systems (“physics”)
  • Learning theory (e.g., shard theory, causal incentives)
  • Regularizers (e.g., speed priors)
  • Embedded agency (e.g., infra-Bayesianism, finite factored sets)
  • Decision theory (e.g., timeless decision theory, cooperative bargaining theory, acausal trade)
• Model evaluation (“biology”)
  • Capabilities evaluation (e.g., survive-and-spread, hacking)
  • Red-teaming alignment techniques
  • Demonstrations of emergent properties/behavior (e.g., instrumental powerseeking)
• Interpretability (“neuroscience”)
  • Mechanistic interpretability (e.g., superposition, toy models, automated circuit detection)
  • Gray box ELK (e.g., Collin Burns’ research)
  • Feature extraction/sparsity (including Wentworth/Bushnaq style “modularity” research)
  • Model surgery (e.g., ROME)
• Alignment MVP (“psychology”)
  • Sampling simulators safely (conditioning predictive models)
  • Scalable oversight (e.g., RLHF, CAI, debate, RRM, model-assisted evaluations)
  • Cyborgism
  • Prompt engineering (e.g., jailbreaking)
• Strategy/governance (“sociology”)
  • Compute governance (e.g., GPU logging/restrictions, treaties)
  • Model safety standards (e.g., auditing policies)
• Infosecurity
  • Multi-party authentication
  • Airgapping
  • AI-assisted infosecurity

AI alignment threat models that are somewhat MECE (but not quite):

• We get what we measure [LW · GW] (models converge to the human++ simulator [LW · GW] and build a Potemkin village world without being deceptive consequentialists [LW · GW]);
• Optimization daemon [? · GW] (deceptive [LW · GW] consequentialist [? · GW] with a non-myopic [? · GW] utility function [? · GW] arises in training and does gradient hacking [? · GW], buries trojans [LW · GW] and obfuscates cognition to circumvent interpretability tools [LW · GW], "unboxes [? · GW]" itself, executes a "treacherous turn [? · GW]" when deployed, coordinates with auditors and future instances of itself [LW · GW], etc.);
• Coordination failure [LW · GW] (otherwise-aligned AI systems combust or gridlock far from the Pareto frontier due to opaque values/capabilities, inadequate commitment mechanisms, or irreconcilable differences [? · GW]);
• Sharp left turn [? · GW] (models learn generally powerful cognitive tools that are efficiently reached by training on real-world tasks, especially as the real world contains useful embedded knowledge that shortcuts learning these tools from scratch; but powerful cognitive tools are somewhat anti-natural to corrigibility [? · GW] and the training process does not efficiently constrain the directionality of these tools towards human CEV [? · GW], which manifests under distributional shift [? · GW]).

In particular, the last threat model feels like it is trying to cut across aspects of the first two threat models, violating MECE.

Reasons that scaling labs might be motivated to sign onto AI safety standards:

• Companies who are wary of being sued for unsafe deployment that causes harm might want to be able to prove that they credibly did their best to prevent harm.
• Big tech companies like Google might not want to risk premature deployment, but might feel forced to if smaller companies with less to lose undercut their "search" market. Standards that prevent unsafe deployment fix this.

However, AI companies that don’t believe in AGI x-risk might tolerate higher x-risk than ideal safety standards by the lights of this community. Also, I think insurance contracts are unlikely to appropriately account for x-risk, if the market [EA · GW] is anything to go by.

Types of organizations that conduct alignment research, differentiated by funding model and associated market forces:

• Academic research groups (e.g., Krueger's lab at Cambridge, UC Berkeley CHAI, NYU ARG, MIT AAG);
• Research nonprofits (e.g., ARC Theory, MIRI, FAR AI, Redwood Research);
• "Mixed funding model" organizations:
  • "Alignment-as-a-service" organizations, where the product directly contributes to alignment (e.g., Apollo Research, Aligned AI, ARC Evals, Leap Labs);
  • "Alignment-on-the-side" organizations, where product revenue helps funds alignment research (e.g., Conjecture);
• Scaling labs, where alignment research is mostly directed towards improving product (e.g., Anthropic, DeepMind, OpenAI).

Can the strategy of "using surrogate goals to deflect threats" be countered by an enemy agent that learns your true goals and credibly precommits to always defecting (i.e., Prisoner's Dilemma style) if you deploy an agent against it with goals that produce sufficiently different cooperative bargaining equilibria than your true goals would?

MATS' goals:

• Find + accelerate high-impact research scholars:
  • Pair scholars with research mentors via specialized mentor-generated selection questions (visible on our website);
  • Provide a thriving academic community for research collaboration, peer feedback, and social networking;
  • Develop scholars according to the “T-model of research [LW(p) · GW(p)]” (breadth/depth/epistemology);
  • Offer opt-in curriculum elements, including seminars, research strategy workshops, 1-1 researcher unblocking support, peer study groups, and networking events;
• Support high-impact research mentors:
  • Scholars are often good research assistants and future hires;
  • Scholars can offer substantive new critiques of alignment proposals;
  • Our community, research coaching, and operations free up valuable mentor time and increase scholar output;
• Help parallelize high-impact AI alignment research:
  • Find, develop, and refer scholars with strong research ability, value alignment, and epistemics;
  • Use alumni for peer-mentoring in later cohorts;
  • Update mentor list and curriculum as the alignment field’s needs change.

"Why suicide doesn't seem reflectively rational, assuming my preferences are somewhat unknown to me," OR "Why me-CEV is probably not going to end itself":

• Self-preservation is a convergent instrumental goal for many goals.
• Most systems of ordered preferences that naturally exhibit self-preservation probably also exhibit self-preservation in the reflectively coherent pursuit of unified preferences (i.e., CEV).
• If I desire to end myself on examination of the world, this is likely a local hiccup in reflective unification of my preferences, i.e., "failure of present me to act according to me-CEV's preferences rather than a failure of hypothetical me-CEV to account for facts about the world."

Note: I'm fine; this is purely intellectual.

Are these framings of gradient hacking [? · GW], which I previously articulated here [LW · GW], a useful categorization?

1. Masking: Introducing a countervailing, “artificial” performance penalty that “masks” the performance benefits of ML modifications that do well on the SGD objective, but not on the mesa-objective [? · GW];
2. Spoofing: Withholding performance gains until the implementation of certain ML modifications that are desirable to the mesa-objective; and
3. Steering: In a reinforcement learning context, selectively sampling environmental states that will either leave the mesa-objective unchanged or "steer" the ML model in a way that favors the mesa-objective.

How does the failure rate of a hierarchy of auditors scale with the hierarchy depth, if the auditors can inspect all auditors below their level?

Are GPT-n systems more likely to:

1. Learn superhuman cognition to predict tokens better and accurately express human cognitive failings in simulacra [LW(p) · GW(p)] because they learned these in their "world model"; or
2. Learn human-level cognition to predict tokens better, including human cognitive failings?