Prosaic Alignment is currently more important to work on than Agent Foundations work.

LLMs as currently trained run ~0 risk of catastrophic instrumental convergence even if scaled up with 1000x more compute

Academia is sufficiently dysfunctional that if you want to make a great scientific discovery you should basically do it outside of academia.

Pursuing plans that cognitively enhance humans while delaying AGI should be our top strategy for avoiding AGI risk

Current progress in AI governance will translate with greater than 50% probability into more than a 2 year counterfactual delay of dangerous AI systems

Ambitious mechanistic interpretability is quite unlikely^[1] to be able to confidently assess^[2] whether AIs^[3] are deceptively aligned (or otherwise have dangerous propensities) in the next 10 years.

It is very unlikely AI causes an existential catastrophe (Bostrom or Ord definition) but doesn't result in human extinction. (That is, non-extinction AI x-risk scenarios are unlikely)

Things will basically be fine regarding job loss and unemployment due to AI in the next several years and those worries are overstated 

The current AI x-risk grantmaking ecosystem is bad and could be improved substantially.

People aged 12 to 18 should basically be treated like adults rather than basically treated like children.

It is critically important for US/EU companies to build AGI before Chinese companies.

EAs and rationalists should strongly consider having lots more children than they currently are

Meaningness's "Geeks Mops and Sociopaths" model is an accurate model of the dynamics that underlie most social movements

Irrefutable evidence of extraterrestrial life would be a good thing.

It was a mistake to increase salaries in the broader EA/Rationality/AI-Alignment ecosystem between 2019 and 2022

Good AGI-notkilleveryoneism-conscious researchers should in general prioritize working at big AGI labs over working independently, for alignment-focused labs, or for academia marginally more than they currently do.

The ratio of good alignment work done at labs vs independently mostly skews toward labs

Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than independent researchers. Your answer should factor out such considerations.

Edit: Also normalize for quantity of researchers.

Someone in the AI safety community (e.g. Yud, Critch, Salamon, you) can currently, within 6 month's effort, write a 20,000 word document that would pass a threshold for a coordination takeoff on Earth, given that 1 million smart Americans and Europeans would read all of it and intended to try out much of the advice (i.e. the doc succeeds given 1m serious reads, it doesn't need to cause 1m serious reads). Copy-pasting already-written documents/posts would count.

There is a greater than 20% chance that the Effective Altruism movement has been net negative for the world.

Empirical agent foundations is currently a good idea for a research direction.

A basic deontological and straightforward morality (such as that exmplified by Hermione in HPMOR) is basically right; this is in contrast with counterintuitive moralities that suggest evil-tinted people (like Quirrell in HPMOR) are also valid ways of being moral.

Just like the last 12 months was the time of the chatbots, the next 12 months will be the time of agent-like AI product releases.

The work of agency-adjacent research communities such as artificial life, complexity science and active inference is at least as relevant to AI alignment as LessWrong-style agent foundations research is.

American intelligence agencies consider AI safety to be substantially more worth watching than most social movements

It is possible to make meaningful progress on deceptive alignment using experiments on current models

Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large compute budgets for safety research teams.

Moloch is winning.

"Polyamory-as-a-default-option" would be a better social standard than "Monogamy-as-a-default-option".

The rationality community will noticeably spill over into other parts of society in the next ten years. Examples: entertainment, politics, media, art, sports, education etc.

At least one American intelligence agency is concerned about the AI safety movement potentially decelerating the American AI industry, against the administration/natsec community's wishes

I broadly agree with the claim that "most people don't do anything and the world is very boring".

On the current margin most people would be better off involving more text-based communication in their lives than in-person communication.

Agent foundations research should become more academic on the margin (for example by increasing the paper to blogpost ratio, and by putting more effort into relating new work to existing literature).

Current progress in AI governance will translate with greater than 20% probability into more than a 2 year counterfactual delay of dangerous AI systems

Rationality should be practiced for Rationality’s sake (rather than for the sake of x-risk).

It is possible to make meaningful progress on ELK using empirical experiments on current models

Language model agents are likely (>20%) to produce AGI (including the generalization to foundation model-based cognitive architectures)

Current AI safety university groups are overall a good idea and helpful, in expectation, for reducing AI existential risk 

Having another $1 billion to prevent AGI x-risk would be useful because we could spend it on large-scale lobbying efforts in DC.

Immersion into phenomena is better for understanding them than trying to think through at the gears-level, on the margin for most people who read LessWrong.

Public mechanistic interpretability research is net positive in expectation.

Among existing alignment research agendas/projects, Superalignment has the highest expected value

Most LWers should rely less on norms of their own (or the LW community's) design, and instead defer to regular societal norms more.

Rationalists would be better off if they were more spiritual/religious

Effective altruism can be well modeled by cynically thinking of it as just another social movement, in the sense that those a part of it are mainly jockeying for in-group status, and making costly demonstrations to their in-group & friends that they care about other sentiences more than others in the in-group. Its just that EA has more cerebral standards than others.

"Agent" is an incoherent concept.

"Open-source LLM-based agent with hacking abilities starts spreading itself over the Internet because some user asked it to do so or to do something like to conquer the world" is a quite probable point-of-no-return regarding AGI risk.

Investing in early-stage AGI companies helps with reducing x-risk (via mission hedging, having board seats, shareholder activism)

Great art is rarely original and mostly copied.

The younger generation of rationalists are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.

At least one of {Anthropic, OpenAI, Deepmind} is net-positive compared to the counterfactual where just before founding the company, its founders were all discretely paid $10B by a time-travelling PauseAI activist not to found the company and to exit the industry for 30 years, and this worked.

One should basically not invest into having "charisma".

The most valuable new people joining AI safety will usually take ~1-3 years of effort to begin to be adequately sorted and acknowledged for their worth, unless they are unusually good at self-promotion e.g. gift of gab, networking experience, and stellar resume.

Poll feature on LW: Yay or Nay?

There is a greater than 80% chance that effective altruism has been net-negative for the world.

If rationality took off in China, it would yield higher EV from potentially spreading to the rest of the world than from potentially accelerating China.

When people try to discuss philosophy, math, or science, especially pre-paradigmatic fields such as ai safety, they use a lot of metaphorical thinking to extend from familiar concepts to new concepts. It would be very helpful and people would stop talking past each other so much if they practiced being explicitly aware of these mental representations and directly shared them rather than pretending that something more rigorous is happening. This is part of Alfred Korzybski's original rationality project, something he called 'consciousness of abstraction.'

Language model agents are very likely (>80%) to produce AGI (including the generalization to foundation model-based cognitive architectures)

Research into getting a mechanistic understanding of the brain for purposes of at least one of: understanding how values/empathy works in people, brain uploading or improving cryonics/plastination is net positive and currently greatly underfunded.

Most persistent disagreements can more usefully be thought of as a difference in priors rather than a difference in evidence or rationality.

A Secular Solstice variation designed to work weekly (akin to Sunday Service or Shabbat) would be positive for rationalists, both for community and for the thought processes of the members.

There is a greater than 50% chance that the Effective Altruism movement has been net negative for the world.

Most LWers are prioritizing their slack too much.

"Intelligence" can be characterized with a similar level of theoretical precision as e.g., heat, motion, and information. (In other words: it's less like a messy, ad-hoc phenomena and more like a deep, general fact about our world). 

You know of a technology that has at least a 10% chance of having a very big novel impact on the world (think the internet or ending malaria) that isn't included in this list, very similar, or downstream from some element of it: AI, mind uploads, cryonics, human space travel, geo-engineering, gene drives, human intelligence augmentation, anti-aging, cancer cures, regenerative medicine, human genetic engineering, artificial pandemics, nuclear weapons, proper nanotech, very good lie detectors, prediction markets, other mind-altering drugs, cryptocurrency, better batteries, BCIs, nuclear fusion, better nuclear fission, better robots, AR, VR, room-temperature superconductors, quantum computers, polynomial time SAT solvers, cultured meat, solutions to antibiotic resistance, vaccines to some disease, optical computers, artificial wombs, de-extinction and graphene.

Bad options included just in case someone thinks they are good.
 

Xi Jinping thinks that economic failure in the US or China, e.g. similar to 2008, is one of the most likely things to change the global balance of power.

Having another $1 billion to prevent AGI x-risk would be pretty useful.

If we had access to a brain upload (and maybe a world simulator too) we could in principle extract something like a utility function, and the theory behind it relates more to agents in general than it does to humans in particular.

Any activity or action taken after drinking coffee in the morning will strongly reward/reinforce that action/activity

Humans are the dominant species on earth primarily because our individual intelligence surpassed the necessary threshold to sustain civilization and take control of our environment.

American intelligence agencies are actively planning to defend the American AI industry against foreign threats (e.g. Russia, China).

If you can write prompt for GPT-2000 such that completion of this prompt results in aligned pivotal act, you can just use knowledge necessary for writing this prompt to Just Build aligned ASI, without necessity to use GPT-2000.

Rationalist rituals like Petrov day or the Secular Solstices should be marginally more emphasized within those collections of people who call themselves rationalists.

Rationality is likely to organically gain popularity in China (e.g. quickly reaching 10,000 people or reaching 100,000 by 2030, e.g. among scientists or engineers, etc).

It is wrong to protest AI labs.

The ratio of good alignment work done at labs vs in academia mostly skews toward labs

Good meaning something different from impactful here. Obviously AGI labs will pay more attention to their researchers or researchers from respectable institutions than academics. Your answer should factor out such considerations.

Edit: Also normalize for quantity of researchers.

The ratio of good alignment work done in academia vs independently mostly skews toward academia

Good meaning something different from impactful here. Possibly AGI labs will pay more attention to academics than independent researchers. Your answer should factor out such considerations.

Edit: Also normalize for quantity of researchers.

At least one mole, informant, or spy has been sent by a US government agency or natsec firm to infiltrate the AI safety community by posing as a new member (even if it's just to ask questions in causal conversations at events about recent happenings or influential people's priorities).

The government should build nuclear-driven helicopters, like nuclear subs.

An alignment technique that can fully align GPT-4 is likely (>50%) to also fully align the first existentially dangerous AGI

Most end-to-end “alignment plans” are bad because research will be incremental. For example, Superalignment's impact will mostly come from adapting to the next ~3 years of AI discoveries and working on relevant subproblems like interp, rather than creating a superhuman alignment researcher.

Conceptual alignment work on concepts like “agency”, “optimization”, “terminal values”, “abstractions”, “boundaries” is mostly intractable at the moment.

Most LWers are not prioritizing their slack enough.

Those who call themselves rationalists or EAs should drink marginally more alcohol at social events.

In human interactions at any scale, it is net good to at least momentarily consider the Elephant and/or the Player [LW · GW], with very few exceptions.

Most of the time, power-seeking behavior in humans is morally good or morally neutral.

Computer science & ML will become lower in relevance/restricted in scope for the purposes of working with silicon-based minds, just as human-neurosurgery specifics are largely but not entirely irrelevant for most civilization-scale questions like economic policy, international relations, foundational research, etc.

Or IOW: Model neuroscience (and to some extent, model psychology) requires more in-depth CS/ML expertise than will the smorgasbord of incoming subfields of model sociology, model macroeconomics, model corporate law, etc.

The virtue of the void [? · GW] is indeed the virtue above all others (in rationality), and fundamentally unformalizable.

There is likely a deep compositional structure to be found for alignment, possibly to the extent that AGI alignment could come from "merely" stacking together "microalignment", even if in non-trivial ways.

If you can't write a program that produces aligned (under whatever definition of alignment you use) output being run on unphysically large computer, you can't deduce from training data or weights of superintelligent neural network if it produces aligned output.

There are arguments for convergent instrumental pressures towards catastrophe, but the required assumptions are too strong for the arguments to clearly go through.

Cultural values are something like preferences over pairs of social environments and things we actually care about. So it makes sense to talk about jointly optimizing them.

It's good for EA orgs to pay well

The younger generation of EAs are less interesting than the older generation was when that old generation had the same experience as the young generation currently does.

In the context of Offense-defense balance, offense has a strong advantage 

Generative AI like LLMs or diffusion will eventually be superseded by human AI researchers coming up with something autonomous.

EA has gotten a little more sympathetic to vibes-based reasoning recently, and will continue to incorporate more of it.

The mind (ie. your mind), and how it is experienced from the inside, is potentially a very rich source of insights for keeping AI minds aligned on the inside.

All else equal, a unit of animal suffering should be accorded the same moral weight as an equivalent unit of human suffering. (i.e. equal consideration for equal interests)

'Descriptive' agent foundations research is currently more important to work on than 'normative' agent foundations research.

Autism is the extreme male version of a male-female difference in systemic vs empathic thinking.

Developing a solid human intelligence/skill evaluation metric would be a high-EV project for AI safety, e.g. to make it easier to invest in moving valuable AI safety people to the Bay Area/London from other parts of the US/UK.

MadHatter is an original and funny satirist. The universally serious reaction to his jokeposts is a quintessential example of rationalist humorlessness.

People should pay an attractiveness tax to the government.

No. This is a one-off fast mockup of a poll just to see what it's like, and that's one of the problems that I didn't think I needed to fix before testing the idea (along with things like agree/disagree not automatically showing up).

Overall I'd say it was like an 75th-80th percentile outcome, I'm pleasantly surprised by how much activity it's had and how much interesting disagreement it has surfaced. So I've updated toward building this as a proper feature.

I work at Manifold, I don't know if this is true but I can easily generate some arguments against:

• Manifold's business model is shaky and Manifold may well not exist in 3 years.
• Manifold's codebase is also shaky and would not survive Manifold-the-company dying right now.
• Manifold is quite short on engineering labor.
• It seems to me that Manifold and LW have quite different values (Manifold has a typical startup focus on prioritizing growth at all costs) and so I expect many subtle misalignments in a substantial integration.

Personally for these reasons I am more eager to see features developed in the LW codebase than the Manifold codebase.

I'd love to do this. I think clarifying the types of work that would fall under agent foundations would be very useful. On one hand, it seems as though most of the useful work has probably been done; on the other, some current discussion seems to be missing important points from that work.

I might want to do this. I'm on the vaguely pro side but think we currently don't have many tractable directions.

I now think that it should go back to the binary yes and no responses, adding bells and whistles will complicate things too much.

Farmed animals are currently inside a non-extinction X-risk. 

Steelmanning a position I don't quite hold: non-extinction AI x-risk scenarios aren't limited to inescapable dystopias as we imagine them.

"Kill all humans" is certainly an instrumental subgoal of "take control of the future lightcone" and it certainly gains an extra epsilon of resources compared to any form of not literally killing all humans, but it's not literally required, and there are all sorts of weird things the AGI could prefer to do with humanity instead depending on what kind of godshatter [LW · GW] it winds up with, most of which are so far outside the realm of human reckoning that I'm not sure it's reasonable to call them dystopian. (Far outside Weirdtopia, for that matter.)

It still seems very likely to me that a non-aligned superhuman AGI would kill humanity in the process of taking control of the future lightcone, but I'm not as sure of that as I'm sure that it would take control.

One of the main questions on which I'd like to understand others' views is something like: Conditional on sentient/conscious humans^[1] continuing to exist in an x-risk scenario^[2], with what probability do you think they will be in an inescapable dystopia^[3]?

(My own current guess is that dystopia is very likely.)