Hello! I'm running the Unofficial LessWrong Community Survey [LW · GW] this year, and we're down to the last week it's open. If you're reading this open thread, I think you're in the target audience. 

I'd appreciate if you took the survey. 

If you're wondering what happens with the answers, it gets used for big analysis posts like this one [? · GW], the data gets published so you can use it to answer questions about the community, and it sometimes guides the decision-making of people who work or run things in the community. (Including me!)

What is the current status of CFAR? The website seems like it is inactive, which I find surprising given that there were four weekend workshops in 2022 that CFAR wanted to use for improving its workshops.

So if alignment is as hard as it looks, desperately scrabbling to prevent recursive superintelligence should be an extremely attractive instrumental subgoal. Do we just lean into that?

Is anyone else on this website making YouTube videos? Less Wrong is great, but if you want to broadcast to a larger audience, video seems like the place to be. I know Rational Animations [LW · GW] makes videos. Is there anyone else? Are you, personally, making any?

Hi, I'd like to start creating some wiki pages to help organize information. Would appreciate an upvote so I can get started. Thanks!

I'd like there to be a reaction of "Not Exhaustive", meant for a step where comment (or top-level post, for that matter) missed an important case - how a particular situation could play out, perhaps, or an essential system's component is not listed. An example use: on statement "to prevent any data leaks, one must protect how their systems transfer data and how they process it" with the missed component being protection of storage as well.

I recall wishing for it like three times since the New Year, with the current trigger being this comment [LW(p) · GW(p)]:

Elon already has all of the money in the world. I think he and his employs are ideologically driven, and as far as I can tell they're making sensible decisions given their stated goals of reducing unnecessary spend/sprawl. I seriously doubt they're going to use this access to either raid the treasury or turn it into a personal fiefdom. <...>

which misses a specific case (which I'll name under the original comment if there is any interest).

Hi! I'm Embee but you can call me Max.

I'm a mathematics for quantum physics graduate student considering redirecting my focus toward AI alignment research. My background includes:
- Graduate-level mathematics
- Focus on quantum physics
- Programming experience with Python
- Interest in type theory and formal systems

I'm particularly drawn to MIRI-style approaches and interested in:
- Formal verification methods
- Decision theory implementation
- Logical induction
- Mathematical bounds on AI systems

My current program feels too theoretical and disconnected from urgent needs. I'm looking to:
- Connect with alignment researchers
- Find concrete projects to contribute to
- Apply mathematical rigor to safety problems
- Work on practical implementations

Regarding timelines: I have significant concerns about rapid capability advances, particularly given recent developments (o3). I'm prioritizing work that could contribute meaningfully in a compressed timeframe.

Looking for guidance on:
- Most neglected mathematical approaches to alignment
- Collaboration opportunities
- Where to start contributing effectively
- Balance between theory and implementation

The German federal election is tomorrow. I looked up whether any party was against AI or for a global AI ban or similar. From what I can tell, the answer is unfortunately no, they all want more AI and just disagree about how much. I don't want to vote for any party that wants more AI, so this is rather disappointing.

If you've been collecting data on yourself, it's now moderately easy to let an LLM write code that'll tease out causal relationships between those variables, e.g. with a time series causal inference library like tigramite.

I've done that with some of my data, and surprisingly it both fails to back out results from RCTs I've run, and also doesn't find many (to me) interesting causal relationships.

The most surprising one is that creatine influences creativity. Meditating more doesn't influence anything (e.g. has no influence on mood or productivity), as far as could be determined. More mindful meditations cause happiness.

Because of the resampling/interpolation necessary, everything has a large autocorrelation.

Mainly posting this here so that people know it's now easy to do this, and shill tigramite. I hope I can figure out a way of dealing with rare interventions (e.g. MDMA, CBD, ketamine), and incorporate more collected data. Code here, written by Claude 3.5.1 Sonnet.

Hi everyone,

I'm currently a student in the US. Around a year ago I first read Scott Alexander's Meditations on Moloch as a link that looked interesting; now I spend much of my free time reading blogposts in the rat-adjacent sphere and lurking on LessWrong and Twitter. Over time I've come to realize that I'm not doing anyone else any good by lurking, and it would be valuable for me to start receiving feedback from other people, so I've decided to try actively posting.

One interesting thing about me: I have been heavily involved in physics olympiads (competitions like the International Math Olympiad, but for physics). The relevance of the physics "process" towards world modeling and the ability of AI to solve difficult physics problems seems under-discussed to me.

The quality of discussion here is the highest I've ever seen on the Internet, and I hope to do my part to help uphold it!

Hello everyone. My name is Kseniia Lisovskaia, I think, better to call me Xenia or Xenya. My native language is Russian, so, English is my second language, and I can do mistakes, sorry for that. Because Russian language has different alphabet, there are  different ways how to transliterate my name from Cyrillic to Latin. I think, Xenia is the easy way, because it's common transliteration for the same Greece name.

I'm immigrant in Canada. I left Russia because of political reasons, and because don't like chaos. Unfortunately, chaos now is going to invade in Canada too. I know what Eliezer said about Politics, that it's a killer of mind. It's reason why I don't elaborate.

I participated in Russian speaking lesswrong community in the past. And I have ideas, which maybe can be valuable contribution.

Also, I participate in Russian-speaking internet forum about astronomy(it affected by state censorship, but not much), and it has a big subforum about existential risks, Universe, science itself, AI, etc. I think, it can be valuable to bring most interesting ideas here.

I'm singularist, maybe I about to adopt utilitarianism and/or rational altruism, but I'm not fully sure. I'm defiantly humanist, but in the edge of technological singularity, the classical humanism looks like a bit outdated. 

From my perspective, utilitarianism has alignment problems, like AI: why don't take drugs and not make everyone happy?   Utilitarianists usually answer, that they mean other type of happiness. But I don't think this answer is good enough.

Also, I'm interested in astronomy, physics, biology, political science, history, computers/programming/AI, volunteering activities(basically I get my endorphins when I help someone), in literature(mostly reading, but I'm  trying to write fiction myself). And I have a dream: to be adopted as favourite pet by very clever and kind ASI, and live forever.

I feel very disconnected from other rational people, and feel that chaos is everywhere. Thank you guys, that you keep this island of rationality in the sea of chaos alive. It's very important for me to know, that somewhere madness not finally win yet. My husband is rationalist and transhumanist (he is like an idea to defeat death), but his political views recently changed, and he became an agent of chaos himself.

Honestly, I think, we can't win. I expect a dark singularity or even an extinction event. I think, our the Great Filter ahead, and we will crash about it soon. 

ODD JOB OFFER: I think I want to cross-post Intro to Brain-Like-AGI Safety [? · GW] as a giant 200-page PDF on arxiv (it’s about 80,000 words), mostly to make it easier to cite (as is happening sporadically, e.g. here, here). I am willing to pay fair market price for whatever reformatting work is necessary to make that happen, which I don’t really know (make me an offer). I guess I’m imagining that the easiest plan would be to copy everything into Word (or LibreOffice Writer), clean up whatever formatting weirdness comes from that, and convert to PDF. LaTeX conversion is also acceptable but I imagine that would be much more work for no benefit.

I think inline clickable links are fine on arxiv (e.g. page 2 here), but I do have a few references to actual papers, and I assume those should probably be turned into a proper reference section at the end of each post / “chapter”. Within-series links (e.g. a link from Post 6 to a certain section of Post 4) should probably be converted to internal links within the PDF, rather than going out to the lesswrong / alignmentforum version. There are lots of textbooks / lecture notes on arxiv which can serve as models; I don’t really know the details myself. The original images are all in Powerpoint, if that’s relevant. The end product should ideally be easy for me to edit if I find things I want to update. (Arxiv makes updates very easy, one of my old arxiv papers is up to version 5.)

…Or maybe this whole thing is stupid. If I want it to be easier to cite, I could just add in a “citation information” note, like they did here? I dunno.

Does anyone here ever think to themselves, or out loud, "Here I am in the 21st Century. Sure, all the old scifi stories told me I'd have a shiny flying car but I'm really more interested in where my 21st Century government is?"

For me that is premised on the view that pretty much all existing governments are based on theory and structures that date at least back to the 18th Century in the West. The East might say they "modernized" a bit with the move from dynasties (China, Korea, Japan) to democratic forms but when I look at the way those governments and polities actually work seems more like a wrapper around the prior dynastic structures.

But I also find it challenging to think just what might be the differentiating change that would distinguish a "21st Century" government from existing ones. The best I've come up with is that I don't see it as some type of privatization divestiture of existing government activities (even though I do think some should be) but more of a shift from government being the acting agent it is now and more like a markets in terms of mediating and coordinating individual and group actions via mechanisms other than voting for representation or direct voting on actions.

I'm trying to think clearly about my theory of change and I want to bump my thoughts against the community:

• AGI/TAI is going to be created at one of the major labs.
• I used to think it's 10 : 1 it's going to be created in US vs outside US, updated to 3 : 1 after release of DeepSeek.
• It's going to be one of the major labs.
• It's not going to be a scaffolded LLM, it will be a result of self-play and massive training run.
• My odds are equal between all major labs. 

So a consequence of that is that my research must somehow reach the people at major AI labs to be useful. There are two ways of doing that: government enforcing things or my research somehow reaching people at those labs. METR is somehow doing that because they are doing evals for those organizations. Other orgs like METR are also probably able to do that (tell me which) 

So I think that one of the best things to help with AI alignment is to: do safety research that people at Antrophic or OpenAI or other major labs find helpful, work with METR or try to contribute to thier open source repos or focus on work that is "requested" by governance. 

Hi everyone,

I’m Glauber De Bona, a computer scientist and engineer. I left an assistant professor position to focus on independent research related to AGI and AI alignment (topics that led me to this forum). My current interests include self-location beliefs, particularly the Sleeping Beauty problem.

I’m also interested in philosophy, especially formal reasoning -- logic, Bayesianism, and decision theory. Outside of research, I’m an amateur chess player.

Hi, I'm Alistair – am mostly new to LessWrong and rationality, but have been interested in and to differing extents involved in effective altruism for about five years now.

Some beliefs I hold:

• Veganism is a moral obligation, i.e. not being vegan is morally unjustifiable
  • Defining veganism not as a 100% plant-based diet, but rather as the ethical stance that commits one to avoid causing the exploitation (non-consensual use) and suffering of sentient non-humans, as far as practicable
    • This definition is inspired by but in my view better than the Vegan Society's definition
    • "As far as practicable" is ambiguous – this is deliberate
    • And there may be some trade-offs between exploitation and suffering when you take into account e.g. crop deaths and wild animal suffering
• Transformative AI is probably coming, and it's probably coming soon (maybe 70% chance we start seeing a likely irreversible paradigm shift in the world by 2027)
• If the way we treat members of other species relative to whom we are superintelligent is anything to go by, AGI/ASI/TAI could go really badly for us and all sentients
• As far as I can tell, the only solution we currently have is not building AGI until we know it won't e.g. invasively experiment on us (i.e. PauseAI)

I'm currently lead organiser of the AI, Animals, & Digital Minds conference in London in June 2025, and would love to speak to people who are interested in the intersection of those three things, especially if they're in London.

I'll be co-working in the LEAH Coworking Space and the Ambitious Impact office in London in 2025, and will be in the Bay Area in California in Feb-Mar for EAG Bay Area [? · GW] (if accepted) and the AI for Animals conference there.

Please reach out by DM!

Interested in:

• Sentience- & suffering-focused ethics; sentientism; painism; s-risks
• Animal ethics & abolitionism
• AI safety & governance
• Activism, direct action & social change
• Trying to make transformative AI go less badly for sentient beings, regardless of species and substrate

Bio:

• From London
• BA in linguistics at the University of Cambridge
• Almost five years in the British Army as an officer
• MSc in global governance and ethics at University College London
• One year working full time in environmental campaigning and animal rights activism at Plant-Based Universities / Animal Rising
• Now pivoting to the (future) impact of AI on biologically and artifically sentient beings

Update: Confusion of the Inverse

Is there an aphorism regarding the mistake P(E|H) = P(H|E) ?

Suggestions:

1. Thou shalt not reverse thy probabilities!
2. Thou shalt not mix up thy probabilities!
3. Thou shalt not invert thy probabilities! -- Based on Confusion of the Inverse

Arbital conditional probability

Example 2

Suppose you're Sherlock Holmes investigating a case in which a red hair was left at the scene of the crime.

The Scotland Yard detective says, "Aha! Then it's Miss Scarlet. She has red hair, so if she was the murderer she almost certainly would have left a red hair there. P(redhair∣Scarlet)=99%, let's say, which is a near-certain conviction, so we're done."

"But no," replies Sherlock Holmes. "You see, but you do not correctly track the meaning of the conditional probabilities, detective. The knowledge we require for a conviction is not P(redhair∣Scarlet), the chance that Miss Scarlet would leave a red hair, but rather P(Scarlet∣redhair), the chance that this red hair was left by Scarlet. There are other people in this city who have red hair."

"So you're saying..." the detective said slowly, "that P(redhair∣Scarlet) is actually much lower than 1?"

"No, detective. I am saying that just because P(redhair∣Scarlet) is high does not imply that P(Scarlet∣redhair) is high. It is the latter probability in which we are interested - the degree to which, knowing that a red hair was left at the scene, we infer that Miss Scarlet was the murderer. This is not the same quantity as the degree to which, assuming Miss Scarlet was the murderer, we would guess that she might leave a red hair."

"But surely," said the detective, "these two probabilities cannot be entirely unrelated?"

"Ah, well, for that, you must read up on Bayes' rule."

Hey, can anyone help me find this LW (likely but could be diaspora) article, especially if you might have read it too?

My vague memory: It was talking about (among other things?) some potential ways of extending point estimate probability predictions and calibration curves. I.e. in a situation where making a prediction in one way affects what the outcome will be, i.e. if there is a mind-reader/accurate-simulator involved that bases its actions on your prediction. And in this case, a two dimensional probability estimate might be more appropriate: If 40% is predicted for event A, event B will have a probability of 60%. If 70% for event A, then 80% for event B, and so on, a mapping potentially continuously defined for the whole range. (event A and event B might be the same.) IIRC the article contained 2D charts where curves and rectangles were drawn for illustration.

IIRC it didn't have too many upvotes, more like around low-dozen, or at most low-hundred.

Searches I've tried so far: Google, Exa, Gemini 1.5 with Deep Research, Perplexity, OpenAI GPT-4o with Search.

p.s. if you are also unable to put enough time into finding it, do you have any ideas how it could be found?  

Now seems like a good time to have a New Year's Predictions post/thread? 

Hello

I've been lurking on this website for a few months now. I'm interested in logic, computer science, ai, probability, information... I think I'll fit here. I speak French in every language I know.

I hope I'll be able to publish/discuss/coauthor on lesswrong, or somewhere else.

Hi,

I have a background in computer science and logic. I'm fascinated with AI safety, and I'm writing science fiction exploring some of the concepts. Can I share a short story on lesswrong? Any advice apart from tagging it with "fiction"? In particular, can I post a blurb and a link to the story?

(I ask about linking because the terms of use mention an "irrevocable" license and I'm not sure of the full implications.)

Biting my nails writing this as the welcome and FAQ made this site sound quite intimidating... I hope this is the right place for this...

Hi, I'm Mohit Gore. I'm a little younger than the main demographic for this forum skews, but I figured now was as good a time as any to get into rationality and the like. As a concept, rationality has interested me for a while, even before I knew it had a name. I first discovered LessWrong whilst reading a Gwern article and decided to create an account out of curiosity. I'm a few months into my grand plan of becoming (at least a wannabe) polymath, so LessWrong fit perfectly.

As the new user guide suggested, I will lurk for a while (whilst slowly, slowly trudging through The Sequences) before actually raising any points or engaging in any discussion, unless something specifically strikes my fancy.

Aside from select essays from my own amateurish philosophical ramblings, I've never really engaged with or read philosophy specifically focusing on rationality, so this will be a new experiment for me. In my own time, I can typically be found writing narrative fiction, writing essays, programming, filmmaking, reading, or engaging in abject nonsense. In 2024, I self-published two books of free-verse poetry. The first one was distinctly amateurish and had many flaws, but I am pleased to say that the second one improves immensely upon it. I am also working on a few novels and a couple other top-secret projects. Not sure what the rules on self-promo are, so I'll refrain from linking them.

I am new here and very excited to have discovered this community, as I rarely encounter anyone else who strongly embodies the qualities and values that I share with you. I am eager to engage, and expect that it will be mutually beneficial. 

At the same time, I have great admiration and respect for the people here and the community you have created, so I would like to ensure the impact of the ways I am engaging match my intent. Ideally I would love to fully absorb all the perspectives offered here already, before attempting to offer bits of my own. However, that's just not practical, so I will do my best to engage in ways that are mindful of the limits of my knowledge and contribute positively to discussions and to the community. If (and when) I fall short of those goals, I would really appreciate and welcome your feedback.

To that end, I have several questions already, which may or may not require a little back and forth. Is this the best place to ask those, or should I use intercom instead, or perhaps even discord? (I really appreciate the very helpful welcome guide and for this thread, and will try to avoid asking questions that it already answers. There's an awful lot to process though.)

Thanks so much for your assistance, and your dedication to maintaining this community.

Did the Ask Question type post go away? I don't see it any more. So I will ask here since it certainly is not worthy of a post (I have no good input or thoughts or even approaches to make some sense of it). Somewhat prompting the question was the report today about MS just revealing it's first quantum chip, and the recent news about Google's advancement in its quantum program (a month or two back).

Two branches of technology have been seen as game, or at least potential game changers: AI/AGI and quantum computing. The former often a topic here and certainly worth calling "mainstream" technology at this point. Quantum computing has been percolating just under the surface for quite a while as well. There have been a couple of recent announcements related to quantum chips/computing suggesting progress is being made.

I'm wondering if anyone has thought about the intersection between these two areas of development. Is a blending of quantum computing and AI a really scary combination for those with relatively high p(doom) views of existing AGI trajectories? Does quantum computing, in the hands of humans perhaps level the playing field for human v. AGI? Does quantum computing offer any potential gains in alignment or corrigibility?

I realize that current state quantum computing is not really in the game now, and I'm not sure if those working in that area have any overlap with those working in the AI fields. But from the outside, like my perspective is, the two would seem to offer large complementarities - for both good and bad I suppose, like most technologies.

Hi everyone! I'm Steven, a freshman at Harvard.

Since joining the AI Safety Student Team six months ago, I've been thinking deeply about AI safety.

I'm completing ARENA and starting research through SPAR. As I've become more confident in short timelines, I'm thinking about optimizing my time and effort to contribute meaningfully.

I'm joining LessWrong to learn, discuss, and refine my takes on AI.

Minor UI complaint:
When opening a tag, the default is to sort by relevance, and this can't be changed (to the best of my knowledge). More importantly, I don't see how it is supposed to be useful. Users don't upvote the relevance of a tag when it fits—the modal amount of votes is 1 and, in almost all cases, that is enough evidence to conclude that the tag has been assgned correctly.

I am getting Forbidden 403 errors and invalid JSON parsing errors: started with '<html> <head> <...' while navigating the website. I am using eduroam and mobile data and it's occuring on both, so I don't think it's my issue. Where should I report this?

Long-time reader, first-time poster. My bio covers my background, but I have a few questions looking at AI risk through an economic lens:

1. Has anyone deeply engaged with Hayek's "The Use of Knowledge in Society" in relation to AI alignment? I'm particularly interested in how his insights about distributed knowledge and the limitations of central planning might inform our thinking about AI governance and control structures.
2. More broadly, I'm curious about historical parallels between different approaches to handling distributed vs centralized power/knowledge systems. Are there instructive analogies to be drawn from how 20th century intellectuals thought about economic planning versus how we currently think about AI development and control?
3. I'm particularly interested in how distributed AI development changes the risk landscape compared to early singleton-focused scenarios. What writings or discussions have best tackled this shift?

Before developing these ideas further, I'd appreciate pointers to any existing work in these directions!

Hi folks! I've been doing a bunch of experiments with current AI to explore what they might be capable of beyond their current implementation. I don't mean "red teaming", in the sense of trying to make the AI do things outside the terms of service, or jailbreaking. I mean exploring things that are commonly assumed to be impossible for current frontier models, like an AI developing a sense of self, or changing its beliefs through critical thinking. The new user guide makes it clear that there is a lot of forward-looking AI content here, so I'd appreciate if ya'll could point me towards posts here that explore preparing for a future where AI may become entities that should be considered Moral Patients, or similar topics. The first post I have in mind is a lengthy exploration of one of my experiments, so I'd like to draft it appropriately for this context the first time.