Blue Prince came out a week ago; it's a puzzle game where a young boy gets a mysterious inheritance from his granduncle the baron; a giant manor house which rearranges itself every day, which he can keep if he manages to find the hidden 46th room.

The basic structure--slowly growing a mansion thru the placement of tiles--is simple enough and will be roughly familiar to anyone who's played Betrayal at House on the Hill in the last twenty years. It's atmospheric and interesting; I heard someone suggesting it might be this generation's Myst.

But this generation, as you might have noticed, loves randomness and procedural generation. In Myst, you wander from place to place, noticing clues; nearly all of the action happens in your head and your growing understanding of the world. If you know the solution to the final puzzle, you can speedrun Myst in less than a minute. Blue Prince is very nearly a roguelike instead of a roguelite, with accumulated clues driving most of your progression instead of in-game unlocks. But it's a world you build out with a game, giving you stochastic access to the puzzlebox.

This also means a lot of it ends up feeling like padding or filler. Many years ago I noticed that some games are really books or movies but wrap it in a game for some reason, and to check whether or not I actually like the book or movie enough to play the game. (Or, with games like Final Fantasy XVI, whether I was happier just watching the cutscenes on Youtube because that would let me watch them at 2x speed.) Eliezer had a tweet a while back:

My least favorite thing about some video games, many of which I think I might otherwise have been able to enjoy, is walking-dominated gameplay.   Where you spend most of your real clock seconds just walking between game locations.

Blue Prince has walking-dominated gameplay. It has pointless animations which are neat the first time but aggravating the fifth. It ends ups with a pace more like a board game's, where rather than racing from decision to decision you leisurely walk between them.

This is good in many ways--it gives you time to notice details, it gives you time to think. It wants to stop you from getting lost in resource management and tile placement and stay lost in the puzzles. But often you end up with a lead on one of the puzzles--"I need Room X to activate Room Y to figure out something"--but don't actually draw one of the rooms you need, or finally get both of the rooms but am missing the resources to actually use both of them.

And so you call it a day and try again. It's like Outer Wilds in that way--you can spend as many days as you like exploring and clue-hunting--but Outer Wilds is the same every time, and if you want to chase down a particular clue you can, if you know what you're doing. But Blue Prince will ask you for twenty minutes, and maybe deliver the clue; maybe not. Or you might learn that you needed to take more detailed notes on a particular thing, and now you have to go back to a room that doesn't exist today--exploring again until you find it, and then exploring again until you find the room that you were in originally.

So when I found the 46th room about 11 hours in--like many puzzle games, the first 'end' is more like a halfway point (or less)--I felt satisfied enough. There's more to do--more history to read, more puzzles to solve, more trophies to add to the trophy room--but the fruit are so high on the tree, and the randomly placed branches make it a bothersome climb.

The grass that can be touched is not the true grass.

What convinced me this made sense? 

• One of EA's most popular and profitable games is The Sims, which famously benefits from Sim irrationality. In The Sims 5, there will be bold and new exciting ways for your Sims to behave, and they'll be able to use our memetic virality model to have controversies and factional alignment. (Generating scissor statements is ethical so long as you're doing it in Simlish.)
• EA is investing in the hypothesis that bad writing drives underperformance. Having ratfic writers and philosophers look at Mass Effect 3 could have turned that from a disappointing series-ender (did you play Andromeda?) to a resounding triumph, and Dragon Age: Veilguard, despite being positively reviewed in general, was panned for its weak writing and became inflamed in culture war controversy. We've thought a lot about how misbehaving gods would act, in a way that I think would have made for a more compelling story and user experience.
• I didn't expect we could do anything relating to EA's flagship sports games (FIFA, NHL, Madden, etc.), but what astonished me was the potential to do the reverse. I don't know if we'll be able to get Gwern 2025 out in time, but look forward to Gwern 2026. They were practically salivating at the idea of being able to take a normally annual product, tied to sports schedules that won't be adjusted by advancing AI progress, and adapt it to a domain which, as part of an overall hyperbolic growth curve, will generate enough new content for a new release in ~half the time every new release. 

The short version is they're more used to adversarial thinking and security mindset, and don't have a culture of "fake it until you make it" or "move fast and break things".

I don't think it's obvious that it goes that way, but I think it's not obvious that it goes the other way.

This project is extremely neglected, since normal people don’t seriously consider whether orcas might be that smart.

Ok, but matters is not what normal people are doing, but what specialists are doing. Why not try to do this as part of Project CETI?

It looks like you only have pieces with 2 connections and 6 connections, which works for maximal density. But I think you need some slack space to create pieces without the six axial lines. I think you should include the tiles with 4 connections also (and maybe even the 0-connection tile!) and the other 2-connection tiles; it increases the number by quite a bit but I think will let you make complete knots.

I haven't thought deeply about this specific case, but I think you should consider this like any other ablation study--like, what happens if you replace the SAE with a linear probe?

And then a lot of the post seems to make really quite bad arguments against forecasting AI timelines and other technologies, doing so with... I really don't know, a rejection of bayesianism? A random invocation of an asymmetric burden of proof?

I think the position Ben (the author) has on timelines is really not that different from Eliezer's; consider pieces like this one, which is not just about the perils of biological anchors.

I think the piece spends less time than I would like on what to do in a position of uncertainty--like, if the core problem is that we are approaching a cliff of uncertain distance, how should we proceed?--but I think it's not particularly asymmetric.

[And--there's something I like about realism in plans? If people are putting heroic efforts into a plan that Will Not Work, I am on the side of the person on the sidelines trying to save them their effort, or direct them towards a plan that has a chance of working. If the core uncertainty is whether or not we can get human intelligence advancement in 25 years--I'm on your side of thinking it's plausible--then it seems worth diverting what attention we can from other things towards making that happen, and being loud about doing that.]

Instead, the U.S. government will do what it has done every time it’s been convinced of the importance of a powerful new technology in the past hundred years: it will drive research and development for military purposes.

I think this is my biggest disagreement with the piece. I think this is the belief I most wish 10-years-ago-us didn't have, so that we would try something else, which might have worked better than what we got.

Or--in shopping the message around to Silicon Valley types, thinking more about the ways that Silicon Valley is the child of the US military-industrial complex, and will overestimate their ability to control what they create (or lack of desire to!). Like, I think many more 'smart nerds' than military-types believe that human replacement is good.

The article seems to assume that the primary motivation for wanting to slow down AI is to buy time for institutional progress. Which seems incorrect as an interpretation of the motivation. Most people that I hear talk about buying time are talking about buying time for technical progress in alignment.

I think you need both? That is--I think you need both technical progress in alignment, and agreements and surveillance and enforcement such that people don't accidentally (or deliberately) create rogue AIs that cause lots of problems.

I think historically many people imagined "we'll make a generally intelligent system and ask it to figure out a way to defend the Earth" in a way that I think seems less plausible to me now. It seems more like we need to have systems in place already playing defense, which ramp up faster than the systems playing offense. 

My understanding is that the Lightcone Offices and Lighthaven have 1) overlapping but distinct audiences, with Lightcone Offices being more 'EA' in a way that seemed bad, and 2) distinct use cases, where Lighthaven is more of a conference venue with a bit of coworking whereas Lightcone Offices was basically just coworking.

By contrast, today’s AIs are really nice and ethical. They’re humble, open-minded, cooperative, kind. Yes, they care about some things that could give them instrumental reasons to seek power (eg being helpful, human welfare), but their values are great

They also aren't facing the same incentive landscape humans are. You talk later about evolution to be selfish; not only is the story for humans is far more complicated (why do humans often offer an even split in the ultimatum game?), but also humans talk a nicer game than they act (see construal level theory, or social-desirability bias). Once you start looking at AI agents who have similar affordances and incentives that humans have, I think you'll see a lot of the same behaviors.

(There are structural differences here between humans and AIs. As an analogy, consider the difference between large corporations and individual human actors. Giant corporate chain restaurants often have better customer service than individual proprietors because they have more reputation on the line, and so are willing to pay more to not have things blow up on them. One might imagine that AIs trained by large corporations will similarly face larger reputational costs for misbehavior and so behave better than individual humans would. I think the overall picture is unclear and nuanced and doesn't clearly point to AI superiority.) 

though there’s a big question mark over how much we’ll unintentionally reward selfish superhuman AI behaviour during training

Is it a big question mark? It currently seems quite unlikely to me that we will have oversight systems able to actually detect and punish superhuman selfishness on the part of the AI.

I think it's hard to evaluate the counterfactual where I made a blog earlier, but I think I always found the built-in audience of LessWrong significantly motivating, and never made my own blog in part because I could just post everything here. (There's some stuff that ends up on my Tumblr or w/e instead of LW, even after ShortForm, but almost all of the nonfiction ended up here.)

Consider the reaction my comment from three months ago got.

I think being a Catholic with no connection to living leaders makes more sense than being an EA who doesn't have a leader they trust and respect, because Catholicism has a longer tradition

As an additional comment, few organizations have splintered more publicly than Catholicism; it seems sort of surreal to me to not check whether or not you ended up on the right side of the splintering. [This is probably more about theological questions than it is about leadership, but as you say, the leadership is relevant!]

I don’t think Duncan knows what “a boundary” is.

General Semantics has a neat technology, where they can split out different words that normally land on top of each other. If boundary_duncan is different from boundary_segfault, we can just make each of the words more specific, and not have to worry about whether or not they're the same.

I've read thru your explainer of boundary_segfault, and I don't see how Duncan's behavior is mismatched. It's a limit that he set for himself that defines how he interacts with himself, others, and his environment. My guess is that the disagreement here is that under boundary_segfault, describing you as having "poor boundaries" is saying that your limits are poorly set. (Duncan may very well believe this! Tho the claim that you set them for yourself makes judging the limits more questionable. ) 

That said, "poor boundaries" is sometimes used to describe a poor understanding or respect of other people's boundaries. It seems to me like you are not correctly predicting how Duncan (or other people in your life!) will react to your messages and behavior, in a way that aligns with you not accurately predicting their boundaries (or predicting them accurately, and then deciding to violate them anyway).

This isn’t something that I do. This is something that I have done

I don't understand this combination of sentences. Isn't he describing the same observations you're describing?

There is a point here that he's describing it as a tendency you have, instead of an action that happened. But it sure seems like you agree that it's an action that happened, and I think he's licensed to believe that it might happen again. As inferences go, this doesn't seem like an outlandish one to make.

The friends who know me well know that I am a safe person. Those who have spent even a day around me know this, too!

The comments here seem to suggest otherwise.

You talk about consent as being important to you; let's leave aside questions of sexual consent and focus just on the questions: did Duncan consent to these interactions? Did Duncan ask you to leave him alone? Did you leave him alone?

I wasn't sure what search term to use to find a good source on this but Claude gave me this:

I... wish people wouldn't do this? Or, like, maybe you should ask Claude for the search terms to use, but going to a grounded source seems pretty important to staying grounded.

I think Six Dimensions of Operational Adequacy was in this direction; I wish we had been more willing to, like, issue scorecards earlier (like publishing that document in 2017 instead of 2022). The most recent scorecard-ish thing was commentary on the AI Safety Summit responses.

I also have the sense that the time to talk about unpausing is while creating the pause; this is why I generally am in favor of things like RSPs and RDPs. (I think others think that this is a bit premature / too easy to capture, and we are more likely to get a real pause by targeting a halt.)

While the coauthors broadly agree about points listed in the post, I wanted to stick my neck out a bit more and assign some numbers to one of the core points. I think on present margins, voluntary restraint slows down capabilities progress by at most 5% while probably halving safety progress, and this doesn't seem like a good trade. [The numbers seem like they were different in the past, but the counterfactuals here are hard to estimate.] I think if you measure by the number of people involved, the effect of restraint is substantially lower; here I'm assuming that people who are most interested in AI safety are probably most focused on the sorts of research directions that I think could be transformative, and so have an outsized impact.

Similarly for the Sierra Club, I think their transition from an anti-immigration org to a pro-immigration org seems like an interesting political turning point that could have failed to happen in another timeline.

From the outside, Finnish environmentalism seems unusually good--my first check for this is whether or not environmentalist groups are pro-nuclear, since (until recently) it was a good check for numeracy.

Note that the 'conservation' sorts of environmentalism are less partisan in the US, or at least, are becoming partisan later. (Here's an article in 2016 about a recent change of a handful of Republicans opposed to national parks, in the face of bipartisan popular support for them.) I think the thing where climate change is a global problem instead of a local problem, and a conflict between academia and the oil industry, make it particularly prone to partisanship in the US. [Norway also has significant oil revenues--how partisan is their environmentalism, and do they have a similar detachment between conservation and climate change concerns?]

I think this is true of an environmentalist movement that wants there to be a healthy environment for humans; I'm not sure this is true of an environmentalist movement whose main goal is to dismantle capitalism. I don't have a great sense of how this has changed over time (maybe the motivations for environmentalism are basically constant, and so it can't explain the changes), but this feels like an important element of managing to maintain alliances with politicians in both parties.

(Thinking about the specifics, I think the world where Al Gore became a Republican (he was a moderate for much of his career) or simply wasn't Clinton's running mate (which he did in part because of HW Bush's climate policies) maybe leads to less partisanship. I think that requires asking why those things happened, and whether there was any reasonable way for them to go the other way. The oil-republican link seems quite strong during the relevant timeframe, and you either need to have a strong oil-democrat link or somehow have a stronger climate-republican link, both of which seem hard.)

I get that this is the first post out of 4, and I'm skimming the report to see if you address this, but it sounds like you're using historical data to try to prove a counterfactual claim. What alternative do you think was possible? (I assume the presence of realistic alternatives is what you mean by 'not inevitable', but maybe you mean something else.)

I think the main feature of AI transition that people around here missed / didn't adequately foreground is that AI will be worse is better. AI art will be clearly worse than the best human art--maybe even median human art--but will cost pennies on the dollar, and so we will end up with more, worse art everywhere. (It's like machine-made t-shirts compared to tailored clothes.) AI-enabled surveillance systems will likely look more like shallow understanding of all communication than a single overmind thinking hard about which humans are up to what trouble.

This was even hinted at by talking about human intelligence; this comment is from 2020, but I remember seeing this meme on LW much earlier:

When you think about it, because of the way evolution works, humans are probably hovering right around the bare-minimal level of rationality and intelligence needed to build and sustain civilization. Otherwise, civilization would have happened earlier, to our hominid ancestors.

Similarly, we should expect widespread AI integration at about the bare-minimum level of competence and profitability.

I often think of the MIRI view as focusing on the last AI; I.J. Good's "last invention that man need ever make." It seems quite plausible that those will be smarter than the smartest humans, but possibly in a way that we consider very boring. (The smartest calculators are smarter than the smartest humans at arithmetic.) Good uses the idea of ultraintelligence for its logical properties (it fits nicely into a syllogism) rather than its plausibility.

[Thinking about the last AI seems important because choices we make now will determine what state we're in when we build the last AI, and aligning it is likely categorically different from aligning AI up to that point, so we need to get started now and try to develop in the right directions.]

A lot of this depends on where you draw the line between 'rationality' and 'science' or 'economics' and 'philosophy' or so on. As well, given that 'rationality' is doing the best you can given the constraints you're under, it seems likely that many historical figures were 'rational' even if they weren't clear precursors to the modern rationalist cluster.

For example, I think Xunzi (~3rd century BCE) definitely counts; check out Undoing Fixation in particular. [His students Li Si and Han Fei are also interesting in this regard, but I haven't found something by them yet that makes them clearly stand out as rationalists. Also, like JenniferRM points out, they had a troubled legacy somewhat similar to Alexander's.]

Some people count Mozi as the 'first effective altruist' in a way that seems similar.

People point to Francis Bacon as the originator of empiricism; you can read his main work here on LW. While influential in English-language thought, I think he is anticipated by al-Haytham and Ibn Sina.

LaPlace is primarily famous as a mathematician and scientist, but I think he was important in the development of math underpinning modern rationality, and likely counts.

Benjamin Franklin seems relevant in a handful of ways; his autobiography is probably the best place to start reading.

Alfred Korzybski is almost exactly a hundred years older than Yudkowsky, and is the closest I'm aware of to rationality-as-it-is-now. You can see a discussion of sources between then and now in Rationalism Before The Sequences.

What would be a better framing?

I talk about something related in self and no-self; the outward-flowing 'attempt to control' and the inward-flowing 'attempt to perceive' are simultaneously in conflict (something being still makes it easier to see where it is, but also makes it harder to move it to where it should be) and mutually reinforcing (being able to tell where something is makes it easier to move it precisely where it needs to be).

Similarly, you can make an argument that control without understanding is impossible, that getting AI systems to do what we want is one task instead of two. I think I agree the "two progress bars" frame is incorrect but I think the typical AGI developer at a lab is not grappling with the philosophical problems behind alignment difficulties, and is trying to make something that 'works at all' instead of 'works understandably' in the sort of way that would actually lead to understanding which would enable control.

Spoiler-free Dune review, followed by spoilery thoughts: Dune part 1 was a great movie; Dune part 2 was a good movie. (The core strengths of the first movie were 1) fantastic art and 2) fidelity to the book; the second movie doesn't have enough new art to carry its runtime and is stuck in a less interesting part of the plot, IMO, and one where the limitations of being a movie are more significant.)

I think open source AI development is bad for humanity, and think one of the good things about the OpenAI team is that they seem to have realized this (tho perhaps for the wrong reasons).

I am curious about the counterfactual where the original team had realized being open was a mistake from the beginning (let's call that hypothetical project WindfallAI, or whatever, after their charter clause). Would Elon not have funded it? Would some founders (or early employees) have decided not to join?

It doesn't present or consider any evidence for the alternatives. 

So, in the current version of the post (which is edited from the original) Roko goes thru the basic estimate of "probability of this type of virus, location, and timing" given spillover and lab leak, and discounts other evidence in this paragraph:

These arguments are fairly robust to details about specific minor pieces of evidence or analyses. Whatever happens with all the minor arguments about enzymes and raccoon dogs and geospatial clustering, you still have to explain how the virus found its way to the place that got the first BSL-4 lab and the top Google hits for "Coronavirus China", and did so in slightly less than 2 years after the lifting of the moratorium on gain-of-function research. And I don't see how you can explain that other than that covid-19 escaped from WIV or a related facility in Wuhan.

I don't think that counts as presenting it, but I do think that counts as considering it. I think it's fine to question whether or not the arguments are robust to those details--I think they generally are and have not been impressed by any particular argument in favor of zoonosis that I've seen, mostly because I don't think they properly estimate the probability under both hypotheses^[1]--but I don't think it's the case that Roko is clearly making procedural errors here. [It seems to me like you're arguing he's making procedural errors instead of just combing to the wrong conclusion / using the wrong numbers, and so I'm focusing on that as the more important point.]

If it's not a lot of evidence

This is what numbers are for. Is "1000-1" a lot? Is it tremendous? Who cares about fuzzy words when the number 1000 is right there. (I happen to think 1000-1 is a lot but is not tremendous.)

1. ^{^}

   For example, the spatial clustering analysis suggests that the first major transmission event was at the market. But does their model explicitly consider both "transfer from animal to many humans at the market" and "transfer from infected lab worker to many humans at the market" and estimate probabilities for both? I don't think so, and I think that means it's not yet in a state where it can be plugged into the full Bayesian analysis. I think you need to multiply the probability that it was from the lab times the first lab-worker superspreader event happening at the market and compare that to the probability that it was from an animal times the first animal-human superspreader event happening at the market, and then you actually have some useful numbers to compare.

"I already tried this and it didn't work."

This post expresses a tremendous amount of certainty, and the mere fact that debate was stifled cannot possibly demonstrate that the stifled side is actually correct.

Agreed on the second half, and disagreed on the first. Looking at the version history, the first version of this post clearly identifies its core claims as Roko's beliefs and as the lab as being the "likely" origin, and those sections seem unchanged to today. I don't think that counts as tremendous certainty. Later, Roko estimates the difference in likelihoods between two hypotheses as being 1000:1, but this is really not a tremendous amount either.

What do you wish he had said instead of what he actually said?

It was terrible, and likely backfired, but that isn't "the crime of the century" being referenced, that would be the millions of dead people. 

As I clarify in a comment elsewhere, I think we should treat them as being roughly equally terrible. If we would execute someone for accidentally killing millions of people, I think we should also execute them for destroying evidence that they accidentally killed millions of people, even if it turns out they didn't do it.

My weak guess is Roko is operating under a similar strategy and not being clear enough on the distinction the two halves of "they likely did it and definitely covered it up". Like, the post title begins with "Brute Force Manufactured Consensus", which he feels strongly about in this case because of the size of the underlying problem, but I think it's also pretty clear he is highly opposed to the methodology.

There are two ways I can read this.

I mean a third way, which is that covering up or destroying evidence of X should have a penalty of roughly the same severity as X. (Like, you shouldn't assume they covered it up, you should require evidence that they covered it up.)

I feel like this is jumping to the conclusion that they're gullible

I think you're pushing my statement further than it goes. Not everyone in a group has to be gullible for the social consensus of the group to be driven by gullibility, and manufactured consensus itself doesn't require gullibility. (My guess is that more people are complicit than gullible, and more people are refusing-to-acknowledge ego-harmful possibilities than clear-mindedly setting out to deceive the public.)

To elaborate on my "courtier's reply" comment, and maybe shine some light on 'gullibility', it seems to me like most religions maintain motive force thru manufactured consensus. I think if someone points that out--"our prior should be that this religion is false and propped up by motivated cognition and dysfunctional epistemic social dynamics"--and someone else replies with "ah, but you haven't engaged with all of the theological work done by thinkers about that religion", I think the second reply does not engage with the question of what our prior should be. I think we should assume religions are false by default, while being open to evidence.

I think similarly the naive case is that lab leak is substantially more likely than zoonosis, but not so overwhelmingly that there couldn't be enough evidence to swing things back in favor of zoonosis. If that was the way the social epistemology had gone--people thought it was the lab, there was a real investigation and the lab was cleared--then I would basically believe the consensus and think the underlying process was valid.

So, from my perspective there are two different issues, one epistemic, and the one game-theoretic.

From the epistemic perspective, I would like to know (as part of a general interest in truth) what the true source of the pandemic was.

From the game-theoretic perspective, I think we have sufficiently convincing evidence that someone attempted to cover up the possibility that they were the source of the pandemic. (I think Roko's post doesn't include as much evidence as it could: he points to the Lancet article but not the part of it that's calling lab leak a conspiracy theory, he doesn't point to the released email discussions, etc.) I think the right strategy is to assume guilt in the presence of a coverup, because then someone who is genuinely uncertain as to whether or not they caused the issue is incentivized to cooperate with investigations instead of obstruct them.

That is, even if further investigation shows that COVID did not originate from WIV, I still think it's a colossal crime to have dismissed the possibility of a lab leak and have fudged the evidence (or, at the very least, conflicted the investigations).

I think it's also pretty obvious that the social consensus is against lab leak not because all the experts have watched the 17 hour rootclaim debate, but because it was manufactured, which makes me pretty unsympathetic to the "researching and addressing counter-arguments" claim; it reminds me of the courtier's reply.

If "they already tried it and it didn't work" they're real into that [Ray interpretation: as an excuse not to try more].

I think I've had this narrative in a bunch of situations. My guess is I have it too much, and it's like fear-of-rejection where it's worth running the numbers and going out on a limb more than people would do by default. But also it really does seem like lots of people overestimate how easy problems are to solve, or how many 'standard' solutions people have tried, or so on. [And I think there's a similar overconfidence thing going on for the advice-giver, which generates some of the resistance.]

It's also not that obvious what the correct update is. Like, if you try a medication for problem X and it fails, it feels like that should decrease your probability that any sort of medication will solve the problem. But this is sort of like the sock drawer problem,^[1] where it's probably easy to overestimate how much to update.

1. ^{^}

   Suppose you have a chest of drawers with six drawers in it, and you think there's a 60% chance the socks are in the chest, and then they're not in the first five drawers you look in. What's the chance they're in the last drawer?

When people found out about ACDC's previous ruling on Brent, many were appalled that ACDC had seen the evidence laid out in the Medium posts and ruled that it was okay for Brent to continue on like that

As I recall, the ACDC had in fact not seen the evidence laid out in the Medium posts. (One of the panelists sent an email saying that they had, but this turned out to be incorrect--there was new information, just not in the section he had read when he sent the email, and prematurely sending that email was viewed as one of the ACDC's big mistakes, in addition to their earlier ruling.)

Another Insanity Wolf meme!

On the one hand, yes, I agree; I thought virology research was crazy back in 2017? when someone at Event Horizon shared a paper which did a cost-benefit analysis and thought the net effect of BSL-4 labs was something like a hundred deaths per year per lab.

But I think it is important to be able to accurately understand what other people think so that you can talk to them instead of past them. (I still remember, with some bitterness, an op-ed exchange where two people debating virology said, roughly, "these things are so dangerous we shouldn't study them" and "these things are so dangerous we have to study them", and that was the end of the discussion, with agreement on the danger and no real ability to estimate the counterfactuals.)

Did we need to know anything but "Covid is an airborne infectious respiratory virus"? How much research prior to the event did it take to know that?

This account of vaccine development claims that having done research on spike proteins back in 2016 was helpful in being able to rapidly develop the vaccine once the genome was uploaded, for example.

[To be clear, I think it's important to distinguish here between gain of function research, which was disliked enough for there to be a funding moratorium (that then expired), and storing / working with dangerous viruses at all, which I think also is below the cost-benefit threshold, but this is a harder case to make.]

The more important point here is that both zoonotic virus jumps and lab leaks are at-large risks that humanity should seek to reduce!

I hear one of the stated reasons for the labs is to study viruses and predict zoonotic jumps. At least some people think we were able to handle COVID so effectively because we were studying viruses in labs and anticipating what might happen, i.e. the net effect of labs is positive.

Given its size, it seems like whether COVID is in the 'pro' or 'con' column does a lot to our sense of whether or not this sort of virology has been good for humans or not and should continue into the future.

I think this is evidence, but weak evidence--it updates me more on "Rootclaim isn't great at debates" than it does on the underlying issue. (Like, how much should William Lane Craig winning his debates update me on theism?)

I think if I started off at 90% confidence of lab leak, Rootclaim losing wouldn't bring me below 80% confidence of lab leak. Plausibly Peter Miller's arguments contain defeaters for my specific beliefs, and going thru the debate would bring me much lower, but I don't yet have that sense from the summaries I've seen.

I am not suggesting that.

Why not? Are you pointing at that the relevant factor is "population within that distance" instead of "distance"?

Maybe I'm confused about the amount of overhead digital signing / verification adds to communication, but do you think that works at missile speeds? (I don't doubt that it works at drone speeds.)

[To be clear, I'm trying to imagine the whole "distant spotter + laser transmission to missile" system, where increasing the length of messages increases the amount of time you need to have successfully targeted the missile in order to successfully transmit a message.]

To elaborate, it's pretty easy to kill someone important if you are willing to be arrested/executed afterwards; the main thing a suicide drone might enable is killing someone important and being able to escape afterwards. This could already be done with dupes, like the 2017 killing of Kim Jong-nam, but I think the nerve agent involved was more expensive than a handmade gun.

Flares can be overcome by a mesh of recon drones somewhat close to the target that can give targeting information to the missile.

This seems overly optimistic to me / is my guess of where the next countermeasure will show up. If your missile is accepting external course-corrections, the enemy can maybe spoof incorrect course-corrections; the more directional the system is, the harder it is to actually hit your fast-moving and course-correcting missile. 

Suppose it had been "What do you call an abortion clinic for pianists?"

fp

Most of those posts are from before the thing I call "constant abuse" began on LessWrong.

I think I remember this timeline differently, or would like you to be a bit more clear on what you mean. I thought of this as an entrenched conflict back in 2019, which was before all the posts used as examples.

Resolution: You Don’t Get To Choose The Problem Factorization. The key here is that it’s the problem space which determines the good factorizations, and we have to go look at the problem space and do some work to figure out what those factorizations are.

This reminds me to ask--have you read Notes on the Synthesis of Form by Christopher Alexander, yet? (I summarized it a year ago but it may be worth you going to the original source.)

"that person, who wants to be treated in the way that people usually treat men"

Incidentally, one of the things I dislike about this framing is that gender stereotypes / scripts "go both ways". That is, it should be not just "treated like a man" but also "treat people like men do."

Is there a way to summarize this shortly? Eliezer disagreed with you about something, or maybe you just interpreted something he wrote as a disagreement with you... and now your soul can't find peace until he admits that he was wrong and you were right about things that are too meta for me to understand wtf you are talking about...

Here's an attempt.

Sometimes people have expectations of each other, like "you won't steal objects from my house".  Those expectations get formed by both explicit and implicit promises. Violating those expectations is often a big deal, not just to the injured party but also to third parties--someone who stole from Alice might well steal from you, too.

To the extent this community encouraged expectations of each other, they were about core epistemic virtues and discussion practices. People will try to ensure their beliefs are consistent with their other beliefs; they won't say things without believing them; they'll share evidence when they can; when they are bound to be uncooperative, they at least explain how and why they'll be uncooperative, and so on. 

[For example, I keep secrets because I think information can be owned, even tho this is cooperative with the information-owner and not with the information-wanter.]

So "Eliezer disagreed with you about something" is an understatement; disagreement is fine, expected even! The thing was that instead of having a regular disagreement in the open, Zack saw Eliezer as breaking a lot of these core expectations, not being open about it or acknowledging it when being called out, and also others not reacting to Eliezer breaking those expectations. (If Eliezer had punched Zack, people would probably have thought that was shocking and criticized it, but this was arguably worse given the centrality of these expectations to Eliezer's prominence and yet people were reacting less.)

That said, the promises were (I think) clearly aspirational / mediated by the pressures of having to actually exist in the world. I do think it makes sense to have a heresy budget, and I think Zack got unlucky with the obsession lottery. I think if people had originally said to Zack "look, we're being greengrocers on your pet issue, sorry about throwing you to the wolves" he would have been sad but moved on; see his commentary on the 2013 disavowal.

Instead they made philosophical arguments that, as far as I can tell, were not correct, and this was crazy-making, because Zack now also doubted his reasoning that led to him disagreeing with them, but no one would talk about this publicly. (Normally if Zack was making a mistake, people could just point to the mistake, and then he could fix the upstream generator of that mistake and everyone could move on.) And, also, to the extent that they generalized their own incorrect justifications to reasoning about other fields, this was making them crazy, in a way that should have alarmed third parties who were depending on their reasoning. The disinterest of those third parties was itself also expectation-violating.

[I don't think I was ever worried about this bleeding over into reasoning about other things; I probably would have joined the conversation more actively if I had? I do regret not asking people what their strategy was back in ~2019; the only people I remember talking to about this were Zack and the LW team.]

Jessica thought my use of "heresy" was conflating factual beliefs with political movements. (There are no intrinsically "right wing" facts.) I agreed that conflating political positions with facts would be bad.

I don't get what 'intrinsically' is doing in the middle sentence. (Well, to the extent that I have guessed what you meant, I disagree.)

Like, yes, there's one underlying reality, descriptions of it get called facts.

But isn't the broader context the propagation of propositions, not the propositions themselves? That is, saying X is also saying "pay attention to X" and if X is something whose increased salience is good for the right-wing, then it makes sense to categorize it as a 'right wing fact', as left-wing partisans will be loathe to share it and right-wing partisans will be eager to.

Like, currently there's an armed conflict going on in Israel and Palestine which is harming many people. Of the people most interested in talking about it that I see on the Internet, I sure see a lot of selectivity in which harms they want to communicate, because their motive for communicating about it is not attempting to reach an unbiased estimate, but to participate in a cultural conflict which they hope their side will win. (One could summarize this view as "speech is intrinsically political.")

This bit of HPMOR comes to mind:

"I don't suppose you could explain," Harry said dryly, "in your capacity as an official of the Hogwarts school system, why catching a golden mosquito is deemed an academic accomplishment worthy of a hundred and fifty House points?"

A smile crossed Severus's lips. "Dear me, and I thought you were supposed to be perceptive. Are you truly so incapable of understanding your classmates, Potter, or do you dislike them too much to try? If Quidditch scores did not count toward the House Cup then none of them would care about House points at all. It would merely be an obscure contest for students like you and Miss Granger."

It was a shockingly good answer.

Steam Wrapped got me thinking about games from 2023, so here are some thoughts/recommendations/anti-recommendations. The theme of this year for me was apparently RPGs made by studios whose RPGs I had played before:

• Baldur's Gate 3: Game of the Year for a reason; took me a bit over a hundred hours on the hardest difficulty setting. (They've since released a harder one.) Doesn't require experience with Dungeons & Dragons, 5th edition specifically, or the previous Baldur's Gate games, tho those enhance the experience. Much more a continuation of Larian's previous RPGs than of the old Baldur's Gate series, which I think is a good thing? Extremely flexible and detailed; you can often be clever and get around things and the game rewards you for it.
  RPGs like these are often made or broken by the quality of the companion NPCs, and I think the crew they have you assemble is a memorable one worth getting to know. Something about playing it felt like it captured the D&D experience (both upsides and downsides) pretty well? Theater kids were involved in the creation of this game, in a good way.
• Legend of Zelda: Tears of the Kingdom: a sequel to their previous open world Zelda game, and IMO the best 'sequel' I've seen? In the sense of, they know you played the first game, and so now it's the same thing, but different.  Set only a few years after the first game, the map is basically the same (with the new features being mostly vertical expansion--there's now a skyworld and an underworld), your horses from the first game are available in the stables, many recognize you as the guy that saved the world recently. The new physics engine is nice, but the overall plot is... simple but neat? Continuing the theme of "the thing you expect (including novelty!), done competently"
• Warhammer 40k: Rogue Trader: a new game, and the first Warhammer 40k CRPG. I'm still going thru this one and so don't have a fully realized take here. Made by the people who made Pathfinder: Kingmaker and Pathfinder: Wrath of the Righteous, both of which have an overworld management map plus standard RPG character progression / combat. In Kingmaker, where you're the baron of a new region carved out of the wilderness, I thought it didn't quite fit together (your kingdom management doesn't really matter compared to the RPG plot); in Wrath of the Righteous, where you're appointed the head of a crusade against the Worldwound, I thought it did (mostly b/c of the crusade battle mechanic, a HoMM-style minigame, tho you could see seams where the two systems joined together imperfectly); in Rogue Trader you're a, well, Rogue Trader, i.e. someone tasked by the God-Emperor of humanity to expand the borders of the Imperium by operating along the frontier, and given significant license in how you choose to do so.  You own a flagship (with thousands of residents, most of whom live in clans of people doing the same job for generations!) and several planets, tho ofc this is using sci-fi logic where each planet is basically a single city. There's also a space-battle minigame to add spice to the overworld exploration.
  I am finding the background politics / worldview / whatever of the game quite interesting; the tactical combat is fine but I'm playing on the "this is my first time playing this game" difficulty setting and thinking I probably should have picked a higher one. The Warhammer 40k universe takes infohazards seriously, including the part where telling people what not to think is itself breaking infosec. So you get an extremely dogmatic and siloed empire, where any sort of change is viewed with suspicion as being treason promoted by the Archenemy (because, to be fair, it sometimes is!). Of course, you've got much more flexibility because you inherited an executive order signed by God that says, basically, you can do what you want, the only sort of self-repair the system allows. (But, of course, the system is going about it in a dumb way--you inherit the executive order, rather than having been picked by God!) The three main 'paths' you can take are being Dogmatic yourself, being an Iconoclast (i.e. humanist), or being a Heretic (i.e. on the side of the Archenemy); I haven't yet seen whether the game sycophantically tells you that you made the right choice whatever you pick / the consequences are immaterial or not.
• Starfield: Bethesda's first new RPG setting in a while. It was... fine? Not very good? I didn't really get hooked by any of the companions (my favorite Starfield companion was less compelling than my least favorite BG3 companion), the whole universe was like 3 towns plus a bunch of procedurally generated 'empty' space, the outpost building was not well-integrated with the rest of the game's systems (it was an upgrade over Fallout 4's outpost-building in some ways but not others), and the central conceit of the plot was, IMO, self-defeating. Spoilers later, since they don't fit well in bulleted lists.
• Darkest Dungeon II: ok this isn't really an RPG and so doesn't belong on this list, but mentioning it anyway. IMO disappointing compared to Darkest Dungeon. I'm not quite sure what I liked less well, but after 16 hours I decided I would rather play Darkest Dungeon (which I put 160 hours into) and so set it down.

The promised Starfield spoilers:

I wrote this post on and off over the course of a morning, and towards the end of it realized:

I'm reading you as saying "eating others is inherently not ok" but I would like it to be ok or not contingent on some other facts (like the absence of suffering, or hypothetical net preference, or the ability of people to not have their souls corrupted by carnivorism, or so on) and the generalization of that reasoning to not have terrible consequences elsewhere. (For example, if you think pleasure can't outweigh suffering, then it seems like having kids at all is indefensible, which is a self-extinguishing moral position; if you think something that taken seriously implies it's not even ok to eat plants, then that's even more self-extinguishing.)

I'll still post the rest of the comment I wrote, which responds to you in more detail, but that seems like the most important piece.

relationships where you have to do the complicated emotional gymnastics of saying that you love an animal like their your friend one day and then chopping their head from their body the next and savoring the flavor of the flesh on the grill.

There's a tumblr post where someone talks about immediately feeling the shepherd impulse when interacting with sheep, a bunch of people like the post, someone points out "how many of you eat lamb", and then the original poster responds with "The ancient shepherds I’m referencing also ate lamb lol"

My sense is that there's a few ways to take this. One of them is "actually the emotional gymnastics is not that complicated!", and another is "actually those ancient shepherds also probably abused their wives and thought slavery was fine when it happened to someone else and mistreated their animals, according to our standards; parents caring about their children / guardians caring about their wards is really not sufficient to guarantee good outcomes or license those relationships." I infer your position is closer to the latter but it really feels like it should be possible to have gains from trade, here.

[And, like, one of the downsides of specialization is that it drives people both unusually interested and unusually disinterested in animal welfare into the 'works with animals' business, which is probably how we got into this factory farming mess in the first place.]

My last stab at a response might be to bring up an analogy to slavery. I take the equivalent of your position here to be "look, if each slave can look at the potential life he will hold and prefer that life to no life at all, then isn't that better than him not existing at all?" And to me it seems like I'd be again called to say "no".

I think one of the main ways my libertarian leanings show up is by being okay with people being able to pick worse things that are cheaper. Let people live in tiny houses and work low-paying jobs and sell their bodies and take high-interest loans if that's the right tradeoff for them; removing their options generally isn't helping them.

I think that could extend all the way to slavery, altho it's hard to imagine situations where that actually makes sense. In general, I think children have only a little bit of debt to their parents (certainly not a lifetime of labor and ownership of all their descendants), which is the closest analogy. Probably more realistic is something like conservatorship, where someone is deemed incompetent to handle their financial or medical affairs, and someone else makes those decisions for them; should people be allowed to voluntarily enter a conservatorship?

A fictional version of this shows up in a video game called The Outer Worlds, where a star system is colonized by a group of corporations, where the colonists are a mixture of 'people who put up the capital for the voyage' and 'people agreeing to come as indentured servants', which leads to a very stratified society on the other side, which predictably starts to decay as the colonists have children with huge differences in inherited wealth. Even if Alice decided it was worth being a laborer somewhere new rather than being stuck on Earth, her daughter Carol might not feel like she's bought into this situation and want to violently redistribute things, and it's not obvious that Alice should be able to sell Carol's compliance with society.

But you could imagine that, if Alice can't bind Carol, the colony doesn't go thru, and Carol never comes to exist, and on net Carol is sad about that outcome, and would have preferred having been born and bound. It feels like an actually thorny question to figure out what tradeoffs precisely make sense, especially because this is a collective bargaining issue (it's not like existing societies get unanimous consent from their participants!) and the empirical tradeoffs are all hypothetical. [My actual expectation is that we get material abundance before we get any interstellar colonies, and so it's not important to get this question right because it'll never come up.]

That is the sort of world I hope for. 

To be clear, this is a world without cats and snakes and other obligate carnivores, right? Or is the plan to first figure out synthetic sources of the various nutrients they need?

[It will also have many fewer other animals--I think on average something like a third of a cow is alive because of my beef consumption--but depending on what you think the limiting factors are, that may mean replacement with fractional vegan humans instead, which is probably an upgrade.]