Consider the reaction my comment from three months ago got.

I think being a Catholic with no connection to living leaders makes more sense than being an EA who doesn't have a leader they trust and respect, because Catholicism has a longer tradition

As an additional comment, few organizations have splintered more publicly than Catholicism; it seems sort of surreal to me to not check whether or not you ended up on the right side of the splintering. [This is probably more about theological questions than it is about leadership, but as you say, the leadership is relevant!]

I don’t think Duncan knows what “a boundary” is.

General Semantics has a neat technology, where they can split out different words that normally land on top of each other. If boundary_duncan is different from boundary_segfault, we can just make each of the words more specific, and not have to worry about whether or not they're the same.

I've read thru your explainer of boundary_segfault, and I don't see how Duncan's behavior is mismatched. It's a limit that he set for himself that defines how he interacts with himself, others, and his environment. My guess is that the disagreement here is that under boundary_segfault, describing you as having "poor boundaries" is saying that your limits are poorly set. (Duncan may very well believe this! Tho the claim that you set them for yourself makes judging the limits more questionable. ) 

That said, "poor boundaries" is sometimes used to describe a poor understanding or respect of other people's boundaries. It seems to me like you are not correctly predicting how Duncan (or other people in your life!) will react to your messages and behavior, in a way that aligns with you not accurately predicting their boundaries (or predicting them accurately, and then deciding to violate them anyway).

This isn’t something that I do. This is something that I have done

I don't understand this combination of sentences. Isn't he describing the same observations you're describing?

There is a point here that he's describing it as a tendency you have, instead of an action that happened. But it sure seems like you agree that it's an action that happened, and I think he's licensed to believe that it might happen again. As inferences go, this doesn't seem like an outlandish one to make.

The friends who know me well know that I am a safe person. Those who have spent even a day around me know this, too!

The comments here seem to suggest otherwise.

You talk about consent as being important to you; let's leave aside questions of sexual consent and focus just on the questions: did Duncan consent to these interactions? Did Duncan ask you to leave him alone? Did you leave him alone?

I wasn't sure what search term to use to find a good source on this but Claude gave me this:

I... wish people wouldn't do this? Or, like, maybe you should ask Claude for the search terms to use, but going to a grounded source seems pretty important to staying grounded.

I think Six Dimensions of Operational Adequacy was in this direction; I wish we had been more willing to, like, issue scorecards earlier (like publishing that document in 2017 instead of 2022). The most recent scorecard-ish thing was commentary on the AI Safety Summit responses.

I also have the sense that the time to talk about unpausing is while creating the pause; this is why I generally am in favor of things like RSPs and RDPs. (I think others think that this is a bit premature / too easy to capture, and we are more likely to get a real pause by targeting a halt.)

While the coauthors broadly agree about points listed in the post, I wanted to stick my neck out a bit more and assign some numbers to one of the core points. I think on present margins, voluntary restraint slows down capabilities progress by at most 5% while probably halving safety progress, and this doesn't seem like a good trade. [The numbers seem like they were different in the past, but the counterfactuals here are hard to estimate.] I think if you measure by the number of people involved, the effect of restraint is substantially lower; here I'm assuming that people who are most interested in AI safety are probably most focused on the sorts of research directions that I think could be transformative, and so have an outsized impact.

Similarly for the Sierra Club, I think their transition from an anti-immigration org to a pro-immigration org seems like an interesting political turning point that could have failed to happen in another timeline.

From the outside, Finnish environmentalism seems unusually good--my first check for this is whether or not environmentalist groups are pro-nuclear, since (until recently) it was a good check for numeracy.

Note that the 'conservation' sorts of environmentalism are less partisan in the US, or at least, are becoming partisan later. (Here's an article in 2016 about a recent change of a handful of Republicans opposed to national parks, in the face of bipartisan popular support for them.) I think the thing where climate change is a global problem instead of a local problem, and a conflict between academia and the oil industry, make it particularly prone to partisanship in the US. [Norway also has significant oil revenues--how partisan is their environmentalism, and do they have a similar detachment between conservation and climate change concerns?]

I think this is true of an environmentalist movement that wants there to be a healthy environment for humans; I'm not sure this is true of an environmentalist movement whose main goal is to dismantle capitalism. I don't have a great sense of how this has changed over time (maybe the motivations for environmentalism are basically constant, and so it can't explain the changes), but this feels like an important element of managing to maintain alliances with politicians in both parties.

(Thinking about the specifics, I think the world where Al Gore became a Republican (he was a moderate for much of his career) or simply wasn't Clinton's running mate (which he did in part because of HW Bush's climate policies) maybe leads to less partisanship. I think that requires asking why those things happened, and whether there was any reasonable way for them to go the other way. The oil-republican link seems quite strong during the relevant timeframe, and you either need to have a strong oil-democrat link or somehow have a stronger climate-republican link, both of which seem hard.)

I get that this is the first post out of 4, and I'm skimming the report to see if you address this, but it sounds like you're using historical data to try to prove a counterfactual claim. What alternative do you think was possible? (I assume the presence of realistic alternatives is what you mean by 'not inevitable', but maybe you mean something else.)

I think the main feature of AI transition that people around here missed / didn't adequately foreground is that AI will be worse is better. AI art will be clearly worse than the best human art--maybe even median human art--but will cost pennies on the dollar, and so we will end up with more, worse art everywhere. (It's like machine-made t-shirts compared to tailored clothes.) AI-enabled surveillance systems will likely look more like shallow understanding of all communication than a single overmind thinking hard about which humans are up to what trouble.

This was even hinted at by talking about human intelligence; this comment is from 2020, but I remember seeing this meme on LW much earlier:

When you think about it, because of the way evolution works, humans are probably hovering right around the bare-minimal level of rationality and intelligence needed to build and sustain civilization. Otherwise, civilization would have happened earlier, to our hominid ancestors.

Similarly, we should expect widespread AI integration at about the bare-minimum level of competence and profitability.

I often think of the MIRI view as focusing on the last AI; I.J. Good's "last invention that man need ever make." It seems quite plausible that those will be smarter than the smartest humans, but possibly in a way that we consider very boring. (The smartest calculators are smarter than the smartest humans at arithmetic.) Good uses the idea of ultraintelligence for its logical properties (it fits nicely into a syllogism) rather than its plausibility.

[Thinking about the last AI seems important because choices we make now will determine what state we're in when we build the last AI, and aligning it is likely categorically different from aligning AI up to that point, so we need to get started now and try to develop in the right directions.]

A lot of this depends on where you draw the line between 'rationality' and 'science' or 'economics' and 'philosophy' or so on. As well, given that 'rationality' is doing the best you can given the constraints you're under, it seems likely that many historical figures were 'rational' even if they weren't clear precursors to the modern rationalist cluster.

For example, I think Xunzi (~3rd century BCE) definitely counts; check out Undoing Fixation in particular. [His students Li Si and Han Fei are also interesting in this regard, but I haven't found something by them yet that makes them clearly stand out as rationalists. Also, like JenniferRM points out, they had a troubled legacy somewhat similar to Alexander's.]

Some people count Mozi as the 'first effective altruist' in a way that seems similar.

People point to Francis Bacon as the originator of empiricism; you can read his main work here on LW. While influential in English-language thought, I think he is anticipated by al-Haytham and Ibn Sina.

LaPlace is primarily famous as a mathematician and scientist, but I think he was important in the development of math underpinning modern rationality, and likely counts.

Benjamin Franklin seems relevant in a handful of ways; his autobiography is probably the best place to start reading.

Alfred Korzybski is almost exactly a hundred years older than Yudkowsky, and is the closest I'm aware of to rationality-as-it-is-now. You can see a discussion of sources between then and now in Rationalism Before The Sequences.

What would be a better framing?

I talk about something related in self and no-self; the outward-flowing 'attempt to control' and the inward-flowing 'attempt to perceive' are simultaneously in conflict (something being still makes it easier to see where it is, but also makes it harder to move it to where it should be) and mutually reinforcing (being able to tell where something is makes it easier to move it precisely where it needs to be).

Similarly, you can make an argument that control without understanding is impossible, that getting AI systems to do what we want is one task instead of two. I think I agree the "two progress bars" frame is incorrect but I think the typical AGI developer at a lab is not grappling with the philosophical problems behind alignment difficulties, and is trying to make something that 'works at all' instead of 'works understandably' in the sort of way that would actually lead to understanding which would enable control.

Spoiler-free Dune review, followed by spoilery thoughts: Dune part 1 was a great movie; Dune part 2 was a good movie. (The core strengths of the first movie were 1) fantastic art and 2) fidelity to the book; the second movie doesn't have enough new art to carry its runtime and is stuck in a less interesting part of the plot, IMO, and one where the limitations of being a movie are more significant.)

I think open source AI development is bad for humanity, and think one of the good things about the OpenAI team is that they seem to have realized this (tho perhaps for the wrong reasons).

I am curious about the counterfactual where the original team had realized being open was a mistake from the beginning (let's call that hypothetical project WindfallAI, or whatever, after their charter clause). Would Elon not have funded it? Would some founders (or early employees) have decided not to join?

It doesn't present or consider any evidence for the alternatives. 

So, in the current version of the post (which is edited from the original) Roko goes thru the basic estimate of "probability of this type of virus, location, and timing" given spillover and lab leak, and discounts other evidence in this paragraph:

These arguments are fairly robust to details about specific minor pieces of evidence or analyses. Whatever happens with all the minor arguments about enzymes and raccoon dogs and geospatial clustering, you still have to explain how the virus found its way to the place that got the first BSL-4 lab and the top Google hits for "Coronavirus China", and did so in slightly less than 2 years after the lifting of the moratorium on gain-of-function research. And I don't see how you can explain that other than that covid-19 escaped from WIV or a related facility in Wuhan.

I don't think that counts as presenting it, but I do think that counts as considering it. I think it's fine to question whether or not the arguments are robust to those details--I think they generally are and have not been impressed by any particular argument in favor of zoonosis that I've seen, mostly because I don't think they properly estimate the probability under both hypotheses^[1]--but I don't think it's the case that Roko is clearly making procedural errors here. [It seems to me like you're arguing he's making procedural errors instead of just combing to the wrong conclusion / using the wrong numbers, and so I'm focusing on that as the more important point.]

If it's not a lot of evidence

This is what numbers are for. Is "1000-1" a lot? Is it tremendous? Who cares about fuzzy words when the number 1000 is right there. (I happen to think 1000-1 is a lot but is not tremendous.)

1. ^{^}

   For example, the spatial clustering analysis suggests that the first major transmission event was at the market. But does their model explicitly consider both "transfer from animal to many humans at the market" and "transfer from infected lab worker to many humans at the market" and estimate probabilities for both? I don't think so, and I think that means it's not yet in a state where it can be plugged into the full Bayesian analysis. I think you need to multiply the probability that it was from the lab times the first lab-worker superspreader event happening at the market and compare that to the probability that it was from an animal times the first animal-human superspreader event happening at the market, and then you actually have some useful numbers to compare.

"I already tried this and it didn't work."

This post expresses a tremendous amount of certainty, and the mere fact that debate was stifled cannot possibly demonstrate that the stifled side is actually correct.

Agreed on the second half, and disagreed on the first. Looking at the version history, the first version of this post clearly identifies its core claims as Roko's beliefs and as the lab as being the "likely" origin, and those sections seem unchanged to today. I don't think that counts as tremendous certainty. Later, Roko estimates the difference in likelihoods between two hypotheses as being 1000:1, but this is really not a tremendous amount either.

What do you wish he had said instead of what he actually said?

It was terrible, and likely backfired, but that isn't "the crime of the century" being referenced, that would be the millions of dead people. 

As I clarify in a comment elsewhere, I think we should treat them as being roughly equally terrible. If we would execute someone for accidentally killing millions of people, I think we should also execute them for destroying evidence that they accidentally killed millions of people, even if it turns out they didn't do it.

My weak guess is Roko is operating under a similar strategy and not being clear enough on the distinction the two halves of "they likely did it and definitely covered it up". Like, the post title begins with "Brute Force Manufactured Consensus", which he feels strongly about in this case because of the size of the underlying problem, but I think it's also pretty clear he is highly opposed to the methodology.

There are two ways I can read this.

I mean a third way, which is that covering up or destroying evidence of X should have a penalty of roughly the same severity as X. (Like, you shouldn't assume they covered it up, you should require evidence that they covered it up.)

I feel like this is jumping to the conclusion that they're gullible

I think you're pushing my statement further than it goes. Not everyone in a group has to be gullible for the social consensus of the group to be driven by gullibility, and manufactured consensus itself doesn't require gullibility. (My guess is that more people are complicit than gullible, and more people are refusing-to-acknowledge ego-harmful possibilities than clear-mindedly setting out to deceive the public.)

To elaborate on my "courtier's reply" comment, and maybe shine some light on 'gullibility', it seems to me like most religions maintain motive force thru manufactured consensus. I think if someone points that out--"our prior should be that this religion is false and propped up by motivated cognition and dysfunctional epistemic social dynamics"--and someone else replies with "ah, but you haven't engaged with all of the theological work done by thinkers about that religion", I think the second reply does not engage with the question of what our prior should be. I think we should assume religions are false by default, while being open to evidence.

I think similarly the naive case is that lab leak is substantially more likely than zoonosis, but not so overwhelmingly that there couldn't be enough evidence to swing things back in favor of zoonosis. If that was the way the social epistemology had gone--people thought it was the lab, there was a real investigation and the lab was cleared--then I would basically believe the consensus and think the underlying process was valid.

So, from my perspective there are two different issues, one epistemic, and the one game-theoretic.

From the epistemic perspective, I would like to know (as part of a general interest in truth) what the true source of the pandemic was.

From the game-theoretic perspective, I think we have sufficiently convincing evidence that someone attempted to cover up the possibility that they were the source of the pandemic. (I think Roko's post doesn't include as much evidence as it could: he points to the Lancet article but not the part of it that's calling lab leak a conspiracy theory, he doesn't point to the released email discussions, etc.) I think the right strategy is to assume guilt in the presence of a coverup, because then someone who is genuinely uncertain as to whether or not they caused the issue is incentivized to cooperate with investigations instead of obstruct them.

That is, even if further investigation shows that COVID did not originate from WIV, I still think it's a colossal crime to have dismissed the possibility of a lab leak and have fudged the evidence (or, at the very least, conflicted the investigations).

I think it's also pretty obvious that the social consensus is against lab leak not because all the experts have watched the 17 hour rootclaim debate, but because it was manufactured, which makes me pretty unsympathetic to the "researching and addressing counter-arguments" claim; it reminds me of the courtier's reply.

If "they already tried it and it didn't work" they're real into that [Ray interpretation: as an excuse not to try more].

I think I've had this narrative in a bunch of situations. My guess is I have it too much, and it's like fear-of-rejection where it's worth running the numbers and going out on a limb more than people would do by default. But also it really does seem like lots of people overestimate how easy problems are to solve, or how many 'standard' solutions people have tried, or so on. [And I think there's a similar overconfidence thing going on for the advice-giver, which generates some of the resistance.]

It's also not that obvious what the correct update is. Like, if you try a medication for problem X and it fails, it feels like that should decrease your probability that any sort of medication will solve the problem. But this is sort of like the sock drawer problem,^[1] where it's probably easy to overestimate how much to update.

1. ^{^}

   Suppose you have a chest of drawers with six drawers in it, and you think there's a 60% chance the socks are in the chest, and then they're not in the first five drawers you look in. What's the chance they're in the last drawer?

When people found out about ACDC's previous ruling on Brent, many were appalled that ACDC had seen the evidence laid out in the Medium posts and ruled that it was okay for Brent to continue on like that

As I recall, the ACDC had in fact not seen the evidence laid out in the Medium posts. (One of the panelists sent an email saying that they had, but this turned out to be incorrect--there was new information, just not in the section he had read when he sent the email, and prematurely sending that email was viewed as one of the ACDC's big mistakes, in addition to their earlier ruling.)

Another Insanity Wolf meme!

On the one hand, yes, I agree; I thought virology research was crazy back in 2017? when someone at Event Horizon shared a paper which did a cost-benefit analysis and thought the net effect of BSL-4 labs was something like a hundred deaths per year per lab.

But I think it is important to be able to accurately understand what other people think so that you can talk to them instead of past them. (I still remember, with some bitterness, an op-ed exchange where two people debating virology said, roughly, "these things are so dangerous we shouldn't study them" and "these things are so dangerous we have to study them", and that was the end of the discussion, with agreement on the danger and no real ability to estimate the counterfactuals.)

Did we need to know anything but "Covid is an airborne infectious respiratory virus"? How much research prior to the event did it take to know that?

This account of vaccine development claims that having done research on spike proteins back in 2016 was helpful in being able to rapidly develop the vaccine once the genome was uploaded, for example.

[To be clear, I think it's important to distinguish here between gain of function research, which was disliked enough for there to be a funding moratorium (that then expired), and storing / working with dangerous viruses at all, which I think also is below the cost-benefit threshold, but this is a harder case to make.]

The more important point here is that both zoonotic virus jumps and lab leaks are at-large risks that humanity should seek to reduce!

I hear one of the stated reasons for the labs is to study viruses and predict zoonotic jumps. At least some people think we were able to handle COVID so effectively because we were studying viruses in labs and anticipating what might happen, i.e. the net effect of labs is positive.

Given its size, it seems like whether COVID is in the 'pro' or 'con' column does a lot to our sense of whether or not this sort of virology has been good for humans or not and should continue into the future.

I think this is evidence, but weak evidence--it updates me more on "Rootclaim isn't great at debates" than it does on the underlying issue. (Like, how much should William Lane Craig winning his debates update me on theism?)

I think if I started off at 90% confidence of lab leak, Rootclaim losing wouldn't bring me below 80% confidence of lab leak. Plausibly Peter Miller's arguments contain defeaters for my specific beliefs, and going thru the debate would bring me much lower, but I don't yet have that sense from the summaries I've seen.

I am not suggesting that.

Why not? Are you pointing at that the relevant factor is "population within that distance" instead of "distance"?

Maybe I'm confused about the amount of overhead digital signing / verification adds to communication, but do you think that works at missile speeds? (I don't doubt that it works at drone speeds.)

[To be clear, I'm trying to imagine the whole "distant spotter + laser transmission to missile" system, where increasing the length of messages increases the amount of time you need to have successfully targeted the missile in order to successfully transmit a message.]

To elaborate, it's pretty easy to kill someone important if you are willing to be arrested/executed afterwards; the main thing a suicide drone might enable is killing someone important and being able to escape afterwards. This could already be done with dupes, like the 2017 killing of Kim Jong-nam, but I think the nerve agent involved was more expensive than a handmade gun.

Flares can be overcome by a mesh of recon drones somewhat close to the target that can give targeting information to the missile.

This seems overly optimistic to me / is my guess of where the next countermeasure will show up. If your missile is accepting external course-corrections, the enemy can maybe spoof incorrect course-corrections; the more directional the system is, the harder it is to actually hit your fast-moving and course-correcting missile. 

Suppose it had been "What do you call an abortion clinic for pianists?"

fp

Most of those posts are from before the thing I call "constant abuse" began on LessWrong.

I think I remember this timeline differently, or would like you to be a bit more clear on what you mean. I thought of this as an entrenched conflict back in 2019, which was before all the posts used as examples.

Resolution: You Don’t Get To Choose The Problem Factorization. The key here is that it’s the problem space which determines the good factorizations, and we have to go look at the problem space and do some work to figure out what those factorizations are.

This reminds me to ask--have you read Notes on the Synthesis of Form by Christopher Alexander, yet? (I summarized it a year ago but it may be worth you going to the original source.)

"that person, who wants to be treated in the way that people usually treat men"

Incidentally, one of the things I dislike about this framing is that gender stereotypes / scripts "go both ways". That is, it should be not just "treated like a man" but also "treat people like men do."

Is there a way to summarize this shortly? Eliezer disagreed with you about something, or maybe you just interpreted something he wrote as a disagreement with you... and now your soul can't find peace until he admits that he was wrong and you were right about things that are too meta for me to understand wtf you are talking about...

Here's an attempt.

Sometimes people have expectations of each other, like "you won't steal objects from my house".  Those expectations get formed by both explicit and implicit promises. Violating those expectations is often a big deal, not just to the injured party but also to third parties--someone who stole from Alice might well steal from you, too.

To the extent this community encouraged expectations of each other, they were about core epistemic virtues and discussion practices. People will try to ensure their beliefs are consistent with their other beliefs; they won't say things without believing them; they'll share evidence when they can; when they are bound to be uncooperative, they at least explain how and why they'll be uncooperative, and so on. 

[For example, I keep secrets because I think information can be owned, even tho this is cooperative with the information-owner and not with the information-wanter.]

So "Eliezer disagreed with you about something" is an understatement; disagreement is fine, expected even! The thing was that instead of having a regular disagreement in the open, Zack saw Eliezer as breaking a lot of these core expectations, not being open about it or acknowledging it when being called out, and also others not reacting to Eliezer breaking those expectations. (If Eliezer had punched Zack, people would probably have thought that was shocking and criticized it, but this was arguably worse given the centrality of these expectations to Eliezer's prominence and yet people were reacting less.)

That said, the promises were (I think) clearly aspirational / mediated by the pressures of having to actually exist in the world. I do think it makes sense to have a heresy budget, and I think Zack got unlucky with the obsession lottery. I think if people had originally said to Zack "look, we're being greengrocers on your pet issue, sorry about throwing you to the wolves" he would have been sad but moved on; see his commentary on the 2013 disavowal.

Instead they made philosophical arguments that, as far as I can tell, were not correct, and this was crazy-making, because Zack now also doubted his reasoning that led to him disagreeing with them, but no one would talk about this publicly. (Normally if Zack was making a mistake, people could just point to the mistake, and then he could fix the upstream generator of that mistake and everyone could move on.) And, also, to the extent that they generalized their own incorrect justifications to reasoning about other fields, this was making them crazy, in a way that should have alarmed third parties who were depending on their reasoning. The disinterest of those third parties was itself also expectation-violating.

[I don't think I was ever worried about this bleeding over into reasoning about other things; I probably would have joined the conversation more actively if I had? I do regret not asking people what their strategy was back in ~2019; the only people I remember talking to about this were Zack and the LW team.]

Jessica thought my use of "heresy" was conflating factual beliefs with political movements. (There are no intrinsically "right wing" facts.) I agreed that conflating political positions with facts would be bad.

I don't get what 'intrinsically' is doing in the middle sentence. (Well, to the extent that I have guessed what you meant, I disagree.)

Like, yes, there's one underlying reality, descriptions of it get called facts.

But isn't the broader context the propagation of propositions, not the propositions themselves? That is, saying X is also saying "pay attention to X" and if X is something whose increased salience is good for the right-wing, then it makes sense to categorize it as a 'right wing fact', as left-wing partisans will be loathe to share it and right-wing partisans will be eager to.

Like, currently there's an armed conflict going on in Israel and Palestine which is harming many people. Of the people most interested in talking about it that I see on the Internet, I sure see a lot of selectivity in which harms they want to communicate, because their motive for communicating about it is not attempting to reach an unbiased estimate, but to participate in a cultural conflict which they hope their side will win. (One could summarize this view as "speech is intrinsically political.")

This bit of HPMOR comes to mind:

"I don't suppose you could explain," Harry said dryly, "in your capacity as an official of the Hogwarts school system, why catching a golden mosquito is deemed an academic accomplishment worthy of a hundred and fifty House points?"

A smile crossed Severus's lips. "Dear me, and I thought you were supposed to be perceptive. Are you truly so incapable of understanding your classmates, Potter, or do you dislike them too much to try? If Quidditch scores did not count toward the House Cup then none of them would care about House points at all. It would merely be an obscure contest for students like you and Miss Granger."

It was a shockingly good answer.

Steam Wrapped got me thinking about games from 2023, so here are some thoughts/recommendations/anti-recommendations. The theme of this year for me was apparently RPGs made by studios whose RPGs I had played before:

• Baldur's Gate 3: Game of the Year for a reason; took me a bit over a hundred hours on the hardest difficulty setting. (They've since released a harder one.) Doesn't require experience with Dungeons & Dragons, 5th edition specifically, or the previous Baldur's Gate games, tho those enhance the experience. Much more a continuation of Larian's previous RPGs than of the old Baldur's Gate series, which I think is a good thing? Extremely flexible and detailed; you can often be clever and get around things and the game rewards you for it.
  RPGs like these are often made or broken by the quality of the companion NPCs, and I think the crew they have you assemble is a memorable one worth getting to know. Something about playing it felt like it captured the D&D experience (both upsides and downsides) pretty well? Theater kids were involved in the creation of this game, in a good way.
• Legend of Zelda: Tears of the Kingdom: a sequel to their previous open world Zelda game, and IMO the best 'sequel' I've seen? In the sense of, they know you played the first game, and so now it's the same thing, but different.  Set only a few years after the first game, the map is basically the same (with the new features being mostly vertical expansion--there's now a skyworld and an underworld), your horses from the first game are available in the stables, many recognize you as the guy that saved the world recently. The new physics engine is nice, but the overall plot is... simple but neat? Continuing the theme of "the thing you expect (including novelty!), done competently"
• Warhammer 40k: Rogue Trader: a new game, and the first Warhammer 40k CRPG. I'm still going thru this one and so don't have a fully realized take here. Made by the people who made Pathfinder: Kingmaker and Pathfinder: Wrath of the Righteous, both of which have an overworld management map plus standard RPG character progression / combat. In Kingmaker, where you're the baron of a new region carved out of the wilderness, I thought it didn't quite fit together (your kingdom management doesn't really matter compared to the RPG plot); in Wrath of the Righteous, where you're appointed the head of a crusade against the Worldwound, I thought it did (mostly b/c of the crusade battle mechanic, a HoMM-style minigame, tho you could see seams where the two systems joined together imperfectly); in Rogue Trader you're a, well, Rogue Trader, i.e. someone tasked by the God-Emperor of humanity to expand the borders of the Imperium by operating along the frontier, and given significant license in how you choose to do so.  You own a flagship (with thousands of residents, most of whom live in clans of people doing the same job for generations!) and several planets, tho ofc this is using sci-fi logic where each planet is basically a single city. There's also a space-battle minigame to add spice to the overworld exploration.
  I am finding the background politics / worldview / whatever of the game quite interesting; the tactical combat is fine but I'm playing on the "this is my first time playing this game" difficulty setting and thinking I probably should have picked a higher one. The Warhammer 40k universe takes infohazards seriously, including the part where telling people what not to think is itself breaking infosec. So you get an extremely dogmatic and siloed empire, where any sort of change is viewed with suspicion as being treason promoted by the Archenemy (because, to be fair, it sometimes is!). Of course, you've got much more flexibility because you inherited an executive order signed by God that says, basically, you can do what you want, the only sort of self-repair the system allows. (But, of course, the system is going about it in a dumb way--you inherit the executive order, rather than having been picked by God!) The three main 'paths' you can take are being Dogmatic yourself, being an Iconoclast (i.e. humanist), or being a Heretic (i.e. on the side of the Archenemy); I haven't yet seen whether the game sycophantically tells you that you made the right choice whatever you pick / the consequences are immaterial or not.
• Starfield: Bethesda's first new RPG setting in a while. It was... fine? Not very good? I didn't really get hooked by any of the companions (my favorite Starfield companion was less compelling than my least favorite BG3 companion), the whole universe was like 3 towns plus a bunch of procedurally generated 'empty' space, the outpost building was not well-integrated with the rest of the game's systems (it was an upgrade over Fallout 4's outpost-building in some ways but not others), and the central conceit of the plot was, IMO, self-defeating. Spoilers later, since they don't fit well in bulleted lists.
• Darkest Dungeon II: ok this isn't really an RPG and so doesn't belong on this list, but mentioning it anyway. IMO disappointing compared to Darkest Dungeon. I'm not quite sure what I liked less well, but after 16 hours I decided I would rather play Darkest Dungeon (which I put 160 hours into) and so set it down.

The promised Starfield spoilers:

I wrote this post on and off over the course of a morning, and towards the end of it realized:

I'm reading you as saying "eating others is inherently not ok" but I would like it to be ok or not contingent on some other facts (like the absence of suffering, or hypothetical net preference, or the ability of people to not have their souls corrupted by carnivorism, or so on) and the generalization of that reasoning to not have terrible consequences elsewhere. (For example, if you think pleasure can't outweigh suffering, then it seems like having kids at all is indefensible, which is a self-extinguishing moral position; if you think something that taken seriously implies it's not even ok to eat plants, then that's even more self-extinguishing.)

I'll still post the rest of the comment I wrote, which responds to you in more detail, but that seems like the most important piece.

relationships where you have to do the complicated emotional gymnastics of saying that you love an animal like their your friend one day and then chopping their head from their body the next and savoring the flavor of the flesh on the grill.

There's a tumblr post where someone talks about immediately feeling the shepherd impulse when interacting with sheep, a bunch of people like the post, someone points out "how many of you eat lamb", and then the original poster responds with "The ancient shepherds I’m referencing also ate lamb lol"

My sense is that there's a few ways to take this. One of them is "actually the emotional gymnastics is not that complicated!", and another is "actually those ancient shepherds also probably abused their wives and thought slavery was fine when it happened to someone else and mistreated their animals, according to our standards; parents caring about their children / guardians caring about their wards is really not sufficient to guarantee good outcomes or license those relationships." I infer your position is closer to the latter but it really feels like it should be possible to have gains from trade, here.

[And, like, one of the downsides of specialization is that it drives people both unusually interested and unusually disinterested in animal welfare into the 'works with animals' business, which is probably how we got into this factory farming mess in the first place.]

My last stab at a response might be to bring up an analogy to slavery. I take the equivalent of your position here to be "look, if each slave can look at the potential life he will hold and prefer that life to no life at all, then isn't that better than him not existing at all?" And to me it seems like I'd be again called to say "no".

I think one of the main ways my libertarian leanings show up is by being okay with people being able to pick worse things that are cheaper. Let people live in tiny houses and work low-paying jobs and sell their bodies and take high-interest loans if that's the right tradeoff for them; removing their options generally isn't helping them.

I think that could extend all the way to slavery, altho it's hard to imagine situations where that actually makes sense. In general, I think children have only a little bit of debt to their parents (certainly not a lifetime of labor and ownership of all their descendants), which is the closest analogy. Probably more realistic is something like conservatorship, where someone is deemed incompetent to handle their financial or medical affairs, and someone else makes those decisions for them; should people be allowed to voluntarily enter a conservatorship?

A fictional version of this shows up in a video game called The Outer Worlds, where a star system is colonized by a group of corporations, where the colonists are a mixture of 'people who put up the capital for the voyage' and 'people agreeing to come as indentured servants', which leads to a very stratified society on the other side, which predictably starts to decay as the colonists have children with huge differences in inherited wealth. Even if Alice decided it was worth being a laborer somewhere new rather than being stuck on Earth, her daughter Carol might not feel like she's bought into this situation and want to violently redistribute things, and it's not obvious that Alice should be able to sell Carol's compliance with society.

But you could imagine that, if Alice can't bind Carol, the colony doesn't go thru, and Carol never comes to exist, and on net Carol is sad about that outcome, and would have preferred having been bound. It feels like an actually thorny question to figure out what tradeoffs precisely make sense, especially because this is a collective bargaining issue (it's not like existing societies get unanimous consent from their participants!) and the empirical tradeoffs are all hypothetical. [My actual expectation is that we get material abundance before we get any interstellar colonies, and so it's not important to get this question right because it'll never come up.]

That is the sort of world I hope for. 

To be clear, this is a world without cats and snakes and other obligate carnivores, right? Or is the plan to first figure out synthetic sources of the various nutrients they need?

[It will also have many fewer other animals--I think on average something like a third of a cow is alive because of my beef consumption--but depending on what you think the limiting factors are, that may mean replacement with fractional vegan humans instead, which is probably an upgrade.]

To explain: Alfred Koryzbski, the guy behind General Semantics, is basically "rationality from 100 years ago". (He lived 1879-1950.) He's ~2 generations before Feynman (1918-1988), who was ~one before Sagan (1934-1996), then there's a 2-3 generation gap to Yudkowsky (1979-). (Of course if you add more names to the list, the gaps disappear; reordering your list, you get James Randi (1928-2020), Dawkins (1941-), Hitchens (1949-2011), Michael Shermer (1954-), and Sam Harris (1967-) which takes you from Feynman to Yudkowsky, basically.)

He features in Rationalism before the Sequences, and is interesting both because 1) you can directly read his stuff, like Science and Sanity, and 2) most of his stuff has already made it to you indirectly, from the student's students. (Yudkowsky apparently wrote the Sequences before reading any Korzybski directly, but read lots of stuff written by people who read Korzybski.)

There are, of course, figures before Korzybski, but I think the gaps get larger / it becomes less obviously "rationalism" instead of something closer to "science". 

To have this sort of love, this commitment to preventing suffering, with animals to me means pretty much just drawing the line at sentient beings and trying to cultivate a basic sense that they matter and that "it's just bad" to eat them.

I feel like the dialogue circled this for a while, and I want to try to poke directly at it again. I think my line is something like "try to only make trades that the other party 'would' consent to," which means eating high-welfare meat if it seems likely that the animals net prefer being raised to be eaten that way to not existing, tho ofc we have to use our judgment instead of theirs. [This article convinced me to avoid chicken products, for example.] It seems to me like you don't accept this, like in this section here:

When I first started to change my diet, I was most appalled at factory farming, and remember that first call with my parents afterwards, trying to soothe their worries by saying "no, no, this doesn't apply to our animals, I'll keep chowing down on Williams family steaks don't you worry. I'll spare you the details, but long story short I came into the thinking of the above slowly and found myself eventually unable to eat even our products, because now it wasn't a steak on my plate, it was a piece of a cow, maybe even a cow like the one I bottlefed for a year growing up and then sold for slaughter, a good cow named Max. 

Like, I think in the picture where you're only willing to eat Max because of a sense that Max was grateful to have been alive at all, this works. Max might have even more preferred to be a pet, but that's not on offer to all hypothetical Maxes. It's only if there's a clear separation between the friends and food category that this doesn't work, and I see how having that category is consistent but it's not obvious to me that it's the right end goal. (I think many historical people have viewed animals as sacred / people / etc. and also as food.)

[Like, you talk about "what if climate change is solved and all the enviro-vegans disappear?", but this feels to me like that worry is somehow broken; like, what if all the factory farms disappear, and on net all farmed animals are grateful to exist? Then it seems like Mission Accomplished to me, even tho I imagine you will still want to be vegan in that world.]

The story that you tell afterwards is mostly about the standards for checking being too high, but I am not sure how practical that is. You bring up the hypothetical of opposing child slavery coal mines, but I think this is actually a problem for cacao production, and so I try to be about as selective in my chocolate sourcing as I am in my steak sourcing--with the understanding that the ethics of "fair trade" or "grass finished" includes some amount of fraud and errors.

ctrl-f korz

hmmm

This how you see it?

Roughly.

Nukes have x-risk but humans couldn't help but build them

I think no one seriously considered the prospect of nuclear winter until well after stockpiles were large, and even now it's not obviously an existential concern instead of merely catastrophic. If you're talking about the 'ignite the atmosphere' concern, I think that's actually evidence for voluntary relinquishment--they came up with a number where if they thought the risk was that high, they would give up on the project and take the risk of Nazi victory. 

I expect the consensus estimate will be that AGI projects have risks in excess of that decision criterion, and that will motivate a halt until the risks are credibly lowered.

What if all the other powers at that time went to the Pope and asked for a bull that firing grapeshot wasn't Christian.  Would this change anything?

I assume you're familiar with Innocent II's prohibition on crossbows, and that it wasn't effectively enforced. I am more interested in, say, the American/Israeli prohibition on Iranian nuclear weapons, which does seem to be effectively enforced on Earth. 

The bottlenecks are in the chip fabrication tooling.

Yeah, I think it is more likely that we get compute restrictions / compute surveillance than restrictions on just AI developers. But even then, I think there aren't that many people involved in AI development and it is within the capacities of intelligence agencies to surveil them (tho I am not confident that a "just watch them all the time" plan works out; you need to be able to anticipate the outcomes of the research work they're doing, which requires technical competence that I don't expect those agencies to have).

Like, you shouldn't work yourself ragged, but my guess is for most people, working on something meaningful (or at least difficult) is actually more fun and rewarding compared to the alternative of doing nothing or hedonism or whatever, even if you ultimately fail. (And on the off-chance you succeed, things can be a lot more fun.)

I think one of the potential cruxes here is how many of the necessary things are fun or difficult in the right way. Like, sure, it sounds neat to work at a geothermal startup and solve problems, and that could plausibly be better than playing video games. But, does lobbying for permitting reform sound fun to you?

The secret of video games is that all of the difficulty is, in some deep sense, optional, and so can be selected to be interesting. ("What is drama, but life with the dull bits cut out?") The thing that enlivens the dull bits of life is the bigger meaning, and it seems to me like the superstructure is what makes the bigger meaning more real and less hallucinatory.

those successes could snowball in lots of different directions pretty quickly, without much meta-level direction.

This seems possible to me, but I think most of the big successes that I've seen have looked more like there's some amount of meta-level direction. Like, I think Elon Musk's projects make more sense if your frame is "someone is deliberately trying to go to Mars and fill out the prerequisites for getting there". Lots of historical eras have people doing some sort of meta-level direction like this.

But also we might just remember the meta-level direction that was 'surfing the wave' instead of pushing the ocean, and many grand plans have failed.

Vaniver is it your belief that a worldwide AI pause - not one limited to a specific geographic area - is a plausible outcome? Could you care to elaborate in more detail why you think it would be possible?

Yes, I think it's plausible. I don't think it's especially likely--my modal scenario still involves everyone dying--but I think especially if you condition on success it seems pretty likely, and it makes sense to play to your outs.

The basic argument for plausibility is that 1) people are mostly awake to risks from advanced AI, 2) current power structures are mostly not enamored with AI / view it as more likely to be destabilizing than stabilizing, 3) the people pushing for unregulated AI development are not particularly charismatic or sympathetic, and 4) current great powers are pretty willing to meddle in other countries when it comes to serious national security issues.

I expect pauses to look more like "significant regulatory apparatus" than "ban"; the sort of thing where building new nuclear plants was legal with approval and yet it takes decades to get NRC approval. Probably this involves a significant change in how chips are constructed and sold. [I note that computer hardware seems like an area where people are pouring gas onto the race instead of trying to slow down.]

I think as the race heats up and AI becomes more and more promising, we might see national total efforts to develop AI faster.

I think this might happen, and is >98% likely to be game over for humanity.

I don't understand why we want to do this.

I want Alice to have help choosing what things to do and not do, in the form of easily understandable prices that turn uncertain badness ("it'll probably be fine, I probably won't break the camera") into certain costs ("hmm, am I really going to get $70 worth of value from using this camera?").

I am most interested in this in contexts where self-insurance is not reasonable to expect. Like, if some satellite company / government agency causes Kessler Syndrome, they're not going to be able to pay back the rest of the Earth on their own, and so there's some temptation to just ignore that outcome; "we'll be bankrupt anyway." But society as a whole very much does not want them to ignore that outcome; society wants avoiding that outcome to be more important to them than the survival of their company, and something like apocalypse insurance seems like the right way to go about that.

But how do you price the apocalypse insurance? You don't want to just kick the can down the road, where now the insurance company is trying to look good to regulators while being cheap enough for customers to get business, reasoning "well, we'll be bankrupt anyway" about the catastrophe happening.

You mention the "unilateralist's curse", but this sounds more like the "auction winner's curse",

I think those are very similar concepts, to the point of often being the same.

which I would expect an insurer to already be taking into account when setting their prices (as that's the insurer's entire core competency).

I probably should have brought up the inexploitability concept from Inadequate Equilbria; I'm arguing that mistaken premiums are inexploitable, because Carol can't make any money from correcting Bob's mistaken belief about Alice, and I want a mechanism to make it exploitable.

Normally insurers just learn from bad bets after the fact and this is basically fine, from society's point of view; when we're insuring catastrophic risks (and using insurance premiums to determine whether or not to embark on those risks) I think it's worth trying to make the market exploitable.

If you buy $1 of synthetic risk for $0.05, does that mean you get $1.00 if Alice breaks the camera, and $0.00 if Alice does not?

Yes, the synthetic risk paying out is always conditional. The sketch I have for that example is Bob has to offer $10 of synthetic risk at each percentage point, except I did the math as tho it were continuous, which you can also do by just choosing midpoints. So there's $10 for sale at $0.55, another $10 for $0.65, and so on; Carol's $40 for $2.80 comes from buying $0.55+0.65+0.75+0.85 (and she doesn't buy the $0.95 one because it looks like a 5 cent loss to her). That is, your tentative guess looks right to me.

The $910 that goes unsold is still held by Bob, so if the camera is wrecked Bob has to pay themselves $910, which doesn't matter. 

As you point out, Bob pays $1.25 for the first $50 of risk, which ends up being a wash. Does that just break the whole scheme, since Bob could just buy all the required synthetic risk and replicate the two-party insurance market? Well... maybe. Maybe you need a tiny sales tax, or something, but I think Bob is incentivized to participate in the market. Why did we need to require it, then? I don't have a good answer there. (Maybe it's easier to have mandatory prediction markets than just legalizing them.)

the problem I most care about

I want markets for x-risks, basically. Suppose someone wants to train an AI and they're pretty sure it'll be fine, but only pretty sure. How do we aggregate estimates to figure out whether or not the AI should be trained? Seems like we should be able to have a risk market. [so8res proposes apocalypse insurance here with "premiums dependent on their behavior", but what's the dependence? Is it just set by the regulator?]

But the standard problems appear; on the "we all die" risk, the market doesn't exist and so people who bet on risk never get paid out.

You could imagine instead using a cap-and-trade system, where, say, only 3 AIs can be trained per year and companies bid for one of the permits, but it seems like this is still tilted towards "who thinks they can make the most money from success?" and not "who does the group think is least likely to fail?". You could have instead an explicit veto/permission system, where maybe you have 11 votes on whether or not to go thru with an AI training run and you need to buy at least 6 'yes'es, but this doesn't transfer resources from worse predictors to better predictors, just from projects that look worse to projects that look better.

And so I think we end up with, as johnswentworth suggests over here, needing to use regular liability / regular insurance, where people are betting on short-term questions  that we expect to resolve ("how will this lawsuit by rightsholders against generative AI companies go?") instead of the unresolvable questions that are most existentially relevant.

"Building an AI that doesn't game your specifications" is the actual "alignment question" we should be doing research on.

Ok, it sounds to me like you're saying:

"When you train ML systems, they game your specifications because the training dynamics are too dumb to infer what you actually want. We just need One Weird Trick to get the training dynamics to Do What You Mean Not What You Say, and then it will all work out, and there's not a demon that will create another obstacle given that you surmounted this one."

That is, training processes are not neutral; there's the bad training processes that we have now (or had before the recent positive developments) and eventually will be good training processes that create aligned-by-default systems.

Is this roughly right, or am I misunderstanding you?

If you created a misaligned AI, then it would be "thinking back", and you'd be in an adversarial position where security mindset is appropriate.

Cool, we agree on this point.

my point in that section is that the fundamental laws governing how AI training processes work are not "thinking back". They're not adversaries.

I think we agree here on the local point but disagree on its significance to the broader argument. [I'm not sure how much we agree-I think of training dynamics as 'neutral', but also I think of them as searching over program-space in order to find a program that performs well on a (loss function, training set) pair, and so you need to be reasoning about search. But I think we agree the training dynamics are not trying to trick you / be adversarial and instead are straightforwardly 'trying' to make Number Go Down.]

In my picture, we have the neutral training dynamics paired with the (loss function, training set) which creates the AI system, and whether the resulting AI system is adversarial or not depends mostly on the choice of (loss function, training set). It seems to me that we probably have a disagreement about how much of the space of (loss function, training set) leads to misaligned vs. aligned AI (if it hits 'AI' at all), where I think aligned AI is a narrow target to hit that most loss functions will miss, and hitting that narrow target requires security mindset.

To explain further, it's not that the (loss function, training set) is thinking back at you on its own; it's that the AI that's created by training is thinking back at you. So before you decide to optimize X you need to check whether or not you actually want something that's optimizing X, or if you need to optimize for Y instead.

So from my perspective it seems like you need security mindset in order to pick the right inputs to ML training to avoid getting misaligned models.

Thanks!

Congrats on another successful program!

Mentors rated their enthusiasm for their scholars to continue with their research at 7/10 or greater for 94% of scholars.

What is it at 9/10 or greater? My understanding is that 7/10 and 8/10 are generally viewed as 'neutral' scores, and this is more like "6% of scholars failed" than it is "94% of scholars succeeded." (It looks like averages of roughly 8 are generally viewed as 'high' in this postmortem so this population might be tougher raters than in other contexts, and so I'm wrong on what counts as 'neutral'.)

I think something like 2015?