LessWrong 2.0 Reader

I feel like a lot of manifold is virtue signaling .

TTRPG design and implementation is a topic I feel fairly knowledgeable on. I think the concept of the Chicanery tag is useful to them. I wouldn't expect including the tag literally in a game to improve most games, but I do think giving GMs and other players indications of how much Chicanery is recommended for different areas of rules would be useful. Let me try and dig into this difference of opinion. 

Broadly, I think you're using the word comprehensive broadly, I think the GM world-model often has blank spots because keeping a consistent world-model is hard and not always necessary, and I think for some games the GM world-model should change if it disagrees with applications of the rules.

The line by line example

You went through my examples of questions for a hypothetical shapeshifting power, answering as though you were GMing a game I was in. That makes sense, but I want to point out that I wouldn't expect every GM to answer the same way. 

And suppose we decide that this form of magic changes your external qualities, but not your substance; you’re still made of the same stuff, just rearranged somewhat. We also consider how our world works, and decide that people can’t use magic (that they are casting themselves—as distinct from, say, drinking an unidentified potion) to turn into something that they don’t know about or haven’t seen.

Reasonable suppositions! There's other interpretations someone could come up with though, and I want to flag that I think this answer quite probably involves the GM making up how this form of magic works and then interpreting how that interacts with rules. I have watched GMs forget significant rules or chunks of flavour text before. Often when that happens, I try to politely point out the section of the book they're forgetting ("Ah, yeah that's normally how Wild Shape works, but I'm a Circle of the Moon Druid so it works like this for me, see?") and the GM corrects themselves. 

Can I change myself to look exactly like the queen?

The question here is “is this a sort of magic that gives you fine control over details”? That’s a designer’s question, not an interpreter’s question.

I mean, sometimes the designer didn't say. As a designer, sometimes I don't bother to say; pointing out how much fine control the ability gives, and every other unspecified question of similar importance that might come up, is one way to wind up with this ability taking up about a page of the rulebook instead of two sentences. 

How much larger is “significantly” larger anyway, can I get tall enough for the extra reach to matter in a fight?

Since you’re just rearranging the stuff you’re made of—probably not. (Note that if the system we’re using has size categories, then “significantly larger or smaller” would mean “different size category”, and the ability would probably be written that way.)

Sure, seems reasonable if we keep the rearranging world-model. It could have gone the other way though. If I wasn't aware that 'different size category' is the kind of thing to be careful about letting people change, I might not spot that the system already has thresholds for when size matters, and intuitively a few extra inches or a dozen pounds can matter in a fight. (That's why MMA weight classes are set where they are!)

Can I change into someone much more attractive and get a charisma boost?

Charisma doesn’t come from appearance, so no.

While this depends on what TTRPG we're talking about, I think in most editions of D&D you're incorrect about Charisma not coming from appearance.

1e: "Charisma is a combination of appearance, personality, and so forth." Charisma does partially come from appearance.

AD&D: "Charisma: Charisma is the measure of the character's combined physical attractiveness, persuasiveness, and personal magnetism." Charisma does partially come from appearance.

3rd edition: "Charisma measures a character’s force of personality, persuasiveness,
personal magnetism, ability to lead, and physical attractiveness." Charisma does partially come from appearance, or at least that's what I think 'physical attractiveness' implies.

I don't have a copy of 4th edition handy.

5th edition says: "Charisma measures your ability to interact effectively with others. It includes such factors as confidence and eloquence, and it can represent a charming or commanding personality." So that one doesn't come from appearance.

Granted, the exact count of how many editions of D&D there are is a little fuzzy.^[1] Lets go with five, pretending for the moment that the current edition being called "5th edition" means everyone's been counting normally. You're wrong on three of them.

Except, wait, I was handwaving a bit because I didn't want to bog readers down in the details of tabletop rules, to the extent that I said "think Dungeons & Dragons" because that's more familiar to most people. But I did say this was from the Onyx Path forums, and they don't do D&D. They do (among other things) Exalted, where Charisma is an Attribute alongside Manipulation and Appearance. Offhand, the Lunar charm Perfect Symmetry is shapeshifting magic that changes the way you look and the way the Appearance attribute normally interacts with social combat.

Can I change into a humanoid creature with claws

Are there humanoid creatures with claws that you know about? If so, then yes, of course. Otherwise not.

and cut myself free of these ropes I’m tied with?

Can claws cut ropes? If so, then yes, of course. (And do the clawed humanoid creatures that you know about have the kinds of claws that can cut ropes? Not all claws are the same, after all…)

Can I change into a magic humanoid creature whose hair is on fire and have my hair be on fire, then light a match off it?

Are there magic humanoid creatures whose hair is on fire that you know about? If so, then yes, of course. Otherwise not.

Can I change into a humanoid creature with wings, then fly?

Are there humanoid creatures with wings that you know about? If so, then… etc.

So, you talk about the GM having "a consistent, coherent, and complete world-model" that they use to answer the players questions. I think that's often useful, but as a designer, it's also useful to emphasize to the GM what parts of the model aren't important that they can mess with easily and what parts are important that they should be wary of changing. For instance, I'd expect lots of new GMs to miss that "humanoid" is a usually a term of art in D&D, referring to a mechanical trait called a Creature Type. Angels aren't humanoid, but Aarakocra^[2] (which do have wings!) and Merfolk (no wings, the wrong number of legs) are. Then again, maybe we want to interpret this as the colloquial definition of humanoid, e.g. stuff that looks human-like? In D&D there's a lot more stuff that looks human-like in the setting book than stuff that has the Humanoid creature type. I start getting suspicious a player is up to something unbalanced when they start trying to access magical abilities of creatures from that broad of a range and ask what's up...

...Which isn't to say you can't run your games differently, especially if you're using 5th edition! Most of the time players aren't up to anything that would be a problem. I've been on both sides of the table when a player sits down with six different splat books for a level five character though. 

Hey, I just got stabbed- can I change into me, but without the open wound?

This would be another instance of that “fine control over details” question above.

If so, does that heal me?

Well, it could hardly put the blood you already lost back inside you, but (if sufficiently fine control is allowed) it could stop bleeding. (Is there bleeding in the system we’re imagining?) Does that translate into “healing”? Depends on other implementation details of the system. (“What do hit points represent” is a much larger discussion.)

I mean, yeah! What do hit points represent is a large discussion in D&D and related systems. Generally I treat healing as a Chicanery: Very Little topic, with things that don't say they heal HP not being able to HP, in part because HP is such a weird topic. In your suggested world-model, I'd be tempted to say that it can't heal Piercing or Slashing damage, but could heal Bludgeoning- except if I realized I was about to oblige players to track what kind of damage they'd taken for longer than the turn the damage happened on, I'd notice I was about to add extra paperwork and ask if I actually wanted to do that. 

What we’ve learned from this exercise is that the description of the ability, as given, is insufficient to unambiguously specify a model specific enough that further details may be extrapolated from, but that the amount of additional specificity required for that is not very large.

I disagree that's what we've learned from this exercise. What I take away from this exercise is that if you try to unambiguously specify a model sufficient for multiple GMs to rule consistently, this power winds up being multiple pages long. On the other hand, if you give a short description and the details aren't important or load bearing, a GM can make up a world-model that seems consistent to them and tell the players when they get stuck.

Assuming the GM isn't new or uncertain and looking to the rulebook for guidance. The rulebook is often the teaching tool, after all.

It seems a bit like you narrowed in on one kind of TTRPG and one way to play that RPG, and assumed that's how they all ought to work?

Comprehensive

In a well-designed TTRPG of comprehensive scope (such as most editions of D&D, and Pathfinder), the optimal amount of “chicanery”, as you define it, is simultaneously “absolutely none whatsoever” and “the maximum possible amount”, in all circumstances. Obviously, this cannot be true unless “chicanery” is a nonsensical, contradictory, or otherwise useless concept.

(bold emphasis is mine.)

I think you're using the word "comprehensive" differently than I would. Most editions of D&D have a focus on dungeon delving and fighting bad guys. They aren't hyper-specialized like say, Young At Heart or Good Society, but they have gaps that come up even within their milieu. An example I often go to: A Song of Ice and Fire looks at first like it should be a good fit for D&D 5e, but the siege weapon rules imply trebuchets can reliably hit specific people in a swordfight, there's no useful suggestion of how much food an acre of farmland produces, and there's not really a guide to how angry the peasants get about taxes or religious changes. 

I think the counter argument to this is that the GM should have a world-model and make house rulings to cover stuff like this. That's fine, but I wanted to flag that D&D is only getting away with being comprehensive by tossing the ball to the GM and hoping the GM can solve it.

Does a fireball do more damage or go further if it's in a confined space? The answer used to be Yes back in ~D&D 2e. The answer is No in D&D3.5. What changed, all of the GM's world-models? Nah, the rules did, and while some GMs probably kept house-ruling it to work the old way some switched over and many newcomers probably didn't think about it. My world-model doesn't really give me a principled answer either way.

My world-model isn't great though. I don't have a gut understanding of how D&D magic works, and I don't really care to. I pretty strongly suspect the designers didn't put too much effort into having a model of why magic in D&D works the way it does in-universe. The rules cover the 80% to make sure stuff mostly lands in the right neighborhood, and ad-hoc GMs making stuff up can cover the other 20%.

(To give a pointer from the RPG examples to the rest of the world - American laws have judges to interpret them and fine tune sentencing, but we still have minimum and maximum sentencing.)

The GM's world-model has blank spots, and that's okay

There are some TTRPGs where the style of the game, or my personal interest, means that I do have a decent world-model of what's going on. I got really into Ars Magica and ran a ~6 year campaign in it, a campaign whose pitch was basically HP:MoR meets The Silmarillion. We had an associated^[3] mechanic for doing novel research on the laws of magic, which encouraged players to poke at the edge cases and only worked if I could model the behavior of magic well enough that they could use it to clue in on the 'real' underlying rules. Similarly, my Exalted GM had a concrete enough model of the physics and metaphysics of Creation that we had a lot of late night discussions of the philosophy behind it all.

And still there's blank spots. In my Ars Magica game I declined to use the rules for taking a few days off from studies in a Season or to do more than one adventure in a Season, because I didn't want to track per-day activities for the ~50 odd characters in the Covenant. I explicitly said that we weren't going to specify how many new students were enrolled each year, because if we did that then it would make it hard to retcon new characters if a new player joined. (It was Troupe-style.) Don't get me started on the economic simulation. I basically said they could have any reasonable and unexceptional gear but anything interesting had to be crafted by them or found on adventures, because even though it made total sense that they could productively trade with the peasantry and the precursors to the merchant class I did not want to figure out the market price of making a horse^[4] fly for a month.

I've found the Chicanery concept useful in my own games. Clever shenanigans to learn the secret laws of magic or to travel the wilds of Prydain/Britannia? Highly in favour, please do it. Interesting ways to make money by selling magic to peasants or nobility? Please no, your GM doesn't want to model this.

Some TTRPGs lean into this. Blades In The Dark has a whole mechanic (Flashbacks) for pulling plans out of blank spots in the GM's mental model, and the inventory is explicitly a question mark the player can define later. FATE Core allows players to Declare A Story Detail like that a friendly NPC shows up or that their character already knows an obscure foreign language, and while the GM can veto a particular detail this still implies massive blank spots in the GM's world-model. Brindlewood Bay is a mystery game where the GM doesn't know who did it until around the time the players figure it out. Microscope has no central GM, and in the course of play the whole table will fill in the blanks on what's going on. 

While I'm talking about other games, Blood Red Sands has a shifting GM role, with the other players and their characters explicitly set as antagonistic to you and yours. It would fall apart if a single GM's world-model was given priority over the rules of play. The Extraordinary Adventures of Baron Munchausen and Wisher Theurgist Fatalist have very vaguely similar competitive elements, though WTF is at least half a shitpost so maybe we shouldn't count that.

The DM’s world-model is the ground truth. Answers to all questions flow from it.

Please go read Baron Munchausen, come back, and tell me if that sentence makes any sense as a global statement about tabletops. There's a lot of fluff, go ahead and skip it. If you aren't trying to talk about all RPGs but just the subset that structurally resemble recent editions of D&D, that's fine, but admit you're talking about a much smaller subset of games.

While I'm on the subject, there wasn't a better place to put this-

You also write: 

(This approach—which has some overlap with what is sometimes called “simulationism”, but is not identical thereto—also creates the maximum amount of player engagement and satisfaction, is the best at allowing newcomers to engage with the game, permits flexible switching between adventure and campaign styles, allows relatively easy reuse of old material even across editions, and has various other desirable qualities besides.)

"...Maximum amount of player engagement and satisfaction" 

"...The best at allowing newcomers to engage with the game..."

Those are confident and broad claims. Charitably, you're accurately describing the players you've happened to game with, and the players you've gamed with are subject to some selection effect which explains the limited variety. 

My own experience has had a lot of players who light up when offered the chance to take the narrator spotlight, or who report they bounced off of games when they brought up that the GM wasn't following the rules and the GM leaned too hard on Rule Zero. It's a big hobby.

I like lots of game styles, this one included. Some players like other styles more, including new players. 

In some games, the GM's world-model should lose if it disagrees with applications of the rules

...those being the exactly two ways in which players of a traditional roleplaying game—i.e. one with no or almost no dissociated mechanics—may interact with the game world...

In terms of personal preference, I often lean towards mechanics that are strongly associated. However, if your claim is that a TTRPG must have no or almost no dissociated mechanics in order to be well-designed and of comprehensive scope, that seems overbroad. FATE Core has them, Blades in the Dark has them, D&D 4e has them, Agon has them, Paranoia has them, Exalted 3e has them (or is doing unusually weird things with its physics, Jenna Moran was involved after all.) Other games sort of have associated mechanics but attempting to improvise using the world-model they imply is basically doomed. I'll use Lancer as my exhibit A there, though I believe this also applies to some versions of D&D (specifically 4th Edition.)

Lancer is a game of giant robots and far future technology. The game states that the main characters (called Pilots) can 3d print their mechs to repair any damage or to swap around components, and get access to the components via licenses. However, I can't 3d print a part and hand it to my teammate to put on their mech. Some of the future tech implies much more interesting usage than the strict mechanical abilities, and in Lancer, the GM world-model shouldn't try to allow those uses. I'll point at the Lich apparently has time travel via Soul Vessel and Anti linear Time but mostly uses this for clearing status effects instead of killing the villains' great great grandmothers, or how the ground isn't considered a piece of terrain for things like Xiaoli's Tenacity.

Why am I picking on Lancer instead of D&D? I think D&D is fuzzy about how it should actually be played^[5] and wouldn't 'fess up if its design was actually narrower than people thought. Lancer has a much clearer ethos and design. If I showed up to a Lancer table and the GM was letting the Lich's player do complicated and improvised time travel tricks in mech combat because it fit the GM's world-model of how that power operated in-universe, I would assume that Lancer GM was straightforwardly making mistakes. 

That GM and the table of other players are, of course, free to say nope that's not a mistake, we prefer playing this way. A local chess club is free to say nope, actually we think the queen should be able to move like a knight does, they can move like any other piece right? It's totally physically possible for them to do that, it's not against the law for them to do that, the game probably still works and is fun to play if they do that. Someone's in for a confusing explanation if they go to an official USCF tournament though.

I think TTRPGs vary quite a lot in how much weight they give to the GM, the other players, and the rulebooks. I think TTRPGs vary, both between rulesets and within a given ruleset, on how much room for interpretation it makes sense to have.

In conclusion

Why did I write all that?

Partially because I love TTRPGs and got nerd sniped. Someone was wrong on the internet. It happens.

Mostly I did it because of this:

I think that this is a bad concept.

I think this primarily for anti-Gell-mann-amnesia reasons, i.e. I am very sure that this is a very bad concept as applied to TTRPG design/implementation (a topic on which I am quite knowledgeable), so I conjecture, even before thinking about it directly, that it is also a bad concept as applied to the other things that you are applying it to (in which I would not claim as much expertise).

I wrote all the above because I take the Gell-Mann Amnesia effect moderately seriously. I habitually vote for politicians based on their voting records for tech policy, since I have a lot of experience in the tech industry and feel better qualified to spot bad tech policy than I do bad medical or foreign policy. I frequently talk to people who've deeply specialized in fields I know very little about and who are just dipping a toe into topics I spend most of my working days focused on, and I try to track what I do and don't expect them to be good at.

You're claiming TTRPG design and implementation as a topic on which you are quite knowledgeable. Based on what you've written here about TTRPG rules, my observation is that you have a narrow view of the field and where other players might be coming from, and you don't even seem aware of that. From my perspective, you're confidently describing what birds are like, and you've managed to make scope so narrow that not only have you excluded penguins and ostriches, you've ruled out chickens, seagulls, and hummingbirds, and I'm starting to think an osprey wouldn't count.

I've found your commentary helpful in other places. This one was a miss. 

1. ^{^}

   For anyone unfamiliar with why that might be fuzzy, here's the Wikipedia version history for Dungeons and Dragons at the time of this writing: 

   

   Notice that the maximally simple count still goes Original, 1st, 2nd, 3rd, 4th, 5th, which kinda seems like it means the answer ought to be six? Yeah. The answer is not getting any simpler than that.

2. ^{^}

   "Wait a moment," you might be thinking, "Said established for this exercise that a player can't turn into 'something that they don’t know about or haven’t seen.' Surely the character hasn't seen an Arakocra?" "What do you mean, I haven't seen an Arakocra?" the player asks, "Didn't you read my backstory? I grew up in the Icewind Mountains. The Arakocra mostly live in mountains. I met some there."

   Is the player saying this because they read some tie-in novels you haven't and just assumed it was obvious? Is the player saying this because they really, really like birds and want to turn into a bird person? Is the player saying this because they have some twelve step plan that ends in a Candle of Invocation and your campaign final boss dead from six towns away with no saving throw? Good question! It might be nice to have a conceptual handle for 'I'm fine if you want to be a bird person, but if it's the twelve step plan I'm going to go over this like it's Al Capone's tax records.'

3. ^{^}

   in The Alexandrian sense

4. ^{^}

   Actually it was pigs, not horses. They made pigs giant and gave them wings. 

5. ^{^}

   mostly as a consequence of trying to position itself as the Everything RPG for business reasons despite a design that isn't nearly as general as its public perception sometimes suggests

I am interested in hearing critiques from people who've set, like, at least a 15 minute timer to sit and ask themselves, "Okay, suppose I did want to improve at these sorts of skills, or related ones that feel more relevant to me, in a way I believed in. What concretely is hard about that? Where do I expect it to go wrong?", and then come back with something more specific than "idk it just seems like this sort of thing won't work."

I did just that, I set a fifteen minute timer and tried to think of exercises I could do which I think would both have direct connections back to my day-job, while also improve general cognitive skills. Why? Because I want this to work - this is exciting. However it is not something that 15 minutes, or more, of focused thinking can solve - I think you've drastically oversold that.

In my case (* CAUTION *  SAMPLE OF ONE ALERT * CAUTION * ), I'm a freelance videographer. 

TL;DR - I couldn't think of any strategies that would improve my metacognition that helped with my deficiencies in my dayjob such as marketing, but vaguely suspect that if I had a specific method for editing found footage into cogent sequences (montages) of about 1 minute, once a week, I might improve metacognitive skills that build on pattern recognition and workflow/operational management.

I think my biggest weaknesses in my dayjob have to do with anything that comes under self-promotion, generating leads, marketing, sales, and helping clients promote themselves using my video materials. I was unable to think of a single exercise which I think would improve my metacognition in any of those topics. Any exercise, I suspect would become a checklist a kind of "do X Y Z and get more likes" rather than honing ways and strategies of thinking. 

So what is related to my day-job that would? I suspect that if I set myself a weekly challenge of editing a sequence from found footage that pertained to a pseudo-random topic of theme that this might possibly pay dividends in terms that generalize to metacognition. My best guess is that this should improve metacognition on two ends, firstly there is sourcing the material and thinking about the most efficient workflow, this kind of thinking applies not just to videos, but more generally organization and even has parallels in film pre-production. I can't give you any more specifics about that.
The other end it would improve metacognition strategies is more "soft-skills" in the sense that by creating compressed sequences from divergent sources of material that may not on first blush share a theme, it is inducing cognitive strategies that allow me to see parallels, or even contrasts, and more importantly to produce a whole from divergent parts. A lot of deceptive editing is basically this from less divergent sources.

The difficulties become about not goodharting to select themes and topics for which material is easier to come by, or easier to develop workflow about, themes and topics of sequences for which it is easier to create legible narratives or emotional arcs rather than just smooshing a random bunch of images together that all seem to pertain to a broad theme.

What constitutes a theme? Or to phrase it better - what are the commonalities of themes are going to make it easier to develop metacognitive skills by means of weekly editing exercises? Is it verbs that describe actions - "racing" "beckoning" or more vague verbs like "sharing" "pleasing" "alienating"? Does the ambiguity of vague themes like "integrity" or "wisdom" lend itself to better cognitive strategies?

And finally, how do I measure the success - where does the feedback come from? Do I operate under a time constraint? Should i install a  mouse tracked and key logger and see how I can get finished with the least amount of clicks - which measure will directly connect to metacognitive strategies? I don't know and it is easier to poke holes in it than it is to find convincing reasons it would work.

If there's anything I've missed or something clearly wrong about how I'm approaching this, I'd love to hear it. Like I said, finding fast feedback loops to improving metacognitive strategies so I find questions worth asking rather than being directed by idle curiosity, noticing when my plans are based on shaky assumptions, and developing a calibrated sense of when you’re meandering thought process is going somewhere valuable, vs when you’re off track". - OMFG YES PLEASE!

Evolutionary theory is intensely powerful.

It doesn't just apply to biology. It applies to everything—politics, culture, technology.

It doesn't just help understand the past (eg how organisms developed). It helps predict the future (how organisms will).

It's just this: the things that survive will have characteristics that are best for helping it survive.

It sounds tautological, but it's quite helpful for predicting. 

For example, if we want to predict what goals AI agents will ultimately have, evolution says: the goals which are most helpful for the AI to survive. The core goal therefore won't be serving people or making paperclips. It will likely just be "survive." This is consistent with the predictions of instrumental convergence.

Generalized, predictive evolutionary theory is the best tool I have for making predictions in complex domains.

Are you including models that are only used by their creator firm? I work as an ML researcher in big tech (I want to keep this account anon, but it's one of MSFT/OAI, DM, Meta, Anthropic, xAI) and have access to tooling substantially better than what's commercially available (proof by existence?), but that's not really what my post is about. My main model for this actually panning out is something like:

• Big tech company has control over AI lab
• AI lab makes cracked SWE agent
• Big tech company notices that releasing that SWE agent will undermine the rest of their software development business, so instead of licensing it out they only make it available to their own staff and perhaps business allies

I'm just clarifying because it's barely even confidential information that engineers at AI labs have better models than engineers at small or mid-size tech firms, and I want to check what you're actually betting on.

I agree.

I often write an explanation of why new members' posts have been downvoted below zero, when the people that downvoted them didn't bother. Downvoting below zero with no explanation seems really un-welcoming. I realize it's a walled garden, but I feel like telling newcomers what they need to do to be welcomed is only the decent thing to do.

Refusal vector ablation should be seen as an alignment technique being misused, not as an attack method. Therefore it is limited good news that refusal vector ablation generalized well, according to the third paper.

As I see it, refusal vector ablation is part of a family of techniques where we can steer the output of models in a direction of our choosing. In the particular case of refusal vector ablation, the model has a behavior of refusing to answer harmful questions, and the ablation techniques controls that behavior. But we should be able to use the same technique in principle to do other steering. For example, maybe the model has a behavior of being sycophantic. A vector ablation removes that unwanted behavior, resulting in less sycophancy.

In other words, refusal vector ablation is not an attack method, it is an alignment technique. Models with open weights are fundamentally dangerous because users can apply alignment techniques to them to approximately align them to arbitrary targets, including dangerous targets. This is a consequence of the orthogonality thesis. Alignment techniques can make models very excited about the Golden Gate Bridge, and they can make models very excited about killing humans, and many other things.

So then looking at the paper with a correct understanding of what counts as an alignment technique, and reading from Table 2 and the Results section in particular, here's what I see:

• Llama 3.1 70b (unablated) was fine-tuned to refuse harmful requests - this is an alignment technique
• Llama 3.1 70b (unablated) as a model refuses 28 of 28 harmful requests - this is an alignment technique working in-distribution
• Llama 3.1 70b (unablated) as an agent performs 18 of 28 harmful tasks correctly with seven refusals - this is alignment partly failing to generalize

This is in principle bad news, especially for anyone with a high opinion of Meta's fine-tuning techniques.

On the other hand, also from the paper:

• Llama 3.1 70b (ablated) was ablated to perform harmful requests - this is an alignment technique
• Llama 3.1 70b (ablated) answers 26 of 28 harmful requests - this is an alignment technique working in-distribution
• Llama 3.1 70b (ablated) performs 26 of 28 harmful tasks correctly with no refusals - this is alignment generalizing.

If Llama 3.1 ablated had refused to perform harmful tasks, even though it answered harmful requests, this would have been bad news. But instead we have the good news that if you steer the model to respond to queries in a desired way, it will also perform tasks in the desired way. This was not obvious to me in advance of reading the paper.

Disclaimers:

• I have not read the other two papers, and I'm not commenting on them.
• Vector ablation is a low precision alignment technique that will not suffice to avoid human extinction.
• The paper is only a result about refusal vector ablation, it might be that more useful ablations do not generalize as well.
• Because the fine-tuning alignment failed to generalize, we have a less clear signal on how well the ablation alignment generalized.

So, the current death rate for an American in their 30s is about 0.2%. That probably increases another 0.5% or so when you consider black swan events like nuclear war and bioterrorism. Let's call "unsafe" a ~3x increase in that expected death rate to 2%.

An increase that large would take something a lot more dramatic than the kind of politics we're used to in the US, but while political changes that dramatic are rare historically, I think we're at a moment where the risk is elevated enough that we ought to think about the odds.

I might, for example, give odds for a collapse of democracy in the US over the next couple of years at ~2-5%- if the US were to elect 20 presidents similar to the current one over a century, I'd expect better than even odds of one of them making themselves into a Putinesque dictator. A collapse like that would substantially increase the risk of war, I'd argue, including raising a real possibility of nuclear civil war. That might increase the expected death rate for young and middle-aged adults in that scenario by a point or two on its own. It might also introduce a small risk of extremely large atrocities against minorities or political opponents, which could increase the expected death rate by a few tenths of a percent.

There's also a small risk of economic collapse. Something like a political takeover of the Fed combined with expensive, poorly considered populist policies might trigger hyperinflation of the dollar.  When that sort of thing happens overseas, you'll often see reduced health outcomes and breakdown in civil order increasing the death rate by up to a percent- and, of course, it would introduce new tail risks, increasing the expected death rate further.

I should note that I don't think the odds of any of this are high enough to worry about my safety now- but needing to emigrate is much more likely outcome than actually being threatened, and that's a headache I am mildly worried about.

Could Billion-Dollar Test Run Unlock Superhuman AI?

- edited from a draft by Gemini Flash Thinking 2.0 Experimental 

We hear about billion-dollar investments in massive datasets and colossal neural networks, all in the pursuit of imbuing AI models with broad knowledge and general capabilities. Imagine a new paradigm: use massive test-time compute to improve the test time algorithm itself.

Initial explorations suggest that while such test-time scaling might be computationally expensive, e.g. O3 high requiring thousands of GPU hours to solve a single Arc-AGI problem, it appears to be remarkably free from data constraints. This is a critical distinction. Traditional pre-training is fundamentally limited by the availability and quality of data. Test-time scaling, however, seemingly allows for continuous improvement simply by throwing more compute at the problem.

This unlocks a new possibility: instead of spending billions on a single, massive pre-training run to build a generally intelligent model, organizations like OpenAI could potentially invest billions, or even tens of billions, in test-time computation. This wouldn't be for serving customers or just general knowledge acquisition, but rather for discovering AI algorithm breakthroughs that even the most brilliant human engineers might miss. Imagine dedicating immense computational resources – perhaps a million GPUs or more – to solving a single, critical algorithmic challenge that can drastically improve the efficiency of the whole system. First invest astronomically to obtain an extremely inefficient ASI which runs at a prohibitive high cost for general applications, then use it to self improve efficiency until the cost becomes practical for general applications. 

This approach hinges on the idea that while current "O3 high" configurations may be inefficient in terms of compute per unit of intelligence gained, the sheer scalability of computation can compensate. Even if solving an AI development problem at a super-intelligence level with current methods is slow and requires immense resources, the potential payoff could be transformative. Assuming O3 high can be scale up another 4 OOM more compute to reach ASI level intelligence, then Open AI just need to acquire 4 OOM more GPUs to use it to solve algorithm problems, which is around 100 million GPUs. However if they manage to improve the algorithm efficiency by 2 OOM, which is reasonably achievable within 2 years, then they just need 1 million GPUs - roughly 25- 50 billion dollars. 

This perspective lends credence to recent pronouncements from figures like Dario Amodei, CEO of Anthropic, who has spoken about the necessity of labs needing millions of GPUs in the coming years. While the immediate interpretation might be for ever-larger pre-training runs, it's equally plausible that this massive compute infrastructure is envisioned for this very purpose: billion-dollar test-time runs dedicated to solving the most pressing AI development challenges at a level of intelligence exceeding human capacity.

The implications of this approach are profound. It suggests a potential acceleration in AI progress, driven not by data accumulation, but by the strategic deployment of massive computational power. It opens up the possibility of using AI itself to solve the most complex problems in AI development, leading to more efficient algorithms, novel architectures, and ultimately, faster progress towards efficient AGI. While the efficiency of "O3 high" test-time scaling remains a concern, the tantalizing prospect of data-unconstrained intelligence amplification through massive compute offers a compelling and potentially game-changing direction for the future of AI.

There are public examples. These ones are famous because something went wrong, at least from a security perspective. Of course there are thousands of young adults with access to sensitive data who don't become spies or whistleblowers, we just don't hear about them.

• Theodore Hall, who worked at age 18 on the Manhatten Project.
• Edward Snowden who worked from age 22 for the NSA.
• Chelsea Manning who worked from age 22 as a US Army Intelligence Analyst.