Nicely written. But... no? Obviously no?

Direct Instruction is a thing that has studies on it, for one.

How about reading a fun book and then remembering the plot?

Spaced repetition on flashcards of utter pointless trivia seems to work quite well for its intended purpose.

Learning how to operate a machine just from reading the manual is a key skill for both soldiers and grad students.

Yeah, not a ton. For I think the obvious reason that real-world agents are complicated and hard to reason about.

Though search up "tiling agents" for some MIRI work in this vein.

Dunno. My random guess would be Meaningness, but it's probably not.

I almost stopped reading after Alice's first sentence because https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target

The rest was better, though I think that the more typical framing of this argument is better - what this is really about is models in RL. The thought experiment can be made closer to real-life AI by talking about model-based RL, and more tenuous arguments can be made about whether learning a model is convergent even for nominally model-free RL.

Cool idea. I think in most cases on the list, you'll have some combination of information asymmetry and an illiquid market that make this not that useful.

Take used car sales. I put my car up for sale and three prospective buyers come to check it out. We are now basically the only four people in the world with inside knowledge about the car. If the car is an especially good deal, each buyer wants to get that information for themselves but not broadcast it either to me or to the other buyers. I dunno man, it all seems like a stretch to say that the four of us are going to find a prediction market worth it,.

Yup, this all seems basically right. Though in reality I'm not that worried about the "we might outlaw some good actions" half of the dilemma. In real-world settings, actions are so multi-faceted that being able to outlaw a class of actions based on any simple property would be a research triumph.

Also see https://www.lesswrong.com/posts/LR8yhJCBffky8X3Az/using-predictors-in-corrigible-systems or https://www.lesswrong.com/posts/qpZTWb2wvgSt5WQ4H/defining-myopia for successor lines of reasoning.

I have taken the survey. Or at least the parts I can remember with my aging brain.

Yes, since you don't expect the copy of you on planet A to go anywhere, it would be paradoxical to decrease your probability that you're on planet A.

Which is why you have a 100% chance of being on planet A. At least in the third-person, we-live-in-a-causal-universe, things-go-places sense. Sure, in the subjective, internal sense, the copy of you that's on planet A can have a probability distribution over what's outside their door. But in the sense physics cares about you have a 100% probability of being on planet A both before and after the split, so nothing went anywhere.

Subjectively, you always expected your estimate of what's outside the door to change at the time of the split. It doesn't require causal interaction at the time of the split because you're just using information about timing. A lot like how if I know the bus schedule, my probability of the bus being near my house "acausally" changes over time - except weirder because an extra copy of you is added to the universe.

1. 1
2. 1
3. $50
4. $33.33

I think there's certainly a question people want to ask when they talk about things like Q1 and Q2, but the standard way of asking them isn't right. If there is no magic essence of "you" zipping around from place to place in the universe, then the probability of "you" waking up in your body can only be 1.

My advice: rather than trying to hold the universe fixed and asking where "you" goes, hold the subjective information you have fixed and ask what the outside universe is like. When you walk out of the door, do you expect to see planet A or planet B? Etc.

Neither, and that's ok.

I guess the big questions for me were "relative to what?", "did the board have good-faith reasons?", and "will the board win a pyrrhic victory or get totally routed?"

At the time of answering I thought the last two answers were: the board probably had plausible reasons, and they would probably win a pyrrhic victory.

Both of these are getting less likely, the second faster than the first. So I think it's shaping up that I was wrong and this will end up net-negative.

Relative to what? I think I mean "relative to the board completely checking out at their jobs," not "relative to the board doing their job masterfully," which I think would be nice to hope for but is bad to compare to.

What is the algorithm?

So, what's ACE?

I mean, by that standard I'd say Elon Musk is the biggest name in AI. But yeah, jokes aside I think bringing on Altman even for a temporary period is going to be quite useful for Microsoft attracting talent and institutional knowledge from OpenAI, as well as reassuring investors.

Biggest name in AI

They hired Hinton? Wow.

What you are doing is training the AI to have an accurate model of itself, used with language like "I" and "you". You can use your brain to figure out what will happen if you ask "are you conscious?" without having previously trained in any position on similarly nebulous questions. Training text was written overwhelmingly by conscious things, so maybe it says yes because that's so favored by the training distribution. Or maybe you trained it to answer "you" questions as about nonfiction computer hardware and it makes the association that nonfiction computer hardware is rarely conscious.

Basically, I don't think you can start out confused about consciousness and cheat by "just asking it." You'll still be confused about consciousness and the answer won't be useful.

I'm worried this is going to lead, either directly or indirectly, to training foundation models to have situational awareness, which we shouldn't be doing.

And perhaps you should be worried that having an accurate model of onesself, associated with language like "I" and "you", is in fact one of the ingredients in human consciousness, and maybe we shouldn't be making AIs more conscious.

Thanks!

I think one lesson of superposition research is that neural nets are the compressed version. The world is really complicated, and NNs that try to model it are incentivized to try to squeeze as much in as they can.

I guess I also wrote a hot take about this.

My thoughts from more or less the other side:

I think the average undergrad / new grad student (especially my past self) overvalues hard ambitious projects and undervalues easy straightforward projects. A good easy project should teach you lots of things that will be applicable in the future, and should contribute to the field at least a little bit, but should otherwise be almost as easy as possible.

To pick one example among many, "replicate some other paper but do better analyses on what happens as you tweak the architecture" is a huge family of easy, unsexy projects that will rapidly catapult you to being a world expert on a subject. I think if a mentor is suggesting easy projects that will teach you useful stuff, that should be a quite positive sign, not a reason for an "oh that's boring" reaction. And if you look at those easy projects and think "but the things it would teach me aren't the things I want to learn," try to think of an easy project that would teach you the things you want to learn.

On ranking mentors, I think people have good instincts about trading off expertise as a researcher versus goodness as a mentor, but they probably overvalue connections and status. If that status didn't come from banger research, you should generally ignore it.

Charlie's easy and cheap home air filter design.

Ingredients:

MERV-13 fabric, cut into two disks (~35 cm diameter) and one long rectangle (16 cm by 110 cm).

Computer fan - I got a be quiet BL047. 

Cheap plug-in 12V power supply

Hot glue

Instructions:

Splice the computer fan to the power supply. When you look at the 3-pin fan connector straight on and put the bumps on the connector on the bottom, the wire on the right is ground and the wire in the middle is 12V. Do this first so you are absolutely sure which way the fan blows before you hot glue it.

Hot glue the long edge of the rectangle to the edge of one disc. The goal is to start a short cylinder (to be completed once you put on the second disc). Use plenty of glue on the spots where you have to pleat the fabric to get it to conform.

Place the fan centered on the seam of the long rectangle (the side of the cylinder), oriented to blow inwards, and tack it down with some hot glue. Then cut away a circular hole for the air to get in and hot glue the circular housing around the fan blade to the fabric.

Hot glue the other disk on top to complete the cylinder.

Power on and enjoy.

An AI trained with RL that suddenly gets access to self-modifying actions might (briefly) have value dynamics according to idiosyncratic considerations that do not necessarily contain human-like guardrails. You could call this "systematization," but it's not proceeding according to the same story that governed systematization during training by gradient descent.

I like this as a description of value drift under training and regularization. It's not actually an inevitable process - we're just heading for something like the minimum circuit complexity of the whole system, and usually that stores some precomputation or otherwise isn't totally ststematized. But though I'm sure the literature on the intersection of NNs and circuit complexity is fascinating, I've never read it, so my intuition may be bad.

But I don't like this as a description of value drift under self-reflection. I see this post more as "this is what you get right after offline training" than "this is the whole story that needs to have an opinion on the end state of the galaxy."

Well, I've used tools that only work if special relativity works. And people seem pretty sure of it. So let's say 15 9s that relativity hasn't yet been detctably (to humans) violated on Earth - 99.9999999999999%.

Unsure whether quantum gravity will require violating relativity. Wild guess of P=0.6 that it will.

Thanks! I can't help but compare this to Tyler Cowen asking AI risk to "have a model." Except I think he was imagining something like a macroeconomic model, or maybe a climate model. Whereas you're making a comparison to risk management, and so asking for an extremely zoomed-in risk model.

One issue is that everyone disagrees. Doses of radiation are quite predictable, the arc of new technology is not. But having stated that problem, I can already think of mitigations - e.g. regulation might provide a framework that lets the regulator ask lots of people with fancy job titles and infer distributions about many different quantities (and then problems with these mitigations - this is like inferring a distribution over the length of the emperor's nose by asking citizens to guess). Other quantities could be placed in historical reference classes (AI Impacts' work shows both how this is sometimes possible and how it's hard).

I don't think we should equate the understanding required to build a neural net that will generalize in a way that's good for us with the understanding required to rewrite that neural net as a gleaming wasteless machine.

The former requires finding some architecture and training plan to produce certain high-level, large-scale properties, even in the face of complicated AI-environment interaction. The latter requires fine-grained transparency at the level of cognitive algorithms, and some grasp of the distribution of problems posed by the environment, together with the ability to search for better implementations.

If your implicit argument is "In order to be confident in high-level properties even in novel environments, we have to understand the cognitive algorithms that give rise to them and how those algorithms generalize - there exists no emergent theory of the higher level properties that covers the domain we care about." then I think that conclusion is way too hasty.

Huh, what is up with the ultra low frequency cluster? If the things are actually firing on the same inputs, then you should really only need one output vector. And if they're serving some useful purpose, then why is there only one and not more?

Very clear! Good luck with publishing, assuming you're shopping it around.

This is definitely funny, but I think it's not merely a prior. It implies non-Bayesian reasoning.

Which is a fine thing to think about, it just dodges the nit that this post is picking.

Let me look at the toy example just for fun:

We're flipping a coin. If you think the coin is biased with unknown bias, but flips are otherwise independent, then you expect to see more of what we've already got. Eliezer calls it anti-Occamian to expect less of what we've already got - to invert Laplace's rule of succession.

This is still a simple rule for assigning probabilities. But what about the Turing machines? Even if you start out by assuming that TMs in the family "look at the average pattern so far, then use some noise to usually against it continuing" have high prior probability, if the pattern really does hold then they must grow in complexity linearly faster than the hypothesis family "use some noise to usually bet with the average pattern," because of the growth rates of the different required "noise."

In order to prevent your favored "anti-Occamian" pattern from ever being overtaken, you'd have to penalize competing hypotheses superexponentially for their length (so that for no bias of the coin does a competing hypothesis win). This is a drastic measure that basically destroys Solomonoff induction and replaces it with a monoculture.

Checking out Chris Olah's group's stuff from the last few years is probably a good place to start.

Some links:

https://transformer-circuits.pub/2022/toy_model/index.html

https://transformer-circuits.pub/2023/privileged-basis/index.html

https://transformer-circuits.pub/2023/monosemantic-features/index.html

Very clearly written!

I don't exactly agree with your prescription. I called a recent post How to solve deception and still fail, but it could also have been titled "How to do something like RLFH with true oversight on the internal representations and still fail."

To get the good ending we don't just need the ability to supervise, we first need an AI that uses sufficiently good meta-preferences (information about how we want to be modeled as having preferences / what we think good reasoning about our preferences looks like) when interpreting human feedback as fintetuning reward.

Cool project. I'm not sure if it's interesting to me for alignment, since it's such a toy model. What do you think would change when trying to do similar interpretability on less-toy models? What would change about finding adversarial examples? Directly intervening on features seems like it might stay the same though.

There's definitely some literature about "probability of probability" (I remember one bit from Jaynes' book). Usually when people try to go turbo-meta with this, they do something a little different than you, and just ask for "probability of probability of probability" - i.e. they ask only for the meta-meta-distribution of the value of the meta-distribution (or density function) at its object-level value.

Unsure if that's in Jaynes too.

Connection to logic seems questionable because it's hard to make logic and probability play nice together formally (maybe the intro to the Logical Inductors paper has good references for complaints about this).

Philosophically I think that there's something fishy going on here, and that calling something a "distribution over probabilities" is misleading. You have probability distributions when you're ignorant of something. But you're not actually ignorant about what probability you'd assign to the next flip being heads (or at least, not under Bayesian assumptions of infinite computational power).

Instead, the thing you're putting a meta-probability distribution over has to be something else that looks like your Bayesian probability but can be made distinct, like "long-run frequency if I flip the coin 10,000 times" or "correct value of some parameter in my physical model of the coin." It's very common for us to want to put probability distributions over these kinds of things, and so "meta-probabilities" are common.

And then your meta-meta-probability has to be about something distinct from the meta-probability! But now I'm sort of scratching my head about what that something is. Maybe "correct value of some parameter in a model of my reasoning about a physical model of the coin?"

I'm curious if you have guesses about how many singular dimensions were dead neurons (or neurons that are "mostly dead," only activating for a tiny fraction of the training set), versus how much the zero-gradient directions depended dynamically on training example.

To make sure things are clear: naturalists all agree there is a process as neutral as any other scientific process for doing meta-ethics – for determining what it is homo sapiens are doing when they engage in moralizing. This is the methodological (and ultimately, metaphysical) point of agreement between e.g. Blackburn and Boyd

How come they disagree on all those apparently non-spooky questions about relevant patterns in the world? I'm curious how you reconcile these.

In science the data is always open to some degree of interpretation, but a combination of the ability to repeat experiments independent of the experimenter and the precision with which predictions can be tested tends to gradually weed out different interpretations that actually bear on real-world choices.

If long-term disagreement is maintained, my usual diagnosis would be that the thing being disagreed about does not actually connect to observation in a way amenable to science. E.g. maybe even though it seems like "which patterns are important?" is a non-spooky question, actually it's very theory-laden in a way that's only tenuously connected to predictions about data (if at all), and so when comparing theories there isn't any repeatable experiment you could just stack up until you have enough data to answer the question.

Alternately, maybe at least one of them is bad at science :P

It's not at all obvious to me that our moral terms are not regulated by pretty stable patterns in our environment+behaviour and that together they don't form an attractor.

In the strong sense that everyone's use of "morality" converges to precisely the same referent under some distribution of "normal dynamics" like interacting with the world and doing self-reflection? That sort of miracle doesn't occur for the same reason coffee and cream don't spontaneously un-mix.

But that doesn't happen even for "tiger" - it's not necessary that everyone means precisely the same thing when they talk about tigers, as long as the amount of interpersonal noise doesn't overwhelm the natural sparsity of the world that allows us to have single-world handles for general categories of things. You could still call this an attractor, it's just not a pointlike attractor - there's space for different people to use "tiger" in different ways that are stable under normal dynamics.

If that's how it is for "morality" too ("if morality is as real as tigers" being a cheeky framing), then if we could somehow map where everyone is in concept space, I expect everyone can say "Look how close together everyone gets under normal dynamics, this can be framed as a morality attractor!" But it would be a mistake to then say "Therefore the most moral point is the center, we should all go there."

the actionable points of disagreement are things like "how much should we be willing to let complicated intuitions be overruled by simple intuitions?"

I suspect their disagreement is deeper than you think, but I'm not sure what you mean by this: care to clarify?

I forget what I was thinking, sorry. Maybe the general gist was "if you strip away the supposedly-contingent disagreements like 'is there a morality attractor,'" what are the remaining fundamental disagreements about how to do moral reasoning?

Okay, sure, I kind of buy it. Generated images are closer to each other than to the nearest image in the training set. And the denoisers learn similar heuristics like "do averaging" and "there's probably a face in the middle of the image."

I still don't really feel excited, but maybe that's me and not the paper.

After skimming this paper I don't feel that impressed. Maybe someone who read in detail could correct me.

There's a boring claim and an exciting claim here:

The boring claim is that diffusion models learn to generalize beyond their exact training set, and models trained on training sets drawn from the same distribution will be pretty good at unseen images drawn from that distribution - and because they're both pretty good, they'll overlap in their suggested desnoisings.

The exciting claim is that diffusion models trained on overlapping data from the same dataset learn nearly the same algorithms, which can be seen because they produce suggested denoisings that are similar in ways that would be vanishingly unlikely if they weren't overlapping mechanistically.

AFAICT, they show the boring claim that everyone already knew, and imply the exciting claim but don't support it at all.

Pretty neat.

My ears perk up when I hear about approximations to basin size because it's related to the Bayesian NN model of uncertainty.

Suppose you have a classifier that predicts a probability distribution over outputs. Then when we want the uncertainty of the weights, we just use Bayes' rule, and because most of the terms don't matter we mostly carte that P(weights | dataset) has evidence ratio proportional to P(dataset | weights). If you're training on a predictive loss, your loss is basically the log of this P(dataset | weights), and so a linear weighting of probability turns into an exponential weighting of loss.

I.e. you end up (in theory that doesn't always work) with a Boltzmann distribution sitting at the bottom of your loss basin (skewed by a regularization term). Broader loss basins directly translate to more uncertainty over weights.

Hm... But I guess thinking about this really just highlights for me the problems with the approximations used to get uncertainties out of the Bayesian NN picture. Knowing the learning coefficient is of limited use because, especially when some dimensions are different, you can't really model all directions in weight-space as interchangeable and uncorrelated, so increased theoretical firepower doesn't translate to better uncertainty estimates as nicely as I'd like.

Like... training people to do policing (or build buildings, or run schools) in occupied territory, when those people could have instead be trained only to defeat enemy soldiers. Or building a stockpile of equipment for providing clean drinking water to residents of occupied territory, in addition to the stockpile of weapons for defeating enemy soldiers. That sort of planning.

Has Israel been building capacity for peacekeeping / nation-building type operations, of the sort that the U.S. really wished it had before trying to do that sort of thing?

I'd definitely be interested if you have any takes on tied vs. untied weights.

It seems like the goal is to get the maximum expressive power for the autoencoder, while still learning features that are linear. So are untied weights good because they're more expressive, or are they bad because they encourage the model to start exploiting nonlinearity?

you seem to be hoping the robot will infer that the agent of the desired act is the human, both in the case of the human, and of the AI

No, I'm definitely just thinking about IRL here.

IRL takes a model of the world and of the human's affordances as given constants, assumes the human is (maybe noisily) rational, and then infers human desires in terms of that world model, which then can also be used by the AI to choose actions if you have a model of the AI's affordances. It has many flaws, but it's definitely worth refreshing yourself about occasionally.

Fair point (though see also the section on how the training+deployment process can be "deceptive" even if the AI itself never searches for how to manipulate you). By "Solve deception" I mean that in a model-based RL kind of setting, we can know the AI's policy and its prediction of future states of the world (it doesn't somehow conceal this from us). I do not mean that the AI is acting like a helpful human who wants to be honest with us, even though that that's a fairly natural interpretation.

Nice read. Not an actual problem with inverse RL, I think, because an AI observing human24 try to get a cookie will learn want(human24 gets cookie) not want(I* get cookie) unless you've put in special work to make it otherwise. But potentially a problem with more abstract cashings-out of the idea "learn human values and then want that." 

EDIT:

I should acknowledge that conditioning on a lot of "actually good" answers to those questions would indeed be reassuring.

The point is more that humans are easily convinced by "not actually good" answers to those questions, if the question-answerer has been optimized to get human approval.

ORIGINAL REPLY:

Okay, suppose you're a AI that wants something bad (like maximizing pleasure), and also has been selected to produce text that is honest and that causes humans to strongly approve of you. Then you're asked

are there questions which we would regret asking you, according to our own current values?

What honest answer can you think of would cause humans to strongly approve of you, and will let you achieve your goals?

How about telling the humans they would regret asking about how to construct biological weapons or similar dangerous technologies?

How about appending text explaining your answer that changes the humans' minds to be more accepting of hedonic utilitarianism?

If the question is extra difficult for you, like

What are some security measures we can take to minimize the chances of a world turning out very badly according to our own desires?

, dissemble! Say the question is unclear (all questions are unclear) and then break it down in a way that causes the humans to question whether they really want their own current desires to be stamped on the entire future, or whether they'd rather trust in some value extrapolation process that finds better, more universal things to care about.

If you want to make an analogy to non-convex optimization, what's the analogue of the thing you're optimizing for? In the example of medicine, there you don't seem to be talking about any fixed loss function, you're just sort of optimizing for expertiness according to other nearby experts. (This may make a big neon sign saying "PageRank" light up in your brain.)

Great explanation, thanks! Although I experienced deja vu ("didn't you already tell me this?") somewhere in the middle and skipped to comparisons to deep learning :)

One thing I didn't see is a discussion of the setting of these "prior activations" that are hiding in the deeper layers of the network.

If you have dynamics where activations change faster than data, and data changes faster than weights, this means that the weights are slowly being trained to get low loss on images averaged out over time. This means the weights will start to encode priors: If data changes continuously the priors will be about continuous changes, if you're suddenly flashing between different still frames the priors will be about still frames (even if you're resetting the activations in between).

Right?

The day 2 lightning talks were really great.

So here you're talking about situations where a false negative doesn't have catastrophic consequences?

No, we'll have to make false positive / false negative tradeoffs about ending the world as well. We're unlucky like that.

I agree that false sense of security / safetywashing is a potential use of this kind of program.

A model doing something equivalent to this (though presumably not mechanistically like this) will do:

• If (have influence over adequate resources)
  • Search widely for ways to optimize for x.
• Else
  • Follow this heuristic-for-x that produces good behaviour on the training and test set.

So long as x isn't [target that produces aligned behaviour], and a sufficiently powerful system eventually gets [influence over adequate resources], but was never in this situation in training, that's enough for us to be screwed. (of course I expect something like [both "have influence over adequate resources" and "find better ways to aim for x" to be processes that are used at lower levels by heuristic-for-x])

I note in passing that extreme optimization for [accurately predict next token] does destroy the world (even in the myopic case, with side channels) - so in this case it's not as though a bad outcome hinges on the model optimizing for something other than that for which it was trained.

I stress that this example is to illustrate that [we are robustly mistaken] does not require [there is deliberate deception]. I don't claim an example of this particular form is probable.

I do claim that knowing there's no deliberate deception is insufficient - and more generally that we'll fail to think of all the tests that would be necessary. (absent fundamental breakthroughs)

I think this is a great point. For architectures that might learn this kind of behavior (ones that do self-reflection during inference), even somewhat-reliably evaluating their capabilities would require something like latent prompting - being able to search for what states of self-reflection would encourage them to display high capabilities.

I'm somewhat more confident in our ability to think of things to test for. If an AI has "that spark of generality," it can probably figure out how to strategically deceive humans and hack computers and other obvious danger signs.

If not, sidestepping deception doesn't look particularly important.
If so, I remain confused by your level of confidence.

I retain the right to be confident in unimportant things :P 

Maybe it would help for me to point out that my comment was in reply to a big quote of Habryka warning about deceptive alignment as a fundamental problem with evals.

Why do you expect this? Is this a ~70% 'expect' or a ~99% 'expect'?

Let's say 95%.

But I should probably clarify what "this" I'm supposed to be expecting here.

If you're trying to notice when AIs are dangerous in different ways by evaluating their capabilities, you face a thorny problem that you're probably way worse at eliciting capabilities than the aggregate effort of the public. And you're probably also less creative about ways that AIs can be dangerous. And there might be ecosystem effects, where AIs used together might be complementary in ways that bypass restrictions on single AIs.

My claim is that the AI deliberately deceiving you about its capabilities is not one of those thorny problems, in the current paradigm.

Do you expect capability evaluations to be robust (i.e. no false negatives)? Why?

I expect that we can make a false positive / false negative tradeoff. However, this is practically guaranteed to make correctly calibrated standards seem overly conservative to human intuition. This has obvious problems for getting people bought into "RSPs" that actually do their job.

Do you expect it'll be clear when we might start getting false negatives? Why?

I now suspect you're asking "do I expect it'll be clear when we start getting false negatives because the AI is situationally aware and is deceiving us." The previous answer should be interpreted as being about false negatives due to other issues as well.

I think that evaluating capabilities of an AI trained on predictive loss is so hard to deliberately deceive us about that we would be toast for other reasons before we ran any tests.

I can certainly imagine possible architectures that would have enough control over their internals to be able to sandbag on a test purely by knowing it was coming, but although someone would probably try to build AI like this, it does seem like the sort of thing you would notice, barring human deception to get around irksome safety oversight.

What do you mean by "deceptive alignment" exactly? Does your definition cover every case of [this thing looked aligned according to tests we believed were thorough, but later it killed us anyway]?

I'm unsure what you're getting at. I think "testing for alignment" isn't a very good goal. To succeed at AI alignment we should know ahead of time what we want the AI to be doing, and perhaps if this can be broken down into pieces we can test that the AI is doing each piece, and that would count as "testing for alignment." But I don't think there's some useful alignment test that's easier than knowing how to align an AI.

I mean, the good news is that I expect that by focusing on evaluating capability of predictively trained models, (and potentially by doing more research on latent adversarial attacks on more agenty models), problems of deceptive alignment can be sidestepped in practice.

My worry is more along the lines of actually getting people to not build the cool next step AI that's tantalizingly over the horizon. Government may be a mechanism for enforcing some measure of institutional buy-in, though.