evhub's Shortform

COI: I work at Anthropic and I ran this by Anthropic before posting, but all views are exclusively my own.

I got a question about Anthropic's partnership with Palantir using Claude for U.S. government intelligence analysis and whether I support it and think it's reasonable, so I figured I would just write a shortform here with my thoughts. First, I can say that Anthropic has been extremely forthright about this internally, and it didn't come as a surprise to me at all. Second, my personal take would be that I think it's actually good that Anthropic is doing this. If you take catastrophic risks from AI seriously, the U.S. government is an extremely important actor to engage with, and trying to just block the U.S. government out of using AI is not a viable strategy. I do think there are some lines that you'd want to think about very carefully before considering crossing, but using Claude for intelligence analysis seems definitely fine to me. Ezra Klein has a great article on "The Problem With Everything-Bagel Liberalism" and I sometimes worry about Everything-Bagel AI Safety where e.g. it's not enough to just focus on catastrophic risks, you also have to prevent any way that the government could possibly misuse your models. I think it's important to keep your eye on the ball and not become too susceptible to an Everything-Bagel failure mode.

This is a list of random, assorted AI safety ideas that I think somebody should try to write up and/or work on at some point. I have a lot more than this in my backlog, but these are some that I specifically selected to be relatively small, single-post-sized ideas that an independent person could plausibly work on without much oversight. That being said, I think it would be quite hard to do a good job on any of these without at least chatting with me first—though feel free to message me if you’d be interested.

• What would be necessary to build a good auditing game [AF · GW] benchmark?
• How would AI safety AI [AF · GW] work? What is necessary for it to go well?
• How do we avoid end-to-end training while staying competitive with it? Can we use transparency on end-to-end models to identify useful modules to train non-end-to-end?
• What would it look like to do interpretability on end-to-end trained probabilistic models instead of end-to-end trained neural networks?
• Suppose you had a language model that you knew was in fact a good generative model of the world and that this property continued to hold regardless of what you conditioned it on. Furthermore, suppose you had some prompt that described some agent for the language model to simulate (Alice) that in practice resulted in aligned-looking outputs. Is there a way we could use different conditionals to get at whether or not Alice was deceptive (e.g. prompt the model with “DeepMind develops perfect transparency tools and provides an opportunity for deceptive models to come clean and receive a prize before they’re discovered.”).
• Argue for the importance of ensuring that the state-of-the-art in “using AI for alignment [AF · GW]” never lags behind as a capability compared to where it could be given just additional engineering effort.
• What does inner alignment look like in the context of models with access to memory (e.g. a retrieval database)?
• Argue for doing scaling laws for phase changes. We have found some phase changes in models—e.g. the induction bump—but we haven’t yet really studied the extent to which various properties—e.g. Honesty—generalize across these sorts of phase changes.
• Humans rewarding themselves for finishing their homework by eating candy suggests a plausible mechanism for gradient hacking [AF · GW].
• If we see precursors to deception (e.g. non-myopia, self-awareness, etc.) but suspiciously don’t see deception itself, that’s evidence for deception.
• The more model’s objectives vary depending on exact setup, randomness, etc., the less likely deceptive models are to want to cooperate with future deceptive models, thus making defection earlier more likely.
• China is not a strategically relevant actor for AI, at least in short timeline scenarios—they are too far behind, their GDP isn’t growing fast enough, and their leaders aren’t very good at changing those things.
• If you actually got a language model that was a true generative model of the world that you could get arbitrary conditionals from, that would be equivalent to having access to a quantum suicide machine.
• Introduce the concept of how factored an alignment solution is in terms of how easy it is to turn up or down alignment relative to capabilities—or just swap out an aligned goal for a misaligned one—as an important axis to pay attention to. Currently, things are very factored—alignment and capabilities are both heavily dependent on dataset, reward, etc.—but that could change in the future.
• Argue that wireheading, unlike many other reward gaming or reward tampering problems, is unlikely in practice because the model would have to learn to value the actual transistors storing the reward, which seems exceedingly unlikely in any natural environment.
• How has transparency changed over time—Chris claims [AF · GW] it's easier to interpret later models; is that true?
• Which AI safety proposals are most likely to fail safely? Proposals which have the property that the most likely way for them to fail is just not to work are better than those that are most likely to fail catastrophically. In the former case, we’ve sacrificed some of our alignment tax [EA · GW], but still have another shot.
• What are some plausible scenarios for how a model might be suboptimality deceptively aligned [AF · GW]?
• What can we learn about AI safety from the domestication of animals? Does the success of domesticating dogs from wolves provide an example of how to train for corrigibility? Or did we just make them dumber via the introduction of something like William’s syndrome?

I'll continue to include more directions like this in the comments here.

Here's a two-sentence argument for misalignment that I think is both highly compelling to laypeople and technically accurate at capturing the key issue:

When we train AI systems to be nice, we're giving a bunch of random programs a niceness exam and selecting the programs that score well. If you gave a bunch of humans a niceness exam where the humans would be rewarded for scoring well, do you think you would get actually nice humans?

I think that sticking to capitalism as an economic system post-singularity would be pretty clearly catastrophic and something to strongly avoid, despite capitalism working pretty well today. I've talked about this a bit previously here [LW(p) · GW(p)], but some more notes on why:

• Currently, our society requires the labor of sentient beings to produce goods and services. Capitalism incentivizes that labor by providing a claim on society's overall production in exchange for it. If the labor of sentient beings becomes largely superfluous as an economic input, however, then having a system that effectively incentivizes that labor also becomes largely superfluous.
• Currently, we rely on the mechanism of price discovery to aggregate and disseminate information about the optimal allocation of society's resources. But it's far from an optimal mechanism for allocating resources, and a superintelligence with full visibility and control could do a much better job of resource allocation without falling prey to common pitfalls of the price mechanism such as externalities.
• Capitalism incentivizes the smart allocation of capital in the same way it incentivizes labor. If society can make smart capital allocation decisions without relying on properly incentivized investors, however, then as with labor there's no reason to keep such an incentive mechanism.
• While very large, the total optimization pressure humanity puts into economic competition today would likely pale in comparison to that of a post-singularity future. In the context of such a large increase in optimization pressure, we should generally expect extremal Goodhart [LW · GW] failures.
• More specifically, competitive dynamics incentivize the reinvestment of all economic proceeds back into resource acquisition lest you be outcompeted by another entity doing so. Such a dynamic results in pushing out actors that reinvest proceeds into the flourishing of sentient beings in exchange for those that disregard any such investment in favor of more resource acquisition.
• Furthermore, the proceeds of post-singularity economic expansion flowing to the owners of existing capital is very far from socially optimal. It strongly disfavors future generations, simulated humans, and overall introduces a huge amount of variance into whether we end up with a positive future, putting a substantial amount of control into a set of people whose consumption decisions need not align with the socially optimal allocation.

Listening to this John Oliver, I feel like getting broad support behind transparency-based safety standards might be more possible than I previously thought. He emphasizes the "if models are doing some bad behavior, the creators should be able to tell us why" point a bunch and it's in fact a super reasonable point. It seems to me like we really might be able to get enough broad consensus on that sort of a point to get labs to agree to some sort of standard based on it.

If you want to better understand counting arguments for deceptive alignment, my comment here [LW(p) · GW(p)] might be a good place to start.

Epistemic status: random philosophical musings.

Assuming a positive singularity, how should humanity divide its resources? I think the obvious (and essentially correct) answer is "in that situation, you have an aligned superintelligence, so just ask it what to do." But I nevertheless want to philosophize a bit about this, for one main reason.

That reason is: an important factor imo in determining the right thing to do in distributing resources post-singularity is what incentives that choice of resource allocation creates for people pre-singularity. For those incentives to work, though, we have to actually be thinking about this now, since that's what allows the choice of resource distribution post-singularity to have its acausal influence on our choices pre-singularity. I will note that this is definitely something that I think about sometimes, and something that I think a lot of other people also implicitly think about sometimes when they consider things like amassing wealth, specifically gaining control over current AIs, and/or the future value of their impact certificates.

So, what are some of the possible options for how to distribute resources post-singularity? Let's go over some of the various possible solutions here and why I don't think any of the obvious things here are what you want:

1. The evolutionary/capitalist solution: divide future resources in proportion to control of current resources (e.g. AIs). This is essentially what happens by default if you keep in place an essentially capitalist system and have all the profits generated by your AIs flow to the owners of those AIs. Another version of this is a more power/bargaining-oriented version where you divide resources amongst agents in proportion to the power those agents could bring to bear if they chose to fight for those resources.
   1. The most basic problem with this solution is that it's a moral catastrophe if the people that get all the resources don't do good things with them. We should not want to build AIs that lead to this outcome--and I wouldn't really call AIs that created this outcome aligned.
   2. Another more subtle problem with this solution is that it creates terrible incentives for current people if they expect this to be what happens, since it e.g. incentivizes people to maximize their personal control over AIs at the expense of spending more resources trying to align those AIs.
      1. I feel like I see this sort of thinking a lot and I think that if we were to make it more clear that this is never what should happen in a positive singularity that then people would do this sort of thing less.
2. The egalitarian/democratic solution: divide resources equally amongst all current humans. This is what naive preference utilitarianism would do.
   1. Though it might be less obvious than with the first solution, I think this solution also leads to a moral catastrophe, since it cements current people as oligarchs over future people, leads to value lock-in, and could create a sort of tyranny of the present.
   2. This solution also creates some weird incentives for trying to spread your ideals as widely as possible and to create as many people as possible that share your ideals.
3. The unilateralist/sovereign/past-agnostic/CEV solution: concentrate all resources under the control of your aligned AI(s), then distribute those resources in accordance with how they generate the most utility/value/happiness/goodness/etc., without any special prejudice given to existing people.
   1. In some sense, this is the "right" thing to do, and it's pretty close to what I would ideally want. However, it has a couple of issues:
      1. Though, unlike the first solution, it doesn't create any perverse incentives right now, it doesn't create any positive incentives either.
   2. Since this solution doesn't give any special prejudice to current people, it might be difficult to get current people to agree to this solution, if that's necessary.
4. The retroactive impact certificate solution: divide future resources in proportion to retroactively-assessed social value created by past agents.
   1. This solution obviously creates the best incentives for current agents, so in that sense it does very well.
   2. However, it still does pretty poorly on potentially creating a moral catastrophe, since the people that created the most social value in the past need not continue doing so in the future.

As above, I don't think that you should want your aligned AI to implement any of these particular solutions. I think some combination of (3) and (4) is probably the best out of these options, though of course I'm sure that if you actually asked an aligned superintelligent AI it would do better than any of these. More broadly, though, I think that it's important to note that (1), (2), and (4) are all failure stories, not success stories, and you shouldn't expect them to happen in any scenario where we get alignment right.

Circling back to the original reason that I wanted to discuss all of this, which is how it should influence our decisions now:

• Obviously, the part of your values that isn't selfish should continue to want things to go well.
• However, for the part of your values that cares about your own future resources, if that's something that you care about, how you go about maximizing that is going to depend on what you most expect between (1), (2), and (4).
  • First, in determining this, you should condition on situations where you don't just die or are otherwise totally disempowered, since obviously those are the only cases where this matters. And if that probability is quite high, then presumably a lot of your selfish values should just want to minimize that probability.
  • However, going ahead anyway and conditioning on everyone not being dead/disempowered, what should you expect? I think that (1) and (2) are possible in worlds where get some parts of alignment right, but overall are pretty unlikely: it's a very narrow band of not-quite-alignment that gets you there. So probably if I cared about this a lot I'd focus more on (4) than (1) and (2).
    • Which of course gets me to why I'm writing this up, since that seems like a good message for people to pick up. Though I expect it to be quite difficult to effectively communicate this very broadly.

Here's a simple argument that I find quite persuasive for why you should have linear returns to whatever your final source of utility is (e.g. human experience of a fulfilling life, which I'll just call "happy humans"). Note that this is not an argument that you should have linear returns to resources (e.g. money). The argument goes like this:

1. You have some returns to happy humans (or whatever else you're using as your final source of utility) in terms of how much utility you get from some number of happy humans existing.
2. In most cases, I think those returns are likely to be diminishing, but nevertheless monotonically increasing and differentiable. For example, maybe you have logarithmic returns to happy humans.
3. We happen to live in a massive multiverse [LW · GW]. (Imo the Everett interpretation is settled science, and I don't think you need to accept anything else to make this go through, but note that we're only depending on the existence of any sort of big multiverse here—the one that the Everett interpretation gives you is just the only one that we know is guaranteed to actually exist.)
4. In a massive multiverse, the total number of happy humans is absolutely gigantic (let's ignore infinite ethics problems, though, and assume it's finite—though I think this argument still goes through in the infinite case, it just then depends on whatever infinite ethics framework you like).
5. Furthermore, the total number of happy humans is mostly insensitive to anything you can do, or anything happening locally within this universe, since this universe is only a tiny fraction of the overall multiverse. (Though you could get out of this by claiming that what you really care about is happy humans per universe, that's a pretty strange thing to care about—it's like caring about happy humans per acre.)
6. As a result, the effective returns to happy humans that you are exposed to within this universe reflect only the local behavior of your overall returns. (Note that this assumes "happy humans" are fungible, which I don't actually believe—I care about the overall diversity of human experience throughout the multiverse. However, I don't think that changes the bottom line conclusion, since, if anything, centralizing the happy humans rather than spreading them out seems like it would make it easier to ensure that their experiences are as diverse as possible.)
7. As anyone who has taken any introductory calculus will know, the local behavior of any differentiable function is linear.
8. Since we assumed that your overall returns were differentiable and monotonically increasing, the local returns must be linear with a positive slope.

I wrote up some of my thoughts on how to think about comparing the complexities of different model classes here [LW(p) · GW(p)].

If you want to produce warning shots for deceptive alignment, you're faced with a basic sequencing question. If the model is capable of reasoning about its training process before it's capable of checking a predicate like RSA-2048, then you have a chance to catch it—but if it becomes capable of checking a predicate like RSA-2048 first, then any deceptive models you build won't be detectable.