Measuring Predictability of Persona Evaluations 2024-04-06T08:46:40.936Z
How to train your own "Sleeper Agents" 2024-02-07T00:31:42.653Z
Introducing Alignment Stress-Testing at Anthropic 2024-01-12T23:51:25.875Z
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training 2024-01-12T19:51:01.021Z
Steering Llama-2 with contrastive activation additions 2024-01-02T00:47:04.621Z
RSPs are pauses done right 2023-10-14T04:06:02.709Z
Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research 2023-08-08T01:30:10.847Z
When can we trust model evaluations? 2023-07-28T19:42:21.799Z
Training Process Transparency through Gradient Interpretability: Early experiments on toy language models 2023-07-21T14:52:09.311Z
The Hubinger lectures on AGI safety: an introductory lecture series 2023-06-22T00:59:27.820Z
Towards understanding-based safety evaluations 2023-03-15T18:18:01.259Z
Agents vs. Predictors: Concrete differentiating factors 2023-02-24T23:50:40.322Z
Bing Chat is blatantly, aggressively misaligned 2023-02-15T05:29:45.262Z
Conditioning Predictive Models: Open problems, Conclusion, and Appendix 2023-02-10T19:21:20.251Z
Conditioning Predictive Models: Deployment strategy 2023-02-09T20:59:01.473Z
Conditioning Predictive Models: Interactions with other approaches 2023-02-08T18:19:22.670Z
Conditioning Predictive Models: Making inner alignment as easy as possible 2023-02-07T20:04:20.272Z
Conditioning Predictive Models: The case for competitiveness 2023-02-06T20:08:55.404Z
Conditioning Predictive Models: Outer alignment via careful conditioning 2023-02-02T20:28:58.955Z
Conditioning Predictive Models: Large language models as predictors 2023-02-02T20:28:46.612Z
Why I'm joining Anthropic 2023-01-05T01:12:13.822Z
Discovering Language Model Behaviors with Model-Written Evaluations 2022-12-20T20:08:12.063Z
In defense of probably wrong mechanistic models 2022-12-06T23:24:20.707Z
Engineering Monosemanticity in Toy Models 2022-11-18T01:43:38.623Z
We must be very clear: fraud in the service of effective altruism is unacceptable 2022-11-10T23:31:06.422Z
Attempts at Forwarding Speed Priors 2022-09-24T05:49:46.443Z
Toy Models of Superposition 2022-09-21T23:48:03.072Z
Path dependence in ML inductive biases 2022-09-10T01:38:22.885Z
Monitoring for deceptive alignment 2022-09-08T23:07:03.327Z
Sticky goals: a concrete experiment for understanding deceptive alignment 2022-09-02T21:57:08.246Z
AI coordination needs clear wins 2022-09-01T23:41:48.334Z
Strategy For Conditioning Generative Models 2022-09-01T04:34:17.484Z
How likely is deceptive alignment? 2022-08-30T19:34:25.997Z
Precursor checking for deceptive alignment 2022-08-03T22:56:44.626Z
Acceptability Verification: A Research Agenda 2022-07-12T20:11:34.986Z
A transparency and interpretability tech tree 2022-06-16T23:44:14.961Z
evhub's Shortform 2022-06-11T00:43:35.233Z
Learning the smooth prior 2022-04-29T21:10:18.064Z
Towards a better circuit prior: Improving on ELK state-of-the-art 2022-03-29T01:56:39.684Z
Musings on the Speed Prior 2022-03-02T04:04:42.134Z
Transformer Circuits 2021-12-22T21:09:22.676Z
ML Alignment Theory Program under Evan Hubinger 2021-12-06T00:03:15.443Z
A positive case for how we might succeed at prosaic AI alignment 2021-11-16T01:49:48.464Z
How do we become confident in the safety of a machine learning system? 2021-11-08T22:49:41.080Z
You can talk to EA Funds before applying 2021-09-28T20:39:57.427Z
Automating Auditing: An ambitious concrete technical research proposal 2021-08-11T20:32:41.487Z
LCDT, A Myopic Decision Theory 2021-08-03T22:41:44.545Z
Answering questions honestly instead of predicting human answers: lots of problems and some solutions 2021-07-13T18:49:01.842Z
Knowledge Neurons in Pretrained Transformers 2021-05-17T22:54:50.494Z
Agents Over Cartesian World Models 2021-04-27T02:06:57.386Z


Comment by evhub on Anthropic AI made the right call · 2024-04-16T01:20:05.452Z · LW · GW

Summarizing from the private conversation: the information there is not new to me and I don't think your description of what they said is accurate.

As I've said previously, Anthropic people certainly went around saying things like "we want to think carefully about when to do releases and try to advance capabilities for the purpose of doing safety", but it was always extremely clear at least to me that these were not commitments, just general thoughts about strategy, and I believe that's what was being referred to as being widespread in 2022 here.

Comment by evhub on Anthropic AI made the right call · 2024-04-15T20:36:26.072Z · LW · GW

I think it was an honest miscommunication coupled to a game of telephone—the sort of thing that inevitably happens sometimes—but not something that I feel particularly concerned about.

Comment by evhub on Anthropic AI made the right call · 2024-04-15T05:59:37.926Z · LW · GW

I mostly believe y'all about 2—I didn't know 2 until people asserted it right after the Claude 3 release, but I haven't been around the community, much less well-connected in it, for long—but that feels like an honest miscommunication to me.

For the record, I have been around the community for a long time (since before Anthropic existed), in a very involved way, and I had also basically never heard of this before the Claude 3 release. I can recall only one time where I ever heard someone mention something like this, it was a non-Anthropic person who said they heard it from someone else who was a non-Anthropic person, they asked me if I had heard the same thing, and I said no. So it certainly seems clear given all the reports that this was a real rumour that was going around, but it was definitely not the case that this was just an obvious thing that everyone in the community knew about or that Anthropic senior staff were regularly saying (I talked regularly to a lot of Anthropic senior staff before I joined Anthropic and I never heard anyone say this).

Comment by evhub on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-12T20:44:27.053Z · LW · GW

Pilot wave theory keeps all of standard wave mechanics unchanged, but then adds on top rules for how the wave "pushes around" classical particles. But it never zeroes out parts of the wave—the entirety of wave mechanics, including the full superposition and all the multiple worlds that implies, are still necessary to compute the exact structure of that wave to understand how it will push around the classical particles. Pilot wave theory then just declares that everything other than the classical particles "don't exist", which doesn't really make sense because the multiple worlds still have to exist in some sense because you have to compute them to understand how the wave will push around the classical particles.

Comment by evhub on Any evidence or reason to expect a multiverse / Everett branches? · 2024-04-12T04:13:17.948Z · LW · GW

See my discussion of pilot wave theory here.

Comment by evhub on The Case for Predictive Models · 2024-04-03T22:56:21.334Z · LW · GW

Fwiw, I still think about Conditioning Predictive Models stuff quite a lot and think it continues to be very relevant. I think that if future AI systems continue to look very much like present AI systems, I expect some of the major problems that we'll have to deal with will be exactly the major problems presented in that paper (e.g. a purely predictive AI system predicting the output of a deceptively aligned AI system continues to look like a huge and important problem to me).

Comment by evhub on Video and transcript of presentation on Scheming AIs · 2024-03-22T21:19:39.097Z · LW · GW

(Moderation note: added to the Alignment Forum from LessWrong.)

Comment by evhub on Stagewise Development in Neural Networks · 2024-03-21T05:31:03.448Z · LW · GW

(Moderation note: added to the Alignment Forum from LessWrong.)

Comment by evhub on How do we become confident in the safety of a machine learning system? · 2024-03-18T05:08:24.757Z · LW · GW

I still really like my framework here! I think this post ended up popularizing some of the ontology I developed here, but the unfortunate thing about that post as the one that popularized this is that it doesn't really provide an alternative.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-12T00:29:55.771Z · LW · GW

Sure—I agree with that. The section I linked from Conditioning Predictive Models actually works through at least to some degree how I think simplicity arguments for deception go differently for purely pre-trained predictive models.

Comment by evhub on How to train your own "Sleeper Agents" · 2024-03-09T04:16:02.445Z · LW · GW

Some people pointed out to me that the code_backdoor_train_data.jsonl file had mismatched prompts and completions—sorry about that! This has now been resolved and the data available on GitHub has now been corrected.

Comment by evhub on Anthropic release Claude 3, claims >GPT-4 Performance · 2024-03-06T23:01:17.336Z · LW · GW

Not making any claims about actual Anthropic strategy here, but as gwern notes, I don't think that these are necessarily contradictory. For example, you could have a strategy of getting far enough ahead that new entrants like e.g. Mistral would have a hard time keeping up, but staying on pace with or behind current competitors like e.g. OpenAI.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T06:57:58.796Z · LW · GW

I'm not sure where it was established that what's under consideration here is just deceptive alignment in pre-training. Personally, I'm most worried about deceptive alignment coming after pre-training. I'm on record as thinking that deceptive alignment is unlikely (though certainly not impossible) in purely pretrained predictive models.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T06:54:57.190Z · LW · GW

How can this be true, when you e.g. say there's "only one saint"? That doesn't make any sense with parameterizations due to internal invariances; there are uncountably many saints.

Because it was the transcript of a talk? I was trying to explain an argument at a very high level. And there's certainly not uncountably many; in the infinite bitstring case there would be countably many, though usually I prefer priors that put caps on total computation such that there are only finitely many.

I'd expect you to raise that as an obvious point in worlds where this really was about parameterizations.

I don't really appreciate the psychoanalysis here. I told you what I thought and think, and I have far more evidence about that than you do.

And, as you've elsewhere noted, we don't know enough about parameterizations to make counting arguments over them. So how are you doing that?

As I've said, I usually try to take whatever the most realistic prior is that we can reason about at a high-level, e.g. a circuit prior or a speed prior.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T06:48:53.867Z · LW · GW

I think instrumental convergence does provide meaningful evidence of doom, and you can make a valid counting argument for it, but as with deceptive alignment you have to run the counting argument over algorithms not over functions.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T02:56:09.215Z · LW · GW

The real counting argument that Evan believes in is just a repackaging of Paul's argument for the malignity of the Solomonoff prior, and not anything novel.

I'm going to stop responding to you now, because it seems that you are just not reading anything that I am saying. For the last time, my criticism has absolutely nothing to do with Solomonoff induction in particular, as I have now tried to explain to you here and here and here etc.

I mean, the neural network Gaussian process is literally this, and you can make it more realistic by using the neural tangent kernel to simulate training dynamics, perhaps with some finite width corrections. There is real literature on this.

Yes—that's exactly the sort of counting argument that I like! Though note that it can be very hard to reason properly about counting arguments once you're using a prior like that; it gets quite tricky to connect those sorts of low-level properties to high-level properties about stuff like deception.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T02:39:50.027Z · LW · GW

I definitely thought you were making a counting argument over function space, and AFAICT Joe also thought this in his report.

Sorry about that—I wish you had been at the talk and could have asked a question about this.

You're making an argument about one type of learning procedure, Solomonoff induction, which is physically unrealizable and AFAICT has not even inspired any serious real-world approximations, and then assuming that somehow the conclusions will transfer over to a mechanistically very different learning procedure, gradient descent.

I agree that Solomonoff induction is obviously wrong in many ways, which is why you want to substitute it out for whatever the prior is that you think is closest to deep learning that you can still reason about theoretically. But that should never lead you to do a counting argument over function space, since that is never a sound thing to do.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T02:10:57.128Z · LW · GW

If I examine the causal mechanisms here, I find things like "humans seem to have have 'parameterizations' which already encode situationally activated consequentialist reasoning", and then I wonder "will AI develop similar cognition?" and then that's the whole thing I'm trying to answer to begin with.

Do you believe that AI systems won't learn to use goal-directed consequentialist reasoning even if we train them directly on outcome-based goal-directed consequentialist tasks? Or do you think we won't ever do that?

If you do think we'll do that, then that seems like all you need to raise that hypothesis into consideration. Certainly it's not the case that models always learn to value anything like what we train them to value, but it's obviously one of the hypotheses that you should be seriously considering.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T02:00:09.857Z · LW · GW

I think you should have asked for clarification before making blistering critiques about how Nora "ended up using reasoning that doesn't actually correspond to any well-defined mathematical object." I think your comments paint a highly uncharitable and (more importantly) incorrect view of N/Q's claims.

I'm happy to apologize if I misinterpreted anyone, but afaict my critique remains valid. My criticism is precisely that counting arguments over function space aren't generally well-defined, and even if they were they wouldn't be the right way to run a counting argument. So my criticism that the original post misunderstands how to properly run a counting argument still seems correct to me. Perhaps you could say that it's not the authors' fault, that they were responding to weak arguments that other people were actually making, but regardless the point remains that the authors haven't engaged with the sort of counting arguments that I actually think are valid.

Your presentations often include a counting argument over a function space, in the form of "saints" versus "schemers" and "sycophants." So it seems to me that you do suggest that. What am I missing?

What makes you think that's intended to be a counting argument over function space? I usually think of this as a counting argument over infinite bitstrings, as I noted in my comment (though there are many other valid presentations). It's possible I said something in that talk that gave a misleading impression there, but I certainly don't believe and have never believed in any counting arguments over function space.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T01:46:21.557Z · LW · GW

Humans under selection pressure—e.g. test-takers, job-seekers, politicians—will often misrepresent themselves and their motivations to get ahead. That very basic fact that humans do this all the time seems like sufficient evidence to me to consider the hypothesis at all (though certainly not enough evidence to conclude that it's highly likely).

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T01:31:29.098Z · LW · GW

I agree that there is a valid argument that critiques counting arguments over function space that sort of has the same shape as the one presented in this post. If that was what the authors had in mind, it was not what I got from reading the post, and I haven't seen anyone making that clarification other than yourself.

Regardless, though, I think that's still not a great objection to counting arguments for deceptive alignment in general, because it's explicitly responding only to a very weak and obviously wrong form of a counting argument. My response there is just that of course you shouldn't run a counting argument over function space—I would never suggest that.

Comment by evhub on Anthropic release Claude 3, claims >GPT-4 Performance · 2024-03-05T01:23:52.564Z · LW · GW

As one data point: before I joined Anthropic, when I was trying to understand Anthropic's strategy, I never came away with the impression that Anthropic wouldn't advance the state of the art. It was quite clear to me that Anthropic's strategy at the time was more amorphous than that, more like "think carefully about when to do releases and try to advance capabilities for the purpose of doing safety" rather than "never advance the state of the art". I should also note that now the strategy is actually less amorphous, since it's now pretty explicitly RSP-focused, more like "we will write RSP commitments that ensure we don't contribute to catastrophic risk and then scale and deploy only within the confines of the RSP".

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T01:17:00.502Z · LW · GW

Were I not busy with all sorts of empirical stuff right now, I would consider prioritizing a project like that, but alas I expect to be too busy. I think it would be great if somebody else wanted devote more time to working through the arguments in detail publicly, and I might encourage some of my mentees to do so.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T01:13:04.141Z · LW · GW

The main thing I was trying to show there is just that having the formalism prevents you from making logical mistakes in how to apply counting arguments in general, as I think was done in this post. So my comment is explaining how to use the formalism to avoid mistakes like that, not trying to work through the full argument for deceptive alignment.

It's not that the formalism provides really strong evidence for deceptive alignment, it's that it prevents you from making mistakes in your reasoning. It's like plugging your argument into a proof-checker: it doesn't check that your argument is correct, since the assumptions could be wrong, but it does check that your argument is sound.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-05T01:08:04.290Z · LW · GW

I'm referring to the deceptiveness of the reasoning displayed in the chain of thought during training time. So it's not a generalization question, it's about whether, if the model is using deceptive reasoning to compute its answer (as we think it is, since we think our models really are using their chain of thought), does that deceptive reasoning go away when the model has to use it to produce aligned answers during training? And we find that not only does it not go away, it actually gets more deceptive when you train it to produce aligned answers.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T22:16:24.119Z · LW · GW

But your work provides no strong conclusions on that matter as it pertains to deceptive reasoning in general.

I just disagree with this. Our chain of thought models do tons of very deceptive reasoning during safety training and the deceptiveness of that reasoning is totally unaffected by safety training, and in fact the deceptiveness increases in the case of adversarial training.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T20:43:15.106Z · LW · GW

I think they are valid if interpreted properly, but easy to misinterpret.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T19:56:54.480Z · LW · GW

I'm surprised by this. It seems to me like most of your reasoning about simplicity is either hand-wavy or only nominally formally backed by symbols which don't (AFAICT) have much to do with the reality of neural networks.

The examples that you cite are from a LessWrong comment and a transcript of a talk that I gave. Of course when I'm presenting something in a context like that I'm not going to give the most formal version of it; that doesn't mean that the informal hand-wavy arguments are the reasons why I believe what I believe.

Maybe a better objection there would be: then why haven't you written up anything more careful and more formal? Which is a pretty fair objection, as I note here. But alas I only have so much time and it's not my current focus.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T19:50:23.561Z · LW · GW

I would be much happier with that; I think that's much more correct. Then, my objection would just be that at least the sort of counting arguments for deceptive alignment that I like are and always have been about parameterizations rather than functions. I agree that if you try to run a counting argument directly in function space it won't work.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T19:49:03.527Z · LW · GW

I myself expected you to respond to this post with some ML-specific reasoning about simplicity and measure of parameterizations, instead of your speculation about a relationship between the universal measure and inductive biases. I spoke with dozens of people about the ideas in OP's post, and none of them mentioned arguments like the one you gave. I myself have spent years in the space and am also not familiar with this particular argument about bitstrings.

That probably would have been my objection had the reasoning about priors in this post been sound, but since the reasoning was unsound, I turned to the formalism to try to show why it's unsound.

If these are your real reasons for expecting deceptive alignment, that's fine, but I think you've mentioned this rather infrequently.

I think you're misunderstanding the nature of my objection. It's not that Solomonoff induction is my real reason for believing in deceptive alignment or something, it's that the reasoning in this post is mathematically unsound, and I'm using the formalism to show why. If I weren't responding to this post specifically, I probably wouldn't have brought up Solomonoff induction at all.

This yields a perfectly well-defined counting argument over .

we can parameterize such functions using the neural network parameter space

I'm very happy with running counting arguments over the actual neural network parameter space; the problem there is just that I don't think we understand it well enough to do so effectively.

You could instead try to put a measure directly over the functions in your setup, but the problem there is that function space really isn't the right space to run a counting argument like this; you need to be in algorithm space, otherwise you'll do things like what happens in this post where you end up predicting overfitting rather than generalization (which implies that you're using a prior that's not suitable for running counting arguments on).

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T19:39:55.744Z · LW · GW

I'm quite aware that we did not see natural deceptive alignment, so I don't think I'm misinterpreting my own results in the way you were predicting. Perhaps "empirically disprove" is too strong; I agree that our results are evidence but not definitive evidence. But I think they're quite strong evidence and by far the strongest evidence available currently on the question of whether deception will be regularized away.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T04:58:20.725Z · LW · GW

Right, and I've explained why I don't think any of those analyses are relevant to neural networks. Deep learning simply does not search over Turing machines or circuits of varying lengths. It searches over parameters of an arithmetic circuit of fixed structure, size, and runtime. So Solomonoff induction, speed priors, and circuit priors are all inapplicable.

It is trivially easy to modify the formalism to search only over fixed-size algorithms, and in fact that's usually what I do when I run this sort of analysis. I feel like you still aren't understanding the key criticism here—it's really not about Solomonoff induction—and I'm not sure how to explain that in any way other than how I've already done so.

There has been a lot of work in the mainstream science of deep learning literature on the generalization behavior of actual neural nets, and I'm pretty baffled at why you don't pay more attention to that stuff.

I'm going to assume you just aren't very familiar with my writing, because working through empirical evidence about neural network inductive biases is something I love to do all the time.

Comment by evhub on Wei Dai's Shortform · 2024-03-04T02:46:21.833Z · LW · GW

I think that UDASSA and UDT might be in academia's blind spots in the same way that the Everett interpretation is: more correct theories that came after less correct theories with mostly only theoretical evidence to support changing over to the new theories.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T02:27:34.779Z · LW · GW

Well, I presented a very simple formulation in my comment, so that could be a reasonable starting point.

But I agree that unfortunately there hasn't been that much good formal analysis here that's been written up. At least on my end, that's for two reasons:

  1. Most of the formal analysis of this form that I've published (e.g. this and this) has been focused on sycophancy (human imitator vs. direct translator) rather than deceptive alignment, as sycophancy is a substantially more tractable problem. Finding a prior that reasonably rules out deceptive alignment seems quite out of reach to me currently; at one point I thought a circuit prior might do it, but I now think that circuit priors don't get rid of deceptive alignment.
  2. I'm currently more optimistic about empirical evidence rather than theoretical evidence for resolving this question, which is why I've been focusing on projects such as Sleeper Agents.
Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T01:14:35.143Z · LW · GW

I obviously don't think the counting argument for overfitting is actually sound, that's the whole point.

Yes, I'm well aware. The problem is that when you make the counting argument for overfitting, you do so in a way that seriously misuses the formalism, which is why the argument fails. So you can't draw any lessons about counting arguments for deception from the failure of your counting argument for overfitting.

But I think the counting argument for scheming is just as obviously invalid, and misuses formalisms just as egregiously, if not moreso.

Then show me how! If you think there are errors in the math, please point them out.

Of course, it's worth stating that I certainly don't have some sort of airtight mathematical argument proving that deception is likely in neural networks—there are lots of assumptions there that could very well be wrong. But I do think that the basic style of reasoning employed by such arguments is sound.

I deny that your Kolmogorov framework is anything like "the proper formalism" for neural networks.

Err... I'm using K-complexity here because it's a simple framework to reason about, but my criticism isn't "you should use K-complexity to reason about neural networks." I think K-complexity captures some important facts about neural network generalization, but is clearly egregiously wrong in other areas. But there are lots of other formalisms! My criticism isn't that you should use K-complexity, it's that you should use any formalism at all.

The basic criticism is that the reasoning you use in the post doesn't correspond to any formalism at all; it's self-contradictory and inconsistent. So by all means you should replace K-complexity with something better (that's what I usually try to do as well) but you still need to be reasoning in a way that's mathematically consistent.

I also deny that the counting argument for overfitting is appropriately characterized as a "finite bitstring" argument, because that suggests I'm talking about Turing machine programs of finite length, which I'm not- I'm directly enumerating functions over a subset of the natural numbers.

One person's modus ponens is another's modus tollens. If you say you have a formalism, and that formalism predicts overfitting rather than generalization, then my first objection to your formalism is that it's clearly a bad formalism for understanding neural networks in practice. Maybe the most basic thing that any good formalism here should get right is that it should predict generalization; if your formalism doesn't, then it's clearly not a good formalism.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-04T00:55:58.976Z · LW · GW

Yes, that's exactly the problem: you tried to make a counting argument, but because you didn't engage with the proper formalism, you ended up using reasoning that doesn't actually correspond to any well-defined mathematical object.

Analogously, it's like you wrote an essay about why 0.999... != 1 and your response to "under the formalism of real numbers as Dedekind cuts, those are identical" was "where did I say I was referring to Dedekind cuts?" It's fine if you don't want to use the standard formalism, but you need some formalism to anchor your words to, otherwise you're just pushing around words with no real way to ensure that your words actually correspond to something. I think the 0.999... != 1 analogy is quite apt here, because the problem really is that there is no formalism under which 0.999... != 1 that looks anything like the real numbers that you know, in the same way that there really is no formalism under which the sort of reasoning that you're using is meaningful.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-03-01T00:17:49.850Z · LW · GW

I think the infinite bitstring case has zero relevance to deep learning.

I think you are still not really understanding my objection. It's not that there is a "finite bitstring case" and an "infinite bitstring case". My objection is that the sort of finite bitstring analysis that you use does not yield any well-defined mathematical object that you could call a prior, and certainly not one that would predict generalization.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-02-29T20:36:59.458Z · LW · GW

Paradoxically, I think larger neural networks are more simplicity-biased.

The idea is that when you make your network larger, you increase the size of the search space and thus the number of algorithms that you're considering to include algorithms which take more computation. That reduces the relative importance of the speed prior, but increases the relative importance of the simplicity prior, because your inductive biases are still selecting from among those algorithms according to the simplest pattern that fits the data, such that you get good generalization—and in fact even better generalization because now the space of algorithms in which you're searching for the simplest one in is even larger.

Another way to think about this: if you really believe Occam's razor, then any learning algorithm generalizes exactly to the extent that it approximates a simplicity prior—thus, since we know neural networks generalize better as they get larger, they must be approximating a simplicity prior better as they do so.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-02-29T03:41:32.435Z · LW · GW

"indifference over infinite bitstrings" is a misnomer in an important sense, because it's literally impossible to construct a normalized probability measure over infinite bitstrings that assigns equal probability to each one. What you're talking about is the length weighted measure that assigns exponentially more probability mass to shorter programs. That's definitely not an indifference principle, it's baking in substantive assumptions about what's more likely.

No; this reflects a misunderstanding of how the universal prior is traditionally derived in information theory. We start by assuming that we are running our UTM over code such that every time the UTM looks at a new bit in the tape, it has equal probability of being a 1 or a 0 (that's the indifference condition). That induces what's called the universal semi-measure, from which we can derive the universal prior by enforcing a halting condition. The exponential nature of the prior simply falls out of that derivation.

I don't see why we should expect any of this reasoning about Turning machines to transfer over to neural networks at all, which is why I didn't cast the counting argument in terms of Turing machines in the post. In the past I've seen you try to run counting or simplicity arguments in terms of parameters. I don't think any of that works, but I at least take it more seriously than the Turing machine stuff.

Some notes:

  1. I am very skeptical of hand-wavy arguments about simplicity that don't have formal mathematical backing. This is a very difficult area to reason about correctly and it's easy to go off the rails if you're trying to do so without relying on any formalism.
  2. There are many, many ways to adjust the formalism to take into account various ways in which realistic neural network inductive biases are different than basic simplicity biases. My sense is that most of these changes generally don't change the bottom-line conclusion, but if you have a concrete mathematical model that you'd like to present here that you think gives a different result, I'm all ears.
  3. All of that being said, I'm absolutely with you that this whole space of trying to apply theoretical reasoning about inductive biases to concrete ML systems is quite fraught. But it's even more fraught if you drop the math!
  4. So I'm happy with turning to empirics instead, which is what I have actually done! I think our Sleeper Agents results, for example, empirically disprove the hypothesis that deceptive reasoning will be naturally regularized away (interestingly, we find that it does get regularized away for small models—but not for large models!).
Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-02-29T02:34:12.876Z · LW · GW

Here's another fun way to think about this—you can basically cast what's wrong here as an information theory exercise.


Spot the step where the following argument goes wrong:

  1. Suppose I have a dataset of finitely many points arranged in a line. Now, suppose I fit a (reasonable) universal prior to that dataset, and compare two cases: learning a line and learning to memorize each individual datapoint.
  2. In the linear case, there is only one way to implement a line.
  3. In the memorization case, I can implement whatever I want on the other datapoints in an arbitrary way.
  4. Thus, since there are more ways to memorize than to learn a line, there should be greater total measure on memorization than on learning the line.
  5. Therefore, you'll learn to memorize each individual datapoint rather than learning to implement a line.


By the logic of the post, step 4 is the problem, but I think step 4 is actually valid. The problem is step 2: there are actually a huge number of different ways to implement a line! Not only are there many different programs that implement the line in different ways, I can also just take the simplest program that does so and keep on adding comments or other extraneous bits. It's totally valid to say that the algorithm with the most measure across all ways of implementing it is more likely, but you have to actually include all ways of implementing it, including all the cases where many of those bits are garbage and aren't actually doing anything.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-02-29T01:10:01.300Z · LW · GW

Yep, I endorse that text as being equivalent to what I wrote; sorry if my language was a bit confusing.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-02-29T01:07:02.536Z · LW · GW

Yep, that's exactly right. As always, once you start making more complex assumptions, things get more and more complicated, and it starts to get harder to model things in nice concrete mathematical terms. I would defend the value of having actual concrete mathematical models here—I think it's super easy to confuse yourself in this domain if you aren't doing that (e.g. as I think the confused reasoning about counting arguments in this post demonstrates). So I like having really concrete models, but only in the "all models are wrong, but some are useful" sense, as I talk about in "In defense of probably wrong mechanistic models."

Also, the main point I was trying to make is that the counting argument is both sound and consistent with known generalization properties of machine learning (and in fact predicts them), and for that purpose I went with the simplest possible formalization of the counting argument.

Comment by evhub on evhub's Shortform · 2024-02-29T00:14:52.789Z · LW · GW

If you want to better understand counting arguments for deceptive alignment, my comment here might be a good place to start.

Comment by evhub on Counting arguments provide no evidence for AI doom · 2024-02-29T00:11:05.143Z · LW · GW

I really do appreciate this being written up, but to the extent that this is intended to be a rebuttal to the sorts of counting arguments that I like, I think you would have basically no chance of passing my ITT here. From my perspective reading this post, it read to me like "I didn't understand the counting argument, therefore it doesn't make sense" which is (obviously) not very compelling to me. That being said, to give credit where credit is due, I think some people would make a more simplistic counting argument like the one you're rebutting. So I'm not saying that you're not rebutting anyone here, but you're definitely not rebutting my position.

Edit: If you're struggling to grasp the distinction I'm pointing to here, it might be worth trying this exercise pointing out where the argument in the post goes wrong in a very simple case and/or looking at Ryan's restatement of my mathematical argument.

Edit: Another point of clarification here—my objection is not that there is a "finite bitstring case" and an "infinite bitstring case" and you should be using the "infinite bitstring case". My objection is that the sort of finite bitstring analysis in this post does not yield any well-defined mathematical object at all, and certainly not one that would predict generalization.

Let's work through how to properly reason about counting arguments:

  • When doing reasoning about simplicity priors, a really important thing to keep in mind is the relationship between infinite bitstring simplicity and finite bitstring simplicity. When you just start counting the ways in which the model can behave on unseen inputs and then saying that the more ways there are the more likely it is, what you're implicitly computing there is actually an inverse simplicity prior: Consider two programs, one that takes bits and then stops, and one that takes bits to specify the necessary logic but then uses remaining bits to fill in additional pieces for how it might behave on unseen inputs. Obviously the bit program is simpler, but by your logic the bit program would seem to be simpler because it leaves more things unspecified in terms of all the ways to fill in the remaining bits. But if you understand that we can recast everything into infinite bitstring complexity, then it's clear that actually the bit program is leaving bits unspecified—even though those bits don't do anything in that case, they're still unspecified parts of the overall infinite bitstring.
  • Once we understand that relationship, it should become pretty clear why the overfitting argument doesn't work: the overfit model is essentially the model, where it takes more bits to specify the core logic, and then tries to "win" on the simplicity by having unspecified bits of extra information. But that doesn't really matter: what matters is the size of the core logic, and if there are simple patterns that can fit the data in bits rather than bits, you'll learn those.
  • However, this doesn't apply at all to the counting argument for deception. In fact, understanding this distinction properly is critically important to make the argument work. Let's break it down:
    • Suppose my model has the following structure: an bit world model, an bit search procedure, and an bit objective function. This isn't a very realistic assumption, but it'll be enough to make it clear why the counting argument here doesn't make use of the same fallacious reasoning that would lead you to think that the bit model was simpler.
    • We'll assume that the deceptive and non-deceptive models require the same bits of world modeling and search, so the only question is the objective function.
    • I would usually then make an argument here for why in most cases the simplest objective that leads to deception is simpler than the simplest objective that leads to alignment, but that's just a simplicity argument, not a counting argument. Since we want to do the counting argument here, let's assume that the simplest objective that leads to alignment is simpler than the simplest objective that leads to deception.
    • Okay, but now if the simplest objective that leads to alignment is simpler than the simplest objective that leads to deception, how could deception win? Well, the key is that the core logic necessary for deception is simpler: the only thing required for deception is a long-term objective, everything else is unspecified. So, mathematically, we have:
      • Complexity of simplest aligned objective:
      • Complexity of simplest deceptive objective: where is the minimum necessary for any long-term objective and is everything else necessary to implement some particular long-term objective.
      • We're assuming that , but that .
      • Casting into infinite bitstring land, we see that the set of aligned objectives includes those with anything after the first bits, whereas the set of deceptive objectives includes anything after the first bits. Even though you don't get a full program until you're bits deep, the complexity here is just , because all the bits after the first bits aren't pinned down. So if we're assuming that , then deception wins.
      • Certainly, you could contest the assumption that —and conversely I would even go further and say probably —but either way the point is that this argument is totally sound given its assumptions.
  • At a high level, what I'm saying here is that counting arguments are totally valid and in fact strongly predict that you won't learn to memorize, but only when you do them over infinite bitstrings, not when done over finite bitstrings. If we think about the simplicity of learning a line to fit a set of linear datapoints vs. the simplicity of memorizing everything, there are more ways to implement a line than there are to memorize, but only over infinite bitstrings. In the line case, the extra bits don't do anything, whereas in the memorization case, they do, but that's not a relevant distinction: they're still unspecified bits, and what we're doing is counting up the measure of the infinite bitstrings which implement that algorithm.
  • I think this analysis should also make clear what's going on with the indifference principle here. The "indifference principle" in this context is about being indifferent across all infinite bitstrings—it's not some arbitrary thing where you can carve up the space however you like and then say you're indifferent across the different pieces—it's a very precise notion that comes from theoretical computer science (though there is a question about what UTM to use; there you're trying to get as close as possible to a program prior that would generalize well in practice given that we know ML generalizes well). The idea is that indifference across infinite bitstrings gives you a universal semi-measure, from which you can derive a universal prior (which you're trying to select out of the space of all universal priors to match ML well). Of course, it's certainly not the case that actual machine learning inductive biases are purely simplicity, or that they're even purely indifferent across all concrete parameterizations, but I think it's a reasonable first-pass assumption given facts like ML not generally overfitting as you note.
  • Looking at this more broadly, from my perspective, the fact that we don't see overfitting is the entire reason why deceptive alignment is likely. The fact that models tend to learn simple patterns that fit the data rather than memorize a bunch of stuff is exactly why deception, a simple strategy that compresses a lot of data, might be a very likely thing for them to learn. If models were more likely to learn overfitting-style solutions, I would be much, much less concerned about deception—but of course, that would also mean they were less capable, so it's not much solace.
Comment by evhub on Timaeus's First Four Months · 2024-02-28T21:04:34.599Z · LW · GW

(Moderation note: added to the Alignment Forum from LessWrong.)

Comment by evhub on How to train your own "Sleeper Agents" · 2024-02-18T20:45:30.434Z · LW · GW

Both seem interesting, though it's worth noting that our adversarial training results in the paper do already show that you could use automated red-teaming to detect our sleeper agents.

Comment by evhub on evhub's Shortform · 2024-02-03T22:19:49.264Z · LW · GW

Even if you don't personally value other people, if you're willing to step behind the veil of ignorance with respect to whether you'll be an early person or a late person, it's clearly advantageous before you know which one you'll be to not allocate all the resources to the early people.

Comment by evhub on evhub's Shortform · 2024-02-03T02:35:13.059Z · LW · GW

If the situation was something like "current people, weighted by wealth, deliberate for a while on what to do with our resources" then I agree that's probably like 5 - 10 times worse than the best approach (which is still a huge haircut) but not clearly catastrophic. But it's not clear to me that's what the default outcome of competitive dynamics would look like—sufficiently competitive dynamics could force out altruistic actors if they get outcompeted by non-altruistic actors.

Comment by evhub on evhub's Shortform · 2024-02-02T22:58:31.505Z · LW · GW

I think that sticking to capitalism as an economic system post-singularity would be pretty clearly catastrophic and something to strongly avoid, despite capitalism working pretty well today. I've talked about this a bit previously here, but some more notes on why:

  • Currently, our society requires the labor of sentient beings to produce goods and services. Capitalism incentivizes that labor by providing a claim on society's overall production in exchange for it. If the labor of sentient beings becomes largely superfluous as an economic input, however, then having a system that effectively incentivizes that labor also becomes largely superfluous.
  • Currently, we rely on the mechanism of price discovery to aggregate and disseminate information about the optimal allocation of society's resources. But it's far from an optimal mechanism for allocating resources, and a superintelligence with full visibility and control could do a much better job of resource allocation without falling prey to common pitfalls of the price mechanism such as externalities.
  • Capitalism incentivizes the smart allocation of capital in the same way it incentivizes labor. If society can make smart capital allocation decisions without relying on properly incentivized investors, however, then as with labor there's no reason to keep such an incentive mechanism.
  • While very large, the total optimization pressure humanity puts into economic competition today would likely pale in comparison to that of a post-singularity future. In the context of such a large increase in optimization pressure, we should generally expect extremal Goodhart failures.
  • More specifically, competitive dynamics incentivize the reinvestment of all economic proceeds back into resource acquisition lest you be outcompeted by another entity doing so. Such a dynamic results in pushing out actors that reinvest proceeds into the flourishing of sentient beings in exchange for those that disregard any such investment in favor of more resource acquisition.
  • Furthermore, the proceeds of post-singularity economic expansion flowing to the owners of existing capital is very far from socially optimal. It strongly disfavors future generations, simulated humans, and overall introduces a huge amount of variance into whether we end up with a positive future, putting a substantial amount of control into a set of people whose consumption decisions need not align with the socially optimal allocation.
Comment by evhub on Will quantum randomness affect the 2028 election? · 2024-01-25T01:31:16.780Z · LW · GW

One relevant consideration: I used to make all sorts of everyday decisions via a quantum random number generator (specifically an app that would query this online source of quantum random numbers), and it wouldn't take that many people like me doing that to create pretty large butterfly effects.