LessWrong 2.0 Reader

Does anyone know why Anthropic doesn't want models with powerful cyber capabilities to be classified as "dual-use foundation models?"

In its BIS comment, Anthropic proposes a new definition of dual-use foundation model that excludes cyberoffensive capabilities. This also comes up in TechNet's response (TechNet is a trade association that Anthropic is a part of).

Does anyone know why Anthropic doesn't want the cyber component of the definition to remain? (I don't think they cover this in the comment).

---

More details– the original criteria for "dual-use foundation model" proposed by BIS are:

(1) Substantially lowering the barrier of entry for non-experts to design, synthesize, acquire, or use chemical, biological, radiological, or nuclear (CBRN) weapons;
(2) Enabling powerful offensive cyber operations through automated vulnerability discovery and exploitation against a wide range of potential targets of cyberattacks; or
(3) Permitting the evasion of human control or oversight through means of deception or obfuscation.

Anthropic's definition includes criteria #1 and #3 in its definition but excludes criterion #2. 

(Separately, Anthropic argues that dual-use foundation models should be defined as those that pose catastrophic risks as opposed to serious risks to national security. This is important too, but I'm less confused about why Anthropic wants this.)

links 10/23/24:

https://roamresearch.com/#/app/srcpublic/page/10-23-2024

• https://eukaryotewritesblog.com/2024/10/21/i-got-dysentery-so-you-dont-have-to/  personal experience at a human challenge trial, by the excellent Georgia Ray
• https://catherineshannon.substack.com/p/the-male-mind-cannot-comprehend-the
  • I...guess this isn't wrong, but it's a kind of Take I've never been able to relate to myself. Maybe it's because I found Legit True Love at age 22, but I've never had that feeling of "oh no the men around me are too weak-willed" (not in my neck of the woods they're not!) or "ew they're too interested in going to the gym" (gym rats are fine? it's a hobby that makes you good-looking, I'm on board with this) or "they're not attentive and considerate enough" (often a valid complaint, but typically I'm the one who's too hyperfocused on my own work & interests) or "they're too show-offy" (yeah it's irritating in excess but a little bit of show-off energy is enlivening).
  • Look: you like Tony Soprano because he's competent and lives by a code? But you don't like it when a real-life guy is too competitive, intense, or off doing his own thing? I'm sorry, but that's not how things work.
    • Tony Soprano can be light-hearted and always have time for the women around him because he is a fictional character. In real life, being good at stuff takes work and is sometimes stressful.
    • My husband is, in fact, very close to this "Tony Soprano" ideal -- assertive, considerate, has "boyish charm", lives by a "code", is competent at lots of everyday-life things but isn't too busy for me -- and I guarantee you would not have thought to date him because he's also nerdy and argumentative and wouldn't fit in with the yuppie crowd.
    • Also like. This male archetype is a guy who fixes things for you and protects you and makes you feel good. In real life? Those guys get sad that they're expected to give, give, give and nobody cares about their feelings. I haven't watched The Sopranos but my understanding is that Tony is in therapy because the strain of this life is getting to him. This article doesn't seem to have a lot of empathy with what it's like to actually be Tony...and you probably should, if you want to marry him.
• https://fas.org/publication/the-magic-laptop-thought-experiment/ from Tom Kalil, a classic: how to think about making big dreams real.
• https://paulgraham.com/yahoo.html Paul Graham's business case studies!
• https://substack.com/home/post/p-150520088 a celebratory reflection on the recent Progress Conference. Yes, it was that good.
• https://en.m.wikipedia.org/wiki/Hecuba  in some tellings (not Homer's), Hecuba turns into a dog from grief at the death of her son.
• https://www.librariesforthefuture.bio/p/lff
  • a framework for thinking about aging: "1st gen" is delaying aging, which is where the field started (age1, metformin, rapamycin), while "2nd gen" is pausing (stasis), repairing (reprogramming), or replacing (transplanting), cells/tissues. 2nd gen usually uses less mature technologies (eg cell therapy, regenerative medicine), but may have a bigger and faster effect size.
  • "function, feeling, and survival" are the endpoints that matter.
    • biomarkers are noisy and speculative early proxies that we merely hope will translate to a truly healthier life for the elderly. apply skepticism.
• https://substack.com/home/post/p-143303463 I always like what Maxim Raginsky has to say. you can't do AI without bumping into the philosophy of how to interpret what it's doing.

This is actually why a short position (a complicated loan) would theoretically work. If we all die, then you, as someone else's counterparty, never need to pay your loan back.

(I think this is a bad idea, but not because of counterparty risk)

I think the idea is that short position pays off up-front, and then you don't need to worry about the loan if everyone's dead.

If by paying off you mean this bet actually working I think you're right though. It seems more likely that the stock market would go up in the short term, forcing you to cover at a higher price and losing a bunch of money. And if the market stays flat, you'll still lose money on interest payments unless doom is coming this year.

the obvious thing to happen is that nvidia realizes it can just build AI itself. if Taiwan is Dune, GPUs are the spice, then nvidia is house Atreides

They've already started... 

Was there ever a time where CEA was focusing on truth-alignment? 

I doesn't seem to me like they used to be truth-aligned and then they did recruiting in a way that caused a value shift is a good explanation of what happened. They always optimized for PR instead of optimizing for truth-alignment. 

It's quite a while since they edited out Leverage Research on the photos that they published with their website, but the kind of organization where people consider it reasonable to edit photos that way is far from truth-aligned. 

Yes, my point here is mainly that the exponential decay seems almost baked into the setup even if we don't explicitly set it up that way, not that the decay is very notably stronger than it looks at first glance.

Given how many words have been spilled arguing over the philosophical validity of putting the decay with program length into the prior, this seems kind of important?

Thanks for writing this up. Some miscellaneous thoughts:

• My guess is that the safety property you'll want to ensure here is something like "the model is unlikely to succeed at a problematically large quantity of sabotage without us first getting a big (20:1) update that it's engaging in intentional sabotage".
• I think your internal usage restrictions are wildly unaffordable; it seems extremely unlikely that a sensible AI lab should agree to not deploying this model as a coding agent, or not use it for day-to-day tasks. My guess is that companies should treat the untrusted model similarly to how they treat external contractors (or how they treat employees at organizations that take insider threat seriously).

We're doing some research related to control measures for this kind of threat model, some of which should be out soon.

houghts, I want to comment on the interpretation of S(A) as an "intention" to do A.

Note that I'm coming back here from section 6. Awakening / Enlightenment / PNSE [LW · GW], so if somebody hasn't read that, this might be unclear.

Using the terminology above, A here is "the patterns of motor control and attention control outputs that would does collectively make my muscles actually execute the standing-up action."

And S(A) is "the patterns of motor control and attention control outputs that would does collectively make my muscles actually execute the standing-up action are in my awareness." Meaning a representation of "awareness" is active together with the container-relationship and a representation of A. (I am still very unsure about how "awareness" is learned and represented.)^[1]

Referring to 2.6.2, I agree with this:

[S(A) and A] are obviously strongly associated with each other. They can activate simultaneously. And even if they don’t, each tends to bring the other to mind, such that the valence of one influences the valence of the other.

and

For any action A where S(A) has positive valence, there’s often a two-step temporal sequence: [S(A) ; A actually happens]"

I agree that in this co-occurence sense "S(X) often summons a follow-on thought of X." But it is not causing it, what "summon" might imply. This choice of word is maybe an indication of the uncertainty here.

Clearly, action A can happen without S(A) being present. In fact, actions are often more effectively executed if you don't think too hard about them[citation needed]. An S(A) is not required. Maybe S(A) and A cooccur often, but that doesn't imply causality. But, indeed, it would seem to be causal in the context of a homunculus model of action. Treating it as causal/vitalistic is predictive. The real reason is the co-occurrence of the thoughts, which can have a common cause, such as when the S(A) thought brings up additional associations that lead to higher valence thoughts/actions later (e.g., chains of S(A), A, S(A)->S(B), B).

Thus, S(A) isn't really an "Intention to do A" per se but just as it says on the tin: "awareness of (expecting) A." I would say it is only an "intention to do A" if the thought S(A) also includes the concept of intention - which is a concept tied to the homunculus and an intuitive model of agency.

1. ^{^}

   I am still very unsure about how "awareness" is learned and represented. Above it says 

   the cortex, which has a limited computational capacity that gets deployed serially [...] When this aspect of the brain algorithm is itself incorporated into a generative model via predictive (a.k.a. self-supervised) learning, it winds up represented as an “awareness” concept,

   but this doesn't say how. The brain needs to observe something (sense, interoception) from which it can infer this. The pattern in what observations would that be? The serial processing is a property the brain can't observe unless there is some way to combine/compare past and present "thoughts." That's why I have long thought that there has to be a feedback from the current thought back as input signal (thoughts as observations). Such a connection is not present in the brain-like model [? · GW], but it might not be the only way. Another way would be via memory. If a thought is remembered, then one way of implementing memory would be to provide a representation of the remembered thought as input. In any case, there must be a relation between successive thoughts, otherwise they couldn't influence each other. 

   It seems plausible that, in a sequence of events, awareness S(A) is a related to a pattern of A having occurred previously in the sequence (or being expected to occur).

I think you are broadly right.

So we're automatically giving $x_1$ ca. $2^{1000}$ higher probability – even before applying the length penalty $\frac{1}{2^{|p|}}$.

But note that under the Solomonoff prior, you will get another $2^{-2000-|G|}$ penalty for these programs with DEADCODE. So with this consideration, the weight changes from $2^{-1000}$ (for normal $p_1$) to $2^{-1000}(1+2^{-|G|})$ (normal $p_1$ plus $2^{1000}$ DEADCODE versions of $p_1$), which is not a huge change.

For your case of "uniform probability until ${10}^{90}$" I think you are right about exponential decay.