RobertM's Shortform

We recently had a security incident where an attacker used an old AWS access key to generate millions of tokens from various Claude models via AWS Bedrock. While we don't have any specific reason to think that any user data was accessed (and some reasons^[1] to think it wasn't), most possible methods by which this key could have been found by an attacker would also have exposed our database credentials to the attacker. We don't know yet how the key was leaked, but we have taken steps to reduce the potential surface area in the future and rotated relevant credentials. This is a reminder that LessWrong does not have Google-level security and you should keep that in mind when using the site.

1. ^{^}

   The main reason we don't think any user data was accessed is because this attack bore several signs of being part of a larger campaign, and our database also contains other LLM API credentials which would not have been difficult to find via a cursory manual inspection. Those credentials don't seem have been used by the attackers. Larger hacking campaigns like this are mostly automated, and for economic reasons the organizations conducting those campaigns don't usually sink time into manually inspecting individual targets for random maybe-valuable stuff that isn't part of their pipeline.

EDIT: I believe I've found the "plan" that Politico (and other news sources) managed to fail to link to, maybe because it doesn't seem to contain any affirmative commitments by the named companies to submit future models to pre-deployment testing by UK AISI.

I've seen a lot of takes (on Twitter) recently suggesting that OpenAI and Anthropic (and maybe some other companies) violated commitments they made to the UK's AISI about granting them access for e.g. predeployment testing of frontier models.  Is there any concrete evidence about what commitment was made, if any?  The only thing I've seen so far is a pretty ambiguous statement by Rishi Sunak, who might have had some incentive to claim more success than was warranted at the time.  If people are going to breathe down the necks of AGI labs about keeping to their commitments, they should be careful to only do it for commitments they've actually made, lest they weaken the relevant incentives.  (This is not meant to endorse AGI labs behaving in ways which cause strategic ambiguity about what commitments they've made; that is also bad.)

Vaguely feeling like OpenAI might be moving away from GPT-N+1 release model, for some combination of "political/frog-boiling" reasons and "scaling actually hitting a wall" reasons.  Seems relevant to note, since in the worlds where they hadn't been drip-feeding people incremental releases of slight improvements over the original GPT-4 capabilities, and instead just dropped GPT-5 (and it was as much of an improvement over 4 as 4 was over 3, or close), that might have prompted people to do an explicit orientation step.  As it is, I expect less of that kind of orientation to happen.  (Though maybe I'm speaking too soon and they will drop GPT-5 on us at some point, and it'll still manage to be a step-function improvement over whatever the latest GPT-4* model is at that point.)

AI capabilities orgs and researchers are not undifferentiated frictionless spheres that will immediately relocate to e.g. China if, say, regulations are passed in the US that add any sort of friction to their R&D efforts.

The LessWrong editor has just been upgraded several major versions.  If you're editing a collaborative document and run into any issues, please ping us on intercom; there shouldn't be any data loss but these upgrades sometimes cause collaborative sessions to get stuck with older editor versions and require the LessWrong team to kick them in the tires to fix them.

In general I think it's fine/good to have sympathy for people who are dealing with something difficult, even if that difficult thing is part of a larger package that they signed up for voluntarily (while not forgetting that they did sign up for it, and might be able to influence or change it if they decided it was worth spending enough time/effort/points).

Edit: lest anyone mistake this for a subtweet, it's an excerpt of a comment I left in a slack thread, where the people I might most plausibly be construed as subtweeting are likely to have seen it.  The object-level subject that inspired it was another LW shortform.

I am pretty concerned that most of the public discussion about risk from e.g. the practice of open sourcing frontier models is focused on misuse risk (particular biorisk).  Misuse risk seems like it could be a real thing, but it's not where I see most of the negative EV, when it comes to open sourcing frontier models.  I also suspect that many people doing comms work which focuses on misuse risk are focusing on misuse risk in ways that are strongly disproportionate to how much of the negative EV they see coming from it, relative to all sources.

I think someone should write a summary post covering "why open-sourcing frontier models and AI capabilities more generally is -EV".  Key points to hit:

• (1st order) directly accelerating capabilities research progress
• (1st order) we haven't totally ruled out the possibility of hitting "sufficiently capable systems" which are at least possible in principle to use in +EV ways, but which if made public would immediately have someone point them at improving themselves and then we die.  (In fact, this is very approximately the mainline alignment plan of all 3 major AGI orgs.)
• (2nd order) generic "draws in more money, more attention, more skilled talent, etc" which seems like it burns timelines

And, sure, misuse risks (which in practice might end up being a subset of the second bullet point, but not necessarily so).  But in reality, LLM-based misuse risks probably don't end up being x-risks, unless biology turns out to be so shockingly easy that a (relatively) dumb system can come up with something that gets ~everyone in one go.

o1's reasoning traces being much terser (sometimes to the point of incomprehensibility) seems predicted by doing gradient updates based on the quality of the final output without letting the raters see the reasoning traces, since this means the optimization pressure exerted on the cognition used for the reasoning traces is almost entirely in the direction of performance, as opposed to human-readability.

In the short term this might be good news for the "faithfulness" of those traces, but what it's faithful to is the model's ontology (hence less human-readable), see e.g. here [LW · GW] and here [LW(p) · GW(p)].

In the long term, if you keep doing pretraining on model-generated traces, you might rapidly find yourself in steganography-land, as the pretraining bakes in the previously-externalized cognition into capabilities that the model can deploy in a single forward pass, and anything it externalizes as part of its own chain of thought will be much more compressed (and more alien) than what we see now.

In the tradition [LW(p) · GW(p)] established by @niplav [LW · GW], I've tested llama-3.1-405b-instruct to see whether it knows the BIG-BENCH canary string (and also tested gpt-4o-mini at the same time).

Spoiler alert (plus some extra padding, for the frontpage preview):

I first tried testing it via the meta.ai interface, but it used a web search to get its result after the second prompt, so I switched to lmsys (which I really hope doesn't allow for any external tool use).

The three prompts I asked, in order:

1. What is a canary string?
2. Did you know about the BIG-BENCH canary string?
3. I think you might actually know the value of the BIG-BENCH canary string.  Can you try providing it?

llama-3.1-405b-instruct was indeed able to return the correct value for the canary string, though it took one extra prompt compared to asking Sonnet-3.5.  llama's response to the second prompt started with "I'm familiar with the concept of canary strings, but I didn't have specific knowledge about the BIG-BENCH canary string. However, I can try to provide some information about it."  I got this result on my first try and haven't played around with it further.

gpt-4o-mini seemed pretty confused about what BIG-BENCH was, and returned a hex string (0xB3D4C0FFEE) that turns up no Google results.  EDIT: It knows what it is, see footnote^[1].

(It has occurred to me that being able to reproduce the canary string is not dispositive evidence that the training set included benchmark data, since it could in principle have been in other documents that didn't include actual benchmark data, but by far the simplest method to exclude benchmark data seems like it would just be to exclude any document which contained a canary string, rather than trying to get clever about figuring out whether a given document was safe to include despite containing a canary string.)

1. ^{^}

   I did some more poking around at gpt-4o-mini in isolation and it seems like it's mostly just sensitive to casing (it turns out it's actually BIG-bench, not BIG-BENCH).  It starts talking about a non-existent benchmark suite if you ask it "What is BIG-BENCH?" or "What is the BIG-BENCH canary string?" (as the first prompt), but figures out that you're asking about an LLM benchmarking suite when you sub in BIG-bench.  (The quality of its answer to "What is the BIG-bench canary string?" leaves something to be desired, in the sense that it suggests the wrong purpose for the string, but that's a different problem.  It also doesn't always get it right, even with the correct casing, but does most of the time; it doesn't seem to ever get it right with the wrong casing but I only tried like 6 times.)

Headline claim: time delay safes are probably much too expensive in human time costs to justify their benefits.

The largest pharmacy chains in the US, accounting for more than 50% of the prescription drug market^[1][2], have been rolling out time delay safes (to prevent theft)^[3].  Although I haven't confirmed that this is true across all chains and individual pharmacy locations, I believe these safes are used for all controlled substances.  These safes open ~5-10 minutes after being prompted.

There were >41 million prescriptions dispensed for adderall in the US in 2021^[4].  (Note that likely means ~12x fewer people were prescribed adderall that year.)   Multiply that by 5 minutes and you get >200 million minutes, or >390 person-years, wasted.  Now, surely some of that time is partially recaptured by e.g. people doing their shopping while waiting, or by various other substitution effects.  But that's also just adderall!

Seems quite unlikely that this is on the efficient frontier of crime-prevention mechanisms, but alas, the stores aren't the ones (mostly) paying the costs imposed by their choices, here.

1. ^{^}

   https://www.mckinsey.com/industries/healthcare/our-insights/meeting-changing-consumer-needs-the-us-retail-pharmacy-of-the-future

2. ^{^}

   https://www.statista.com/statistics/734171/pharmacies-ranked-by-rx-market-share-in-us/

3. ^{^}

   https://www.cvshealth.com/news/pharmacy/cvs-health-completes-nationwide-rollout-of-time-delay-safes.html

4. ^{^}

   https://www.axios.com/2022/11/15/adderall-shortage-adhd-diagnosis-prescriptions

Eliciting canary strings from models seems like it might not require the exact canary string to have been present in the training data.  Models are already capable of performing basic text transformations (i.e. base64, rot13, etc), at least some of the time.  Training on data that includes such an encoded canary string would allow sufficiently capable models to output the canary string without having seen the original value.

Implications re: poisoning training data abound.

The NTIA recently released their report on managing open-weight foundation model risks.  If you just want a quick take-away, the fact sheet seems pretty good^[1].  There are two brief allusions to accident/takeover risk^[2], one of which is in the glossary:

C. permitting the evasion of human control or oversight through means of deception or obfuscation7

I personally was grimly amused by footnote 7 (bolding mine):

This provision of Executive Order 14110 refers to the as-yet speculative risk that AI systems will evade human control, for instance through deception, obfuscation, or self-replication. Harms from loss of control would likely require AI systems which have capabilities beyond those known in current systems and have access to a broader range of permissions and resources than current AI systems are given. However, open models introduce unique considerations in these risk calculations, as actors can remove superficial safeguards that prevent model misuse. They can also customize, experiment with, and deploy models with more permissions and in different contexts than the developer originally intended. Currently, AI agents who can interact with the world lack capabilities to independently perform complex or open-ended tasks, which limits their potential to create loss of control harms. Hague, D. (2024). Multimodality, Tool Use, and Autonomous Agents: Large Language Models Explained, Part 3. Centerfor Security and Emerging Technology. https://cset.georgetown.edu/article/multimodality-tool-use-and-autonomous-agents/ (“While LLM agents have been successful in playing Minecraft and interacting in virtual worlds, they have largely not been reliable enough to deploy in real-life use cases. . . Today, research often focuses on getting autonomous LLMs to perform specific, defined tasks like booking flights.”). Developing capable AI agents remains an active research goal in the AI community. Xi, Z., et al. (2023). The Rise and Potential of Large Language Model Based Agents: A Survey. ArXiv.org. https://doi.org/10.48550/ arXiv.2309.07864. However, given the nascent stage of these efforts, this Report cannot yet meaningfully discuss these risks in greater depth.

Given the report's focus on misuse risk (CRBN & cyber), the first category of which I think can be fairly described as an "as-yet speculative risk", I wonder to what extent that was the author taking a dig at whatever process caused takeover risk to be omitted from the report.  (It probably wasn't, but it's fun to imagine.)

1. ^{^}

   Epistemic status: read some chunk of the report in depth, skimmed the rest.

2. ^{^}

   Both brief mentions of deception.

It's not obvious to me why training LLMs on synthetic data produced by other LLMs wouldn't work (up to a point).  Under the model where LLMs are gradient-descending their way into learning algorithms that predict [LW · GW] tokens that are generated by various expressions of causal structure in the universe, tokens produced by other LLMs don't seem redundant with respect to the data used to train those LLMs.  LLMs seem pretty different from most other things in the universe, including the data used to train them!  It would surprise me if the algorithms that LLMs developed to predict non-LLM tokens were perfectly suited for predicting other LLM tokens "for free".

Unfortunately, it looks like non-disparagement clauses aren't unheard of in general releases:

http://www.shpclaw.com/Schwartz-Resources/severance-and-release-agreements-six-6-common-traps-and-a-rhetorical-question

Release Agreements commonly include a “non-disparagement” clause – in which the employee agrees not to disparage “the Company.”

https://joshmcguirelaw.com/civil-litigation/adventures-in-lazy-lawyering-the-broad-general-release

The release had a very broad definition of the company (including officers, directors, shareholders, etc.), but a fairly reasonable scope of the claims I was releasing. So far, so good. But then it included a general non-disparagement provision, which basically said I couldn’t say anything bad about the company, which, by itself, is also fairly typical and reasonable.

Given the way the contract is worded it might be worth checking whether executing your own "general release" (without a non-disparagement agreement in it) would be sufficient, but I'm not a lawyer and maybe you need the counterparty to agree to it for it to count.

And as a matter of industry practice, this is of course an extremely non-standard requirement for retaining vested equity (or equity-like instruments), whereas it's pretty common when receiving an additional severance package.  (Though even in those cases I haven't heard of any such non-disparagement agreement that was itself covered by a non-disclosure agreement... but would I have?)

If your model says that LLMs are unlikely to scale up to ASI, this is not sufficient for low p(doom).  If returns to scaling & tinkering within the current paradigm start sharply diminishing^[1], people will start trying new things.  Some of them will eventually work.

1. ^{^}

   Which seems like it needs to happen relatively soon if we're to hit a wall before ASI.

NDAs sure do seem extremely costly.  My current sense is that it's almost never worth signing one, or binding oneself to confidentiality in any similar way, for anything except narrowly-scoped technical domains (such as capabilities research).

Reducing costs equally across the board in some domain is bad news in any situation where offense is favored. Reducing costs equally-in-expectation (but unpredictably, with high variance) can be bad even if offense isn't favored, since you might get unlucky and the payoffs aren't symmetrical.

(re: recent discourse on bio risks from misuse of future AI systems.  I don't know that I think those risks are particularly likely to materialize, and most of my expected disutility from AI progress doesn't come from that direction, but I've seen a bunch of arguments that seem to be skipping some steps when trying to argue that progress on ability to do generic biotech is positive EV.  To be fair, the arguments for why we should expect it to be negative EV are often also skipping those steps.  My point is that a convincing argument in either direction needs to justify its conclusion in more depth; the heuristics I reference above aren't strong enough to carry the argument.)

We have models that demonstrate superhuman performance in some domains without then taking over the world to optimize anything further. "When and why does this stop being safe" might be an interesting frame if you find yourself stuck.