LessWrong 2.0 Reader

However, this would not address the underlying pattern of alignment failing to generalize.

Is there proof that this is an overall pattern? It would make sense that models are willing to do things they're not willing to talk about, but that doesn't mean there's a general pattern that e.g. they wouldn't be willing to talk about things, and wouldn't be willing to do them, but WOULD be willing to some secret third option.

The reasoning for the indistinguishability criterion is as a precaution against a model sandbagging when used as the red team attack model. If the model can tell it’s in a control eval it could pretend to be worse at attacking, making us think the control measures are better than they actually are. If the red team elicitation is robust to this, I agree this isn’t necessary. (See the note below [P2.4]).

Fuck Google, seriously. About once a week it asks me whether I want to "backup my photos in the cloud", and I keep clicking no, because fuck you why would I want to upload my private photos on your company servers.

But apparently I accidentally once clicked yes (maybe), because suddenly Google sends me a notification about how it created a beautiful animation of my recent photos in the cloud, offering me the option to download them. I don't want to download my private photos from the fucking Google cloud, I never wanted them to be there in the first place! I want to click the delete button, but it's not there: it's either download the animation from the cloud, or close the dialog.

Of course, turning off the functionality is at least 10x more difficult than turning it on, so I get ready to spend this evening finding the advice online and configuring my phone to stop uploading my private photos to Google servers, and preferably to delete all the photos that are already there despite my wishes. Does the "delete" option even exist anymore, or is there just "move to recycle bin (where it stays for as long as we want it to stay there)"? Today I will find out.

Again, fuck Google. I hope the company burns down. I wonder what other things I have already accidentally "consented" to. Google's idea of consent is totally rapist. And I only found this out by accident. In future, I expect to accidentally find this or some other "optional" feature turned on again.

J. D. Vance's (may he live forever) tweets about AI safety and open source (from March 3, 2024), replying to Vinod Khosla's advocacy for more centralized control:

link

There are undoubtedly risks related to AI. One of the biggest: 

A partisan group of crazy people use AI to infect every part of the information economy with left wing bias. Gemini can’t produce accurate history. ChatGPT promotes genocidal concepts. 

The solution is open source

and link

If Vinod really believes AI is as dangerous as a nuclear weapon, why does ChatGPT have such an insane political bias? If you wanted to promote bipartisan efforts to regulate for safety,  it's entirely counterproductive. 

Any moderate or conservative who goes along with this obvious effort to entrench insane left-wing businesses is a useful idiot. 

I'm not handing out favors to industrial-scale DEI bullshit because tech people are complaining about safety.

1. Yepp, seems reasonable. Though FYI I think of this less as some special meta argument, and more as the common-sense correction that almost everyone implicitly does when giving credences, and rationalists do less than most. (It's a step towards applying outside view, though not fully "outside view".)

2. Yepp, agreed, though I think the common-sense connotations of "if this became" or "this would have a big effect" are causal, especially in the context where we're talking to the actors who are involved in making that change. (E.g. the non-causal interpretation of your claim feels somewhat analogous to if I said to you "I'll be more optimistic about your health if you take these pills", and so you take the pills, and then I say "well the pills do nothing but now I'm more optimistic, because you're the sort of person who's willing to listen to recommendations". True, but it also undermines people's willingness/incentive to listen to my claims about what would make the world better.)

3. Here are ten that affect AI risk as much one way or the other:

1. The US government "waking up" a couple of years earlier or later (one operationalization: AISIs existing or not right now).
2. The literal biggest names in the field of AI becoming focused on AI risk.
3. The fact that Anthropic managed to become a leading lab (and, relatedly, the fact that Meta and other highly safety-skeptical players are still behind).
4. Trump winning the election.
5. Elon doing all his Elon stuff (like founding x.AI, getting involved with Trump, etc).
6. The importance of transparency about frontier capabilities (I think of this one as more of a logical update that I know you've made).
7. o1-style reasoning as the next big breakthrough.
8. Takeoff speeds (whatever updates you've made in the last three years).
9. China's trajectory of AI capabilities (whatever updates you've made about that in last 3 years).
10. China's probability of invading Taiwain (whatever updates you've made about that in last 3 years).

And then I think in 3 years we'll be able to publish a similar list of stuff that mostly we just hadn't predicted or thought about before now.

I expect you'll dispute a few of these; happy to concede the ones that are specifically about your updates if you disagree (unless you agree that you will probably update a bunch on them in the next 3 years).

But IMO the easiest way for safety cases to become the industry-standard thing is for AISI (or internal safety factions) to specifically demand it, and then the labs produce it, but kinda begrudgingly, and they don't really take them seriously internally (or are literally not the sort of organizations that are capable of taking them seriously internally—e.g. due to too much bureaucracy). And that seems very much like the sort of change that's comparable to or smaller than the things above.

I think I would be more sympathetic to your view if the claim were "if AI labs really reoriented themselves to take these AI safety cases as seriously as they take, say, being in the lead or making profit". That would probably halve my P(doom), it's just a very very strong criterion.

links 11/08/2024: https://roamresearch.com/#/app/srcpublic/page/11-08-2024

• https://agingbiotech.info/about/ a database of aging biotech companies compiled by Karl Pfleger
• https://longevitylist.com/longevity-industry-database/ a database of aging biotech companies compiled by Nathan Cheng, includes somewhat different picks
• GLP-1 receptor agonist drugs reduce all-cause mortality -- so what diseases or causes of death do they prevent?
  • https://www.nature.com/articles/s41467-024-50199-y kidney disease (in type-2 diabetes patients with kidney disease)
  • https://www.ajmc.com/view/glp-1s-reduce-cardiovascular-risk-equally-in-patients-with-overweight-obesity-regardless-of-diabetes cardiovascular disease (in overweight or obese patients)
    • https://journals.sagepub.com/doi/pdf/10.1177/17562864241281903
  • https://www.science.org/doi/10.1126/science.adn4128 (sadly I couldn't find the full article)
  • https://www.ingentaconnect.com/content/ben/cdr/2018/00000014/00000003/art00008 cardiovascular disease (in diabetics)
• https://wonder.cdc.gov/controller/datarequest/D176;jsessionid=C53D7110417D14C262ECD70F0091 what are the leading causes of death in 2023?
  • heart disease, cancer, accidents, stroke, COPD, Alzheimer's, diabetes, kidney disease, liver disease, COVID-19, suicide, influenza & pneumonia, hypertension, septicemia, Parkinson's
  • surprised suicide was so high and that COVID-19 was still so deadly (I assume mostly in the elderly)
• https://www.fiercebiotech.com/biotech/bioage-brings-almost-200m-ipo-obesity-biotech-joins-nasdaq BioAge IPO
• I forgot that Sam Altman invested in Retro Bio
  • https://www.technologyreview.com/2023/03/08/1069523/sam-altman-investment-180-million-retro-biosciences-longevity-death/
  • the man has good taste. like, it's not blindingly original to appreciate Retro, but it is eminently reasonable.
• there's a lot of moderate-Democrat post-election resignation to the effect of "this is what the country wanted; the median voter is in fact pretty OK with Trump" and "the progressive apparatus was more interested in staying in its comfort zone than winning elections"
  • https://substack.com/home/post/p-151278372 Jesse Singal
    • he was saying similar things all along: https://jessesingal.substack.com/p/democrats-should-acknowledge-reality
• I'm also seeing a fair number of women going "ok, sure, there are things to criticize about feminist dogma, but actually I have experienced traditionalist religious mores and they were Not Good", which I think is a needed corrective these days
  • https://substack.com/home/post/p-141175575 here's Audrey Horne
• https://backofmind.substack.com/p/incompetence-is-a-form-of-bias Dan Davies says incompetence is a form of bias -- the people who have the social skills and clout to get their problems fixed, will.
• Dan Davies on politics and populism...i'm not sure where he's going here but this is intriguing.
  • https://substack.com/home/post/p-151264334
• https://esmeralda.org/ Esmerelda, Devon Zeugel's Chautauqua-inspired village in California

Why do most humans in 2041 still need to work 40 hours a week? The answer is complicated, but to keep this comment simple, let's focus on a few factors that even a hypothetical reader from 2024 would understand.

In most countries, government regulation requires humans in the loop. These might seem like bullshit jobs, but that doesn't make the competition for them any less fierce. An average person cannot get a good job without good credentials (required for regulatory reasons), and good credentials are expensive; it often takes a lifetime to pay back the school debt. It doesn't matter whether the things taught at school are useful in any practical sense (the few remaining human teachers mostly agree that they are not), but they are required by law. The official reasoning is that general education keeps us human (note: this is simplified to the level of strawman, but I am trying to keep it simple for the hypothetical 2024 reader unfamiliar with the culture wars of 2041).

With the exception of a few things such as rent, most things today are significantly cheaper than they used to be in 2024. On the other hand, there are new expenses, many of them related to AI. Some aspects of life got complicated, for example contracts of all kinds. To put it bluntly, you need the latest AI to safely navigate the legal minefield created by the latest AI. Trying to save money by using a cheaper version of AI that is several weeks obsolete is generally considered a very bad idea, and will probably cost you more in long run, because you have no idea what you sign (and you should generally assume that the form was optimized to extract as much value from you as legally possible, otherwise the company would be leaving money on table). You either spend a large part of your income on AI services... or you risk joining the underclass at the first accident; there is not much of a middle way. If you can't afford the "business version" of the latest AI, you can get one that is supported by advertising -- the less you pay for it, the more you should expect the AI agent to optimize for the goals of the advertisers rather than your personal goals. (Oh, "advertisement" today no longer means trying to influence the humans. Humans are mostly irrelevant. It means influencing the AI agents that make most of the everyday decisions. As a simple example, you can pay the AI agents to buy your products rather than your competitor's products, even if they are somewhat more expensive or worse, and to defend this choice to human users using individually optimized arguments.)

There is increasingly addictive... well, basically everything. I am afraid that a far [? · GW]-mode description will fail to convey how strong the effect is when experienced in near mode, but basically: The salesmen of old have used only a few dozen simple techniques (such as smiling at you, looking in your eyes, repeating your name, trying to anchor you to a higher price and then giving you a discount, creating a false sense of urgency, etc.) which were only statistically effective and often failed or backfired for you, the modern ones come to you with a full AI-powered analysis of your personality (yes, there are regulations against this, but they are trivially circumvented), and they have probably already spent a few previous months trying to influence you in all known ways (bots pretending to be humans contacting you on social networks and nudging you in the desired direction, advertising in your AI agent if you use the cheaper version, subliminal advertising on the streets flashing when the screen detects you looking at it, etc.) which makes is almost impossible to resist; in many cases the humans believe that the interaction was actually their own idea, and quite often they fall in love with the salesperson.

Some people suggest that this is a problem humanity should focus on solving, but the respected economists (and more importantly, their AI advisors) mostly shrug and say: "revealed preferences".

Yudkowsky has written about it:

(...) In standard metaethical terms, we have managed to rescue 'moral cognitivism' (statements about rightness have truth-values) and 'moral realism' (there is a fact of the matter out there about how right something is). We have not however managed to rescue the pretheoretic intuition underlying 'moral internalism' (...)

You can't trust exit polls on demographics crosstabs. From Matt Yglesias on Slow Boring:

Over and above the challenge inherent in any statistical sampling exercise, the basic problem exit pollsters have is that they have no way of knowing what the electorate they are trying to sample actually looks like, but they do know who won the election. They end up weighting their sample to match the election results, which is good because otherwise you’d have polling error about the topline outcome, which would look absurd. But this weighting process can introduce major errors in the crosstabs.

For example, the 2020 exit poll sample seems to have included too many college- educated white people. That was a Biden-leaning demographic group, so in a conventional poll, it would have simply exaggerated Biden’s share of the total vote. But the exit poll knows the “right answer” for Biden’s aggregate vote share, so to compensate for overcounting white college graduates in the electorate, it has to understate Biden’s level of support within this group. That is then further offset by overstating Biden’s level of support within all other groups. So we got a lot of hot takes in the immediate aftermath of the election about Biden’s underperformance with white college graduates, which was fake, while people missed real trends, like Trump doing better with non-white voters.

To get the kind of data that people want exit polls to deliver, you actually need to wait quite a bit for more information to become available from the Census and the voter files about who actually voted. Eventually, Catalist produced its “What Happened in 2020” document, and Pew published its “Behind Biden’s 2020 Victory” report. But those take months to assemble, and unfortunately, conventional wisdom can congeal in the interim.

So just say no to exit poll demographic analysis!

Finally, note to self, probably still don’t use SQLite if you have a good alternative? Twice is suspicious, although they did fix the bug same day and it wasn’t ever released.

SQLite is well-known for its incredibly thorough test suite and relatively few CVEs, and with ~156kloc (excluding tests) it's not a very large project, so I think this would be an over-reaction. I'd guess that other databases have more and worse security vulnerabilities due to their attack surface—see MySQL with its ~4.4mloc (including tests). Big Sleep was probably now used on SQLite because it's a fairly small project of which large parts can fit into an LLMs' context window.

Maybe someone will try to translate the SQLite code to Rust or Zig using LLMs—until then we're stuck.