LessWrong 2.0 Reader

According to Eliezar Yudkowsky [? · GW], your thoughts should reflect reality.

I expect that the more your beliefs track reality, the better you'll get at decision making, yes.

According to Paul Graham, the most successful people are slightly overconfident.

Ah but VCs benefit from the ergodicity of the startup founders! From the perspective of the founder, its a non-ergodic situation. Its better to make Kelly bets instead if you prefer to not fall into gambler's ruin, given whatever definition of the real world situation maps onto the abstract concept of being 'ruined' here.

It usually pays to have a better causal model of reality than relying on what X person says to inform your actions.

Can you think of anyone who has changed history who wasn’t a little overconfident?

Survivorship bias.

It is advantageous to be friends with the kind of people who do things and never give up.

I think I do things and never give up in general, while I can be pessimistic about specific things and tasks I could do. You can be generally extremely confident in yourself and your ability to influence reality, while also being specifically pessimistic about a wide range of existing possible things you could be doing.

Here's a Nate post that provides his perspective on this specific orientation to reality that leads to a sort of generalized confidence that has social benefits. [LW · GW]

Welcome! One gateway for you might be the LW Concepts page about it [? · GW]!

Most of the posts discuss, of course, infohazard policy and properties of information that would be harmful to know, or think about. Directly sharing blatantly harmful information would be irresponsible.

I would find it valuable if someone could gather an easy-to-read bullet point list of all the questionable things Sam Altman has done throughout the years.

I usually link to Gwern’s comment thread (https://www.lesswrong.com/posts/KXHMCH7wCxrvKsJyn/openai-facts-from-a-weekend?commentId=toNjz7gy4rrCFd99A [LW(p) · GW(p)]), but I would prefer if there was something more easily-consumable.

This was fun to see:

More than a quarter of the applications answered it anyway.

I wonder how many of that 25% simply missed the note. People make mistakes like this all the time. And I also wonder how many people noticed this before feeding it to their AI.

It's fun to see until you think about how many of that 25% - already jobless and kinda depressed and with very little ability to tell Contra to take any exploitative or manipulative hiring practices and shove them - saw the note and concluded it was an oversight, an obvious arbitrary filter for applicants who wanted to put in less effort, or some other stupid-clever scheme by Contra's hiring managers; and thus decided that it'd be safest to just answer it anyway.

+1. Also curious about Jade Leung (formerly Open AI)– she's currently the CTO for the UK AI Safety Institute. Also Geoffrey Irving (formerly DeepMind), who is a research director at the UKAISI.

It’s a funny comment because legally Conjecture is for-profit and OpenAI is not. It just goes to show that the various internal and external pressures and incentives on an organization and its staff are not encapsulated by glancing at their legal status—see also my comment here [LW(p) · GW(p)].

Anyway, I don’t think Connor is being disingenuous in this particular comment, because he has always been an outspoken advocate for government regulation of all AGI-related companies including his own.

I don’t think it’s crazy or disingenuous in general to say “This is a terrible system, so we’re gonna loudly criticize it and advocate to change it. But meanwhile / in parallel, we’re gonna work within the system we got, and do the best we can.” And “the system we got” is that private organizations are racing to develop AGI.

We also recently implemented a non-compliance reporting policy that allows employees to anonymously report concerns to our Responsible Scaling Officer about our implementation of our RSP.

This seems great, and I think it would be valuable for other labs to adopt a similar system. 

What about whistleblowing or anonymous reporting to governments? If an Anthropic employee was so concerned about RSP implementation (or more broadly about models that had the potential to cause major national or global security threats), where would they go in the status quo? 

If Anthropic is supportive of this kind of mechanism, it might be good to explicitly include this (e.g., "We also recently implemented non-compliance reporting policy that allows employees to anonymously report concerns to specific officials at the [Department of Homeland Security, Department of Commerce].") 

Minor note: I think we should probably avoid using the term "RSP" to refer to the broad class of things. I would prefer "voluntary safety commitments" or something like that. 

RSP is both a bad name IMO (it has "responsible" in the name, whereas whether the scaling is actually "responsible" is something for people & governments to evaluate) and also gets a little confusing because Anthropic's doc is called an RSP.

I like "voluntary safety commitments" as the broad class of things, and then you can use the company-specific terminology when referring to the company's particular flavor (RSP for Anthropic, PF for OpenAI, etc.)

Commentary on the content of the commitments: shrug. Good RSPs are great but probably require the right spirit to be implemented well, and most of these companies don't employ people who work on scalable alignment, evaluating dangerous capabilities, etc. And people have mostly failed to evaluate existing RSP-ish plans well; if a company makes a basically meaningless RSP, people might not notice.

Strong agree. I mostly see these commitments as useful foundations for domestic and international regulation. If the government is checking or evaluating these criteria, they matter. If not, it seems very easy for companies to produce nice-sounding documents that don't contain meaningful safety plans (see also affirmative safety, safety cases).

With that in mind, Alex Lawsen (on X) noticed some good things about the specifics of the commitments:

From my understanding, the bayesian aspect of this agenda doesn't add much value.

A core advantage of Bayesian methods is the ability to handle out-of-distribution situations more gracefully, and this is arguably as important as (if not more important than) interpretability. In general, most (?) AI safety problems can be cast as an instance of a case where a model behaves as intended on a training distribution, but generalises in an unintended way when placed in a novel situation. Traditional ML has no straightforward way of dealing with such cases, since it only maintains a single hypothesis at any given time. However, Bayesian methods may make it less likely that a model will misgeneralise, or should at least give you a way of detecting when this is the case.

I also don't agree with the characterisation that "almost all the interesting work is in the step where we need to know whether a hypothesis implies harm" (if I understand you correctly). Of course, creating a formal definition or model of "harm" is difficult, and creating a world model is difficult, but once this has been done, it may not be very hard to detect if a given action would result in harm. But I may not understand your point in the intended way.

manually build an (interpretable) infra-bayesian world model which is sufficiently predictive of the world (as smart as our AI)

I don't think this is an accurate description of davidad's plan. Specifically, the world model does not necessarily have to be built manually, and it does not have to be as good at prediction as our AI. The world model only needs to be good at predicting the variables that are important for the safety specification(s), within the range of outputs that the AI system may produce.

Proof checking on this world model also seems likely to be unworkable

I agree that this is likely to be hard, but not necessarily to the point of being unworkable. Similar things are already done for other kinds of software deployed in complex contexts, and ASL-2/3 AI may make this substantially easier.