LessWrong 2.0 Reader

Thanks for these clarifications. I didn't realize that the 30% was for the new yellow-line evals rather than the current ones.

That's how I was thinking about the predictions that I was making; others might have been thinking about the current evals where those were more stable.

I'm having trouble parsing this sentence. What you mean by "doing so only risks the costs of pausing when we could have instead prepared mitigations or better evals"? Doesn't pausing include focusing on mitigations and evals?

Of course, but pausing also means we'd have to shuffle people around, interrupt other projects, and deal with a lot of other disruption (the costs of pausing). Ideally, we'd continue updating our yellow-line evals to stay ahead of model capabilities until mitigations are ready.

I suspect there is much more to this thread, and it may tie back to Superalignment & broken promises about compute-quotas.

The Superalignment compute-quota flashpoint is now confirmed. Aside from Jan Leike explicitly calling out compute-quota shortages post-coup (which strictly speaking doesn't confirm shortages pre-coup), Fortune is now reporting that this was a serious & longstanding issue:

...According to a half-dozen sources familiar with the functioning of OpenAI’s Superalignment team, OpenAI never fulfilled its commitment to provide the team with 20% of its computing power.

Instead, according to the sources, the team repeatedly saw its requests for access to graphics processing units, the specialized computer chips needed to train and run AI applications, turned down by OpenAI’s leadership, even though the team’s total compute budget never came close to the promised 20% threshold.

The revelations call into question how serious OpenAI ever was about honoring its public pledge, and whether other public commitments the company makes should be trusted. OpenAI did not respond to requests to comment for this story.

...It was a task so important that the company said in its announcement that it would commit “20% of the compute we’ve secured to date over the next four years” to the effort. But a half-dozen sources familiar with the Superalignment team’s work said that the group was never allocated this compute. Instead, it received far less in the company’s regular compute allocation budget, which is reassessed quarterly.

One source familiar with the Superalignment team’s work said that there were never any clear metrics around exactly how the 20% amount was to be calculated, leaving it subject to wide interpretation. For instance, the source said the team was never told whether the promise meant “20% each year for four years” or “5% a year for four years” or some variable amount that could wind up being “1% or 2% for the first three years, and then the bulk of the commitment in the fourth year.” In any case, all the sources Fortune spoke to for this story confirmed that the Superalignment team was never given anything close to 20% of OpenAI’s secured compute as of July 2023.

OpenAI researchers can also make requests for what is known as “flex” compute—access to additional GPU capacity beyond what has been budgeted—to deal with new projects between the quarterly budgeting meetings. But flex requests from the Superalignment team were routinely rejected by higher-ups, these sources said.

Bob McGrew, OpenAI’s vice president of research, was the executive who informed the team that these requests were being declined, the sources said, but others at the company, including chief technology officer Mira Murati, were involved in making the decisions. Neither McGrew nor Murati responded to requests to comment for this story.

While the team did carry out some research—it released a paper detailing its experiments in successfully getting a less powerful AI model to control a more powerful one in December 2023—the lack of compute stymied the team’s more ambitious ideas, the source said. After resigning, Leike on Friday published a series of posts on Twitter in which he criticized his former employer, saying “safety culture and processes have taken a backseat to shiny products.” He also said that “over the past few months my team has been sailing against the wind. Sometimes we were struggling for compute and it was getting harder and harder to get this crucial research done.”

5 sources familiar with the Superalignment team’s work backed up Leike’s account, saying that the problems with accessing compute worsened in the wake of the pre-Thanksgiving showdown between Altman and the board of the OpenAI nonprofit foundation.

...One source disputed the way the other sources Fortune spoke to characterized the compute problems the Superalignment team faced, saying they predated Sutskever’s participation in the failed coup, plaguing the group from the get-go.

While there have been some reports that Sutskever was continuing to co-lead the Superalignment team remotely, sources familiar with the team’s work said this was not the case and that Sutskever had no access to the team’s work and played no role in directing the team after Thanksgiving. With Sutskever gone, the Superalignment team lost the only person on the team who had enough political capital within the organization to successfully argue for its compute allocation, the sources said.

...The people who spoke to Fortune did so anonymously, either because they said they feared losing their jobs, or because they feared losing vested equity in the company, or both. Employees who have left OpenAI have been forced to sign separation agreements that include a strict non-disparagement clause that says the company can claw back their vested equity if they criticize the company publicly, or if they even acknowledge the clause’s existence. And employees have been told that anyone who refuses to sign the separation agreement will forfeit their equity as well.

Supposing he was a serious contributor to the paper (which seems unlikely IMO), it seems bad to cut his contribution just because he died.

So, I think the right choice here will depend on how much being an author on this paper is about endorsement or about contribution.

(Even if he didn't contribute much of the content, I still think it might be fine to keep him as an author.)

It's unfortunate that authorship can mean these two pretty different things.

I think Yann LeCun thinks "AGI in 2040 is perfectly plausible", AND he believes "AGI is so far away it's not worth worrying about all that much". It's a really insane perspective IMO. As recently as like 2020, "AGI within 20 years" was universally (correctly) considered to be a super-soon forecast, as contrasted with the people who say "centuries".

The target audience for Soylent is much weirder. Although TBF I originally thought the Soylent branding was a bad idea and I was probably wrong.

I feel kind of conflicted about including Daniel Kahneman on this paper. I feel like by the standards of scientific publication, you can't really have a co-author who wasn't alive or available to give their consent to the contents of the paper when the final version was released. Like, I don't know what edits were made to this paper since Kahneman's death, and it was of course not possible to get his consent for any of these edits.

Plausible the paper was actually mostly written when they got Daniel Kahneman's input, but I do assign some probability to it now saying things that Daniel would not have actually endorsed, and that seems bad.

it seems to me that Anthropic has so far failed to apply its interpretability techniques to practical tasks and show that they are competitive

Do you not consider the steering examples in the recent paper to be a practical task, or do you think that competitiveness hasn't been demonstrated (because people were already doing activation steering without SAEs)? My understanding of the case for activation steering with unsupervisedly-learned features is that it could circumvent some failure modes of RLHF [LW(p) · GW(p)].

50% I'll do this in the next two months if nobody else does. But not right now, and someone else should do it too.

Off the top of my head (this is not the list you asked for, just an outline):

• Loopt stuff
• YC stuff
• YC removal
• NDAs
  • And deceptive communication recently
  • And maybe OpenAI's general culture of don't publicly criticize OpenAI
• Profit cap non-transparency
• Superalignment compute
• Two exoduses of safety people; negative stuff people-who-quit-OpenAI sometimes say
• Telling board members not to talk to employees
• Board crisis stuff
  • OpenAI executives telling the board Altman lies
  • The board saying Altman lies
  • Lying about why he wanted to remove Toner
  • Lying to try to remove Toner
  • Returning
  • Inadequate investigation + spinning results

Stuff not worth including:

• Reddit stuff - unconfirmed
• Financial conflict-of-interest stuff - murky and not super important
• Misc instances of saying-what's-convenient (e.g. OpenAI should scale because of the prospect of compute overhang and the $7T chip investment thing) - idk, maybe, also interested in more examples
• Johansson & Sky - not obvious that OpenAI did something bad, but it would be nice for OpenAI to say "we had plans for a Johansson voice and we dropped that," but if that was true they'd have said it...

What am I missing? Am I including anything misleading or not-worth-it?

BBC Tech News as far as I can tell has not covered any of the recent OpenAI drama about NDAs or employees leaving.

But Scarlett Johansson 'shocked' by AI chatbot imitation is now the main headline.

Ideological adherence to open source seems to act like a religion, arguing against universal applicability of its central tenets won't succeed with only reasonable effort. Unless you state something very explicitly, it will be ignored, and probably even then.

Enforcement of mitigations when it's someone else who removes them won't be seen as relevant, since in this religion a contributor is fundamentally not responsible for how the things they release will be used by others. Arguments to the contrary in particular very unusual cases slide right off.