LessWrong 2.0 Reader

I think both "jog, don't sprint" and "sprint, don't jog" is too low-dimensional as advice. It's good to try to spend 100% of one's resources on doing good—sorta tautologically. What allows Johannes to work as hard as he does, I think, is not (just) that he's obsessed with the work, it's rather that he understands his own mind well enough to navigate around its limits. And that self-insight is also what enables him aim his cognition at what matters—which is a trait I care more about than ability to work hard.

People who are good at aiming their cognition at what matters sometimes choose to purposefwly flout^[1] various social expectations in order to communicate "I see through this distracting social convention and I'm willing to break it in order to aim myself more purely at what matters". Readers who haven't noticed that some of their expectations are actually superfluous or misaligned with altruistic impact, will mistakenly think the flouter has low impact-potential or is just socially incompetent.

By writing the way he does, Johannes signals that he's distancing himself from status-related putative proxies-for-effectiveness, and I think that's a hard requirement for aiming more purely at the conjunction of multipliers [EA · GW]^[2] that matter. But his signals will be invisible to people who aren't also highly attuned to that conjunction.

1. ^{^}

   "flouting a social expectation": choosing to disregard it while being fully aware of its existence, in a not-mean-spirited way.

2. ^{^}

   I hadn't seen this post before, thank you. My term for this bag of nearby considerations is "costs of compromise [EA(p) · GW(p)]":

   there are exponential costs to compromising what you are optimizing for in order to appeal to a wider variety of interests

It would be bad, I agree. (An NDA about what he worked on at OA, sure, but then being required to never say anything bad about OA forever, as a regulator who will be running evaluations etc...?) Fortunately, this is one of those rare situations where it is probably enough for Paul to simply say his OA NDA does not cover that - then either it doesn't and can't be a problem, or he has violated the NDA's gag order by talking about it and when OA then fails to sue him to enforce it, the NDA becomes moot.

[Not very confident, but just saying my current view.]

I'm pretty skeptical about integrated gradients.

As far as why, I don't think we should care about the derivative at the baseline (zero or the mean).

As far as the axioms, I think I get off the train on "Completeness" which doesn't seem like a property we need/want.

I think you just need to eat that there isn't any sensible way to do something reasonable that gets Completeness.

The same applies with attribution in general (e.g. in decision making).

Integrated gradients is a computationally efficient attribution method (compared to activation patching / ablations) grounded in a series of axioms.

Maybe I'm confused, but isn't integrated gradients strictly slower than an ablation to a baseline?

It seems really quite bad for Paul to work in the U.S. government on AI legislation without having disclosed that he is under a non-disparagement clause for the biggest entity in the space the regulator he is working at is regulating. And if he signed an NDA that prevents him from disclosing it, then it was IMO his job to not accept a position in which such a disclosure would obviously be required. 

I am currently like 50% Paul has indeed signed such a lifetime non-disparagement agreement, so I do think I don't buy that a "presumably" is appropriate here (though I am not that far away from it).

Sam Altman has apparently provided a statement to NPR apropos of https://www.npr.org/2024/05/20/1252495087/openai-pulls-ai-voice-that-was-compared-to-scarlett-johansson-in-the-movie-her , quoted on Twitter by the NPR journalist (second):

...In response, Sam Altman has now issued a statement saying “Sky is not Scarlett Johansson’s, and it was never intended to resemble hers.”

“We cast the voice actor behind Sky’s voice before any outreach to Ms. Johansson. Out of respect for Ms. Johansson, we have paused using Sky’s voice in our products. We are sorry to Ms. Johansson that we didn’t communicate better”, Sam Altman wrote.”

To clarify a few things about the letters from ScarJo’s lawyers: They weren’t cease and desist notices. It’s not initiating a lawsuit. The letters sought clarity about what exactly was fed to the model to produce the distinctive “Sky” voice.

(Yeah, the Altman statement seems to be emailed to journalists, despite being short. Not sure why he's not just tweeting like previously.)

Sam Altman appears to have been using non-disparagements at least as far back as 2017-2018, even for things that really don't seem to have needed such things at all, like a research nonprofit arm of YC. It's unclear if that example is also a lifetime non-disparagement (I've asked), but nevertheless, given that track record, you should assume the OA research non-profit also tried to impose it, followed by the OA LLC (obviously), and so Paul F. Christiano (and all of the Anthropic founders) would presumably be bound.

This would explain why Anthropic executives never say anything bad about OA, and refuse to explain exactly what broken promises by Altman triggered their failed attempt to remove Altman and subsequent exodus.

(I have also asked Sam Altman on Twitter, since he follows me, apropos of his vested equity statement, how far back these NDAs go, if the Anthropic founders are still bound, and if they are, whether they will be unbound.)

Go all-in on lobbying the US and other governments to fully prohibit the training of frontier models beyond a certain level, in a way that OpenAI can't route around (so probably block Altman's foreign chip factory initiative, for instance).

I think you're right that I missed their point, thanks for pointing it out.

I have had experiences similar to Johannes' anecdote re: ignoring broken glass to not lose fragile threads of thought; they usually entailed extended deep work periods past healthy thresholds for unclear marginal gain, so the quotes above felt personally relevant as guardrails. But also my experiences don't necessarily generalize (as your hypothetical shows).

I'd be curious to know your model, and how it compares to some of John Wentworth's posts on the same IIRC.

I'd like to know what Holden did while serving on the board, and what OpenAI would have done if he hadn't joined. That's crucial for assessing the grant's impact.

But since board meetings are private, this will remain unknown for a long time. Unfortunately, the best we can do is speculate.