LessWrong 2.0 Reader

Enforcing social norms to prevent scapegoating also destroys information that is valuable for accurate credit assignment and causally modelling reality.

I read the Ben Hoffman post you linked. I'm not finding it very clear, but the gist seems to be something like: Statements about others often import some sort of good/bad moral valence; trying to avoid this valence can decrease the accuracy of your statements.

If OP was optimizing purely for descriptive accuracy, disregarding everyone's feelings, that would be one thing. But the discussion of "repercussions" before there's been an investigation goes into pure-scapegoating territory if you ask me.

I do not read any mention of a 'moral failing' in that comment.

If OP wants to clarify that he doesn't think there was a moral failing, I expect that to be helpful for a post-mortem. I expect some other people besides me also saw that subtext, even if it's not explicit.

You can be empathetic to people having flawed decision making and care about them, while also wanting to keep them away from certain decision-making positions.

"Keep people away" sounds like moral talk to me. If you think someone's decisionmaking is actively bad, i.e. you'd better off reversing any advice from them, then maybe you should keep them around so you can do that! But more realistically, someone who's fucked up in a big way will probably have learned from that, and functional cultures don't throw away hard-won knowledge.

Imagine a world where AI is just an inherently treacherous domain, and we throw out the leadership whenever they make a mistake. So we get a continuous churn of inexperienced leaders in an inherently treacherous domain -- doesn't sound like a recipe for success!

Oh, interesting. Who exactly do you think influential people like Holden Karnofsky and Paul Christiano are accountable to, exactly? This "detailed investigation" you speak of, and this notion of a "blameless culture", makes a lot of sense when you are the head of an organization and you are conducting an investigation as to the systematic mistakes made by people who work for you, and who you are responsible for. I don't think this situation is similar enough that you can use these intuitions blandly without thinking through the actual causal factors involved in this situation.

I agree that changes things. I'd be much more sympathetic to the OP if they were demanding an investigation or an apology.

Edit: nevermind; seems like this tweet is narrow and just about vested equity; I'm not sure what that means in the context of OpenAI's pseudo-equity but this tweet isn't a big deal.

@gwern [LW · GW] I'm interested in your take on this new Altman tweet:

we have never clawed back anyone's vested equity, nor will we do that if people do not sign a separation agreement (or don't agree to a non-disparagement agreement). vested equity is vested equity, full stop.

there was a provision about potential equity cancellation in our previous exit docs; although we never clawed anything back, it should never have been something we had in any documents or communication. this is on me and one of the few times i've been genuinely embarrassed running openai; i did not know this was happening and i should have.

the team was already in the process of fixing the standard exit paperwork over the past month or so. if any former employee who signed one of those old agreements is worried about it, they can contact me and we'll fix that too. very sorry about this.

In particular "i did not know this was happening"

1. Gwern's theories make sense to me. The data was roughly 50/50 on <= 30 vs > 30, so that's where I split it (and I'm only asking the model to pick one of those two options). Sexuality in the dataset is just male/female; they must have added the other options later (35829 male, 24117 female, and 2 blanks which I ignored). Agreed that this is very much a lower bound, also that I applied zero optimization to the system prompt and user prompts. This is 'if you do the simplest possible thing, how good is it?'
2. No, unfortunately it's all lowercased already in the dataset.
3. I agree! Dating site data is somewhat easy mode. I compared gender accuracy on the [Persuade 2.0](https://www.kaggle.com/datasets/nbroad/persaude-corpus-2) corpus of students writing essays on a fixed topic, which I consider very much hard mode, and it was still 80% accurate. So I do think it's getting some advantage from being in easy mode but not that much. I'll note also that I'm removing a bunch of words that are giveaways for gender, and it only lost 2 percentage points of accuracy. So I do think it's mostly working from implicit cues and distributional differences here rather than easy giveaways. [Staab et al](https://arxiv.org/abs/2310.07298) (thanks @gwern [LW · GW] for pointing that paper out to me) looks more at explicit cues and compares to human investigators looking for explicit cues, so you may find that interesting as well.

Our current big stupid

Epistemic status: lukewarm take from the gut (not brain) that feels rightish

The "Big Stupid" of the AI doomers 2013-2023 was AI nerds' solution to the problem "How do we stop people from building dangerous AIs?" was "research how to build AIs".  Methods normal people would consider to stop people from building dangerous AIs, like asking governments to make it illegal to build dangerous AIs, were considered gauche.  When the public turned out to be somewhat receptive to the idea of regulating AIs, doomers were unprepared.

Take: The "Big Stupid" of right now is still the same thing.  (We've not corrected enough).  Between now and transformative AGI we are likely to encounter a moment where 40% of people realize AIs really could take over (say if every month another 1% of the population loses their job).  If 40% of the world were as scared of AI loss-of-control as you, what could the world do? I think a lot!  Do we have a plan for then?

Almost every LessWrong post on AIs are about analyzing AIs.  Almost none are about how, given widespread public support, people/governments could stop bad AIs from being built.

[Example: if 40% of people were as worried about AI as I was, the US would treat GPU manufacture like uranium enrichment.  And fortunately GPU manufacture is hundreds of time harder than uranium enrichment!  We should be nerding out researching integrated circuit supply chains, choke points, foundry logistics in jurisdictions the US can't unilaterally sanction, that sort of thing.]

TLDR, to figure out how to stop deadly AIs from being built, we need comparatively less research on AIs and more research on how to stop AIs from being built.

*My research included 😬

I downvoted this comment because it felt uncomfortably scapegoat-y to me.

Enforcing social norms to prevent scapegoating also destroys information that is valuable for accurate credit assignment and causally modelling reality. [LW(p) · GW(p)]

If you start with the assumption that there was a moral failing on the part of the grantmakers, and you are wrong, there’s a good chance you’ll never learn that.

I think you are misinterpreting the grandparent comment. I do not read any mention of a 'moral failing' in that comment. You seem worried because of the commenter's clear description of what they think would be a sensible step for us to take given what they believe are egregious flaws in the decision-making processes of the people involved. I don't think there's anything wrong with such claims.

Again: You can care about people while also seeing their flaws and noticing how they are hurting you and others you care about. You can be empathetic to people having flawed decision making and care about them, while also wanting to keep them away from certain decision-making positions.

If you think the OpenAI grant was a big mistake, it’s important to have a detailed investigation of what went wrong, and that sort of detailed investigation is most likely to succeed if you have cooperation from people who are involved.

Oh, interesting. Who exactly do you think influential people like Holden Karnofsky and Paul Christiano are accountable to, exactly? This "detailed investigation" you speak of, and this notion of a "blameless culture", makes a lot of sense when you are the head of an organization and you are conducting an investigation as to the systematic mistakes made by people who work for you, and who you are responsible for. I don't think this situation is similar enough that you can use these intuitions blandly without thinking through the actual causal factors involved in this situation.

Note that I don't necessarily endorse the grandparent comment claims. This is a complex situation and I'd spend more time analyzing it and what occurred.

In the spirit of trying to understand what actually went wrong here -- IIRC, OpenAI didn't expect ChatGPT to blow up the way it did. Seems like they were playing a strategy of "release cool demos" as opposed to "create a giant competitive market".

The Internet seems to agree with you. I wonder why I remember "edit time addition".

I downvoted this comment because it felt uncomfortably scapegoat-y to me. If you think the OpenAI grant was a big mistake, it's important to have a detailed investigation of what went wrong, and that sort of detailed investigation is most likely to succeed if you have cooperation from people who are involved. I've been reading a fair amount about what it takes to instill a culture of safety in an organization, and nothing I've seen suggests that scapegoating is a good approach.

Writing a postmortem is not punishment—it is a learning opportunity for the entire company.

...

Blameless postmortems are a tenet of SRE culture. For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior. A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. If a culture of finger pointing and shaming individuals or teams for doing the "wrong" thing prevails, people will not bring issues to light for fear of punishment.

Blameless culture originated in the healthcare and avionics industries where mistakes can be fatal. These industries nurture an environment where every "mistake" is seen as an opportunity to strengthen the system. When postmortems shift from allocating blame to investigating the systematic reasons why an individual or team had incomplete or incorrect information, effective prevention plans can be put in place. You can’t "fix" people, but you can fix systems and processes to better support people making the right choices when designing and maintaining complex systems.

...

Removing blame from a postmortem gives people the confidence to escalate issues without fear. It is also important not to stigmatize frequent production of postmortems by a person or team. An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization [Boy13].

...

We can say with confidence that thanks to our continuous investment in cultivating a postmortem culture, Google weathers fewer outages and fosters a better user experience.

https://sre.google/sre-book/postmortem-culture/

If you start with the assumption that there was a moral failing on the part of the grantmakers, and you are wrong, there's a good chance you'll never learn that.

In my mainline model there are only a few innovations needed, perhaps only a single big one to product an AGI which just like the Turing Machine sits at the top of the Chomsky Hierarchy will be basically the optimal architecture given resource constraints. There are probably some minor improvements todo with bridging the gap between theoretically optimal architecture and the actual architecture, or parts of the algorithm that can be indefinitely improved but with diminishing returns (these probably exist due to Levin and possibly.matrix.multiplication is one of these). On the whole I expect AI research to be very chunky.

Indeed, we've seen that there was really just one big idea to all current AI progress: scaling, specifically scaling GPUs on maximally large undifferentiated datasets. There were some minor technical innovations needed to pull this off but on the whole that was the clinger.

Of course, I don't know. Nobody knows. But I find this the most plausible guess based on what we know about intelligence, learning, theoretical computer science and science in general.

What I didn't know is how immediately thought-provoking it would be to learn even the most basic things about playing music. Maybe it's like learning to program, if you used a computer all the time but you never had one thought about how it might work.

That comparison is also thought-provoking) Thinking for a minute yielded that programming may be considered quite similar to playing music, but differs that in programming you do not need to do most things in any specific order. For example, if I have a dataset of a competition participants, it doesn't matter whether I deduplicate names or remove disqualified entries first.