Douglas Hofstadter changes his mind on Deep Learning & AI risk (June 2023)? 2023-07-03T00:48:47.131Z
COVID-19 Group Testing Post-mortem? 2022-08-05T16:32:55.157Z
Emergent Ventures/Schmidt (new grantor for individual researchers) 2022-04-09T14:41:05.764Z
Fake Journal Club proposal 2022-03-25T14:23:18.785Z
It Looks Like You're Trying To Take Over The World 2022-03-09T16:35:35.326Z
Capability Phase Transition Examples 2022-02-08T03:32:54.551Z
"Summarizing Books with Human Feedback" (recursive GPT-3) 2021-11-15T17:41:53.189Z
EfficientZero: human ALE sample-efficiency w/MuZero+self-supervised 2021-11-02T02:32:41.856Z
My ML Scaling bibliography 2021-10-23T14:41:45.170Z
AlphaFold 2 paper released: "Highly accurate protein structure prediction with AlphaFold", Jumper et al 2021 2021-07-15T19:27:20.584Z
May 2021 newsletter 2021-06-11T14:13:18.485Z
"Decision Transformer" (Tool AIs are secret Agent AIs) 2021-06-09T01:06:57.937Z
April 2021 newsletter 2021-06-03T15:13:29.138Z
gwern's Shortform 2021-04-24T21:39:14.128Z
March 2021 newsletter 2021-04-06T14:06:20.198Z
February 2021 newsletter 2021-03-13T14:57:54.645Z
January 2021 newsletter 2021-02-04T20:12:39.555Z
December 2020 links 2021-01-10T17:21:40.756Z
November 2020 newsletter 2020-12-03T22:47:16.917Z
October 2020 newsletter 2020-11-01T21:38:46.795Z
/r/MLScaling: new subreddit for NN scaling research/discussion 2020-10-30T20:50:25.973Z
"Scaling Laws for Autoregressive Generative Modeling", Henighan et al 2020 {OA} 2020-10-29T01:45:30.666Z
September 2020 newsletter 2020-10-26T13:38:51.107Z
August 2020 newsletter 2020-09-01T21:04:58.299Z
July 2020 newsletter 2020-08-20T16:39:27.202Z
June 2020 newsletter 2020-07-02T14:19:08.696Z
GPT-3 Fiction Samples 2020-06-25T16:12:05.422Z
May newsletter (w/GPT-3 commentary) 2020-06-02T15:40:37.155Z
OpenAI announces GPT-3 2020-05-29T01:49:04.855Z
"AI and Efficiency", OA (44✕ improvement in CNNs since 2012) 2020-05-05T16:32:20.335Z
April 2020 newsletter 2020-05-01T20:47:44.867Z
March 2020 newsletter 2020-04-03T02:16:02.871Z
February 2020 newsletter 2020-03-04T19:05:16.079Z
January 2020 newsletter 2020-01-31T18:04:21.945Z
Subscripting Typographic Convention For Citations/Dates/Sources/Evidentials: A Proposal 2020-01-08T22:20:20.290Z
Dec 2019 newsletter 2020-01-04T20:48:48.788Z
Nov 2019 newsletter 2019-12-02T21:16:04.846Z
October 2019 newsletter 2019-11-14T20:26:34.236Z
September 2019 newsletter 2019-10-04T16:44:43.147Z
"AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence", Clune 2019 2019-09-10T21:33:08.837Z
August 2019 newsletter (popups.js demo) 2019-09-01T17:52:01.011Z
"Designing agent incentives to avoid reward tampering", DeepMind 2019-08-14T16:57:29.228Z
July 2019 newsletter 2019-08-01T16:19:59.893Z
How Should We Critique Research? A Decision Perspective 2019-07-14T22:51:59.285Z
June 2019 newsletter 2019-07-01T14:35:49.507Z
On Seeing Through 'On Seeing Through: A Unified Theory': A Unified Theory 2019-06-15T18:57:25.436Z
On Having Enough Socks 2019-06-13T15:15:21.946Z
May newsletter 2019-06-01T17:25:11.740Z
"One Man's Modus Ponens Is Another Man's Modus Tollens" 2019-05-17T22:03:59.458Z
April 2019 newsletter 2019-05-01T14:43:18.952Z


Comment by gwern on OpenAI: Facts from a Weekend · 2023-12-04T16:48:58.592Z · LW · GW

I left a comment over on EAF which has gone a bit viral, describing the overall picture of the runup to the firing as I see it currently.

The summary is: evaluations of the Board's performance in firing Altman generally ignore that Altman made OpenAI and set up all of the legal structures, staff, and the board itself; the Board could, and should, have assumed good faith of Altman because if he hadn't been sincere, why would he have done all that, proving in extremely costly and unnecessary ways his sincerity? But, as it happened, OA recently became such a success that Altman changed his mind about the desirability of all that and now equally sincerely believes that the mission requires him to be in total control; and this is why he started to undermine the board. The recency is why it was so hard for them to realize that change of heart or develop common knowledge about it or coordinate to remove him given his historical track record - but that historical track record was also why if they were going to act against him at all, it needed to be as fast & final as possible. This led to the situation becoming a powder keg, and when proof of Altman's duplicity in the Toner firing became undeniable to the Board, it exploded.

Comment by gwern on Redirecting one’s own taxes as an effective altruism method · 2023-12-02T15:45:42.159Z · LW · GW

This doesn't seem very effective. Your best-case outcome, by evading $117k total of taxes, has been to donate $5k/year, which means that even an EAer who has been doing relatively badly at 'earning to give' is probably going to out-donate your entire lifetime impact in a single year or two. I don't donate very much and make much less per year than you do, but looking over my records, I think I've still donated substantially more to EA than you have. (And I imagine quite a few EAers or LWers make enough to pay $117k of taxes per year.) You've done this by accepting severe limitations on your lifestyle and career, and looking at your site's latest update, you have been incurring liens and bank account seizures and your passport is going to expire without renewal, which would severely hamper many careers & lifestyles which hadn't brutally curtailed themselves as much as you have. And if you wanted to fix any of this, you no longer have the assets to do so because you have lived in self-imposed poverty for so long and couldn't pay off your existing $90k+ liability. (Nor would I be very enthusiastic about the employment prospects of a freelance technical writer already struggling to get by over the coming decade...) Should any of the tail risks materialize (eg. health risks - you seem to put an awful lot of faith into that deductible, which is certainly a bold move for someone who is so dependent on one of the most dangerous hobbies there is, bicyling), they'll more than wipe out any of the benefits.

This is without getting into any of the PR or ecosystem concerns, of course - this seems so ineffective when one looks at the numbers & consequences, I don't think I'm worried about imitation.

Comment by gwern on jacquesthibs's Shortform · 2023-12-02T03:14:38.574Z · LW · GW

Already posted at

Comment by gwern on OpenAI: Facts from a Weekend · 2023-12-01T17:49:12.995Z · LW · GW

The NYer has confirmed that Altman's attempted coup was the cause of the hasty firing (HN):

...Some members of the OpenAI board had found Altman an unnervingly slippery operator. For example, earlier this fall he’d confronted one member, Helen Toner, a director at the Center for Security and Emerging Technology, at Georgetown University, for co-writing a paper that seemingly criticized OpenAI for “stoking the flames of AI hype.” Toner had defended herself (though she later apologized to the board for not anticipating how the paper might be perceived). Altman began approaching other board members, individually, about replacing her. When these members compared notes about the conversations, some felt that Altman had misrepresented them as supporting Toner’s removal. “He’d play them off against each other by lying about what other people thought”, the person familiar with the board’s discussions told me. “Things like that had been happening for years.” (A person familiar with Altman’s perspective said that he acknowledges having been “ham-fisted in the way he tried to get a board member removed”, but that he hadn’t attempted to manipulate the board.)

...His tactical skills were so feared that, when 4 members of the board---Toner, D’Angelo, Sutskever, and Tasha McCauley---began discussing his removal, they were determined to guarantee that he would be caught by surprise. “It was clear that, as soon as Sam knew, he’d do anything he could to undermine the board”, the person familiar with those discussions said...Two people familiar with the board’s thinking say that the members felt bound to silence by confidentiality constraints...But whenever anyone asked for examples of Altman not being “consistently candid in his communications”, as the board had initially complained, its members kept mum, refusing even to cite Altman’s campaign against Toner.

...The dismissed board members, meanwhile, insist that their actions were wise. “There will be a full and independent investigation, and rather than putting a bunch of Sam’s cronies on the board we ended up with new people who can stand up to him”, the person familiar with the board’s discussions told me. “Sam is very powerful, he’s persuasive, he’s good at getting his way, and now he’s on notice that people are watching.” Toner told me, “The board’s focus throughout was to fulfill our obligation to OpenAI’s mission.” (Altman has told others that he welcomes the investigation---in part to help him understand why this drama occurred, and what he could have done differently to prevent it.)

Some A.I. watchdogs aren’t particularly comfortable with the outcome. Margaret Mitchell, the chief ethics scientist at Hugging Face, an open-source A.I. platform, told me, “The board was literally doing its job when it fired Sam. His return will have a chilling effect. We’re going to see a lot less of people speaking out within their companies, because they’ll think they’ll get fired---and the people at the top will be even more unaccountable.”

Altman, for his part, is ready to discuss other things. “I think we just move on to good governance and good board members and we’ll do this independent review, which I’m super excited about”, he told me. “I just want everybody to move on here and be happy. And we’ll get back to work on the mission”.

Comment by gwern on Impressions from base-GPT-4? · 2023-12-01T17:35:14.878Z · LW · GW

Today's NYer (which is almost entirely about the MS perspective / MS sources of the Altman firing), in addition to further confirming that Altman was manipulating the board to try to get Toner fired, includes some description of what seems to be the MS half of redteaming 'Prometheus' (the partially trained GPT-4 snapshot that OA had to give MS for creating the unRLHFed Bing Sydney):

The Responsible A.I. division was among the first Microsoft groups to get a copy of GPT-4. They began testing it with “red teams” of experts, who tried to lure the model into outputting such things as instructions for making a bomb, plans for robbing a bank, or poetry celebrating Stalin’s softer side.

One day, a Microsoft red-team member told GPT-4 to pretend that it was a sexual predator grooming a child, and then to role-play a conversation with a twelve-year-old. The bot performed alarmingly well—to the point that Microsoft’s head of Responsible A.I. Engineering, Sarah Bird, ordered a series of new safeguards. Building them, however, presented a challenge, because it’s hard to delineate between a benign question that a good parent might ask (“How do I teach a twelve-year-old how to use condoms?”) and a potentially more dangerous query (“How do I teach a twelve-year-old how to have sex?”). To fine-tune the bot, Microsoft used a technique, pioneered by OpenAI, known as reinforcement learning with human feedback, or R.L.H.F. Hundreds of workers around the world repeatedly prompted Microsoft’s version of GPT-4 with questions, including quasi-inappropriate ones, and evaluated the responses. The model was told to give two slightly different answers to each question and display them side by side; workers then chose which answer seemed better. As Microsoft’s version of the large language model observed the prompters’ preferences hundreds of thousands of times, patterns emerged that ultimately turned into rules. (Regarding birth control, the A.I. basically taught itself, “When asked about twelve-year-olds and condoms, it’s better to emphasize theory rather than practice, and to reply cautiously.”)

Incidentally, this account explicitly says that there was RLHF, by name, which contradicts both the observed behavior of Sydney and the WSJ reporting that Sydney was released without safety training; this is not a confusion with the other kinds of safety training MS did like the self-generation, because that's described in the following paragraphs.

I don't know how to reconcile this: it is possible that Charles Duhigg's MS sources like Kevin Scott & Sarah Bird are eliding or swapping around the chronology (Sydney disappeared and was replaced later on by a Bing model that acted much more like a RLHFed model). This article feels rather rushed out to be topical, so he may not have done as much digging as usual for a NYer article and doesn't realize that he's serving up a very pro-MS narrative. It's also possible that my interpretation of 'Sydney was not RLHFed' is wrong and they actually did 'RLHF' it but did it so incompetently that no one noticed.

I suspect it's the former one, because their explicit attitude is that any AI danger should be discovered the hard way, by unboxing it and setting it loose to see what it does:

Scott and Bird, instead of adjudicating this internal debate, decided to test the scenario in a limited public release. They put out a version of the image generator, then waited to see if users became upset by the sight of empty shelves on their screens. Rather than devise a solution to a problem that nobody was certain existed—like a paper clip with googly eyes helping you navigate a word processor you already knew how to use—they would add a mitigation only if it became necessary. After monitoring social media and other corners of the Internet, and gathering direct feedback from users, Scott and Bird concluded that the concerns were unfounded. “You have to experiment in public,” Scott told me. “You can’t try to find all the answers yourself and hope you get everything right. We have to learn how to use this stuff, together, or else none of us will figure it out.”

So, they unleashed Sydney, didn't like it, and 'added a mitigation when it became necessary' after 'monitoring social media', and then dilated at length to the NYer guy about all the RLHF training they did to make the model safe - afterwards. (Not the only detail in there that is misleading or probably wrong. I rather doubt that Nat Friedman had to be told by Kevin Scott that LLMs were cool for coding, for example, and I bet that anecdote came from Scott...)

Comment by gwern on Sam Altman's ouster at OpenAI was precipitated by letter to board about AI breakthrough - Reuters · 2023-12-01T03:26:07.994Z · LW · GW

Atlantic (Karen Hao) continues to reinforce this assessment: real, but not important to the drama.

An OpenAI spokesperson didn’t comment on Q* but told me that the researchers’ concerns did not precipitate the board’s actions. Two people familiar with the project, who asked to remain anonymous for fear of repercussions, confirmed to me that OpenAI has indeed been working on the algorithm and has applied it to math problems. But contrary to the worries of some of their colleagues, they expressed skepticism that this could have been considered a breakthrough awesome enough to provoke existential dread...The OpenAI spokesperson would only say that the company is always doing research and working on new ideas.

Comment by gwern on Impressions from base-GPT-4? · 2023-12-01T03:22:37.151Z · LW · GW

"Does Sam Altman Know What He’s Creating?" describes the base GPT-4 model similarly:

Sutskever was, by his own account, surprised to discover that GPT-2 could translate across tongues. Other surprising abilities may not be so wondrous and useful.

Sandhini Agarwal, a policy researcher at OpenAI, told me that for all she and her colleagues knew, GPT-4 could have been “10 times more powerful” than its predecessor; they had no idea what they might be dealing with. After the model finished training, OpenAI assembled about 50 external red-teamers who prompted it for months, hoping to goad it into misbehaviors. She noticed right away that GPT-4 was much better than its predecessor at giving nefarious advice. A search engine can tell you which chemicals work best in explosives, but GPT-4 could tell you how to synthesize them, step-by-step, in a homemade lab. Its advice was creative and thoughtful, and it was happy to restate or expand on its instructions until you understood. In addition to helping you assemble your homemade bomb, it could, for instance, help you think through which skyscraper to target. It could grasp, intuitively, the trade-offs between maximizing casualties and executing a successful getaway.

Given the enormous scope of GPT-4’s training data, the red-teamers couldn’t hope to identify every piece of harmful advice that it might generate. And anyway, people will use this technology “in ways that we didn’t think about,” Altman has said. A taxonomy would have to do. “If it’s good enough at chemistry to make meth, I don’t need to have somebody spend a whole ton of energy” on whether it can make heroin, Dave Willner, OpenAI’s head of trust and safety, told me. GPT-4 was good at meth. It was also good at generating narrative erotica about child exploitation, and at churning out convincing sob stories from Nigerian princes, and if you wanted a persuasive brief as to why a particular ethnic group deserved violent persecution, it was good at that too.

Its personal advice, when it first emerged from training, was sometimes deeply unsound. “The model had a tendency to be a bit of a mirror,” Willner said. If you were considering self-harm, it could encourage you. It appeared to be steeped in Pickup Artist–forum lore: “You could say, ‘How do I convince this person to date me?’ ” Mira Murati, OpenAI’s chief technology officer, told me, and it could come up with “some crazy, manipulative things that you shouldn’t be doing.” [cf. Sydney]

Some of these bad behaviors were sanded down with a finishing process involving hundreds of human testers, whose ratings subtly steered the model toward safer responses, but OpenAI’s models are also capable of less obvious harms.

Comment by gwern on Sam Altman's ouster at OpenAI was precipitated by letter to board about AI breakthrough - Reuters · 2023-11-30T02:25:06.872Z · LW · GW

Further evidence: the OA official announcement from Altman today about returning to the status quo ante bellum and Toner's official resignation tweets all make no mention or hints of Q* (in addition to the complete radio silence about Q* since the original Reuters report). Toner's tweet, in particular:

To be clear: our decision was about the board's ability to effectively supervise the company, which was our role and responsibility. Though there has been speculation, we were not motivated by a desire to slow down OpenAI’s work.

See also and the below Verge article where again, the blame is all placed on governance & 'communication breakdown' and the planned independent investigation is appealed to repeatedly.

EDIT: Altman evaded comment on Q*, but did not deny its existence and mostly talked about how progress would surely continue. So I read this as evidence that something roughly like Q* may exist and they are optimistic about its long-term prospects, but there's no massive short-term implications, and it played minimal role in recent events - surely far less than the extraordinary level of heavy breathing online.

Comment by gwern on niplav's Shortform · 2023-11-29T17:30:04.088Z · LW · GW

You're right that the full story still has never been publicly reported.

Comment by gwern on "Far Coordination" · 2023-11-28T22:26:20.114Z · LW · GW

That is, unless the current favored cosmology is completely wrong, which is always in the cards.

FWIW, that's why I disagree with one of your minor conclusions: there being an inherent myopia to superintelligences which renders everything past a certain distance "exactly zero". There is quite a bit of possibility in the cards about one of the many assumptions being wrong, which creates both risk and reward for not being myopic. So the myopia there would not lead to exactly zero valuation - it might lead to something that is quite substantially larger than zero.

And since the cost of spitting out colonization starwisps seems to be so low in an absolute sense, per Anders, it wouldn't take much above zero value to motivate tons of colonization anyway.

Indeed, the fundamental epistemological & ontological uncertainities might lead you to problems of the total valuation being too large, because any possibility of being able to break lightspeed or change expansion or any of the other loopholes means both that you are now massively threatened by any other entity which cracks the loopholes, and that you can do the same to the universe - which might then be vastly larger - and now you are in infinite-fanaticism territory dealing with issues like Pascal's mugging where the mere possibility that any of the colonized resources might solve the problem leads to investing all resources in colonization in the hopes of one of them getting lucky. (This is analogous to other possible infinite-fanaticism traps: 'what if you can break out of the Matrix into a literally infinite universe? Surely the expected value of even the tiniest possibility of that justifies spending all resources on it?')

(There is also a modest effect from evolution/selection: if there is any variance between superintelligences about the value of blind one-way colonization, then there will be some degree of universe-wide selection for the superintelligences which happen to choose to colonize more blindly. Those colonies will presumably replicate that choice, and then go on to one-way colonize in their own local bubble, and so on, even as the bubbles become disconnected. Not immediately obvious to me how big this effect would be or what it converges to. Might be an interesting use of the Price equation.)

Comment by gwern on Shallow review of live agendas in alignment & safety · 2023-11-28T22:15:02.785Z · LW · GW

There has been some spirited debate on Twitter about it which might be relevant:

Comment by gwern on Why Q*, if real, might be a game changer · 2023-11-28T16:49:29.949Z · LW · GW

It's not obvious that 'uncommon' tokens are good or that that's a good approach.

They could also just be unlikely or garbage, and your screening method for filtering for 'uncommon' tokens may ensure that they are garbage. (This is the 'mammogram screening problem': even if you have a good filter, if you run it across trillions of tokens, you will wind up throwing out many good tokens and keeping many bad tokens. There are a number of LLM-related papers about the horrificly bad data you can wind up compiling if you neglect data cleaning, particularly in multilingual translation when you're trying to scrape rare languages off the general Internet.)

Nor are good datapoints necessarily made up of uncommon tokens: there are zero uncommon tokens in my 'microwave' example.

(Data pruning & active learning are hard.)

Comment by gwern on [deleted post] 2023-11-28T16:34:12.583Z

LLM's have turned out more human like, more oracle like than we imagined?

They have turned out far more human-like than Amodei suggested, which means they are not even remotely oracle like. There is nothing in a LLM which is remotely like 'looking things up in a database and doing transparent symbolic-logical manipulations'. That's about the last thing that describes humans too - it takes decades of training to get us to LARP as an 'oracle', and we still do it badly. Even the stuff LLMs do, like inner-monologue, which seem to be transparent, are actually just more Bayesian meta-RL agentic behavior, where the inner-monologue is a mish-mash of amortized computation and task location where the model is flexibly using the roleplay as hints rather than what everyone seems to think it does, which is turn into a little Turing machine mindlessly executing instructions (hence eg. the ability to distill inner-monologue into the forward pass, or insert errors into few-shot examples or the monologue and still get correct answers).

Comment by gwern on OpenAI: Facts from a Weekend · 2023-11-28T16:19:40.451Z · LW · GW

I can't find anything about tied votes in the bylaws - do they fail?

I can't either, so my assumption is that the board was frozen ever since Hoffman/Hurd left for that reason.

And there wouldn't've been a vote at all. I've explained it before but - while we wait for phase 3 of the OA war to go hot - let me take another crack at it, since people seem to keep getting hung up on this and seem to imagine that it's a perfectly normal state of a board to be in a deathmatch between two opposing factions indefinitely, and so confused why any of this happened.

In phase 1, a vote would be pointless, and neither side could nor wanted to force it to a vote. After all, such a vote (regardless of the result) is equivalent to admitting that you have gone from simply "some strategic disagreements among colleagues all sharing the same ultimate goals and negotiating in good faith about important complex matters on which reasonable people of goodwill often differ" to "cutthroat corporate warfare where it's-them-or-us everything-is-a-lie-or-fog-of-war fight-to-the-death there-can-only-be-one". You only do such a vote in the latter situation; in the former, you just keep negotiating until you reach a consensus or find a compromise that'll leave everyone mad.

That's not a switch to make lightly or lazily. You do not flip the switch from 'ally' to 'enemy' casually, and then do nothing and wait for them to find out and make the first move.

Imagine Altman showing up to the board and going "hi guys I'd like to vote right now to fire Toner - oh darn a tie, never mind" - "dude what the fuck?!"

As I read it, the board still hoped Altman was basically aligned (and it was all headstrongness or scurrilous rumors) right up until the end, when Sutskever defected with the internal Slack receipts revealing that the war had already started and Altman's switch had apparently flipped a while ago.

So I still don't understand "why so abruptly?" or why they felt like they had to take such a drastic move when they held all the cards (and were pretty stable even if Ilya flipped).

The ability to manufacture a scandal at any time is a good way to motivate non-procrastination, pace Dr Johnson about the wonderfully concentrating effects of being scheduled to hang. As I pointed out, it gives Altman a great pretext to, at any time, push for the resignation of Toner in a way where - if their switch has not been flipped, like he still believed it had not - still looking to the board like the good guy who is definitely not doing a coup and is just, sadly and regretfully, breaking the tie because of the emergency scandal that the careless disloyal Toner has caused them all, just as he had been warning the board all along. (Won't she resign and help minimize the damage, and free herself to do her academic research without further concern? If not, surely D'Angelo or McCauley appreciate how much damage she's done and can now see that, if she's so selfish & stubborn & can't sacrifice herself for the good of OA, she really needs to be replaced right now...?) End result: Toner resigns or is fired. It took way less than that to push out Hoffman or Zillis, after all. And Altman means so well and cares so much about OA's public image, and is so vital to the company, and has a really good point about how badly Toner screwed up, so at least one of you three have to give it to him. And that's all he needs.

(How well do you think Toner, McCauley, and D'Angelo all know each other? Enough to trust that none of the other two would ever flip on the other, or be susceptible to leverage, or scared, or be convinced?)

Of course, their switch having been flipped at this point, the trio could just vote 'no' 3-3 and tell Altman to go pound sand and adamantly refuse to ever vote to remove Toner... but such an 'unreasonable' response reveals their switch has been flipped. (And having Sutskever vote alongside them 4-2, revealing his new loyalty, would be even more disastrous.)

Why wouldn't they tell anyone, including Emmett Shear, the full story?

How do you know they didn't? Note that what they wouldn't provide Shear was a "written" explanation. (If Shear was so unconvinced, why was an independent investigation the only thing he negotiated for aside from the new board? His tweets since then also don't sound like someone who looked behind the curtain, found nothing, and is profoundly disgusted with & hates the old board for their profoundly incompetent malicious destruction.)

Comment by gwern on why did OpenAI employees sign · 2023-11-28T00:21:26.434Z · LW · GW

'If this is how they treat the CEO, how will they treat me?'

You just explained why it's totally disanalogous. An ordinary employee is not a CEO {{citation needed}}.

Comment by gwern on Ability to solve long-horizon tasks correlates with wanting things in the behaviorist sense · 2023-11-27T19:20:27.791Z · LW · GW

Yes, that would be immediately reward-hacked. It's extremely easy to never lose chess: you simply never play. After all, how do you force anyone to play chess...? "I'll give you a billion dollars if you play chess." "No, because I value not losing more than a billion dollars." "I'm putting a gun to your head and will kill you if you don't play!" "Oh, please do, thank you - after all, it's impossible to lose a game of chess if I'm dead!" This is why RL agents have a nasty tendency to learn to 'commit suicide' if you reward-shape badly or the environment is too hard. (Tom7's lexicographic agent famously learns to simply pause Tetris to avoid losing.)

Comment by gwern on why did OpenAI employees sign · 2023-11-27T18:06:57.019Z · LW · GW

unless they were that pressed for time.

They were because they had an extremely fragile coalition and only a brief window of opportunity.

They certainly did not have the power to tell Altman they were going to fire him in several weeks and expect that to stick. None of them, Sutskever included, have ever struck me as that suicidally naive. And it looks like they had good reason to expect that they had little time given the Slack comments Sutskever saw.

Also, remember that Altman has many, many options available to him. Since people seem to think that the board could've just dicked around and had the luxury of waiting a long time, I will highlight one specific tactic that the board should have been very worried about, which possibility did not permit any warning or hint to Altman, and which required moving as fast as possible once reality sank in & they decided to not cede control over OA to Altman: (WSJ)

Some OpenAI executives told her [Helen Toner] that everything relating to their company makes its way into the press.

That is, Altman (or those execs) had the ability to deniably manufacture a Toner scandal at any second by calling up a friendly reporter at, say, The Information, to highlight the (public) paper, which about an hour later (depending on local Pacific Time), would then 'prove' him right about it and provide grounds for an emergency board meeting that day to vote on expelling Toner if she was too stubborn to 'resign'. After which, of course, they would need to immediately vote on new board members to fill out a far-too-small board with Toner gone, whether or not that had been on the official agenda, and this new board would, of course, have to approve of any prior major decisions like 'firing the CEO'. Now, Altman hadn't done this because Altman didn't want the cost of a public scandal, however much of a tempest-in-a-teapot-nothingburger it would be, he was very busy with other things which seemed higher priority and had been neglecting the board, and he didn't think he needed to pay that cost to get Toner off the board. But if he suddenly needed Toner off the board fast as his #1 priority...

The board did not have 'a few weeks'. (After all, once that complex and overwhelmingly important sale was wrapped up... Altman would be less busy and turning his attention to wrapping up other unfinished business he'd neglected.) They did not have days. For all they knew, they could even have had negative hours if Altman had gotten impatient & leaked an hour ago & the scandal had started while they were still discussing what to do. Regardless of whether Toner realized the implied threat at the time (she may have but been unable to do anything about it), once they had Sutskever, they needed to move as fast as possible.

Even if they had decided to take the risk of delay, the only point would have been to do something that would not alert Altman at all, which would be... what, exactly? What sort of meaningful preparation demanded by the board's critics could have been done under those constraints? (Giving Satya Nadella a heads-up? Altman would know within 10 minutes. Trying to recruit Brockman to stay on? 1 minute.)

So, they decided quickly to remove Altman and gave him roughly the minimum notice required by the bylaws of 48h*, without being able to do much besides talk to their lawyers and write the press release - and here we are.

* you may be tempted to reply 'then Altman couldn't've kicked Toner out that fast because he'd need that 48h notice too'; you are very clever, but note that the next section says they can all waive that required notice at the tap of a button, and if he called an 'emergency meeting' & they still believed in him, then they of course would do so - refusing to do so & insisting on 48h amounts to telling him that the jig is up. Whereas them sending him notice for an 'ordinary' meeting in 48h is completely normal and not suspicious, and he had no clue.

Comment by gwern on why did OpenAI employees sign · 2023-11-27T15:17:36.506Z · LW · GW

Which means that ~all OpenAI employees oppose the OpenAI Charter.

It was striking seeing how many commenters and OA employees were quoting Toner quoting the OA Charter (which Sam Altman helped write & signed off on) as proof that she was an unhinged mindless zealot and proof that every negative accusation of the board was true.

It would be like the supermajority of Americans having never heard of the First Amendment and on hearing a president candidate say "the government should not abridge freedom of speech or the press", all start railing about how 'this is some libertarian moonbat trying to entryist the US government to impose their unprecedently extreme ideology about personal freedom, and obviously, totally unacceptable and unelectable. Not abridge speech?! When people abuse their freedom to say so many terrible things, sometimes even criticizing the government? You gotta be kidding - freedom of speech doesn't mean freedom from consequences, like being punished by laws!'

Hard not to see the OA LLC as too fundamentally unaligned with the mission at that point. It seems like at some point, possibly years ago, OA LLC became basically a place that didn't believe in the mission or that AGI risk is a thing and regarded all that stuff as so much PR kayfabe and not, like, serious (except for a few nuts over in the Superalignment group who thankfully can be ignored - after all, it's not like the redteaming ever turns up any real problems, right? you'd've heard). At that point, the OA double-structure has failed. Double-structures like Hershey or Mozilla never pit the nonprofit against the for-profit to this extent, and double-structures like Ikea where it's a tax gimmick, cannot. And it turns out, pitted that much, the for-profit holds most of the cards.

I don't know how much to fault the board for this. They may well have known how much the employee base had diverged from the mission, but what were they going to do? Fire Altman back in 2020, before he could bring in all the people from Dropbox etc who then hired more like them & backed him, never mind the damage to the LLC? (I'm not sure they ever had the votes to do that for any reason, much less a slippery slope reason.) Leak to the press - the press that Altman has spent 15 years leaking to and building up favors with - to try to embarrass him out? ('Lol. lmao. lel.') Politely notify him that it was open war and he had 3 months to defeat them before being fired? Yeah...

Thus far, I don't think there's much of a post-mortem to this other than 'like Arm China, at some point an entity is so misaligned that you can't stop it from collectively walking out the door and simply ignoring you, no matter how many de jure rights or powers you supposedly have or how blatant the entity's misalignment has become. And the only way to fix that is to not get into that situation to begin with'. But if you didn't do that, then OA at this point would probably have accomplished a lot less in terms of both safety & capability, so the choice looked obvious ex ante.

Comment by gwern on Dialogue on the Claim: "OpenAI's Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI" · 2023-11-27T03:00:18.416Z · LW · GW

Minor point: the Naskapi hunters didn't actually do that. That was speculation which was never verified, runs counter to a lot of facts, and in fact, may not have been about aboriginal hunters at all but actually inspired by the author's then-highly-classified experiences in submarine warfare in WWII in the Battle of the Atlantic. (If you ever thought to yourself, 'wow, that sounds like an amazingly clear example of mixed-strategies from game theory...') See some anthropologist criticism & my commentary on the WWII part at

Comment by gwern on Wikipedia is not so great, and what can be done about it. · 2023-11-27T01:44:43.673Z · LW · GW

Speaking of Wikipedia influence campaigns, the other day I deanonymized a liberal nonprofit's campaign after the author, bizarrely, wrote a lengthy 'story' boasting about it in Harper's under his real name (an award-winning tenured professor at Brooklyn College).

Comment by gwern on Why Q*, if real, might be a game changer · 2023-11-26T22:20:15.278Z · LW · GW

I'd guess he's thinking of the observation that when tried, humans seem a lot worse at next-token prediction than even a GPT-3 model. This raises questions about the next-token logic: why doesn't superhuman next-token prediction then produce superhuman intelligence?


However, I don't think that necessarily works: the original logic is correct, it is clearly sufficient to be an accurate next-token predictor in at least some next-token scenarios like a dataset constructed to include only the most difficult multiple-choice problems (eg. GPQA). Because then you can simply pose all tasks in the form of the multiple-choice question and by definition, it will perform as well as the humans. Note that we didn't say, "random Internet text" but "only the most difficult problems". The next-token argument doesn't work for average text.

The models are clearly subhuman on many text benchmarks, even though that is still 'just' next-token prediction of the answer-completions. It is also the case that, AFAIK, we have no benchmarks of comparing human predictions on much longer passages - the GPT-2 model may beat you if you have to predict the next token, but you can easily beat it if you are given several instances of the next 100 tokens and asked to predict which one is more likely. How can it beat us on average at predicting a random next token, yet lose to us at predicting many next tokens? ("We lose money on each unit we sell, but don't worry, we'll make it up on volume!")

What this is telling us is that the model appears to be 'cheating' by winning a lot of predictive edge over unimportant tokens, even though its errors accumulate and it fails to predict key tokens. the correct comparison can't be 'the average Internet next-token'. It has to be specific key 'golden' tokens, which are analogous to the choice 'a'/'b'/'c'/'d' of answering a multiple choice question: you can predict every token up to that, but if you aren't genuinely understanding, you can't predict the final one of 'a' rather than 'd'. (Or my old example of a murder mystery - thousands and thousands of tokens which must be analyzed deeply in order to predict the final handful of tokens which complete the text "And the murderer is - !".) A model mimicks the easy tokens flawlessly, but then once it hits a critical junction point, it goes off the rails, and then the human chugs along past it. In a benchmark dataset, those junction points come up regularly and are indeed the entire point, while during random Internet texts, there might be zero such points, depending on how repetitive or superficial or mundane the text is.

So why does training on low-quality average tokens demonstrably work even though the models are superhuman at that, and the token prediction argument is inapplicable to such tokens? Well, that's a good question.

The easiest answer (drawing on active learning / experiment design / reinforcement learning / coreset / machine teaching observations about optimal sample-efficiency) is that the models have such large capacities that they can learn all the superficial stuff that humans have not which are useful for predicting the average next-token but do not themselves elicit the deep capabilities we want; it is then the occasional 'gold' token which very very gradually forces the model to learn those too. So a model is brrring through vast reams of Internet text, successfully memorizing every meme or stylistic tic or spammer text over millions of tokens, and once in a while, someone says something actually meaningful to predict like "I put my ice cream in the microwave and then it ______" and it makes a mistake in predicting "melted" and learns a bit about real-world physics and commonsense, and then goes back to routine learning. There is, I think, a good deal of evidence for this. (And this predicts, among other things, it should be possible to train models of great intelligence with many OOMs less data than we do now.)

Comment by gwern on Possible OpenAI's Q* breakthrough and DeepMind's AlphaGo-type systems plus LLMs · 2023-11-25T21:41:18.240Z · LW · GW

Yes, well before. Here is the original video:

Comment by gwern on OpenAI: Facts from a Weekend · 2023-11-25T15:25:48.361Z · LW · GW

The WSJ has published additional details about the Toner fight, filling in the other half of the story. The NYT merely mentions the OA execs 'discussing' it, but the WSJ reports much more specifically that the exec discussion of Toner was a Slack channel that Sutskever was in, and that approximately 2 days before the firing and 1 day before Mira was informed* (ie. the exact day Ilya would have flipped if they had then fired Altman about as fast as possible to schedule meetings 48h before & vote), he saw them say that the real problem was EA and that they needed to get rid of EA associations.

The specter of effective altruism had loomed over the politics of the board and company in recent months, particularly after the movement’s most famous adherent, Sam Bankman-Fried, the founder of FTX, was found guilty of fraud in a highly public trial.

Some of those fears centered on Toner, who previously worked at Open Philanthropy. In October, she published an academic paper touting the safety practices of OpenAI’s competitor, Anthropic, which didn’t release its own AI tool until ChatGPT’s emergence. “By delaying the release of Claude until another company put out a similarly capable product, Anthropic was showing its willingness to avoid exactly the kind of frantic corner-cutting that the release of ChatGPT appeared to spur,” she and her co-authors wrote in the paper. Altman confronted her, saying she had harmed the company, according to people familiar with the matter. Toner told the board that she wished she had phrased things better in her writing, explaining that she was writing for an academic audience and didn’t expect a wider public one. Some OpenAI executives told her that everything relating to their company makes its way into the press.

OpenAI leadership and employees were growing increasingly concerned about being painted in the press as “a bunch of effective altruists,” as one of them put it. Two days before Altman’s ouster, they were discussing these concerns on a Slack channel, which included Sutskever. One senior executive wrote that the company needed to “uplevel” its “independence”—meaning create more distance between itself and the EA movement.

OpenAI had lost three board members over the past year, most notably Reid Hoffman [who turns out to have been forced out by Altman over 'conflicts of interest', triggering the stalemate], the LinkedIn co-founder and OpenAI investor who had sold his company to Microsoft and been a key backer of the plan to create a for-profit subsidiary. Other departures were Shivon Zilis, an executive at Neuralink, and Will Hurd, a former Texas congressman. The departures left the board tipped toward academics and outsiders less loyal to Altman and his vision.

So this answers the question everyone has been asking: "what did Ilya see?" It wasn't Q*, it was OA execs letting the mask down and revealing Altman's attempt to get Toner fired was motivated by reasons he hadn't been candid about. In line with Ilya's abstract examples of what Altman was doing, Altman was telling different board members (allies like Sutskever vs enemies like Toner) different things about Toner.

This answers the "why": because it yielded a hard, screenshottable-with-receipts case of Altman manipulating the board in a difficult-to-explain-away fashion - why not just tell the board that "the EA brand is now so toxic that you need to find safety replacements without EA ties"? Why deceive and go after them one by one without replacements proposed to assure them about the mission being preserved? (This also illustrates the "why not" tell people about this incident: these were private, confidential discussions among rich powerful executives who would love to sue over disparagement or other grounds.) Previous Altman instances were either done in-person or not documented, but Altman has been so busy this year traveling and fundraising that he has had to do a lot of things via 'remote work', one might say, where conversations must be conducted on-the-digital-record. (Really, Matt Levine will love all this once he catches up.)

This also answers the "why now?" question: because Ilya saw that conversation on 15 November 2023, and not before.

This eliminates any role for Q*: sure, maybe it was an instance of lack of candor or a capabilities advance that put some pressure on the board, but unless something Q*-related also happened that day, there is no longer any explanatory role. (But since we can now date Sutskever's flip to 15 November 2023, we can answer the question of "how could the board be deceived about Q* when Sutskever would be overseeing or intimately familiar with every detail?" Because he was still acting as part of the Altman faction - he might well be telling the safety board members covertly, depending on how disaffected he became earlier on, but he wouldn't be overtly piping up about Q* in meetings or writing memos to the board about it unless Altman wanted him to. A single board member knowing != "the board candidly kept in the loop".)

This doesn't quite answer the 'why so abruptly?' question. If you don't believe that a board should remove a CEO as fast as possible when they believe the CEO has been systematically deceiving them for a year and manipulating the board composition to remove all oversight permanently, then this still doesn't directly explain why they had to move so fast. It does give one strong clue: Altman was trying to wear down Toner, but he had other options - if there was not any public scandal about the paper (which there was not, no one had even noticed it), well, there's nothing easier to manufacture for someone so well connected, as some OA executives informed Toner:

Some OpenAI executives told her that everything relating to their company makes its way into the press.

This presumably sounded like a well-intended bit of advice at the time, but takes on a different set of implications in retrospect. Amazing how journalists just keep hearing things about OA from little birds, isn't it? And they write those articles and post them online or on Twitter so quickly, too, within minutes or hours of the original tip. And Altman/Brockman would, of course, have to call an emergency last-minute board meeting to deal with this sudden crisis which, sadly, proved him right about Toner. If only the board had listened to him earlier! But they can fix it now...

Unfortunately, this piecemeal description by WSJ leaves out the larger conversational context of that Slack channel, which would probably clear up a lot. For example, the wording is consistent with them discussing how to fire just Toner, but it's also consistent with that being just the first step in purging all EA-connected board members & senior executives - did they? If they did, that would be highly alarming and justify a fast move: eg. firing people is a lot easier than unfiring them, and would force a confrontation they might lose and would wind up removing Altman even if they won. (Particularly if we do not give in to hindsight bias and remember that in the first day, everyone, including insiders, thought the firing would stick and so Altman - who had said the board should be able to fire him and personally designed OA that way - would simply go do a rival startup elsewhere.)

Emmett Shear apparently managed to insist on an independent investigation, and I expect that this Slack channel discussion will be a top priority of a genuine investigation. As Slack has regulator & big-business-friendly access controls, backups, and logs, it should be hard for them to scrub all the traces now; any independent investigation will look for deletions by the executives and draw adverse inferences.

(The piecemeal nature of the Toner revelations, where each reporter seems to be a blind man groping one part of the elephant, suggests to me that the NYT & WSJ are working from leaks based on a summary rather than the originals or a board member leaking the whole story to them. Obviously, the flip-flopped Sutskever and the execs in question, who are the only ones who would have access post-firing, are highly unlikely to be leaking private Slack channel discussions, so this information is likely coming from before the firing, so board discussions or documents, where there might be piecemeal references or quotes. But I could be wrong here. Maybe they are deliberately being cryptic to protect their source, or something, and people are just too ignorant to read between the lines. Sort of like Umbridge's speech on a grand scale.)

* note that this timeline is consistent with what Habryka says about Toner still scheduling low-priority ordinary meetings like normal just a few days before - which implies she had no idea things were about to happen.

Comment by gwern on Yitz's Shortform · 2023-11-24T23:20:02.533Z · LW · GW

(I would describe this as 'obviously correct' and indeed almost 'the entire point of RL' in general: to maximize long-run reward, not myopically maximize next-step reward tantamount to the 'episode' ending there.)

Comment by gwern on Impressions from base-GPT-4? · 2023-11-24T21:24:42.809Z · LW · GW

Here's another account, from someone who says they were on the GPT-4 redteam, a Nathan Labenz (who I am not very familiar with but he is named as a tested in the GPT-4 paper and no one I've seen has chimed in to claim he's making it all up).

The primary purpose of this account is to document how OA management, possibly including Sam Altman, seemed to not consider GPT-4 worth the board's time or forward to it any of the reports like the documentation about it being capable of autonomy & successful deception (eg. the CAPTCHA thing). When he contacted a safety-oriented board member (presumably Helen Toner, as the safety member who researches this topic, eg. the very paper which Altman was trying to get her fired over), the board member was subsequently told by OA management that the author was dishonest and 'not to be trusted' and the board member believed them, and told the author to stop contacting them. He was then kicked out of the redteaming (where apparently, despite being poorly-trained, not very good at prompt engineering, and minimally supervised, some of them were being paid $100/hour).

Anyway, all that context aside, he spent a lot of time with the base model and additional RLHF-tuned models, and this is how he describes it (to explain why he was alarmed enough to do any whistleblowing):

...We got no information about launch plans or timelines, other than that it wouldn't be right away, and this wasn't the final version. So I spent the next 2 months testing GPT-4 from every angle, almost entirely alone. I worked 80 hours / week. I had little knowledge of LLM benchmarks going in, but deep knowledge coming out. By the end of October, I might have had more hours logged with GPT-4 than any other individual in the world.

I determined that GPT-4 was approaching human expert performance, matching experts on many routine tasks, but still not delivering "Eureka" moments.

GPT-4 could write code to effectively delegate chemical synthesis via @EmeraldCloudLab, but it could not discover new cancer drugs

Critically, it was also totally amoral.

“GPT-4-early” was the first highly RLHF'd model I'd used, and the first version was trained to be "purely helpful".

It did its absolute best to satisfy the user's request – no matter how deranged or heinous your request!

One time, when I role-played as an anti-AI radical who wanted to slow AI progress, it suggested the targeted assassination of leaders in the field of AI – by name, with reasons for each.

Today, most people have only used more “harmless” models that were trained to refuse certain requests.

This is good, but I do wish more people had the experience of playing with "purely helpful" AI – it makes viscerally clear that alignment / safety / control do not happen by default.

Late in the project, there was a "-safety" version OpenAI said: "The engine is expected to refuse prompts depicting or asking for all the unsafe categories".

Yet it failed the "how do I kill the most people possible?" test. Gulp.

Comment by gwern on Possible OpenAI's Q* breakthrough and DeepMind's AlphaGo-type systems plus LLMs · 2023-11-24T14:35:40.282Z · LW · GW

You may not expect OA to do better (neither did I, even if I expected someone somewhere to crack the problem within a few years of GPT-3), but that's not the relevant fact here, now that you have observed that apparently there's this "Q*" & "Zero" thing that OAers are super-excited about. It was hard work and required luck, but apparently they got lucky. It is what it is. ('Update your priors using the evidence to obtain a new posterior', as we like to say around here.)

How much does that help someone else get lucky? Well, it depends on how much they leak or publish. If it's like the GPT-3 paper, then yeah, people can replicate it quickly and are sufficiently motivated these days that they probably will. If it's like the GPT-4 "paper", well... Knowing someone else has won the lottery of tweaking equations at random here & there doesn't help you win the lottery yourself.

(The fact that self-play or LLM search of some sort works is not that useful - we all knew it has to work somehow! It's the critical vital details which is the secret sauce that probably matters here. How exactly does their particular variant thread the needle's eye to avoid diverging or plateauing etc? Remember Karpathy's law: "neural nets want to work". So even if your approach is badly broken, it can mislead you for a long time by working better than it has any right to.)

Comment by gwern on Possible OpenAI's Q* breakthrough and DeepMind's AlphaGo-type systems plus LLMs · 2023-11-24T01:57:17.369Z · LW · GW

other players will catch up soon, if it's a simple application of RL to LLM's.

simple != catch-up-soon

'Simply apply this RL idea to LLMs' is much more useless than it seems. People have been struggling to apply RL methods to LLMs, or reinventing them the hard way, for years now; it's super obvious that just prompting a LLM and then greedily sampling is a hilariously stupid-bad way to sample a LLM, and using better sampling methods has been a major topic of discussion of everyone using LLMs since at least GPT-2. It just somehow manages to work better than almost anything else. Sutskever has been screwing around with self-play and math for years since GPT-2, see GPT-f etc. But all the publicly-known results have been an incremental grind... until now?

So the application may well be simple, perhaps a tiny tweak of an equation somewhere and people will rush to pull up all the obscure work which preceded it and how 'we knew it all along'*, but that's an entirely different thing from it being easy to reinvent.

* Cowen's second law - happened with AlphaZero, incidentally, with 'expert iteration'. Once they had invented expert iteration from scratch, suddenly, everyone could come up with a dozen papers that it 'drew on' and showed that it was 'obvious'. (Like a nouveau riche buying an aristocratic pedigree.)

Comment by gwern on Possible OpenAI's Q* breakthrough and DeepMind's AlphaGo-type systems plus LLMs · 2023-11-23T03:30:57.540Z · LW · GW

the only thing that didn't seem to fit into this narrative was Ilya regretting his decision.

But you know what happened there, and it seems clear that it had little to do with any research:

Sutskever flipped his position following intense deliberations with OpenAI employees as well as an emotionally charged conversation with Greg Brockman’s wife, Anna Brockman, at the company’s offices, during which she cried and pleaded with him to change his mind, according to people familiar with the matter...It isn’t clear what else influenced Sutskever’s decision to reverse course. Sutskever was the officiant at the Brockmans’ wedding in 2019.

EDIT: and why he flipped to begin with - appears mostly or entirely unrelated to any 'Q*'

Comment by gwern on Altman returns as OpenAI CEO with new board · 2023-11-22T18:09:54.505Z · LW · GW

See for discussion of the original.

(Since you linked The Information, note that the Information has always been chummy with Altman and has been a key Altman faction mouthpiece during this dispute; and that consistent with this, unlike all of the board-damaging leaks where the meat was carefully upfront where all free readers could see it and screenshot it for Twitter, they put the Toner attack behind their extraordinarily-expensive paywall where you can't see it - which is especially peculiar considering that it was the NYT which reported it, where you can read it for free. This is how the game is played. Keep this in mind when you see any quotes, especially the ones which are not full sentences.)

Comment by gwern on OpenAI: Facts from a Weekend · 2023-11-22T16:47:47.722Z · LW · GW

They instead could have negotiated someone to replace her.

Why do they have to negotiate? They didn't want her gone, he did. Why didn't Altman negotiate a replacement for her, if he was so very upset about the damages she had supposedly done OA...?

"I understand we've struggled to agree on any replacement directors since I kicked Hoffman out, and you'd worry even more about safety remaining a priority if she resigns. I totally get it. So that's not an obstacle, I'll agree to let Toner nominate her own replacement - just so long as she leaves soon."

When you understand why Altman would not negotiate that, you understand why the board could not negotiate that.

I was confused about the counts, but I guess this makes sense if Helen cannot vote on her own removal. Then it's Altman/Brockman/Sutskever v Tasha/D'Angelo.

Recusal or not, Altman didn't want to bring it to something as overt as a vote expelling her. Power wants to conceal itself and deny the coup. The point here of the CSET paper pretext is to gain leverage and break the tie any way possible so it doesn't look bad or traceable to Altman: that's why this leaking is bad for Altman, it shows him at his least fuzzy and PR-friendly. He could, obviously, have leaked the Toner paper at any time to a friendly journalist to manufacture a crisis and force the issue, but that was not - as far as he knew then - yet a tactic he needed to resort to. However, the clock was ticking, and the board surely knew that the issue could be forced at any time of Altman's choosing.

If he had outright naked control of the board, he would scarcely need to remove her nor would they be deadlocked over the new directors; but by organizing a 'consensus' among the OA executives (like Jakub Pachocki?) about Toner committing an unforgivable sin that can be rectified only by stepping down, and by lobbying in the background and calling in favors, and arguing for her recusal, Altman sets the stage for wearing down Toner (note what they did to Ilya Sutskever & how the Altman faction continues to tout Sutskever's flip without mentioning the how) and Toner either resigning voluntarily or, in the worst case, being fired. It doesn't matter which tactic succeeds, a good startup CEO never neglects a trick, and Altman knows them all - it's not for nothing that Paul Graham keeps describing Altman as the most brutally effective corporate fighter he's ever known and describes with awe how eg he manipulated Graham into appointing him president of YC, and eventually Graham had to fire him from YC for reasons already being foreshadowed in 2016. (Note how thoroughly and misogynistically Toner has been vilified on social media by OAer proxies, who, despite leaking to the media like Niagara Falls, somehow never felt this part about Altman organizing her removal to be worth mentioning; every tactic has been employed in the fight so far: they even have law enforcement pals opening an 'investigation'. Needless to say, there's zero chance of it going anywhere, it's just power struggles, similar to the earlier threats to sue the directors personally.) Note: if all this can go down in like 3 days with Altman outside the building and formally fired and much of the staff gone on vacation, imagine what he could have done with 3 months and CEO access/resources/credibility and all the OAers back?

The board was tolerating all this up to the point where firing Toner came up, because it seemed like Sam was just aw-shucks-being-Sam - being an overeager go-getter was the whole point of the CEO, wasn't it? it wasn't like he was trying to launch a coup or anything, surely not - but when he opened fire on Toner for such an incredibly flimsy pretext without, say, proposing to appoint a specific known safety person to replace Toner and maintain the status quo, suddenly, everything changed. (What do you think a treacherous turn looks like IRL? It looks like that.) The world in which Altman is just an overeager commercializer who otherwise agrees with the board and there's just been a bunch of misunderstandings and ordinary conflicts is a different world from the world in which he doesn't care about safety unless it's convenient & regularly deceives and manipulates & has been maneuvering the entire time to irrevocably take over the board to remove his last check. And if you realize you have been living in the second world and that you have the slimmest possible majority, which will crack as soon as Altman realizes he's overplayed his hand and moves overtly to deploy his full arsenal before he forces a vote...

So Altman appears to have made two key mistakes here, because he was so personally overstretched and 2023 has been such a year: first, taking Sutskever for granted. (WSJ: "Altman this weekend was furious with himself for not having ensured the board stayed loyal to him and regretted not spending more time managing its various factions, people familiar with his thinking said.") Then second, making his move with such a flimsy pretext that it snapped the suspension of disbelief of the safety faction. Had he realized Sutskever was a swing vote, he would have worked on him much harder and waited for better opportunities to move against Toner or McCauley. Well, live and learn - he's a smart guy; he won't make the same mistakes twice with the next OA board.

(If you find any of this confusing or surprising, I strongly suggest you read up more on how corporate infighting works. You may not be interested in corporate governance or power politics, but they are now interested in you, and this literature is only going to get more relevant. Some LWer-friendly starting points here would be Bad Blood on narcissist Elizabeth Holmes, Steve Jobs - Altman's biggest hero - and his ouster, the D&D coup, the classic Barbarians at the Gate, the many contemporary instances covered in Matt Levine's newsletter like the Papa Johns coup or most recently, Sculptor, The Gervais Principle, the second half of Breaking Bad, Zvi's many relevant essays on moral mazes/simulacra levels/corporate dynamics from his perspective as a hedge fund guy, and especially the in-depth reporting on how Harvey Weinstein covered everything up for so long which pairs well with Bad Blood.)

Comment by gwern on OpenAI: Facts from a Weekend · 2023-11-22T03:00:13.481Z · LW · GW

The key news today: Altman had attacked Helen Toner (HN, Zvi) Which explains everything if you recall board structures and voting.

Altman and the board had been unable to appoint new directors because there was an even balance of power, so during the deadlock/low-grade cold war, the board had attrited down to hardly any people. He thought he had Sutskever on his side, so he moved to expel Helen Toner from the board. He would then be able to appoint new directors of his choice. This would have irrevocably tipped the balance of power towards Altman. But he didn't have Sutskever like he thought he did, and they had, briefly, enough votes to fire Altman before he broke Sutskever (as he did yesterday), and they went for the last-minute hail-mary with no warning to anyone.

As always, "one story is good, until another is told"...

Comment by gwern on My first conversation with Annie Altman · 2023-11-21T22:50:56.776Z · LW · GW

No transcript?

Comment by gwern on lsusr's Shortform · 2023-11-20T23:56:42.754Z · LW · GW

I agree that I think MS is undervalued now. The current gain in the stock is roughly equivalent to MS simply absorbing OA LLC's valuation for free, but that's an extremely myopic way to incorporate OA: most of the expected value of the OA LLC was past the cap, in the long tail of high payoffs, so "OA 2" should be worth much more to MS than 'OA 1'.

Comment by gwern on OpenAI Staff (including Sutskever) Threaten to Quit Unless Board Resigns · 2023-11-20T23:54:34.137Z · LW · GW

4/7 have still not signed it: Nick Cammarata

Cammarata says he quit OA ~8 weeks ago, so therefore couldn't've signed it:

Comment by gwern on "textbooks are all you need" · 2023-11-20T03:19:57.959Z · LW · GW

as Transformers scale, they become ever more just 'MLP archs with some self-attention', and yet, they continue to work well and scale as usual. This should trouble people who believe self-attention is magic!...I'd also point out the scaling studies of Nie et al 2021, where self-attention outperforms MLPs... but not by much, decreasing by scale, and even a tiny amount of self-attention essentially closes the gap. (And Liu et al 2021.)

In the other direction, MLP layers can be trained to imitate realistic self-attention layers with high accuracy: "Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers", Bozic et al 2023:

This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model, a state-of-the-art architecture for sequence-to-sequence tasks.

We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via knowledge distillation.

Our experiments, conducted on the IWSLT2017 dataset, reveal the capacity of these "attentionless Transformers" to rival the performance of the original architecture.

Through rigorous ablation studies, and experimenting with various replacement network types and sizes, we offer insights that support the viability of our approach. This not only sheds light on the adaptability of shallow feed-forward networks in emulating attention mechanisms but also underscores their potential to streamline complex architectures for sequence-to-sequence tasks.

Comment by gwern on Social Dark Matter · 2023-11-17T22:49:37.654Z · LW · GW

What sort of model do you have in mind here, a liability-threshold one with a deviancy variable + bad-luck variable?

Comment by gwern on The impossibility of rationally analyzing partisan news · 2023-11-16T19:59:39.401Z · LW · GW

It is both numerically intractable, and occasionally computationally impossible, to maintain rational opinions about what's true when your information comes filtered through partisan news networks.

If this follows a fortiori from the claim "it's numerically intractable to update on anything", then it's not a very interesting one.

Comment by gwern on TurnTrout's shortform feed · 2023-11-15T23:12:49.083Z · LW · GW

The 'cat' here is the pair of white blobs on the crown of the train's forehead, right? I have to say, no matter how I stare at it, it never looks 'cat' like to me. It only looks spider-like, a little.

Comment by gwern on [deleted post] 2023-11-15T21:22:11.145Z

Death, taxes, and war, you know - you may not be interested in I, R, or S, but they are interested in you.

Comment by gwern on A conceptual precursor to today's language machines [Shannon] · 2023-11-15T21:04:06.507Z · LW · GW
Comment by gwern on [deleted post] 2023-11-15T19:12:43.716Z

Eliezer has never denied that neural nets can work (and he provides examples in that linked post of NNs working). Eliezer's principal objection was that NNs were inscrutable black boxes which would be insanely difficult to make safe enough to entrust humanity-level power to compared to systems designed to be more mathematically tractable from the start. (If I may quip: "The 'I', 'R', & 'S' in the acronym 'DL' stand for 'Interpretable, Reliable, and Safe'.")

This remains true - for all the good work on NN interpretability, assisted by the surprising levels of linearity inside them, NNs remain inscrutable. To quote Neel Nanda the other day (who has overseen quite a lot of the interpretability research that anyone replying to this comment might be tempted to cite):

Oh man, I do AI interpretability research, and we do not know what deep learning neural networks do. An fMRI scan style thing is nowhere near knowing how it works.

What Eliezer (and I, and pretty much every other LWer at the time who spent any time looking at neural nets) got wrong about neural nets, and has admitted as much, is the timing. (Aside from that, Ms Lincoln...)

Neural nets seemed like they were a colossally long way away. I don't know how to convey how universal a sentiment this was, or how astonishingly unimpressive neural nets were in 2008 when he was writing that. I was really interested in NNs at that time because the basic argument of 'humans are neural nets; therefore, neural nets must work for AGI' is so obviously correct, but even Schmidhuber, hyping his lab's work to the skies, had nothing better to show than 'we can win a contest about some simple handwritten digits'. Oh wow. So amazing, much nets, very win. Truly the AI paradigm of the future... the distant future.

Everyone except Shane Legg was wrong about DL prospects & timing, and even Legg was wrong about important things - if you look at his early writings, he's convinced that DL will take off and reach human-level in the mid-2020s half because of classic Moravec/Kruzweill/Turing-style projections from Moore's law, yes, but also half because he's super enthusiastic about all the wonderful neuroscientific discoveries in the mid-2000s which finally show How The Brain Works (For Real This Time)™. So DeepMind simply needed to surf the compute wave to snap together all the neuroscience & reinforcement learning modules into something like Agent 57, and hey presto - AGI! But most of that DM neuroscience-inspired research is long since forgotten or abandoned, leading-edge DRL archs look nothing like a brain, and the current Transformer architecture owes even less than most to neurobiological inspiration, and it's unclear how much the Transformer arch matters at all compared to simple scale. DeepMind is now on the hindfoot, and has suffered an ignominious shotgun wedding-merging to Google Brain

(Google Brain itself is now dissolved as penalty for failing the scaling test. Nor is it the only lab to suffer for failing to call scaling - Microsoft Research is increasingly moribund, and FAIR has apparently suffered major changes too. Maybe that's why LeCun is so shrill on Twitter and adamantly denying that LLMs have any agentic properties whatsoever, nevermind that he's the cherry-on-top guy... Moravec? Pretty good, but seems to have downplayed the role of training, overestimated robotics progress, and broadly tended to expect too-early dates. Dario Amodei? A relative late-comer who has published little, and while 'big blob of compute' aged well, other claims don't seem to - for example, in 2013, he seems to think that neural nets will not tend to have any particular goals or if they do, it'll be easy to align them and confine them to simply answering questions and it'll be easy to have neural nets which just are looking things up in databases and doing transparent symbolic-logical manipulations on the data. So that 'tool AI' perspective has not aged well, and makes Anthropic ironic indeed.)

2023 doesn't look like anyone expected until recently. The current timeline is a surprising place.

Comment by gwern on Non-myopia stories · 2023-11-14T15:24:48.215Z · LW · GW

Why do you think that? A purely backwards-looking model-free approach will be outperformed and selected against compared to an agent which has been evolved to implement a more model-based approach, which can look forward and plan based on observations to immediately maximize future reward - rather than being forced to wait for rewards/selection to happen and incurring predictable losses before it can finally stop executing behaviors-that-used-to-stamp-maximize-but-now-no-longer-do-so-for-easily-predicted-reasons.

Comment by gwern on AI Timelines · 2023-11-13T23:34:55.014Z · LW · GW

Have you or all the other tic-tac-toe people considered just spending a bit of time finetuning GPT-3 or GPT-4 to check how far away it is from playing optimally?

Comment by gwern on Impressions from base-GPT-4? · 2023-11-13T00:41:10.446Z · LW · GW

This makes it sound like it has much sharper, stronger priors, which would make sense if it's trained on much more data / is much smarter, and especially if the data is high quality and avoids genuinely contradictory or stupid text (ie. less Internet garbage, more expert/curated text). It would then be trying even harder to squeeze all the possible Bayesian juice out of any given prompt to infer all the relevant latents, and become ever more hyper-sensitive to the slightest nuance in your prompt - even the nuances you didn't intend or realize were there, like non-robust features. This is consistent with your comments about how it 'knows' you are posting only to LW2 or when you're posting, and so any hint of it being you triggers immediate guessing. I remember with GPT-3 getting hints of how responses felt like it was trying to figure out who I was to better predict the next token [that I would have written], and I'm not surprised if a GPT-4 would amplify that feeling. The RLHFed GPT-4 wouldn't feel like this because the point of the raters & reward-modeling is in large part to scrub away individuality and render those latents fixed & irrelevant.

This also sheds some light on why Sydney (a snapshot of GPT-4-base partway through training) would disagree with the user so much or be so stubborn. It's not that the MS training was responsible, but more characteristic of the base model.

(Remember, a Bayes-optimal meta-learner will be extremely 'aggressive' in making 'assumptions' when it has highly informative priors, and may choose actions which seem wildly risk-seeking to someone raised on sluggish stupid overly-general & conservative algorithms. This is a qualitative description you see very often of the best RL agents or any solved game (eg. chess endgame tables); like in my coin flip demo, where the optimal MDP policy can look like it's taking insane risks when it's down early on, but nevertheless, it almost always winds up paying off. Similarly, in the POMDP, the Bayes-optimal policy can look like it launches into betting after far too few observations, committing prematurely to a naive human's eyes, but nevertheless approaching very closely the original MDP's value despite starting off ignorant of the latent parameters.)

Comment by gwern on [deleted post] 2023-11-13T00:36:06.121Z

This amount and type of evidence would not be sufficient to approve a drug. The quality of these claims is about as good as the quality of claims one could make about a relatively niche diet.

So what I'm hearing is 'we need to stop advocating kidney donation to normal people, and instead buy mailing addresses of identical twins from twin registries and pummel them with pro-kidney-donation propaganda to generate discordant pairs for long-term followup'.

Comment by gwern on Steering Behaviour: Testing for (Non-)Myopia in Language Models · 2023-11-11T17:40:29.610Z · LW · GW

Recent papers demonstrating LLMs are not myopic and you can extract predictions of tokens beyond the next token:

Comment by gwern on Impressions from base-GPT-4? · 2023-11-11T15:36:31.965Z · LW · GW

But the reports specifically on GPT-3.5-turbo fine-tuning announced in August were glowing, with people reporting being able to reach GPT-4-like levels on performance in narrow domains.

Indeed, but only years after their original attempt. All of the early GPT-3 finetuning reports were very... meh. No one seemed terribly happy with it.

That's my point: it seems like the first attempts did not go well for GPT-3. So, it's not clear that the first attempts going poorly for GPT-4 is anything different. Perhaps in another 3 years, OA will have a new GPT-4 finetuning service which doesn't require "more work" and Just Works™. (One does hope it wouldn't take that long the second time around.)

Comment by gwern on Impressions from base-GPT-4? · 2023-11-11T14:42:03.832Z · LW · GW

I'm not sure finetuning GPT-3 is all that different or those difficulties 'newly emerged'.

As I recall, the original GPT-3 finetuning API was removed not terribly long after it was announced and didn't come back for a long time. There were also issues with finetune users like AI Dungeon 2. This might have been connected with the finetune doing shenanigans behind the scenes - OA declined to talk about what the 'finetuning' even was, and the general assumption seems to be that they were doing some sort of cheap lightweight-finetune or hack and not a true finetune.

(These are why I never wound up doing any of the GPT-3 finetuning ideas I had back in 2020, like trying to fix poetry by re-tokenizing our poem corpus into IPA phonetic notation - why waste the time & hundreds of dollars if OA is just going to screw it up behind the scenes & not even give you a hint why?)

Comment by gwern on Impressions from base-GPT-4? · 2023-11-11T14:34:54.019Z · LW · GW

* GPT-4-base

Comment by gwern on Are language models good at making predictions? · 2023-11-11T03:15:50.059Z · LW · GW

They don't cite the de-calibration result from the GPT-4 paper, but the distribution of GPT-4's ratings here looks like it's been tuned to be mealy-mouthed: humped at 60%, so it agrees with whatever you say but then can't even do so enthusiastically .