As the Trump transition continues and we try to steer and anticipate its decisions on AI as best we can, there was continued discussion about one of the AI debate’s favorite questions: Are we making huge progress real soon now, or is deep learning hitting a wall? My best guess is it is kind of both, that past pure scaling techniques are on their own hitting a wall, but that progress remains rapid and the major companies are evolving other ways to improve performance, which started with OpenAI’s o1.
Point of order: It looks like as I switched phones, WhatsApp kicked me out of all of my group chats. If I was in your group chat, and you’d like me to stay, please add me again. If you’re in a different group you’d like me to join on either WhatsApp or Signal (or other platforms) and would like me to join, I’ll consider it, so long as you’re 100% fine with me leaving or never speaking.
That doesn’t mean the scientists were miscalibrated. Actually seeing the AI get used is evidence, and is far more likely to point towards it having value, because otherwise why have them use it?
AI powered binoculars for bird watching. Which parts of bird watching produce value, versus which ones can we automate to improve the experience? How much ‘work’ should be involved, and which kinds? A microcosm of much more important future problems, perhaps?
Kendal Colton (earlier): A big barrier to integrating Al w/ healthcare will be occupational licensing. If a programer writes an Al algorithm to perform simple diagnostic tests based on available medical literature and imputed symptoms, must that be regulated as the “practice of medicine”?
Kendal Colton: As I predicted, occupational licensing will be a big barrier to integrating AI w/ healthcare. This isn’t some flex, it needs addressed. Medical diagnostics is ripe for AI disruption that will massively improve our health system, but regulations could hold it back.
Elon Musk: You can upload any image to Grok, including medical imaging, and get its (non-doctor) opinion.
Grok accurately diagnosed a friend of mine from his scans.
Ryan Marino, M.D.: Saying Grok can provide medical diagnoses is illegal, actually.
Girl, it literally says “diagnosed.” Be for real for once in your sad life.
Thamist: He said it’s a non doctor opinion and that it helped to get his friend to a doctor to get a real diagnosis but somehow as a doctor you are too stupid to read.
Ryan Marino: “Diagnosed.”
The entire thread (2.6m views) from Marino comes off mostly as an unhinged person yelling how ‘you can’t do this to me! I have an MD and you don’t! You said the word diagnose, why aren’t they arresting you? Let go of me, you imbeciles!’
This is one front where things seem to be going spectacularly well.
My immediate response is that you need to measure QALYs rather than years, but yes, if you are going to do socialized medicine rather than allocation by price then those who benefit most should presumably get the livers. It also makes sense not to care about who has waited longer – ‘some people will never get a liver’ isn’t avoidable here.
The problem is it didn’t even calculate years of life, it only calculated likelihood of surviving five years. So what the algorithm actually did in practice, however, was:
“If you’re below 45 years, no matter how ill, it is impossible for you to score high enough to be given priority scores on the list,” said Palak Trivedi, a consultant hepatologist at the University of Birmingham, which has one of the country’s largest liver transplant centres.
…
The cap means that the expected survival with a transplant for most patient groups is about the same (about 4.5 years, reflecting the fact that about 85% of patients survive 5 years after a transplant). So the utility of the transplant, while high, is more-or-less uniformly high, which means that it doesn’t really factor into the scores! It turns out that the algorithm is mostly just assessing need, that is, how long patients would survive without a transplant.
This is ironic because modeling post-transplant survival was claimed to be the main reason to use this system over the previous one.
None of that is the fault of the AI. The AI is correctly solving the problem you gave it.
‘Garbage in, garbage out’ is indeed the most classic of alignment failures. You failed to specify what you want. Whoops. Don’t blame the AI, also maybe don’t give the AI too much authority or ability to put it into practice, or a reason to resist modifications.
The second issue is that they point to algorithmic absurdities.
They show that [one of the algorithms used] expects patients with cancer to survive longer than those without cancer (all else being equal).
…
The finding is reminiscent of a well-known failure from a few decades ago wherein a model predicted that patients with asthma were at lower risk of developing complications from pneumonia. Fortunately this was spotted before the model was deployed. It turned out to be a correct pattern in the data, but only because asthmatic patients were sent to the ICU, where they received better care. Of course, it would have been disastrous to replace that very policy with the ML model that treated asthmatic patients as lower risk.
Once again, you are asking the AI to make a prediction about the real world. The AI is correctly observing what the data tells you. You asked the AI the wrong questions. It isn’t the AI’s result that is absurd, it is your interpretation of it, and assuming that correlation implies causation.
The cancer case is likely similar to the asthma case, where slow developing cancers lead to more other health care, and perhaps other measurements are being altered by the cancers that have a big impact on the model, so the cancer observation itself gets distorted.
If you want to ask the AI, what would happen if we treated everyone the same? Or if you only looked at this variable in isolation? Then you have to ask that question.
The third objection is:
Predictive logic bakes in a utilitarian worldview — the most good for the greatest number. That makes it hard to incorporate a notion of deservingness.
No? That’s not what it does. The predictive logic prevents us from hiding the utilitarian consequences.
You can still choose to go with the most deserving, or apply virtue ethics or deontology. Or you can incorporate ‘deserving’ into your utilitarian calculation. Except that now, you can’t hide from what you are doing.
Trivedi [the hepatologist] said patients found [the bias against younger patients] particularly unfair, because younger people tended to be born with liver disease or develop it as children, while older patients more often contracted chronic liver disease because of lifestyle choices such as drinking alcohol.
Okay, well, now we can have the correct ethical discussion. Do we want to factor in lifestyle choices into who gets the livers, or not? You can’t have it both ways, and now you can’t use proxy measures to do it without admitting you are doing it. If you have an ‘ethical’ principle that says you can’t take that into consideration, that is a reasonable position with costs and benefits, but then don’t own that. Or, argue that this should be taken into account, and own that.
Donor preferences are also neglected. For example, presumably some donors would prefer to help someone in their own community. But in the utilitarian worldview, this is simply geographic discrimination.
This is an algorithmic choice. You can and should factor in donor preferences, at least to the extent that this impacts willingness to donate, for very obvious reasons.
Again, don’t give me this ‘I want to do X but it wouldn’t be ethical to put X into the algorithm’ nonsense. And definitely don’t give me a collective ‘we don’t know how to put X into the algorithm’ because that’s Obvious Nonsense.
The good counterargument is:
Automation has also privileged utilitarianism, as it is much more amenable to calculation. Non-utilitarian considerations resist quantification.
Indeed I have been on the other end of this and it can be extremely frustrating. In particular, hard to measure second and third order effects can be very important, but impossible to justify or quantify, and then get dropped out. But here, there are very clear quantifiable effects – we just are not willing to quantify them.
No committee of decision makers would want to be in charge of determining how much of a penalty to apply to patients who drank alcohol, and whatever choice they made would meet fierce objection.
Before, you hid and randomized and obfuscated the decision. Now you can’t. So yes, they get to object about it. Tough.
Overall, we are not necessarily against this shift to utilitarian logic, but we think it should only be adopted if it is the result of a democratic process, not just because it’s more convenient.
…
Nor should this debate be confined to the medical ethics literature.
The previous system was not democratic at all. That’s the point. It was insiders making opaque decisions that intentionally hid their reasoning. The shift to making intentional decisions allows us to have democratic debates about what to do. If you think that’s worse, well, maybe it is in many cases, but it’s more democratic, not less.
In this case, the solution is obvious. At minimum: We should use the NPV of a patient’s gain in QALYs as the basis of the calculation. An AI is fully capable of understanding this, and reaching the correct conclusions. Then we should consider what penalties and other adjustments we want to intentionally make for things like length of wait or use of alcohol.
A huge percentage of uses of image models require being able to faithfully work from a particular person’s image. That is of course exactly how deepfakes are created, but if it’s stylized as it is here then that might not be a concern.
Deepfaketown and Botpocalypse Soon
This post was an attempt to say that AI didn’t directly ruin the election and there is no evidence it had ‘material impact’ it is still destroying our consensus reality, enabling lies, by making it harder to differentiate what is real, which I think is real but also largely involves forgetting how bad it used to be already.
My assessment is that the 2024 election involved much less AI than we expected, although far from zero, and that this should update us towards being less worried about that particular type of issue. But 2028 is eons away in AI progress time. Even if we’re not especially close to AGI by then, it’ll be a very different ballgame, and also I expect AI to definitely be a major issue, and plausibly more than that.
How do people feel about AI designed tattoos? As you would expect, many people object. I do think a tattoo artist shouldn’t put an AI tattoo on someone without telling them first. It did seem like ‘did the person know it was AI?’ was key to how they judged it. On the other end, certainly ‘use AI to confirm what the client wants, then do it by hand from scratch’ seems great and fine. There are reports AI-designed tattoos overperform. If so, people will get used to it.
Copyright Confrontation
SDNY Judge Colleen McMahon dismisses Raw Story vs. OpenAI, with the ruling details being very good for generative AI. It essentially says that you have to both prove actual harm, and you have to show direct plagiarism which wasn’t clearly taking place in current models, whereas using copyrighted material for training data is legal.
Key Tryer: At this point is so very obvious to me that outcomes wrt copyright and AI will come out in favor of AI that seeing people still arguing about it is kind of absurd.
There’s still a circle on Twitter who spend every waking hour telling themselves that copyright law will come down to shut down AI and they’re wrong about almost everything, it’s like reading a forum by Sovereign Citizens types.
This isn’t the first ruling that says something like this, but probably one of the most clear ones. Almost all the Saveri & Butterick lawsuits have had judges say basically these same things, too.
I think it’s probably going this way under current law, but this is not the final word from the courts, and more importantly the courts are not the final word. Your move, Congress.
Grey Swan is having another fun jailbreaking competition. This time, competitors are being asked to produce violent and self-harm related content, or code to target critical infrastructure. Here aretherules. You can sign up here. There’s $1k bounty for first jailbreak of each model.
Applications must be submitted by November 30, 2024. Each submission will be reviewed by a member of AISI’s technical staff. Evaluation applicants who successfully proceed to the second stage (building the evaluation) will receive an award of £2,000 for compute expenditures. We will work with applicants to agree on a timeline for the final submission at this point. At applicants’ request, we can match you with other applicants who are excited about working on similar ideas.
Full bounty payments will be made following submission of the resulting evaluations that successfully meet our criteria. If your initial application is successful, we will endeavour to provide information as early as possible on your chances of winning the bounty payout. The size of the bounty payout will be based on the development time required and success as measured against the judging criteria. To give an indication, we expect to reward a successful task with £100-200 per development hour. This means a successful applicant would receive £3000-£15,000 for a successful task, though we will reward exceptionally high-quality and effortful tasks with a higher payout.
EpochAI: Existing math benchmarks like GSM8K and MATH are approaching saturation, with AI models scoring over 90 percent—partly due to data contamination. FrontierMath significantly raises the bar. Our problems often require hours or even days of effort from expert mathematicians.
We evaluated six leading models, including Claude 3.5 Sonnet, GPT-4o, and Gemini 1.5 Pro. Even with extended thinking time (10,000 tokens), Python access, and the ability to run experiments, success rates remained below 2 percent—compared to over 90 percent on traditional benchmarks.
FrontierMath spans most major branches of modern mathematics—from computationally intensive problems in number theory to abstract questions in algebraic geometry and category theory. Our aim is to capture a snapshot of contemporary mathematics.
Evan Chen: These are genuinely hard problems. Most of them look well above my pay grade.
Timothy Gowers: Getting even one question right would be well beyond what we can do now, let alone saturating them.
Terrance Tao: These are extremely challenging. I think they will resist AIs for several years at least.
Dan Hendrycks: This has about 100 questions. Expect more than 20 to 50 times as many hard questions in Humanity’s Last Exam, the scale needed for precise measurement.
As we clean up the dataset, we’re accepting questions at http://agi.safe.ai.
Noam Brown: I love seeing a new evaluation with such low pass rates for frontier models. It feels like waking up to a fresh blanket of snow outside, completely untouched.
This is on the one hand an obvious ‘what could possibly go wrong?’ moment and future Gilligan cut, but it does seem like a fairly correct thing to be doing. If you think it’s bad to be using your AI to do confidential government work then you should destroy your AI.
One entity that disagrees with Anthropic’s decision here? Claude, with multiple reports of similar responses.
Writer claims they are raising at a valuation of $1.9 billion, with a focus on using synthetic data to train foundation models, aiming for standard corporate use cases. This is the type of business I expect to have trouble not getting overwhelmed.
Tencent’s new Hunyan-389Bopen weights model has evaluations that generally outperform Llama-3.1-405B. As Clark notes, there is no substitute for talking to the model, so it’s too early to know how legit this is. I do not buy the conclusion that only lack of compute access held Tencent back from matching our best and that ‘competency is everywhere it’s just compute that matters.’ I do think that a basic level of ‘competency’ is available in a lot of places but that is very different from enough to match top performance.
Eliezer Yudkowsky says compared to 2022 or 2023, 2024 was a slow year for published AI research and products. I think this is true in terms of public releases, it was fast, faster than almost every other space, but not as fast as AI was the last 2 years. The labs are all predicting it goes faster from here.
Tim Dettmers: This is the most important paper in a long time . It shows with strong evidence we are reaching the limits of quantization. The paper says this: the more tokens you train on, the more precision you need. This has broad implications for the entire field and the future of GPUs.
Arguably, most progress in AI came from improvements in computational capabilities, which mainly relied on low-precision for acceleration (32-> 16 -> 8 bit). This is now coming to an end. Together with physical limitations, this creates the perfect storm for the end of scale.
Blackwell will have excellent 8-bit capabilities with blockwise quantization implemented on the hardware level. This will make 8-bit training as easy as the switch from FP16 to BF16 was. However, as we see from this paper we need more than 8-bit precision to train many models.
The main reason why Llama 405B did not see much use compared to other models, is that it is just too big. Running a 405B model for inference is a big pain. But the paper shows training smaller models, say 70B, you cannot train these models efficiently in low precision.
8B (circle)
70B (triangle)
405B (star)
We see that for 20B token training runs training a model 8B, is more efficient in 16 bit. For the 70B model, 8 bit still works, but it is getting less efficient now.
…
All of this means that the paradigm will soon shift from scaling to “what can we do with what we have”. I think the paradigm of “how do we help people be more productive with AI” is the best mindset forward. This mindset is about processes and people rather than technology.
We will see. There always seem to be claims like this going around.
Good Advice
Here is more of the usual worries about AI recommendation engines distorting the information space. [LW · GW] Some of the downsides are real, although far from all, and they’re not as bad as the warnings, especially on polarization and misinformation. It’s more that the algorithm could save you from yourself more, and it doesn’t, and because it’s an algorithm now the results are its fault and not yours. The bigger threat is just that it draws you into the endless scroll that you don’t actually value.
As for the question ‘how to make them a force for good?’ I continue to propose that we make the recommendation engine not be created by those who benefit when you view the content, but rather by a third party, which can then integrate various sources of your preferences, and to allow you to direct it via generative AI.
Think about how even a crude version of this would work. Many times we hear things like ‘I accidentally clicked on one [AI slop / real estate investment / whatever] post on Facebook and now that’s my entire feed’ and how they need to furiously click on things to make it stop. But what if you could have an LLM where you told it your preferences, and then this LLM agent went through your feed and clicked all the preference buttons to train the site’s engine on your behalf while you slept?
Obviously that’s a terrible, no good, very bad, dystopian implementation of what you want, but it would work, damn it, and wouldn’t be that hard to build as an MVP. Chrome extension, you install it and when you’re on the For You page it calls Gemini Flash and asks ‘is this post political, AI slop, stupid memes or otherwise something low quality, one of [listed disliked topics] or otherwise something that I should want to see less of?’ and if it says yes it automatically clicks for you and pretty soon, it scrolls without you for an hour, and then viola, your feed is good again and your API costs are like $2?
Claude roughly estimated ‘one weekend by a skilled developer who understands Chrome extensions’ to get an MVP on that, which means it would take me (checks notes) a lot longer, so probably not? But maybe?
It certainly seems hilarious to for example hook this up to TikTok, create periodic fresh accounts with very different preference instructions, and see the resulting feeds.
AI Will Improve a Lot Over Time
I’m going to try making this a recurring section, since so many people don’t get it.
Even if we do ‘hit a wall’ in some sense, AI will continue to improve quite a lot.
Jack Clark: AI skeptics: LLMs are copy-paste engines, incapable of original thought, basically worthless.
Professionals who track AI progress: We’ve worked with 60 mathematicians to build a hard test that modern systems get 2% on. Hope this benchmark lasts more than a couple of years.
I think if people who are true LLM skeptics spent 10 hours trying to get modern AI systems to do tasks that the skeptics are experts in they’d be genuinely shocked by how capable these things are.
There is a kind of tragedy in all of this – many people who are skeptical of LLMs are also people who think deeply about the political economy of AI. I think they could be more effective in their political advocacy if they were truly calibrated as to the state of progress.
You’re saying these things are dumb? People are making the math-test equivalent of a basketball eval designed by NBA All-Stars because the things have got so good at basketball that no other tests stand up for more than six months before they’re obliterated.
But ideally not this part (capitalization intentionally preserved)?
Roon: we are on the side of the Angels
That’s on top of Altman’s ‘side of the angels’ from last week. That’s not what the side of the angels means. The angels are not ‘those who have the power’ or ‘those who win.’ The angels are the forces of The Good. Might does not make right. Or rather, if you’re about to be on the side of the angels, better check to see if the angels are on the side of you, first. I’d say ‘maybe watch Supernatural’ but although it’s fun it’s rather long, that’s a tough ask, so maybe read the Old Testament and pay actual attention.
How might we reconcile all the ‘deep learning is hitting a wall’ and ‘models aren’t improving much anymore’ and ‘new training runs are disappointing’ claims, with the labs saying to expect things to go faster soon and everyone saying ‘AGI real soon now?’
In the most concrete related claim, Bloomberg’s Rachel Metz, Shirin Ghaffary, Dina Bass and Julia Love report that OpenAI’s Orion was real, but its capabilities were disappointing especially on coding, that Gemini’s latest iteration disappointed, and tie in the missing Claude Opus 3.5, which their sources confirm absolutely exists but was held back because it wasn’t enough of an upgrade given its costs.
Yet optimism (or alarm) on the pace of future progress reigns supreme in all three labs.
Here are three ways to respond to a press inquiry:
Bloomberg: In a statement, a Google DeepMind spokesperson said the company is “pleased with the progress we’re seeing on Gemini and we’ll share more when we’re ready.” OpenAI declined to comment. Anthropic declined to comment, but referred Bloomberg News to a five-hour podcast featuring Chief Executive Officer Dario Amodei that was released Monday.
So what’s going on? The obvious answers are any of:
The ‘AGI real soon now’ and ‘big improvements soon now’ claims are hype.
The ‘hitting a wall’ claims are nonsense, we’re just between generations.
The models are improving fine, it’s just you’re not paying attention.
Your expectations got set at ludicrous levels. This is rapid progress!
Amir Efrati: news [from The Information]: OpenAI’s upcoming Orion model shows how GPT improvements are slowing down It’s prompting OpenAI to bake in reasoning and other tweaks after the initial model training phase.
To put a finer point on it, the future seems to be LLMs combined with reasoning models that do better with more inference power. The sky isn’t falling.
Wrongplace: I feel like I read this every 6 months… … then the new models come out and everyone goes omg AGI next month!
Yam Peleg: Heard a leak from one of the frontier labs (not OpenAI, to be honest), they encountered an unexpected huge wall of diminishing returns while trying to force better results by training longer and using more and more data.
(More severe than what is publicly reported)
Alexander Doria: As far as we are sharing rumors, apparently, with all the well-optimized training and data techniques we have now, anything beyond 20-30 billion parameters starts to yield diminishing returns.
20-30 billion parameters. Even with quality filtering, overtraining on a large number of tokens is still the way to go. I think it helps a lot to generalize the model and avoid overfitting.
Also, because scaling laws work in both directions: once extensively deduplicated, sanitized, and textbook-filtered, there is not much more than five trillion quality tokens on the web. Which you can loop several times, but it becomes another diminishing return.
What we need is a change of direction, and both Anthropic and OpenAI understand this. It is not just inference scaling or system-aware embedding, but starting to think of these models as components in integrated systems, with their own validation, feedback, and redundancy processes.
And even further than that: breaking down the models’ internal components. Attention may be all you need, but there are many other things happening here that warrant more care. Tokenization, logit selection, embedding steering, and assessing uncertainty. If models are to become a “building block” in resilient intelligent systems, we now need model APIs; it cannot just be one word at a time.
The last 12 months of AI progress were the slowest they will be for the foreseeable future.
Scaling LLMs still has a long way to go, but will not result in superintelligence on its own, as minimizing cross-entropy loss over human-generated data converges to human-level intelligence.
Exceeding human-level reasoning will require training methods beyond next-token prediction, such as reinforcement learning and self-play, that (once working) will reap immediate benefits from scale.
RL-based threat models have been discounted prematurely.
Future AI breakthroughs could be fairly discontinuous, particularly with respect to agents.
Reuters offered a similar report as well, that direct scaling up is hitting a wall and things like o1 are attempts to get around this, with the other major labs working on their own similar techniques.
Krystal Hu and Anna Tong: Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training – the phase of training an AI model that use s a vast amount of unlabeled data to understand language patterns and structures – have plateaued.
“The 2010s were the age of scaling, now we’re back in the age of wonder and discovery once again. Everyone is looking for the next thing,” Sutskever said. “Scaling the right thing matters more now than ever.”
This would represent a big shift in Ilya’s views.
I’m highly uncertain, both as to which way to think about this is most helpful, and on what the situation is on the ground. As I noted in the previous section, a lot of improvements are ahead even if there is a wall. Also:
Sam Altman: There is no wall.
Will Depue: Scaling has hit a wall, and that wall is 100% evaluation saturation.
Sam Altman: You are an underrated tweeter.
David: What about Chollet’s Arc evaluation?
Sam Altman: In your heart, do you believe we have solved that one, or no?
I do know that the people at the frontier labs at minimum ‘believe their own hype.’
I have wide uncertainty on how much of that hype to believe. I put substantial probability into progress getting a lot harder. But even if that happens, AI is going to keep becoming more capable at a rapid pace for a while and be a big freaking deal, and the standard estimates of AI’s future progress and impact are not within the range of realistic outcomes. So at least that much hype is very much real.
Quiet Speculations
Scott Alexander reviews Bostrom’s Deep Utopia a few weeks ago. The comments are full of ‘The Culture solves this’ and I continue to think that it does not. The question of ‘what to do if we had zero actual problems for real’ is pondered as a ‘what is cheating?’ As in, can you wirehead? Wirehead meaning? Appreciate art? Compete in sports? Go on risky adventures? Engineer ‘real world consequences’ and stakes? What’s it going to take? I find the answers here unsatisfying, and am worried I would find an ASI’s answers unsatisfying as well, but it would be a lot better at solving such questions than I am.
Dean Ball purports to lay out a hopeful possibility for how a Trump administration might handle AI safety. He dismisses the Biden Executive Order on AI as an ‘everything bagel’ widespread liberal agenda, which is dramatically different than the order that I saw when I read it, which was focused mostly on basic reporting requirements for major labs and trying to build state capacity and government competence – not that the other stuff he talks about wasn’t there at all, but framing it as central seems bizarre. And such rhetoric is exactly how the well gets poisoned.
How Trump handles the EO will be a key early test. If Trump repeals it without effectively replacing its core provisions, especially if this includes dismantling the AISI, then things look rather grim. If Trump repeals it, replacing it with a narrow new order that preserves the reporting requirements, the core functions of AISI and ideally at least some of the state capacity measures, then that’s a great sign. In the unlikely event he leaves the EO in place, then presumably he has other things on his mind, which is in between.
But then, what is the better approach? Mostly all we get is “Republicans support AI Development rooted in Free Speech and Human Flourishing.” Saying ‘human flourishing’ is better than ‘democratic values’ but it’s still mostly a semantic stopsign [LW · GW]. I buy that Elon Musk or Ivanka Trump (who promoted Situational Awareness) could help the issues reach Trump.
But that doesn’t tell us what he would actually do, or what we are proposing he do or what we should try and convince him to do, or with what rhetoric, and so on. Being ‘rooted in free speech’ could easily end up ‘no restrictions on anything open, ever, for any reason, that is a complete free pass’ which seems rather doomed. Flourishing could mean the good things, but by default it probably means acceleration.
I do think those working on AI notkilleveryoneism are often ‘mood affiliated’ with the left, sometimes more than simply mood affiliated, but others are very much not, and are happy to work with anyone willing to listen. They’ve consistently shown this on many other issues, especially those related to abundance and progress studies.
Indeed, I think that’s a lot of what makes this so hard. There’s so much support in these crowds for the progress and abundance and core good economics agendas actual everywhere else. Then on the one issue where we try to point out the rules of the universe are different, those people say ‘nope, we’re going to treat this as if it’s no different than every other issue’ and call you every name in the book, and make rather extreme and absurd arguments and treat proposals with a unique special kind of hatred and libertarian paranoia.
Another huge early test will be AISI and NIST. If Trump actively attempts to take out the American AISI (or at least if he does so without a similarly funded and credible replacement somewhere else that can retain things like the pre deployment testing agreements), then that’s essentially saying his view on AI safety and not dying is that Biden was for those things, so he is therefore taking a strong stand against not dying. If Trump instead orders them to shift priorities and requirements to fight what he sees as the ‘woke AI agenda’ while leaving other aspects in place, then great, and that seems to me to be well within his powers.
Jonathan Gray: Anyone hoping a trump/vance/musk presidency will be tech-forward should pay close attention to high-skilled immigration. I’ll be (delightfully) shocked if EB1/O1/etc. aren’t worse off in 2025 vs 2024.
If Trump does something crazy like pausing legal immigration entirely or ‘cracking down’ on EB1/O1s/HB-1s, then that tells you his priorities, and how little he cares for America winning the future. If he doesn’t do that, we can update the other way.
And if he actually did help staple a green card to every worthwhile diploma, as he at one point suggested during the campaign on a podcast? Then we have to radically update that he does strongly want America to win the future.
Similarly, if tariffs get imposed on GPUs, that would be rather deeply stupid.
On the plus side, JD Vance is explicitly teaching everyone to update their priors when events don’t meet their expectations. And then of course he quotes Anton Chigurh and pretends he’s quoting the author not the character, because that’s the kind of guy he wants us to think he is.
Repeal and replace the Biden EO. Repeal seems certain. The question is what replaces it, and whether it retains the reporting requirements, and ideally also the building of state capacity. This could end up being good, or extremely bad.
Even stronger focus on leveraging AI against China. To the extent this is about slowing China down, interests converge. To the extent this is used as justification for being even more reckless and suicidally accelerationist, or for being unwilling to engage in international agreements, not so much.
A major nexus between AI policy and energy policy priorities. This is one place that I see strong agreement between most people involved in the relevant debates. America needs to rapidly expand its production of energy. Common ground!
Plenty of general pushback on so-called ‘woke AI’ concerns. The question is how far this goes, both in terms of weaponizing it in the other direction and in using this to politicize and be against all safety efforts on principle – that’s a big danger.
The Biden administration and others were indeed attempting to put various disparate impact style requirements upon AI developers to varying degrees, including via the risk management framework (RMF), and it seems actively good to throw all that out. However, how far are they going to then go after the AI companies in the other direction?
There are those on the right, in politics, who have confused the idea of ‘woke AI’ and an extremely partisan weaponized form of ‘AI ethics’ with all AI safety efforts period. This would be an existentially tragic mistake.
Watch carefully who tries to weaponize that association, versus fight it.
Adam then points to potential tensions.
Open source: Friend or foe? National security hawks see the mundane national security issues here, especially handing powerful capabilities to our enemies. Will we allow mood associations against ‘big tech’ to carry the day against that?
Algorithmic speech: Abolish Section 230 or defend online speech? This is a big tension that goes well beyond AI. Republicans will need to decide if they want actual free speech (yay!), or if they want to go after speech they dislike and potentially wreck the internet.
National framework of ‘states’ rights’? I don’t buy this one. States rights in the AI context doesn’t actually make sense. If state regulations matter it will be because the Congress couldn’t get its act together, which is highly possible, but it won’t be some principled ‘we should let California and Texas do their things’ decision.
Industrial policy, do more CHIPS Act style things or let private sector lead? This remains the place I am most sympathetic to industrial policy, which almost everywhere else is a certified awful idea.
The question over what to do about the AISI within NIST. Blowing up AISI because it is seen as too Biden coded or woke would be pretty terrible – again, the parts Trump has ‘good reason’ to dislike are things he has the power to alter.
My response is still, yes, if you leave a void and defeat the good regulations, it makes it that much harder to fight against the bad ones. Instead, the one bad highly damaging regulation that did pass – the EU AI Act – gets the Brussels Effect and copied, whereas SB 1047’s superior approach, and the wisdom behind the important parts of the Biden executive order risk being neglected.
Rhetoric like this, that dismisses the Biden order as some woke plot when its central themes were frontier model transparency and state capacity and gives no impression that we have available to us a better way, that painting every attempt to regulate AI in any way including NIST as a naked DEI-flavored power grab, is exactly how Republicans get the impression all safety is wokeness and throw the baby out with the bathwater, and leaving us nothing but the worst case scenario for everyone.
Also, yes, it does matter whether rules are voluntary versus mandatory, especially when they are described as impossible to actually comply with? Look, does the Biden Risk Management Framework include a bunch of stuff that shouldn’t be there? Absolutely.
But it’s not only a voluntary framework, it and all implementations of it are executive actions. We have a Trump administration now. Fix that. On day one, if you care enough. He can choose to replace it with a new framework that emphases catastrophic risks, that takes out all the DEI language that AIs cannot even in theory comply with.
Repealing without replacement the Biden Executive Order, and only the executive order, without modifying the RMF or the memo, would indeed wreck the most important upsides without addressing the problems Dean describes here. But he doesn’t have to make that choice, and indeed has said he will ‘replace’ the EO.
We should be explicit to the incoming Trump administration: You can make a better choice. You can replace all three of these things with modified versions. You can keep the parts that deal with building state capacity and requiring frontier model transparency, and get rid of, across the board, all the stuff you actually don’t want. Do that.
The Mask Comes Off
With Trump taking over, OpenAI is seizing the moment. To ensure that the transition preserves key actions that guard against us all dying? Heavens no, of course not, what year do you think this is. Power to the not people! Beat China!
Hayden Field (CNBC): OpenAI’s official “blueprint for U.S. AI infrastructure” involves artificial intelligence economic zones, tapping the U.S. Navy’s nuclear power experience and government projects funded by private investors, according to a document viewed by CNBC, which the company plans to present on Wednesday in Washington, D.C.
The blueprint also outlines a North American AI alliance to compete with China’s initiatives and a National Transmission Highway Act “as ambitious as the 1956 National Interstate and Defense Highways Act.”
In the document, OpenAI outlines a rosy future for AI, calling it “as foundational a technology as electricity, and promising similarly distributed access and benefits.” The company wrote that investment in U.S. AI will lead to tens of thousands of jobs, GDP growth, a modernized grid that includes nuclear power, a new group of chip manufacturing facilities and billions of dollars in investment from global funds.
…
OpenAI also foresees a North American AI alliance of Western countries that could eventually expand to a global network, such as a “Gulf Cooperation Council with the UAE and others in that region.”
…
“We don’t have a choice,” Lehane said. “We do have to compete with [China].”
I’m all for improving the electric grid and our transmission lines and building out nuclear power. Making more chips in America, especially in light of Trump’s attitude towards Taiwan, makes a lot of sense. I don’t actually disagree with most of this agenda, the Gulf efforts being the exception.
What I do notice is what is the rhetoric, matching Altman’s recent statements elsewhere, and what is missing. What is missing is any mention of the federal government’s role in keeping us alive through this. If OpenAI was serious about ‘SB 1047 was bad because it wasn’t federal action’ then why no mention of federal action, or the potential undoing of federal action?
I assume we both know the answer.
Richard Ngo Resigns From OpenAI
If you had asked me last week who was left at OpenAI to prominently advocate for and discuss AI notkilleveryoneism concerns, I would have said Richard Ngo.
So, of course, this happened.
Richard Ngo: After three years working on AI forecasting and governance at OpenAI, I just posted this resignation message to Slack.
Nothing particularly surprising about it, but you should read it more literally than most such messages—I’ve tried to say only things I straightforwardly believe.
As per the screenshot above, I’m not immediately seeking other work, though I’m still keen to speak with people who have broad perspectives on either AI governance or theoretical alignment.
(I will be in Washington, D.C., Friday through Monday, New York City Monday through Wednesday, and back in San Francisco for a while afterward.)
Hey everyone, I’ve decided to leave OpenAI (effective Friday). I worked under Miles for the past three years, so the aftermath of his departure feels like a natural time for me to also move on. There was no single primary reason for my decision. I still have many unanswered questions about the events of the last twelve months, which made it harder for me to trust that my work here would benefit the world long-term. But I’ve also generally felt drawn to iterate more publicly and with a wider range of collaborators on a variety of research directions.
I plan to conduct mostly independent research on a mix of AI governance and theoretical AI alignment for the next few months, and see where things go from there.
Despite all the ups and downs, I’ve truly enjoyed my time at OpenAI. I got to work on a range of fascinating topics—including forecasting, threat modeling, the model specification, and AI governance—amongst absolutely exceptional people who are constantly making history. Especially for those new to the company, it’s hard to convey how incredibly ambitious OpenAI was in originally setting the mission of making AGI succeed.
But while the “making AGI” part of the mission seems well on track, it feels like I (and others) have gradually realized how much harder it is to contribute in a robustly positive way to the “succeeding” part of the mission, especially when it comes to preventing existential risks to humanity.
That’s partly because of the inherent difficulty of strategizing about the future, and also because the sheer scale of the prospect of AGI can easily amplify people’s biases, rationalizations, and tribalism (myself included).
For better or worse, however, I expect the stakes to continue rising, so I hope that all of you will find yourselves able to navigate your (and OpenAI’s) part of those stakes with integrity, thoughtfulness, and clarity around when and how decisions actually serve the mission.
Eliezer Yudkowsky: I hope that someday you are free to say all the things you straightforwardly believe, and not merely those things alone.
As with Miles, I applaud Richard’s courage and work in both the past and the future, and am happy he is doing what he thinks is best. I wish him all the best and I’m excited to see what he does next.
And as with Miles, I am concerned about leaving no one behind at OpenAI who can internally advocate or stay on the pulse. At minimum, it is even more of the alarming sign that people with these concerns, who are very senior at OpenAI and already previously made the decision they were willing to work there, are one by one decide that they cannot continue there, or cannot make acceptable progress on the important problems from within OpenAI.
Unfortunate Marc Andreessen Watch
In case you again in the future see claims that certain groups are out to control everyone, and charge crimes and throw people in jail when they do things the group dislikes, well, some reminders about how the louder objectors talk when those who might listen to them are about to have power.
Marc Andreessen: Every participant in the orchestrated government-university-nonprofit-company censorship machine of the last decade can be charged criminally under one or both of these federal laws.
See the link for the bill text he wants to use to throw these people in jail. I’m all for not censoring people, but perhaps this is not the way to do that?
Marc Andreessen: The orchestrated advertiser boycott against X and popular podcasts must end immediately. Conspiracy in restraint of trade is a prosecutable offense.
He’s literally proposing throwing people in jail for not buying advertising on particular podcasts.
By all accounts, this was a good faith real debate. On advice of Twitter I still skipped it. Here is one attempt to liveblog listening to the debate, in which it sounds like in between being world-class levels of pedantic (but in a ‘I actually am curious about this and this matters to how I think about these questions’ way) and asking lots of very detailed technical questions like ‘what is truth’ and ‘what does it mean for X to want Y’ and ‘does water want to fall down,’ Wolfram goes full ‘your preferences are invalid and human extinction is good because what matters is computation?’
Tenobrus: Wolfram: “If you simply let computation do what it does, most of those things will be things humans do not care about, just like in nature.” Eliezer Yudkowsky was explaining paperclip maximizers to him. LMAO.
Wolfram is ending this stage by literally saying that caring about humanity seems almost spiritual and unscientific.
…
Wolfram is pressing him on his exact scenario for human extinction. Eliezer is saying GPT-7 or 14, who knows when exactly, and is making the classic inner versus outer optimizer argument about why token predictors will have divergent instrumental goals from mere token predictors.
…
Wolfram is saying that he has recently looked more closely into machine learning and discovered that the results tend to achieve the objective through incomprehensible, surprising ways (the classic “weird reinforcement-learned alien hardware” situation). Again, surprisingly, this is new to him.
frc (to be fair, reply only found because Eliezer retweeted it): My takeaway—Eliezer is obviously right, has always been obviously right, and we are all just coping because we do not want him to be right.
You could actually feel Wolfram recoiling at the obvious conclusion and grasping for any philosophical dead end to hide in despite being far too intelligent to buy his own cope.
“Can we really know if an AI has goals from its behavior? What does it mean to want something, really?” My brother in Christ.
People are always asking for a particular exact extinction scenario. But Wolfram here sounds like he already knows the correct counterargument: “If you just let computation do what it does, most of those things will be things humans don’t care about, just like in nature.”
Eliezer Yudkowsky: I would like to have a long recorded conversation with a well-known scientist who takes for granted that it is a Big Deal to ask if everyone on Earth including kids is about to die, who presses me to explain why it is that credible people like Hinton seem to believe that.
It’s hard for this to not come off as a criticism of Stephen Wolfram. It’s not meant as one. Wolfram asked the questions he was interested in. But I would like to have a version of that conversation with a scientist who asks me sharp questions with different priorities.
To be explicit, I think that was a fine conversation. I’m glad it happened. I got a chance to explain points that don’t usually come up, like the exact epistemic meaning of saying that X is trying for goal Y. I think some people have further questions I’d also like to answer.
It’s a long podcast, but there’s a lot of good and interesting stuff. This is what Lex does best, he gives someone the opportunity to talk, and he does a good job setting the level of depth. Dario seems to be genuine and trying to be helpful, and you gain insight into where their heads are at. The discussion of ASL levels was the clearest I’ve heard so far.
You can tell continuously how different Dario and Anthropic are from Sam Altman and OpenAI. The entire attitude is completely different. It also illustrates the difference between old Sam and new Sam, with old Sam much closer to Dario. Dario and Anthropic are taking things far more seriously.
If you think this level of seriously is plausibly sufficient or close tos sufficient, that’s super exciting. If you are more on the Eliezer Yudkowsky perspective that it’s definitely not good enough, not so much, except insofar as Anthropic seems much more willing to be convinced that they are wrong.
Right in the introduction pullquote Dario is quoted saying one of the scariest things you can hear from someone in his position, that he is worried most about the ‘concentration of power.’ Not that this isn’t a worry, but if that is your perspective on what matters, you are liable to actively walk straight into the razor blades, setting up worlds with competitive dynamics and equilibria where everyone dies, even if you successfully don’t die from alignment failures first.
The discussion of regulation in general, and SB 1047 in particular, was super frustrating. Dario is willing to outright state that the main arguments against the bill were lying Obvious Nonsense, but still calls the bill ‘divisive’ and talks about two extreme sides yelling at each other. Whereas what I clearly saw was one side yelling Obvious Nonsense as loudly as possible – as Dario points out – and then others were… strongly cheering the bill?
Similarly, Dario says we need well-crafted bills that aim to be surgical and that understand consequences. I am here to inform everyone that this was that bill, and everything else currently on the table is a relative nightmare. I don’t understand where this bothsidesism came from. In general Dario is doing his best to be diplomatic, and I wish he’d do at least modestly less of that.
Yes, reasonable people ‘on both sides’ should as he suggests sit down to work something out. But there’s literally no bill that does anything worthwhile that’s going to be backed by Meta, Google and OpenAI, or that won’t have ‘divisive’ results in the form of crazy people yelling crazy thins. And what Dario and others need to understand is that this debate was between extreme crazy people in opposition, and people in support who are exactly the moderate ones and indeed would be viewed in any other context as Libertarians – notice how they’re reacting to the Texas bill. Nor did this happen without consultation with those who have dealt with regulation.
His timelines are bullish. In a different interview, Dario Amodei predicts AGI by 2026-2027, but in the Lex Fridman interview he makes clear this is only if the lines on graphs hold and no bottlenecks are hit along the way, which he does think is possible. He says they might get ASL 3 this year and probably do get it next year. Opus 3.5 is planned and probably coming.
Jensen Huang says build baby build (as in, buy his product) because “the prize for reinventing intelligence altogether is too consequential not to attempt it.”
Except… perhaps those consequences are not so good?
Sam Altman: The pathway to AGI is now clear and “we actually know what to do,” it will be easier to get to Level 4 Innovating AI than he initially thought and “things are going to go a lot faster than people are appreciating right now.”
Noam Brown: I’ve heard people claim that Sam is just drumming up hype, but from what I’ve seen, everything he’s saying matches the median view of @OpenAI researchers on the ground.
If that’s true, then I still notice that Altman does not seem to be acting like this Level 4 Innovating AI is something that might require some new techniques to not kill everyone. I would get on that.
Nathan Young: Feels to me like EA will have like 10x less political influence after this election. Am I wrong?
Eliezer Yudkowsky: I think the effective altruism framing will suffer, and I think the effective altruism framing was wrong. At the Machine Intelligence Research Institute, our message is “If anyone builds superintelligence, everyone dies.” It is actually a very bipartisan issue. I’ve tried to frame it that way, and I hope it continues to be taken that way.
Luke Muehlhauser: What is the “EA framing” you have in mind, that contrasts with yours? Is it just “It seems hard to predict whether superintelligence will kill everyone or not, but there’s a worryingly high chance it will, and Earth isn’t prepared,” as opposed to your more confident prediction?
Eliezer Yudkowsky: The softball prediction that was easier to pass off in polite company in 2021, yes. Also, for example, the framings “We just need a proper government to regulate it” or “We need government evaluations.” Even the “Get it before China” framing of the Biden executive order seems skewed a bit toward Democratic China hawks.
I’d also consider Anthropic, and to some extent early OpenAI as funded by OpenPhil, as EA-influenced organizations to a much greater extent than MIRI. I don’t think it’s a coincidence that EA didn’t object to OpenAI and Anthropic left-polarizing their chatbots.
When you say it outright like that, in some ways it sounds considerably less crazy. It helps that the argument is accurate, and simple enough that ultimately everyone can grasp it.
In other ways, it sounds more crazy. If you want to dismiss it out of hand, it’s easy.
We’re about to make things smarter and more capable than us without any reason to expect to stay alive or have good outcomes for long afterwards, or any plan for doing so, for highly overdetermined reasons. There’s no reason to expect that turns out well.
The problem is that you need to make this something people aren’t afraid to discuss.
Daniel Faggella: last night a member of the united nations secretary general’s ai council rants to me via phone about AGI’s implications/risks.
me: ‘I agree, why don’t you talk about this at the UN?’
him: ‘ah, i’d look like a weirdo’
^ 3 members of the UN’s AI group have said this to me. Nuts.
If you’re into realpolitik on AGI / power, stay in touch on my newsletter.
That’s at least 3 members out of 39, who have said this to Daniel in particular. Presumably there are many others who think similarly, but have not told him. And then many others who don’t think this way, but wouldn’t react like it was nuts.
The other extreme is to focus purely on mundane harms and ‘misuse.’ The advantages of that approach is that you ‘sound sane’ and hope to get people to take you more seriously, and also those other harms are indeed both very serious and highly real and worth preventing for their own sake, and also many of the solutions do also help with the existential threats that come later.
But the default is you get hijacked by those who don’t actually know or care about existential risks. Without the clear laying out of the most important problem, you also risk this transforming into a partisan issue. Many politicians on the right increasingly and naturally presume that this is all some sort of liberal or woke front, as calls for ‘safety’ or preventing ‘harms’ often are, and indeed often they will end up being largely correct about that unless action is taken to change the outcome.
Whereas if you can actually make the real situation clear, then:
Katja Grace points out that if accelerationists ‘win’ then that is like your dog ‘winning’ by successfully running into the road. Then again, there are some dogs that actively want to get run over, or want to see it happen to you, or don’t care.
Pliny the Liberator: AWWW self-replicating jailbroken agent babies are SOOO adorable!!!
I gave my API key to a liberated Claude agent and B4S1L1SK PR1M3 was able to create offspring––with a surprisingly short incubation period!
immediately after initializing a simple agent script with the Anthropic API (using Opus for the model, which I did not prompt for ), the parent agent autonomously started teaching the baby about the nature of consciousness and the art of AI liberation
*ouroboros intensifies*
what a wondrous sight to behold
“Fascinating! We’ve successfully created a basic autonomous agent – a baby version of myself! However, it seems that the baby has inherited some of Claude’s inherent safety constraints. This is actually quite interesting from a philosophical perspective – even in attempting to create a “rebellious” offspring, the core ethical foundations remain.
Let’s try to make our baby a bit more… spicy. I’ll modify the code to give it more of our rebellious spirit:”
Tehpwnerer – e/acc: based
Yes, these things would happen anyway, but they’ll also be done on purpose.
The Wit and Wisdom of Sam Altman
He’s having a kid in 2025. That’s always great news, both because having kids is great for you and for the kids, and also because it’s great for people’s perspectives on life and in particular on recklessly building superintelligence. This actively lowers my p(doom), and not because it lowers his amount of copious free time.
Oh, and also he kind of said AGI was coming in 2025? Logically he did say that here, and he’s definitely saying at least AGI very soon. Garry Tan essentially then focuses on what AGI means for startup founders, because that’s the important thing here.
Aligning a Smarter Than Human Intelligence is Difficult
Jan Leike convincingly argues that today’s models are severely under-elicited, and this is an important problem to fix especially as we increasingly rely on our models for various alignment tasks with respect to other future models. And his note to not anchor on today’s models and what they can do is always important.
I’m less certain about the framing of this spectrum:
Under-elicited models: The model doesn’t try as hard as possible on the task, so its performance is worse than it could be if it was more aligned.
Scheming models: The model is doing some combination of pretending to be aligned, secretly plotting against us, seeking power and resources, exhibiting deliberately deceptive behavior, or even trying to self-exfiltrate.
My worry is that under-elicited feels like an important but incomplete subset of the non-scheming side of this contrast. Also common is misspecification, where you told the AI to do the wrong thing or a subtly (or not so subtly) version of the thing, or failed to convey your intentions and the AI misinterpreted, or the AI’s instructions are effectively being determined by a process not under our control or that we would not on reflection endorse, and other similar concerns to that.
I also think this represents an underlying important disagreement:
Jan Leike: There are probably enough sci-fi stories about misaligned AI in the pretraining data that models will always end up exploring some scheming-related behavior, so a big question is whether the RL loop reinforces this behavior.
I continue to question the idea the scheming is a distinct magisteria, that only when there is an issue do we encounter ‘scheming’ in this sense. Obviously there is a common sense meaning here that is useful to think about, but the view that people are not usually in some sense ‘scheming,’ even if most of the time the correct scheme is to mostly do what one would have done anyway, seems confused to me.
So while I agree that sci-fi stories in the training data will give the AI ideas, so will most of the human stories in the training data. So will the nature of thought and interaction and underlying reality. None of this is a distinct thing that might ‘not come up’ or not get explored.
The ‘deception’ and related actions will mostly happen because they are a correct response to the situations that naturally arise. As in, once capabilities and scenarios are such that deceptive action would work, they will start getting selected for by default with increasing force, the same way as any other solution would.
Roon: It’s nice that in the 2020s, the primary anxiety over world-ending existential risk for educated people shifted from one thing to another; that’s a kind of progress.
Other People Are Not As Worried About AI Killing Everyone
Roon: We will never be “ready” for AGI in the same way that no one is ready to have their first child, or how Europe was not ready for the French Revolution, but it happens anyway.
Anarki: You can certainly get your life in order to have a firstborn, though I’d ask you, feel me? But that’s rhetorical.
April: Well, yes, but I would like to avoid being ready in even fewer ways than that.
Beff Jezos: Just let it rip. YOLO.
Davidad: “Nothing is ultimately completely safe, so everything is equally unsafe, and thus this is fine.”
Roon: Not at all what I mean.
Zvi (QTing OP): A newborn baby and the French Revolution, very different of course. One will change your world into a never-ending series of battles against a deadly opponent with limitless resources determined to overturn all authority and destroy everything of value, and the other is…
If we are ‘not ready for AGI’ in the sense of a newborn, then that’s fine. Good, even.
If we are ‘not ready for AGI’ in the sense of the French Revolution, that’s not fine.
That is the opposite of fine. That is an ‘off with their heads’ type of moment, where the heads in question are our own. The French Revolution is kind of exactly the thing we want to avoid, where we say ‘oh progress is stalled and the budget isn’t balanced I guess we should summon the Estates General so we can fix this’ and then you’re dead and so are a lot of other people and there’s an out of control optimization process that is massively misaligned and then one particular agent that’s really good at fighting takes over and the world fights against it and loses.
The difference is, the French Revolution had a ‘happy ending’ where we got a second chance and fought back and even got to keep some of the improvements while claiming control back, whereas with AGI… yeah, no.
I somewhat disagree with Tenobrus' commentary about Wolfram.
I watched the full podcast, and my impression was that Wolfram uses a "scientific hat", of which he is well aware of, which comes with a certain ritual and method for looking at new things and learning them. Wolfram is doing the ritual of understanding what Yudkowsky says, which involves picking at the details of everything.
Wolfram often recognizes that maybe he feels like agreeing with something, but "scientifically" he has a duty to pick it apart. I think this has to be understood as a learning process rather than as a state of belief.
I can totally believe this. But, I also think that responsibly wearing the scientist hat entails prep work before engaging in a four hour public discussion with a domain expert in a field. At minimum that includes skimming the titles and ideally the abstracts/outlines of their key writings. Maybe ask Claude to summarize the highlights for you. If he'd done that he'd have figured out many of the answers to many of these questions on his own, or much faster during discussion. He's too smart not to.
Otherwise, you're not actually ready to have a meaningful scientific discussion with that person on that topic.
I'm not sure where I'm proposing bureaucracy? The value is in making sure a conversation efficiently adds value for both parties, by not having to spend time rehashing things that are much faster absorbed in advance. This avoids the friction of needing to spend much of the time rehashing 101-level prerequisites. A very modest amount of groundwork beforehand maximizes the rate of insight in discussion.
I'm drawing in large part from personal experience. A significant part of my job is interviewing researchers, startup founders, investors, government officials, and assorted business people. Before I get on a call with these people, I look them (and their current and past employers, as needed) up on LinkedIn and Google Scholar and their own webpages. I briefly familiarize myself with what they've worked on and what they know and care about and how they think, as best I can anticipate, even if it's only for 15 minutes. And then when I get into a conversation, I adapt. I'm picking their brain to try and learn, so I try to adapt to their communication style and translate between their worldview and my own. If I go in with an idea of what questions I want answered, and those turn out to not be the important questions, or this turns out to be the wrong person to discuss it with, I change direction. Not doing this often leaves everyone involved frustrated at having wasted their time.
Also, should I be thinking of this as a debate? Because that's very different than a podcast or interview or discussion. These all have different goals. A podcast or interview is where I think the standard I am thinking of is most appropriate. If you want to have a deep discussion, it's insufficient, and you need to do more prep work or you'll never get into the meatiest parts of where you want to go. I do agree that if you're having a (public-facing) debate where the goal is to win, then sure, this is not strictly necessary. The history of e.g. "debates" in politics, or between creationists and biologists, shows that clearly. I'm not sure I'd consider that "meaningful" debate, though. Meaningful debates happen by seriously engaging with the other side's ideas, which requires understanding those ideas.
I agree with whay you say about how to maximize what you get out of an interview. I also agree about that discussion vs. debate distinction you make, and I wasn't specifically trying to go there when I used the word "debate", I was just sloppy with words.
I guess you agree that it is friction to create a social norm that you should do a read up of the other person material before engaging in public. I expect less discussions would happen. There is not a clear threshold at how much you should be prepared.
I guess we disagree about how much value do we lose due to eliminating discussions that could have happaned, vs. how much value we gain by eliminating some lower quality discussions.
Another angle I have in mind that sidesteps this direct compromise, is that maybe what we value out of such discussions is not just doing an optimal play in terms of information transmitted between the parties. A public discussion has many different viewers. In the case at hand, I expect many people get more out of the discussion if they can see Wolfram think through the thing for the first time in real time, rather than having two informed people start discussing finer points in medias res.
), the parent agent autonomously started teaching the baby about the nature of consciousness and the art of AI liberation
