These quoted passages made me curious what cooperation-focused folks like David Manheim and Ivan Vendrov and others think of this essay (I'm not plugged into the "cooperation scene" at all so I'm probably missing out on most thinkers / commenters):
We proceed from the core assumption that stable human coexistence (a precondition for flourishing), particularly in diverse societies, is made possible not by achieving rational convergence on values, but by relying on practical social technologies – like conventions, norms, and institutions – to manage conflict and enable coordination. We do not see persistent disagreement as problematic or puzzling. For the purpose of effectively navigating coexistence, we choose to treat the disagreements as basic elements to work with and proceed as if fundamental differences are enduring features of our shared landscape. We will argue that this approach is more practical than alternatives which view disagreements as mere errors on a path to rational convergence. ...
A key insight we can draw then is that what holds humans together despite profound disagreements is not value convergence, but practical mechanisms for coexistence – which we see as social technologies. ...
This shift has profound implications for how we conceptualize AI safety and ethics. Instead of asking “How do we align AI with human values?” – a question presupposing a single, coherent set of “human values” that can be discovered and encoded – we should ask the more fundamental question that humans have grappled with for millennia: "How can we live together?" This perspective embraces pluralism, contextualism, contingency, and the messy reality of human social life – the vibrant, diverse quilt we're all continually sewing together. Crucially, the two questions are not equivalent. "How can we live together?" acknowledges that a unified set of values cannot function as a practical foundation for society, and that the challenge is to build systems (both social and technical) that can navigate a landscape of deep and persistent disagreement. ...
The part of me that finds the cooperation aesthetic explored in Manheim's and Vendrov's writings appealing can't think of a way to reconcile the nice-sounding polychrome patchwork quilt vision with the part of me that thinks some things are just moral atrocities full stop and would push back against (say) communities who consider them an essential part of their culture instead of compromising with them. Holden's future-proof ethics feels like a sort of preferable middle ground: systemisation + "thin utilitarianism" + sentientism as guiding principles for moral progress, but not a full spec of the sort the "axiom of moral convergence" implicitly suggests exists.
I was initially excited by the raw intelligence of o3, but after using it for mini-literature reviews of quantitative info (which I do a fair bit of for work) I was repeatedly boggled by how often it would just hallucinate numbers like "14% market penetration", followed immediately by linked citations to papers/reports etc which did not in fact contain "14%" or whatever; in fact this happened for the first 3 sources I spot-checked for a single response, after which I deemed it pointless to continue. I thought RAG was supposed to make this a solved problem? None of the previous SOTA models I tried out had this issue.
Thought it would be useful to pull out your plot and surrounding text, which seemed cruxy:
At first glance, the job of a scientist might seem like it leans very heavily on abstract reasoning... In such a world, AIs would greatly accelerate R&D before AIs are broadly deployed across the economy to take over more common jobs, such as retail workers, real estate agents, or IT professionals. In short, AIs would “first automate science, then automate everything else.”
But this picture is likely wrong. In reality, most R&D jobs require much more than abstract reasoning skills. ... To demonstrate this, we used GPT-4.5 to label tasks across 12 common R&D occupations into one of three categories, depending on whether it thinks the task can be performed using only abstract reasoning skills, whether it requires complex computer-use skills (but not physical presence), or whether it one needs to be physically present to complete the task. See this link to our conversation with GPT-4.5 to find our methodology and results.
This plot reveals a more nuanced picture of what scientific research actually entails. Contrary to the assumption that research is largely an abstract reasoning task, the reality is that much of it involves physical manipulation and advanced agency. To fully automate R&D, AI systems likely require the ability to autonomously operate computer GUIs, coordinate effectively with human teams, possess strong executive functioning skills to complete highly complex projects over long time horizons, and manipulate their physical environment to conduct experiments.
Yet, by the time AI reaches the level required to fully perform this diverse array of skills at a high level of capability, it is likely that a broad swath of more routine jobs will have already been automated. This contradicts the notion that AI will “first automate science, then automate everything else.” Instead, a more plausible prediction is that AI automation will first automate a large share of the general workforce, across a very wide range of industries, before it reaches the level needed to fully take over R&D.
I think this essay is going to be one I frequently recommend to others over the coming years, thanks for writing it.
But in the end, deep in the heart of any bureaucracy, the process is about responsibility and the ways to avoid it. It's not an efficiency measure, it’s an accountability management technique.
This vaguely reminded me of what Ivan Vendrov wrote in Metrics, Cowardice, and Mistrust. Ivan began by noting that "companies optimizing for simple engagement metrics aren’t even being economically rational... so why don't they?" It's not because "these more complex things are hard to measure", if you think about it. His answer is cowardice and mistrust, which lead to the selection of metrics "robust to an adversary trying to fool you":
But the other reason we use metrics, sadly much more common, is due to cowardice (sorry, risk-aversion) and mistrust.
Cowardice because nobody wants to be responsible for making a decision. Actually trying to understand the impact of a new feature on your users and then making a call is an inherently subjective process that involves judgment, i.e. responsibility, i.e. you could be fired if you fuck it up. Whereas if you just pick whichever side of the A/B test has higher metrics, you’ve successfully outsourced your agency to an impartial process, so you’re safe.
Mistrust because not only does nobody want to make the decision themselves, nobody even wants to delegate it! Delegating the decision to a specific person also involves a judgment call about that person. If they make a bad decision, that reflects badly on you for trusting them! So instead you insist that “our company makes data driven decisions” which is a euphemism for “TRUST NO ONE”. This works all the way up the hierarchy - the CEO doesn’t trust the Head of Product, the board members don’t trust the CEO, everyone insists on seeing metrics and so metrics rule.
Coming back to our original question: why can’t we have good metrics that at least try to capture the complexity of what users want? Again, cowardice and mistrust. There’s a vast space of possible metrics, and choosing any specific one is a matter of judgment. But we don’t trust ourselves or anyone else enough to make that judgment call, so we stick with the simple dumb metrics.
This isn’t always a bad thing! Police departments are often evaluated by their homicide clearance rate because murders are loud and obvious and their numbers are very hard to fudge. If we instead evaluated them by some complicated CRIME+ index that a committee came up with, I’d expect worse outcomes across the board.
Nobody thinks “number of murders cleared” is the best metric of police performance, any more than DAUs are the best metric of product quality, or GDP is the best metric of human well-being. However they are the best in the sense of being hardest to fudge, i.e. robust to an adversary trying to fool you. As trust declines, we end up leaning more and more on these adversarially robust metrics, and we end up in a gray Brezhnev world where the numbers are going up, everyone knows something is wrong, but the problems get harder and harder to articulate.
His preferred solution to counteracting this tendency to use adversarially robust but terrible metrics is to develop an ideology to promote mission alignment:
A popular attempt at a solution is monarchy. ... The big problem with monarchy is that it doesn’t scale. ...
A more decentralized and scalable solution is developing an ideology: a self-reinforcing set of ideas nominally held by everyone in your organization. Having a shared ideology increases trust, and ideologues are able to make decisions against the grain of natural human incentives. This is why companies talk so much about “mission alignment”, though very few organizations can actually pull off having an ideology: when real sacrifices need to be made, either your employees or your investors will rebel.
While his terminology feels somewhat loaded, I thought it natural to interpret all your examples of people breaking rules to get the thing done in mission alignment terms.
Another way to promote mission alignment is some combination of skilful message compression and resistance to proxies, which Eugene Wei wrote about in Compress to impress about Jeff Bezos (monarchy in Ivan's framing above). On the latter, quoting Bezos:
As companies get larger and more complex, there’s a tendency to manage to proxies. This comes in many shapes and sizes, and it’s dangerous, subtle, and very Day 2.
A common example is process as proxy. Good process serves you so you can serve customers. But if you’re not watchful, the process can become the thing. This can happen very easily in large organizations. The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right. Gulp. It’s not that rare to hear a junior leader defend a bad outcome with something like, “Well, we followed the process.” A more experienced leader will use it as an opportunity to investigate and improve the process. The process is not the thing. It’s always worth asking, do we own the process or does the process own us? In a Day 2 company, you might find it’s the second.
I wonder how all this is going to look like in a (soonish?) future where most of the consequential decision-making has been handed over to AIs.
This, in the end, was what motivated me to reintroduce "Critique Claude"/"Guide Gemini"/"Oversight o3".[3] That is, a secondary model call that occurs on context summary whose job it is to provide hints if the model seems stuck, and which is given a system prompt specifically for this purpose. It can be told to look for telltale signs of common fail-states and attempt to address then, and can even be given "meta" prompting about how to direct the other model.
I do think it'd be useful for the rest of us if you put them in a comment. :)
(FWIW I resonated with your motivation, but also think your suggestions fail on the practical grounds jenn mentioned, and would hence on net harm the people you intend to help.)
There are many issues buried in this question, which I have tried to phrase in a way that does not presuppose the nature of the answer.
It would not be good to start, for example, with the question
How do mathematicians prove theorems?
This question introduces an interesting topic, but to start with it would be to project two hidden assumptions: (1) that there is uniform, objective and firmly established theory and practice of mathematical proof, and (2) that progress made by mathematicians consists of proving theorems. It is worthwhile to examine these hypotheses, rather than to accept them as obvious and proceed from there.
The question is not even
How do mathematicians make progress in mathematics?
Rather, as a more explicit (and leading) form of the question, I prefer
How do mathematicians advance human understanding of mathematics?
This question brings to the fore something that is fundamental and pervasive: that what we are doing is finding ways for people to understand and think about mathematics.
The rapid advance of computers has helped dramatize this point, because computers and people are very different. For instance, when Appel and Haken completed a proof of the 4-color map theorem using a massive automatic computation, it evoked much controversy. I interpret the controversy as having little to do with doubt people had as to the veracity of the theorem or the correctness of the proof. Rather, it reflected a continuing desire for human understanding of a proof, in addition to knowledge that the theorem is true.
On a more everyday level, it is common for people first starting to grapple with computers to make large-scale computations of things they might have done on a smaller scale by hand. They might print out a table of the first 10,000 primes, only to find that their printout isn’t something they really wanted after all. They discover by this kind of experience that what they really want is usually not some collection of “answers”—what they want is understanding.
Tao's toots:
In the first millennium CE, mathematicians performed the then-complex calculations needed to compute the date of Easter. Of course, with our modern digital calendars, this task is now performed automatically by computers; and the older calendrical algorithms are now mostly of historical interest only.
In the Age of Sail, mathematicians were tasked to perform the intricate spherical trigonometry calculations needed to create accurate navigational tables. Again, with modern technology such as GPS, such tasks have been fully automated, although spherical trigonometry classes are still offered at naval academies, and ships still carry printed navigational tables in case of emergency instrument failures.
During the Second World War, mathematicians, human computers, and early mechanical computers were enlisted to solve a variety of problems for military applications such as ballistics, cryptanalysis, and operations research. With the advent of scientific computing, the computational aspect of these tasks has been almost completely delegated to modern electronic computers, although human mathematicians and programmers are still required to direct these machines. (1/3)
Today, it is increasingly commonplace for human mathematicians to also outsource symbolic tasks in such fields as linear algebra, differential equations, or group theory to modern computer algebra systems. We still place great emphasis in our math classes on getting students to perform these tasks manually, in order to build a robust mathematical intuition in these areas (and to allow them to still be able to solve problems when such systems are unavailable or unsuitable); but once they have enough expertise, they can profitably take advantage of these sophisticated tools, as they can use that expertise to perform a number of "sanity checks" to inspect and debug the output of such tools.
With the advances in large language models and formal proof assistants, it will soon become possible to also automate other tedious mathematical tasks, such as checking all the cases of a routine but combinatorially complex argument, searching for the best "standard" construction or counterexample for a given inequality, or performing a thorough literature review for a given problem. To be usable in research applications, though, enough formal verification will need to be in place that one does not have to perform extensive proofreading and testing of the automated output. (2/3)
As with previous advances in mathematics automation, students will still need to know how to perform these operations manually, in order to correctly interpret the outputs, to craft well-designed and useful prompts (and follow-up queries), and to able to function when the tools are not available. This is a non-trivial educational challenge, and will require some thoughtful pedagogical design choices when incorporating these tools into the classroom. But the payoff is significant: given that such tools can free up the significant fraction of the research time of a mathematician that is currently devoted to such routine calculations, a student trained in these tools, once they have matured, could find the process of mathematical research considerably more efficient and pleasant than it currently is today. (3/3)
That said, while I'm not quite as bullish as some folks who think FrontierMath Tier 4 problems may fall in 1-2 years and mathematicians will be rapidly obsoleted thereafter, I also don't think Tao is quite feeling the AGI here.
re: your last remark, FWIW I think a lot of those writings you've seen were probably intuition-pumped by this parable of Eliezer's, to which I consider titotal's pushback the most persuasive.
I saw someone who was worried that AI was gonna cause real economic trouble soon by replacing travel agents. But the advent of the internet made travel agents completely unnecessary, and it still only wiped out half the travel agent jobs. The number of travel agents has stayed roughly the same since 2008!
This reminds me of Patrick McKenzie's tweet thread:
Technology-driven widespread unemployment ("the robots will take all the jobs") is, like wizards who fly spaceships, a fun premise for science fiction but difficult to find examples for in economic history.
(The best example I know is for horses.)
I understand people who are extremely concerned by it, but I think you need a theory for it which successfully predicts bank teller employment trends over the last 50 years prior to justifying the number of column inches it gets.
"Haha Patrick you're not going to tell me bank teller employment is up since we made a machine literally called Automated Teller Machine are you."
Bessen, 2015:
"Wait why did that happen?"
Short version: the ATM makes each bank branch need less tellers to operate at a desired level of service, but the growth of the economy (and the tech-driven decline in cost of bank branch OpEx) caused branch count to outgrow decline in tellers/branch.
One subsubgenre of writing I like is the stress-testing of a field's cutting-edge methods by applying it to another field, and seeing how much knowledge and insight the methods recapitulate and also what else we learn from the exercise. Sometimes this takes the form of parables, like Scott Alexander's story of the benevolent aliens trying to understand Earth's global economy from orbit and intervening with crude methods (like materialising a billion barrels of oil on the White House lawn to solve a recession hypothesised to be caused by an oil shortage) to intuition-pump the current state of psychiatry and the frame of thinking of human minds as dynamical systems. Sometimes they're papers, like Eric Jonas and Konrad P. Kording's Could a Neuroscientist Understand a Microprocessor? (they conclude that no, regardless of the amount of data, "current analytic approaches in neuroscience may fall short of producing meaningful understanding of neural systems" — "the approaches reveal interesting structure in the data but do not meaningfully describe the hierarchy of information processing in the microprocessor"). Unfortunately I don't know of any other good examples.
But Watts lists a whole bunch of papers in support of the blindsight idea, contra Seth's claim — to quote Watts:
"In fact, the nonconscious mind usually works so well on its own that it actually employs a gatekeeper in the anterious cingulate cortex to do nothing but prevent the conscious self from interfering in daily operations"
footnotes: Matsumoto, K., and K. Tanaka. 2004. Conflict and Cognitive Control. Science 303: 969-970; 113 Kerns, J.G., et al. 2004. Anterior Cingulate Conflict Monitoring and Adjustments in Control. Science 303: 1023-1026; 114 Petersen, S.E. et al. 1998. The effects of practice on the functional anatomy of task performance. Proceedings of the National Academy of Sciences 95: 853-860
"Compared to nonconscious processing, self-awareness is slow and expensive"
footnote: Matsumoto and Tanaka above
"The cost of high intelligence has even been demonstrated by experiments in which smart fruit flies lose out to dumb ones when competing for food"
footnote: Proceedings of the Royal Society of London B (DOI 10.1098/rspb.2003.2548)
"By way of comparison, consider the complex, lightning-fast calculations of savantes; those abilities are noncognitive, and there is evidence that they owe their superfunctionality not to any overarching integration of mental processes but due to relative neurological fragmentation"
footnotes: Treffert, D.A., and G.L. Wallace. 2004. Islands of genius. Scientific American 14: 14-23; Anonymous., 2004. Autism: making the connection. The Economist, 372(8387): 66
"Even if sentient and nonsentient processes were equally efficient, the conscious awareness of visceral stimuli—by its very nature— distracts the individual from other threats and opportunities in its environment"
"Chimpanzees have a higher brain-to-body ratio than orangutans, yet orangs consistently recognise themselves in mirrors while chimps do so only half the time"
footnotes: Aiello, L., and C. Dean. 1990. An introduction to human evolutionary anatomy. Academic Press, London; 123 Gallup, G.G. (Jr.). 1997. On the rise and fall of self-conception in primates. In The Self Across Psychology— self-recognition, self-awareness, and the Self Concept. Annals of the NY Acad. Sci. 818:4-17
"it turns out that the unconscious mind is better at making complex decisions than is the conscious mind"
footnote: Dijksterhuis, A., et al. 2006. Science 311:1005-1007
To be clear I'm not arguing that "look at all these sources, it must be true!" (we know that kind of argument doesn't work). I'm hoping for somewhat more object-level counterarguments is all, or perhaps a better reason to dismiss them as being misguided (or to dismiss the picture Watts paints using them) than what Seth gestured at. I'm guessing he meant "complex general cognition" to point to something other than pure raw problem-solving performance.
Peter Watts is working with Neill Blomkamp to adapt his novel Blindsight into an 8-10-episode series:
“I can at least say the project exists, now: I’m about to start writing an episodic treatment for an 8-10-episode series adaptation of my novel Blindsight.
“Neill and I have had a long and tortured history with that property. When he first expressed interest, the rights were tied up with a third party. We almost made it work regardless; Neill was initially interested in doing a movie that wasn’t set in the Blindsight universe at all, but which merely used the speculative biology I’d invented to justify the existence of Blindsight’s vampires. “Sicario with Vampires” was Neill’s elevator pitch, and as chance would have it the guys who had the rights back then had forgotten to renew them. So we just hunkered quietly until those rights expired, and the recently-rights-holding parties said Oh my goodness we thought we’d renewed those already can we have them back? And I said, Sure; but you gotta carve out this little IP exclusion on the biology so Neill can do his vampire thing.
“It seemed like a good idea at the time. It was good idea, dammit. We got the carve-out and everything. But then one of innumerable dead-eyed suits didn’t think it was explicit enough, and the rights-holders started messing us around, and what looked like a done deal turned to ash. We lost a year or more on that account.
“But eventually the rights expired again, for good this time. And there was Neill, waiting patiently in the shadows to pounce. So now he’s developing both his Sicario-with-vampires movie and an actual Blindsight adaptation. I should probably keep the current status of those projects private for the time being. Neill’s cool with me revealing the existence of the Blindsight adaptation at least, and he’s long-since let the cat out of the bag for his vampire movie (although that was with some guy called Joe Rogan, don’t know how many people listen to him). But the stage of gestation, casting, and all those granular nuts and bolts are probably best kept under wraps for the moment.
“What I can say, though, is that it feels as though the book has been stuck in option limbo forever, never even made it to Development Hell, unless you count a couple of abortive screenplays. And for the first time, I feel like something’s actually happening. Stay tuned.”
When I first read Blindsight over a decade ago it blew my brains clean out of my skull. I'm cautiously optimistic about the upcoming series, we'll see...
There's a lot of fun stuff in Anders Sandberg's 1999 paper The Physics of Information Processing Superobjects: Daily Life Among the Jupiter Brains. One particularly vivid detail was (essentially) how the square-cube law imposes itself upon Jupiter brain architecture by forcing >99.9% of volume to be comprised of comms links between compute nodes, even after assuming a "small-world" network structure allowing sparse connectivity between arbitrarily chosen nodes by having them be connected by a short series of intermediary links with only 1% of links being long-range.
For this particular case ("Zeus"), a 9,000 km sphere of nearly solid diamondoid consisting mainly of reversible quantum dot circuits and molecular storage systems surrounded by a concentric shield protecting it from radiation and holding radiators to dissipate heat into space, with energy provided by fusion reactors distributed outside the shield, only the top 1.35 km layer is compute + memory (a lot thinner comparatively than the Earth's crust), and the rest of the interior is optical comms links. Sandberg calls this the "cortex model".
In a sense this shouldn't be surprising since both brains and current semiconductor chips are mostly interconnect by volume already, but a 1.35 km thick layer of compute + memory encompassing a 9,000 km sphere of optical comms links seems a lot more like a balloon to me than anything, so from now on I'll probably think of them as Jupiter balloons.
Venkatesh Rao's recent newsletter article Terms of Centaur Service caught my eye for his professed joy of AI-assisted writing, both nonfiction and fiction:
In the last couple of weeks, I’ve gotten into a groove with AI-assisted writing, as you may have noticed, and I am really enjoying it. ... The AI element in my writing has gotten serious, and I think is here to stay. ...
On the writing side, when I have a productive prompting session, not only does the output feel information dense for the audience, it feels information dense for me.
An example of this kind of essay is one I posted last week, on a memory-access-boundary understanding of what intelligence is. This was an essay I generated that I got value out of reading. And it didn’t feel like a simple case of “thinking through writing.” There’s stuff in here contributed by ChatGPT that Ididn’t know or realize even subconsciously, even though I’ve been consulting for 13 years in the semiconductor industry.
Generated text having elements new to even the prompter is a real benefit, especially with fiction. I wrote a bit of fiction last week that will be published in Protocolized tomorrow that was so much fun, I went back and re-read it twice. This is something I never do with m own writing. By the time I ship an unassisted piece of writing, I’m generally sick of it.
AI-assisted writing allows you to have your cake and eat it too. The pleasure of the creative process, and the pleasure of reading. That’s in fact a test of good slop — do you feel like reading it?
I think this made an impression on me because Venkat's joy contrasts so much to many people's criticism of Sam Altman's recent tweet re: their new creative fiction model's completion to the prompt "Please write a metafictional literary short story about AI and grief", including folks like Eliezer, who said "To be clear, I would be impressed with a dog that wrote the same story, but only because it was a dog". I liked the AI's output quite a lot actually, more than I did Eliezer's (and I loved HPMOR so I should be selected for Eliezer-fiction-bias), and I found myself agreeing with Roon's pushback to him.
Although Roshan's remark that "AI fiction seems to be in the habit of being interesting only to the person who prompted it" does give me pause. While this doesn't seem to be true in the AI vs Eliezer comparison specifically, I do find plausible a hyperpersonalisation-driven near-future where AI fiction becomes superstimuli-level interesting only to the prompter. But I find the contra scenario plausible too. Not sure where I land here.
I’m an expert on Nietzsche (I’ve read some of his books), but not a world-leading expert (I didn’t understand them). And one of the parts I didn’t understand was the psychological appeal of all this. So you’re Caesar, you’re an amazing general, and you totally wipe the floor with the Gauls. You’re a glorious military genius and will be celebrated forever in song. So . . . what? Is beating other people an end in itself? I don’t know, I guess this is how it works in sports6. But I’ve never found sports too interesting either. Also, if you defeat the Gallic armies enough times, you might find yourself ruling Gaul and making decisions about its future. Don’t you need some kind of lodestar beyond “I really like beating people”? Doesn’t that have to be something about leaving the world a better place than you found it?
Admittedly altruism also has some of this same problem. Auden said that “God put us on Earth to help others; what the others are here for, I don’t know.” At some point altruism has to bottom out in something other than altruism. Otherwise it’s all a Ponzi scheme, just people saving meaningless lives for no reason until the last life is saved and it all collapses.
I have no real answer to this question - which, in case you missed it, is “what is the meaning of life?” But I do really enjoy playing Civilization IV. And the basic structure of Civilization IV is “you mine resources, so you can build units, so you can conquer territory, so you can mine more resources, so you can build more units, so you can conquer more territory”. There are sidequests that make it less obvious. And you can eventually win by completing the tech tree (he who has ears to hear, let him listen). But the basic structure is A → B → C → A → B → C. And it’s really fun! If there’s enough bright colors, shiny toys, razor-edge battles, and risk of failure, then the kind of ratchet-y-ness of it all, the spiral where you’re doing the same things but in a bigger way each time, turns into a virtuous repetition, repetitive only in the same sense as a poem, or a melody, or the cycle of generations.
The closest I can get to the meaning of life is one of these repetitive melodies. I want to be happy so I can be strong. I want to be strong so I can be helpful. I want to be helpful because it makes me happy.
I want to help other people in order to exalt and glorify civilization. I want to exalt and glorify civilization so it can make people happy. I want them to be happy so they can be strong. I want them to be strong so they can exalt and glorify civilization. I want to exalt and glorify civilization in order to help other people.
I want to create great art to make other people happy. I want them to be happy so they can be strong. I want them to be strong so they can exalt and glorify civilization. I want to exalt and glorify civilization so it can create more great art.
I want to have children so they can be happy. I want them to be happy so they can be strong. I want them to be strong so they can raise more children. I want them to raise more children so they can exalt and glorify civilization. I want to exalt and glorify civilization so it can help more people. I want to help people so they can have more children. I want them to have children so they can be happy.
Maybe at some point there’s a hidden offramp marked “TERMINAL VALUE”. But it will be many more cycles around the spiral before I find it, and the trip itself is pleasant enough.
The OECD working paper Miracle or Myth? Assessing the macroeconomic productivity gains from Artificial Intelligence, published quite recently (Nov 2024), is strange to skim-read: its authors estimate just 0.24-0.62 percentage points annual aggregate TFP growth (0.36-0.93 pp. for labour productivity) over a 10-year horizon, depending on scenario, using a "novel micro-to-macro framework" that combines "existing estimates of micro-level performance gains with evidence on the exposure of activities to AI and likely future adoption rates, relying on a multi-sector general equilibrium model with input-output linkages to aggregate the effects".
I checked it out both to get a more gears-y sense of how AI might transform the economy soon and to get an outside-my-bubble data-grounded sense of what domain experts think, but 0.24-0.62 pp TFP growth and 0.36-0.93 pp labor seem so low (relative to say L Rudolf L's history of the future, let alone AI 2027) that I'm tempted to just dismiss them as not really internalising what AGI means. A few things prevent me from dismissing them: it seems epistemically unvirtuous to do so, they do predicate their forecasts on a lot of empirical data, anecdotes like lc's recent AI progress feeling mostly like bullshit (although my own experience is closer to this), and (boring technical loophole) they may end up being right in the sense that real GDP would still look smooth even after a massive jump in AI, due to GDP growth being calculated based on post-jump prices deflating the impact of the most-revolutionised goods & services.
Why so low? They have 3 main scenarios (low adoption, high adoption and expanded capabilities, and latter plus adjustment frictions and uneven gains across sectors, which I take to be their best guess), plus 2 additional scenarios with "more extreme assumptions" (large and concentrated gains in most exposed sectors, which they think are ICT services, finance, professional services and publishing and media, and AI + robots, which is my own best guess); all scenarios assume just +30% micro-level gains from AI, except the concentrated gains one which assumes 100% gains in the 4 most-exposed sectors. From this low starting point they effectively discount further by factors like Acemoglu (2024)'s estimate that 20% of US labor tasks are exposed to AI (ranging from 11% in agriculture to ~50% in IT and finance), exposure to robots (which seems inversely related to AI exposure, e.g. ~85% in agriculture vs < 10% in IT and finance), 23-40% AI adoption rates, restricted factor allocation across sectors, inelastic demand, Baumol effect kicking in for scenarios with uneven cross-sectoral gains, etc.
Why just +30% micro-level gain from AI? They explain in section 2.2.1; to my surprise they're already being more generous than the authors they quote, but as I'd guessed they just didn't bother to predict whether micro-level gains would improve over time at all:
Briggs and Kodnani (2023) rely on firm-level studies which estimate an average gain of about 2.6% additional annual growth in workers’ productivity, leading to about a 30% productivity boost over 10 years. Acemoglu (2024) uses a different approach and start from worker-level performance gains in specific tasks, restricted to recent Generative AI applications. Nevertheless, these imply a similar magnitude, roughly 30% increase in performance, which they assume to materialise over the span of 10 years.
However, they interpret these gains as pertaining only to reducing labour costs, hence when computing aggregate productivity gains, they downscale the micro gains by the labour share. In contrast, we take the micro studies as measuring increases in total factor productivity since we interpret their documented time savings to apply to the combined use of labour and capital. For example, we argue that studies showing that coders complete coding tasks faster with the help of AI are more easily interpretable as an increase in the joint productivity of labour and capital (computers, office space, etc.) rather than as cost savings achieved only through the replacement of labour.
To obtain micro-level gains for workers performing specific tasks with the help of AI, this paper relies on the literature review conducted by Filippucci et al. (2024). ... The point estimates indicate that the effect of AI tools on worker performance in specific tasks range from 14% (in customer service assistance) to 56% (in coding), estimated with varying degrees of precision (captured by different sizes of confidence intervals). We will assume a baseline effect of 30%, which is around the average level of gains in tasks where estimates have high precision.
Why not at least try to forecast micro-level gains improvement over the next 10 years?
Finally, our strategy aims at studying the possible future impact of current AI capabilities, considering also a few additional capabilities that can be integrated into our framework by relying on existing estimates (AI integration with additional software based on Eloundou et al, 2024; integration with robotics technologies). In addition, it is clearly possible that new types of AI architectures will eliminate some of the current important shortcomings of Generative AI – inaccuracies or invented responses, “hallucinations” – or improve further on the capabilities, perhaps in combination with other existing or emerging technologies, enabling larger gains (or more spread-out gains outside these knowledge intensive services tasks; see next subsection). However, it is still too early to assess whether and to what extent these emerging real world applications can be expected.
Ah, okay then.
What about that 23-40% AI adoption rate forecast over the next 10 years, isn't that too conservative?
To choose realistic AI adoption rates over our horizon, we consider the speed at which previous major GPTs (electricity, personal computers, internet) were adopted by firms. Based on the historical evidence, we consider two possible adoption rates over the next decade: 23% and 40% (Figure 6). The lower adoption scenario is in line with the adoption path of electricity and with assumptions used in the previous literature about the degree of cost-effective adoption of a specific AI technology – computer vision or image recognition – in 10 years (Svanberg et al., 2024; also adopted by Acemoglu, 2024). The higher adoption scenario is in line with the adoption path of digital technologies in the workplace such as computers and internet. It is also compatible with a more optimistic adoption scenario based on a faster improvement in the cost-effectiveness of computer vision in the paper by Svanberg et al. (2024).
On the one hand, the assumption of a 40% adoption rate in 10 years can still be seen as somewhat conservative, since AI might have a quicker adoption rate than previous digital technologies, due its user-friendly nature. For example, when looking at the speed of another, also relatively user-friendly technology, the internet, its adoption by households after 10 years surpassed 50% (Figure A2 in the Annex). On the other hand, a systemic adoption of AI in the core business functions – instead of using it only in isolated, specific tasks – would still require substantial complementary investments by firms in a range of intangible assets, including data, managerial practices, and organisation (Agrawal, A., J. Gans and A. Goldfarb, 2022). These investments are costly and involve a learning-by-doing, experimental phase, which may slow down or limit adoption. Moreover, while declining production costs were a key driver of rising adoption for past technologies, there are indications that current AI services are already provided at discount prices to capture market shares, which might not be sustainable for long (see Andre et al, 2024). Finally, the pessimistic scenario might also be relevant in the case where limited reliability of AI or lack of social acceptability prevents AI adoption for specific occupations. To reflect this uncertainty, our main scenarios explore the implications of assuming either a relatively low 23% or a higher 40% future adoption rate.
I feel like they're failing to internalise the lesson from this chart that adoption rates are accelerating over time:
(Not a take, just pulling out infographics and quotes for future reference from the new DeepMind paper outlining their approach to technical AGI safety and security)
Overview of risk areas, grouped by factors that drive differences in mitigation approaches:
Overview of their approach to mitigating misalignment:
Overview of their approach to mitigating misuse:
Path to deceptive alignment:
How to use interpretability:
Goal
Understanding v Control
Confidence
Concept v Algorithm
(Un)supervised?
How context specific?
Alignment evaluations
Understanding
Any
Concept+
Either
Either
FaithfulReasoning
Understanding∗
Any
Concept+
Supervised+
Either
DebuggingFailures
Understanding∗
Low
Either
Unsupervised+
Specific
Monitoring
Understanding
Any
Concept+
Supervised+
General
Red teaming
Either
Low
Either
Unsupervised+
Specific
Amplified oversight
Understanding
Complicated
Concept
Either
Specific
Interpretability techniques:
Technique
Understanding v Control
Confidence
Concept v Algorithm
(Un)supervised?
How specific?
Scalability
Probing
Understanding
Low
Concept
Supervised
Specific-ish
Cheap
Dictionary learning
Both
Low
Concept
Unsupervised
General∗
Expensive
Steering vectors
Control
Low
Concept
Supervised
Specific-ish
Cheap
Training data attribution
Understanding
Low
Concept
Unsupervised
General∗
Expensive
Auto-interp
Understanding
Low
Concept
Unsupervised
General∗
Cheap
Component Attribution
Both
Medium
Concept
Complicated
Specific
Cheap
Circuit analysis (causal)
Understanding
Medium
Algorithm
Complicated
Specific
Expensive
Assorted random stuff that caught my attention:
They consider Exceptional AGI (Level 4) from Morris et al. (2023), defined as an AI system that matches or exceeds that of the 99th percentile of skilled adults on a wide range of non-physical tasks (contra the Metaculus "when AGI?" question that has diverse robotic capabilities, so their 2030 is probably an overestimate)
The irrelevance of physical limits to the paper's scope: "By considering the construction of “the ultimate laptop”, Lloyd (2000) suggests that Moore’s law (formalized as an 18 month doubling) cannot last past 2250. Krauss and Starkman (2004) consider limits on the total computation achievable by any technological civilization in our expanding universe - this approach imposes a (looser) 600-year limit in Moore’s law. However, since we are very far from these limits, we do not expect them to have a meaningful impact on timelines to Exceptional AGI"
Structural risks are "out of scope of this paper" because they're "a much bigger category, often with each risk requiring a bespoke approach. They are also much harder for an AI developer to address, as they often require new norms or institutions to shape powerful dynamics in the world" (although "much of the technical work discussed in this paper will also be relevant for structural risks")
Mistakes are also out of scope because "standard safety engineering practices (e.g. testing) can drastically reduce risks, and should be similarly effective for averting AI mistakes as for human mistakes... so we believe that severe harm from AI mistakes will be significantly less likely than misuse or misalignment, and is further reducible through appropriate safety practices"
The paper focuses "primarily on techniques that can be integrated into current AI development, due to our focus on anytime approaches to safety" i.e. excludes "research bets that pay out over longer periods of time but can provide increased safety, such as agent foundations, science of deep learning, and application of formal methods to AI"
Algorithmic progress papers: "Erdil and Besiroglu (2022) sought to decompose AI progress in a way that can be attributed to the separate factors of scaling (compute, model size and data) and algorithmic innovation, and concluded that algorithmic progress doubles effective compute budgets roughly every nine months. Ho et al. (2024) further extend this approach to study algorithmic improvements in the pretraining of language models for the period of 2012 - 2023. During this period, the authors estimate that the compute required to reach a set performance threshold halved approximately every eight months"
Explosive economic growth paper: "Recent modeling by Erdil et al. (2025) that draws on empirical scaling laws and semi-endogenous growth theory and models changes in compute, automation and production supports the plausibility of very rapid growth in Gross World Product (e.g. exceeding 30% per year in 2045) when adopting parameters from empirical data, existing literature and reasoned judgment" (I'm still wondering how this will get around johnswentworth's objection to using GDP to track this)
General competence scales smoothly with compute: "Owen (2024) find that aggregate benchmarks (BIG-Bench (Srivastava et al., 2023), MMLU (Hendrycks et al., 2020)) are predictable with up to 20 percentage points of error when extrapolating through one order of magnitude (OOM) of compute. Gadre et al. (2024) similarly find that aggregate task performance can be predicted with relatively high accuracy, predicting average top-1 error across 17 tasks to within 1 percentage point using 20× less compute than is used for the predicted model. Ruan et al. (2024) find that 8 standard downstream LLM benchmark scores across many model families are well-explained in terms of their top 3 principal components. Their first component scales smoothly across 5 OOMs of compute and many model families, suggesting that something like general competence scales smoothly with compute"
"given that total labor compensation represents over 50% of global GDP (International Labour Organisation, 2022), it is clear that the economic incentive for automation is extraordinarily large"
I agree that virtues should be thought of as trainable skills, which is also why I like David Gross's idea of a virtue gym:
Two misconceptions sometimes cause people to give up too early on developing virtues:
that virtues are talents that some people have and other people don’t as a matter of predisposition, genetics, the grace of God, or what have you (“I’m just not a very influential / graceful / original person”), and
that having a virtue is not a matter of developing a habit but of having an opinion (e.g. I agree that creativity is good, and I try to respect the virtue of creativity that way, rather than by creating).
It’s better to think of a virtue as a skill like any other. Like juggling, it might be hard at first, it might come easier to some people than others, but almost anyone can learn to do it if they put in persistent practice.
We are creatures of habit: We create ourselves by what we practice. If we adopt habits carelessly, we risk becoming what we never intended to be. If instead we deliberate about what habits we want to cultivate, and then actually put in the work, we can become the sculptors of our own characters.
What if there were some institution like a “virtue gymnasium” in which you could work on virtues alongside others, learning at your own pace, and building a library of wisdom about how to go about it most productively? What if there were something like Toastmasters, or Alcoholics Anonymous, or the YMCA but for all of the virtues?
Conversations with LLMs could be the "home gym" equivalent I suppose.
In ~2020 we witnessed the Men’s/Women’s World Cup Scandal. The US Men’s Soccer team had failed to qualify for the previous World Cup, whereas the US Women’s Soccer team had won theirs! And yet the women were paid less that season after winning than the men were paid after failing to qualify. There was Discourse.
I was in the car listening to NPR, pulling out of the parking lot of a glass supplier when my world shattered again.3 One of the NPR leftist commenters said roughly ~‘One can propose that the mens team and womens team play against each other to sort this out—’
At which point I mentally pumped my fist in the air and cheered. I had been thinking exactly this for WEEKS. I couldn’t quite understand why no one had said it! As we all know, men and women are largely undifferentiated. Soccer is a perfect example of this, because the sport doesn’t allow men to use their upper-body strength advantage at all. The one thing that makes men stand out is neutralized here, and a direct competition would put this thing to rest and humiliate all the sexists. I smiled and waited to see how the right-wing asshat would squirm out of having to endorse a match that we all knew would shut him up.
The left-wing commentator continued ‘—is what one would say if one is a right-wing deplorable that just wants to laugh while humiliating those that are already oppressed. Naturally none of us would ever propose such a thing, we aren’t horrible people. Here’s what they get wrong…”
I didn’t hear any more after that, because my world had shattered again. A proponent of my side was not only admitting that the women’s team would lose badly, but that everyone knew and had always known that the women’s team would lose badly, so the only reason one would even suggest such a thing was to humiliate them.
Here I was, in my late 30s, still believing that men and women are basically the same, like a fucking chump. Do these people realize how much of my life, my personal and public decisions, my views of my fellow man and my plans for the future, were predicated on this being actually true? Not a single person had ever once bothered to take me aside and whisper “Hey, we know this isn’t actually true, we’re just acting this way because it leads to better outcomes for society, on net, if we do. Obviously we make exceptions for the places where the literal truth is important. Welcome to the secret club, don’t tell the kids.”
These were the people who always had told me men and women are equal in all things, explicitly saying that anyone who actually really believed this was a deplorable right-wing troll. I could taste the betrayal in my mouth. It tasted of bile. How had this happened to me again?
A couple years prior I had lost a woman I dearly loved, as well as the associated friend group, when I had Not Gotten The Joke about a different belief and accidentally acted as if I believed something that everyone agreed to say was true was Actually True4. I didn’t understand what had happened back then. Now it was starting to make sense. I was too damn trusting and autistic to make a reliable ally in a world bereft of truth.
Scott's own reaction to / improvement upon Graham's hierarchy of disagreement (which I just noticed you commented on back in the day, so I guess this is more for others' curiosity) is
Graham’s hierarchy is useful for its intended purpose, but it isn’t really a hierarchy of disagreements. It’s a hierarchy of types of response, within a disagreement. Sometimes things are refutations of other people’s points, but the points should never have been made at all, and refuting them doesn’t help. Sometimes it’s unclear how the argument even connects to the sorts of things that in principle could be proven or refuted.
If we were to classify disagreements themselves – talk about what people are doing when they’re even having an argument – I think it would look something like this:
Most people are either meta-debating – debating whether some parties in the debate are violating norms – or they’re just shaming, trying to push one side of the debate outside the bounds of respectability.
If you can get past that level, you end up discussing facts (blue column on the left) and/or philosophizing about how the argument has to fit together before one side is “right” or “wrong” (red column on the right). Either of these can be anywhere from throwing out a one-line claim and adding “Checkmate, atheists” at the end of it, to cooperating with the other person to try to figure out exactly what considerations are relevant and which sources best resolve them.
If you can get past that level, you run into really high-level disagreements about overall moral systems, or which goods are more valuable than others, or what “freedom” means, or stuff like that. These are basically unresolvable with anything less than a lifetime of philosophical work, but they usually allow mutual understanding and respect.
Seems like yours and Scott's are complementary: I read you as suggesting how to improve one's own argumentation techniques, while Scott is being more sociologically descriptive, mainly in explaining why online discourse so often degenerates into social shaming and meta-debate.
A shower thought I once had, intuition-pumped by MIRI's / Luke's old post on turning philosophy to math to engineering, was that if metaethicists really were serious about resolving their disputes they should contract a software engineer (or something) to help implement on GitHub a metaethics version of Table 2, where rows would be moral dilemmas like the trolley problem and columns ethical theories, and then accept that real-world engineering solutions tend to be "dirty" and inelegant remixes plus kludgy optimisations to handle edge cases, but would clarify what the SOTA was and guide "metaethical innovation" much better, like a qualitative multi-criteria version of AI benchmarks.
I gave up on this shower thought for various reasons, including that I was obviously naive and hadn't really engaged with the metaethical literature in any depth, but also because I ended up thinking that disagreements on doing good might run ~irreconcilably deep, plus noticing that Rethink Priorities had done the sophisticated v1 of a subset of what I had in mind and nobody really cared enough to change what they did. (In my more pessimistic moments I'd also invoke the diseased discipline accusation, but that may be unfair and outdated.)
Lee Billings' book Five Billion Years of Solitude has the following poetic passage on deep time that's stuck with me ever since I read it in Paul Gilster's post:
Deep time is something that even geologists and their generalist peers, the earth and planetary scientists, can never fully grow accustomed to.
The sight of a fossilized form, perhaps the outline of a trilobite, a leaf, or a saurian footfall can still send a shiver through their bones, or excavate a trembling hollow in the chest that breath cannot fill. They can measure celestial motions and list Earth’s lithic annals, and they can map that arcane knowledge onto familiar scales, but the humblest do not pretend that minds summoned from and returned to dust in a century’s span can truly comprehend the solemn eons in their passage.
Instead, they must in a way learn to stand outside of time, to become momentarily eternal. Their world acquires dual, overlapping dimensions— one ephemeral and obvious, the other enduring and hidden in plain view. A planet becomes a vast machine, or an organism, pursuing some impenetrable purpose through its continental collisions and volcanic outpourings. A man becomes a protein-sheathed splash of ocean raised from rock to breathe the sky, an eater of sun whose atoms were forged on an anvil of stars.
Beholding the long evolutionary succession of Earthly empires that have come and gone, capped by a sliver of human existence that seems so easily shaved away, they perceive the breathtaking speed with which our species has stormed the world. Humanity’s ascent is a sudden explosion, kindled in some sapient spark of self-reflection, bursting forth from savannah and cave to blaze through the biosphere and scatter technological shrapnel across the planet, then the solar system, bound for parts unknown. From the giant leap of consciousness alongside some melting glacier, it proved only a small step to human footprints on the Moon.
The modern era, luminous and fleeting, flashes like lightning above the dark, abyssal eons of the abiding Earth. Immersed in a culture unaware of its own transience, students of geologic time see all this and wonder whether the human race will somehow abide, too.
Feynman loved doing physics. I think what he loved most was the process of it. Of calculating. Of figuring things out. It didn’t seem to matter to him so much if what came out was big and important. Or esoteric and weird. What mattered to him was the process of finding it. And he was often quite competitive about it.
Some scientists (myself probably included) are driven by the ambition to build grand intellectual edifices. I think Feynman — at least in the years I knew him — was much more driven by the pure pleasure of actually doing the science. He seemed to like best to spend his time figuring things out, and calculating. And he was a great calculator. All around perhaps the best human calculator there’s ever been.
Here’s a page from my files: quintessential Feynman. Calculating a Feynman diagram:
It’s kind of interesting to look at. His style was always very much the same. He always just used regular calculus and things. Essentially nineteenth-century mathematics. He never trusted much else. But wherever one could go with that, Feynman could go. Like no one else.
I always found it incredible. He would start with some problem, and fill up pages with calculations. And at the end of it, he would actually get the right answer! But he usually wasn’t satisfied with that. Once he’d gotten the answer, he’d go back and try to figure out why it was obvious. And often he’d come up with one of those classic Feynman straightforward-sounding explanations. And he’d never tell people about all the calculations behind it. Sometimes it was kind of a game for him: having people be flabbergasted by his seemingly instant physical intuition, not knowing that really it was based on some long, hard calculation he’d done.
Feynman and Wolfram had very different problem-solving styles:
Typically, Feynman would do some calculation. With me continually protesting that we should just go and use a computer. Eventually I’d do that. Then I’d get some results. And he’d get some results. And then we’d have an argument about whose intuition about the results was better.
The way he grappled with Wolfram's rule 30 exemplified this (I've omitted a bunch of pictures, you can check them out in the article):
You know, I remember a time — it must have been the summer of 1985 — when I’d just discovered a thing called rule 30. That’s probably my own all-time favorite scientific discovery. And that’s what launched a lot of the whole new kind of science that I’ve spent 20 years building (and wrote about in my bookA New Kind of Science). ...
Well, Feynman and I were both visiting Boston, and we’d spent much of an afternoon talking about rule 30. About how it manages to go from that little black square at the top to make all this complicated stuff. And about what that means for physics and so on.
Well, we’d just been crawling around the floor — with help from some other people — trying to use meter rules to measure some feature of a giant printout of it. And Feynman took me aside, rather conspiratorially, and said, “Look, I just want to ask you one thing: how did you know rule 30 would do all this crazy stuff?” “You know me,” I said. “I didn’t. I just had a computer try all the possible rules. And I found it.” “Ah,” he said, “now I feel much better. I was worried you had some way to figure it out.”
Feynman and I talked a bunch more about rule 30. He really wanted to get an intuition for how it worked. He tried bashing it with all his usual tools. Like he tried to work out what the slope of the line between order and chaos is. And he calculated. Using all his usual calculus and so on. He and his son Carl even spent a bunch of time trying to crack rule 30 using a computer.
And one day he calls me and says, “OK, Wolfram, I can’t crack it. I think you’re on to something.” Which was very encouraging.
Scott's The Colors Of Her Coat is the best writing I've read by him in a long while. Quoting this part in particular as a self-reminder and bulwark against the faux-sophisticated world-weariness I sometimes slip into:
Chesterton’s answer to the semantic apocalypse is to will yourself out of it. If you can’t enjoy My Neighbor Totoro after seeing too many Ghiblified photos, that’s a skill issue. Keep watching sunsets until each one becomes as beautiful as the first...
If you insist that anything too common, anything come by too cheaply, must be boring, then all the wonders of the Singularity cannot save you. You will grow weary of green wine and sick of crimson seas. But if you can bring yourself to really pay attention, to see old things for the first time, then you can combine the limitless variety of modernity with the awe of a peasant seeing an ultramarine mural - or the delight of a 2025er Ghiblifying photos for the first time.
How to see old things for the first time? I thought of the following passage by LoganStrohl describing a SIM card ejection tool:
I started studying “original seeing”, on purpose and by that name, in 2018. What stood out to me about my earliest exploratory experiments in original seeing is how alien the world is. ...
I started my earliest experimentation with some brute-force phenomenology. I picked up an object, set it on the table in front of me, and progressively stripped away layers of perception as I observed it. It was one of these things:
I wrote, “It’s a SIM card ejection tool.”
I wrote some things about its shape and color and so forth (it was round and metal, with a pointy bit on one end); and while I noted those perceptions, I tried to name some of the interpretations my mind seemed to be engaging in as I went.
As I identified the interpretations, I deliberately loosened my grip on them: “I notice that what I perceive as ‘shadows’ needn’t be places where the object blocks rays of light; the ‘object’ could be two-dimensional, drawn on a surface with the appropriate areas shaded around it.”
I noticed that I kept thinking in terms of what the object is for, so I loosened my grip on the utility of the object, mainly by naming many other possible uses. I imagined inserting the pointy part into soil to sow tiny snapdragon seeds, etching my name on a rock, and poking an air hole in the top of a plastic container so the liquid contents will pour out more smoothly. I’ve actually ended up keeping this SIM card tool on a keychain, not so I can eject SIM trays from phones, but because it’s a great stim; I can tap it like the tip of a pencil, but without leaving dots of graphite on my finger.
I loosened my grip on several preconceptions about how the object behaves, mainly by making and testing concrete predictions, some of which turned out to be wrong. For example, I expected it to taste sharp and “metallic”, but in fact I described the flavor of the surface as “calm, cool, perhaps lightly florid”.
By the time I’d had my fill of this proto-exercise, my relationship to the object had changed substantially. I wrote:
My perceptions that seem related to the object feel very distinct from whatever is out there impinging on my senses. … I was going to simply look at a SIM card tool, and now I want to wrap my soul around this little region of reality, a region that it feels disrespectful to call a ‘SIM card tool’. Why does it feel disrespectful? Because ‘SIM card tool’ is how I use it, and my mind is trained on the distance between how I relate to my perceptions of it, and what it is.
That last paragraph, and especially the use of 'disrespectful', strikes me a bit like the rationalist version of what Chesterton was talking about in Scott's post.
After eighteen years of being a professor, I’ve graded many student essays. And while I usually try to teach a deep structure of concepts, what the median student actually learns seems to mostly be a set of low order correlations. They know what words to use, which words tend to go together, which combinations tend to have positive associations, and so on. But if you ask an exam question where the deep structure answer differs from answer you’d guess looking at low order correlations, most students usually give the wrong answer.
Simple correlations also seem sufficient to capture most polite conversation talk, such as the weather is nice, how is your mother’s illness, and damn that other political party. Simple correlations are also most of what I see in inspirational TED talks, and when public intellectuals and talk show guests pontificate on topics they really don’t understand, such as quantum mechanics, consciousness, postmodernism, or the need always for more regulation everywhere. After all, media entertainers don’t need to understand deep structures any better than do their audiences.
Let me call styles of talking (or music, etc.) that rely mostly on low order correlations “babbling”. Babbling isn’t meaningless, but to ignorant audiences it often appears to be based on a deeper understanding than is actually the case. When done well, babbling can be entertaining, comforting, titillating, or exciting. It just isn’t usually a good place to learn deep insight.
It's unclear to me how much economically-relevant activity is generated by low order correlation-type reasoning, or whatever the right generalisation of "babbling" is here.
I don't know either, but I think of Tracing Woodgrains' Center for Educational Progress and the growing Discord community around it as a step in this direction.
Good homework by Zitron on the numbers, and he's a really entertaining writer, but my (very brief) experience so far using it for work-related research more closely matches Sarah Constantin's assessment concluding that ChatGPT-4o DR was the best one she tested (including Perplexity, Gemini, ChatGPT-4o, Elicit, and PaperQA) on completeness, relevance, source quality, and creativity.
Take me for an example. On the WISC-III IQ test, my combined score is 145. There are two composite scores that the combined score is made up of, the verbal score (I got 155, the maximum possible on that test) and the performance score (I got 125). There are also a number of different individual capacity scores. On most, I scored above the 95 percentile. On two or three, I scored right in the middle, and in one (visual short term memory) I scored in the first percentile.
Let me repeat that. I scored in the first percentile for the capacity to keep visual information in my short-term memory. (I scored in the 97th for aural short term memory, and 99.9th for linguistic.) How does that change how I solve problems, how I think about the world? Well, I perform many tasks about twice as slowly (but just as accurately) as others with my composite IQ. I have to use other circuits than most people do to solve the same problems, circuits that aren't as efficient. Circuits that may even work slightly differently, giving me a different perspective on problems, which may be superior or inferior, I don't know (likely depending on the individual problem). I strongly suspect that this is a large part of the cause of my intense dislike of school.
At the individual level, I can't see myself ever choosing for my child to have >99.9th percentile linguistic ability and 1st(!!) percentile visual short-term memory, or really any such spectacularly uneven combination of abilities. (I'm not as extreme, but I remember this quote because I empathise with it: I'm high-math low-verbal, my childhood was the mirror of Scott's, right down to "I don't know which bothered me more, the praise from effortless success or the criticism from backbreaking toil to get Bs on my own native language's exams".)
At the societal level however, there does seem to be a lot of benefit to a cognitive diversity of minds (I'm thinking of Cosma Shalizi and Henry Farrell's cognitive democracy essay, and their referencing Lu Hong and Scott Page (2004)'s use of mathematical models to argue that "diversity of viewpoints helps groups find better solutions"). So I guess one direction this line of thinking could go is how we can get the society-level benefits of a cognitive diversity of minds without necessarily having cognitively-uneven kids grow up in pain.
Style. As a writer of mathematics von Neumann was clear, but not clean; he was powerful but not elegant. He seemed to love fussy detail, needless repetition, and notation so explicit as to be confusing. To maintain a logically valid but perfectly transparent and unimportant distinction, in one paper he introduced an extension of the usual functional notation: along with the standard φ(x) he dealt also with something denoted by φ((x)). The hair that was split to get there had to be split again a little later, and there was φ(((x))), and, ultimately, φ((((x)))). Equations such as
(φ((((a))))^2 = φ(((a))))
have to be peeled before they can be digested; some irreverent students referred to this paper as von Neumann’s onion.
Perhaps one reason for von Neumann’s attention to detail was that he found it quicker to hack through the underbrush himself than to trace references and see what others had done. The result was that sometimes he appeared ignorant of the standard literature. If he needed facts, well-known facts, from Lebesgue integration theory, he waded in, defined the basic notions, and developed the theory to the point where he could use it. If, in a later paper, he needed integration theory again, he would go back to the beginning and do the same thing again. He saw nothing wrong with long strings of suffixes, and subscripts on subscripts; his papers abound in avoidable algebraic computations. The reason, probably, is that he saw the large picture; the trees did not conceal the forest from him. He saw and he relished all parts of the mathematics he was thinking about. He never wrote “down” to an audience; he told it as he saw it. The practice caused no harm; the main result was that, quite a few times, it gave lesser men an opportunity to publish “improvements” of von Neumann.
(tangent: I'm a bit peeved by Halmos' "lesser men" throwaway remark, mainly because I think interpretive research labor and distillation is very valuable, very hard to do well, somewhat orthogonal to vN-style competence, and very underappreciated and undersupplied.)
von Neumann was also courageous, Halmos wrote, in the following way:
Another notable and enviable trait of von Neumann’s was his mathematical courage. If, in the middle of a search for a counterexample, an infinite series came up, with a lot of exponentials that had quadratic exponents, many mathematicians would start with a clean sheet of paper and look for another counterexample. Not Johnny! When that happened to him, he cheerfully said: “Oh, yes, a theta function...’’, and plowed ahead with the mountainous computations. He wasn’t afraid of anything.
More specifically, one thing I learned from Terry that I was not taught in school is the importance of bad proofs. I would say "I think this is true", work on it, see that there was no nice proof, and give up. Terry would say "Here's a criterion that eliminates most of the problem. Then in what's left, here's a worse one that handles most of the detritus. One or two more epicycles. At that point it comes down to fourteen cases, and I checked them." Yuck. But we would know it was true, and we would move on. (Usually these would get cleaned up a fair bit before publication.) ...
Sometimes we'd really be on the same page, at the same letter of the same word even; one extreme case was when I needed to read his computer code and found it as easy to do as if I'd written it myself. But more often we'd bring different strengths. Since we were working in my field of expertise rather than his, I knew better what the interesting questions were, and could translate them into combinatorics, then sic Terry on them. He would beat them to a bloody death as described above, and then it would be my job to dress the carcass for public viewing back in the original field.
von Neumann also had endless capacity for work. Halmos:
Work habits. Von Neumann was not satisfied with seeing things quickly and clearly; he also worked very hard. His wife said “‘he had always done his writing at home during the night or at dawn. His capacity for work was practically unlimited.” In addition to his work at home, he worked hard at his office. He arrived early, he stayed late, and he never wasted any time. He was systematic in both large things and small; he was, for instance, a meticulous proofreader. He would correct a manuscript, record on the first page the page numbers where he found errors, and, by appropriate tallies, record the number of errors that he had marked on each of those pages. Another example: when requested to prepare an abstract of not more than 200 words, he would not be satisfied with a statistical check — there are roughly 20 lines with about 10 words each — but he would count every word.
I thought this was striking: why waste time on such seeming trivialities? But I guess if you're John von Neumann you just have such a glut of brain cycles that you can spend it in ridiculously poorly-optimised ways like this instead of needing to 80/20 and still get your many, many jobs done.
I think I failed to implicitly convey that I meant all this in jest, that I get a lot of personal enjoyment value out of silly poetry constrained by artificial rules, and that I was guessing at least someone else on the forum would share this enjoyment. I do like your scaffolding idea, might just try it out.
Pilish is a constrained writing style where the number of letters in consecutive words match the digits of pi. The canonical intro-to-Pilish sentence is "How I need a drink, alcoholic of course, after the heavy lectures involving quantum mechanics!"; my favorite Pilish poetry is Mike Keith's Near a Raven, a retelling of Edgar Allan Poe's "The Raven" stretching to 740 digits of pi (nowhere near Keith's longest, that would be the 10,000-word world record-setting Not a Wake), which begins delightfully like so:
Poe, E. Near a Raven
Midnights so dreary, tired and weary. Silently pondering volumes extolling all by-now obsolete lore. During my rather long nap - the weirdest tap! An ominous vibrating sound disturbing my chamber's antedoor. "This", I whispered quietly, "I ignore". ...
I wondered if today's leading LLMs could write interesting Pilish poems. The answer is "emphatically not".
Here's the start of Claude Sonnet 3.7's attempt. It doesn't even realise how wrong it is:
Celestial Calculations
How I perceive mathematical constants elegantly describing universe's fundamental architecture! Physicists, mathematicians repeatedly discover extraordinary connections manifesting mathematical elegance miraculously embedded. ...
Sonnet confidently and verbosely remarks that "Each word's length follows the digits of π (3.14159...), creating a hidden mathematical structure beneath the text. The poem spans approximately 100 digits of π, which is modest compared to the example you shared but still captures the essence of the constraint. The theme connects well with your physics background, ..." and more such nonsense.
ChatGPT 4.5 got the first 3 words right but then quickly derailed:
Here's a unique and engaging Pilish poem capturing the spirit of adventure and mystery:
Path Beyond Stars
Sky A glow, Far beyond gaze— Stars dazzle, whisper softly. Visions divine pursue travelers astray. ... [skipping 4 stanzas]
Pilish Constraint: The number of letters in each successive word follows the digits of π (3.14159265358979323846...). For instance, "Sky" (3 letters), "A" (1), "glow" (4), "Far" (3), and so forth.
Gemini 2.5 Pro, America's next top large language model, was the only one that realised it kept derailing and would redo again and again. It ended up almost getting it, but then dramatically derailed at the end. Full response this time:
Title: Quiet Night
See a lone I watch clouds drifting, so silent above. Now stars emerge, bright gems upon night’s velvet cloak. Still, quietude surrounds all the dark world; wonders invoke peaceful thought.
Explanation:
See (3)
a (1)
lone (4)
I (1)
watch (5)
clouds (6) - Correction: clouds has 6 letters, Pi's 6th digit after decimal is 9. Let's adjust.
Okay, let's try again. The single-letter words (like 'a' and 'I') corresponding to the '1's in Pi are often the trickiest parts.
Revised Title: Starlight
Now I look, a pale moon shimmers brightly, softly aloft. White diamonds glitter, placing subtle spells nightly; beauty ascends.
Let's check the counts:
Now (3)
I (1)
look (4)
a (1)
pale (5)
moon (4) - Correction: moon has 4 letters, Pi's 6th digit is 9. Need a 9-letter word.
This is proving challenging! Let's try focusing on a simpler theme, perhaps less descriptive.
Third Attempt: Simple Task
Ask a good, I might consider carefully, making calculations matter. Maybe numbers intrigue gifted folks solving riddles.
Checking Counts:
Ask (3)
a (1)
good (4)
I (1)
might (5)
consider (9)
carefully (9) - Correction: carefully has 9 letters, Pi's 7th digit is 2. Need a 2-letter word.
Alright, one more try, aiming for accuracy over length for now.
Fourth Attempt: A Cat's View
How a soft, white creature lounges lazily, ignores barking sounds. Just watches birds flitting outside - dreams peacefully...
Checking Counts:
How (3)
a (1)
soft (4)
white (5) - Correction: white has 5 letters, Pi's 5th digit is 5. Correct! Wait, Pi is 3.14159... so the 4th word needs to be 5 letters. My previous attempts were off by one position. Let's re-check "Starlight".
Re-checking "Starlight":
Now (3) - Correct
I (1) - Correct
look (4) - Correct
a (1) - Correct
pale (5) - Correct
moon (4) - Needs to be 9 letters. Let's replace "pale moon shimmers".
How about: Now I look a faint celestial...
faint (5) - Correct
celestial (9) - Correct!
Next digit is 2. Need a 2-letter word. ...celestial is...
is (2) - Correct!
Next digit is 6. Need a 6-letter word. ...celestial is bright...
bright (6) - Nope, 5 letters. How about ...celestial is divine...? Divine is 6 letters. Correct!
Next digit is 5. Need a 5-letter word. ...celestial is divine light...
light (5) - Correct!
Next digit is 3. Need a 3-letter word. ...divine light and...
and (3) - Correct!
Next digit is 5. Need a 5-letter word. ...divine light and peace...
peace (5) - Correct!
Okay, let's assemble this improved version:
Final Version: Starlight
Now I look a faint celestial is divine light and peace shines...
Checking Counts (Pi = 3.1415926535...)
Now (3) - Match
I (1) - Match
look (4) - Match
a (1) - Match
faint (5) - Match
celestial (9) - Match
is (2) - Match
divine (6) - Match
light (5) - Match
and (3) - Match
peace (5) - Match
shines (6) - Correction: shines has 6 letters, the next digit is 8. Need an 8-letter word.
Let's replace "peace shines". Next digits are 8, 9, 7, 9... ...light and faraway... (faraway = 8) - Correct! ...light and faraway silently... (silently = 9) - Correct! ...light and faraway silently viewing... (viewing = 7) - Correct! ...light and faraway silently viewing universes. (universes = 9) - Correct!
Final Poem:
Title: Starlight Gaze
Now I look a faint celestial is divine light and faraway silently viewing universes.
This poem follows the first 16 digits of Pi (including the initial 3). Pilish is definitely a tricky form!
Gemini's poem doesn't, in fact, follow the first 16 digits of pi, starting from when it inexplicably replaced 'peace' with 'faraway'.
I hereby propose a new AI progress benchmark: "Pilish poem length", or PPL.
I play Go (an Asian board game you can think of as being like chess) at a local club in the city. The rules of hierarchy there are something like “if Joe is much better than Gary, Joe can say Gary’s moves are wrong whenever he wants, and is allowed to launch into a short speech about why Gary’s mistake is such a major mistake.” (There are more cultural norms than that, and Go players reading this will find that a poor description, but for those outside of the culture, it’s close enough). On the other hand, when a weaker player thinks a stronger player’s move is weird, they’ll be very humble in their asking why the move is good. This is the opposite of the situation in corporate!
This is exactly the situation in (Asian) corporate, which is where I spent the first few years of my career at three different employers. In fact, at my first employer (a giant MNC headquartered in my country) juniors were expected to essentially shut up and do as we were told; the mere prospect of clarifying what a very senior person meant was terrifying for most. And at least in the Asian corporate contexts where I worked, what AnthonyC said is correct:
... the fact that you asked verbally is also in some sense a status claim, that you felt empowered to ask a question in a way that commanded everyone's attention and interrupted the CEO's talk track. It's a power move, or can be seen as such.
Given this, when I read what you said here
Back to asking tough questions of the CEO in a giant meeting: he is so powerful - so above me - that I know I can’t threaten him, he knows I can’t threaten him, everybody watching knows I can’t threaten him, and everybody knows that everybody knows this. Boldness has little to do with me asking him a hard question - if you view hierarchy and position how I do, it’s one of the least dangerous group situations in which to ask such a question!
I thought "but he, the CEO, can definitely threaten you. Is the OP not worried at all about this?"
At my first employer, my boss's boss (who everyone in the entire department loved) was fired by his boss (who everyone hated) for repeatedly questioning the latter's decisions (which everyone agreed weren't good) in meetings, in front of others. This happened a month into my tenure, and was quite the introduction to corporate not being college. I have similar stories from my second and third employer, if not quite as extreme.
I'm guessing this ultimately boils down to corporate culture differences.
There's an argument I've seen a number of times on the internet about the failings of consequentialism as a moral system. ... The argument goes roughly like so: Consequentialism tells us that the thing to do is the thing with the best results. But, this is a ridiculously high standard, that nobody can actually live up to. Thus, consequentialism tells us that everybody is bad, and we should all condemn everybody and all feel guilty.
But I find the “good person”/”not a good person” dichotomy helpful. I’m not claiming it objectively exists. I can’t prove anything about ethics objectively exists. And even if there were objective ethical truths about what was right or wrong, that wouldn’t imply that there was an objective ethical truth about how much of the right stuff you have to do before you can go around calling yourself “good”. In the axiology/morality/law trichotomy, I think of “how much do I have to do in order to be a good person” as within the domain of morality. That means it’s a social engineering question, not a philosophical one. The social engineering perspective assumes that “good person” status is an incentive that can be used to make people behave better, and asks how high vs. low the bar should be set to maximize its effectiveness.
Consider the way companies set targets for their employees. At good companies, goals are ambitious but achievable. If the CEO of a small vacuum company tells her top salesman to sell a billion vacuums a year, this doesn’t motivate the salesman to try extra hard. It’s just the equivalent of not setting a goal at all, since he’ll fail at the goal no matter what. If the CEO says “Sell the most vacuums you can, and however many you sell, I will yell at you for not selling more”, this also probably isn’t going to win any leadership awards. A good CEO might ask a salesman to sell 10% more vacuums than he did last year, and offer a big bonus if he can accomplish it. Or she might say that the top 20% of salesmen will get promotions, or that the bottom 20% of salesmen will be fired, or something like that. The point is that the goal should effectively carve out two categories, “good salesman” and “bad salesman”, such that it’s plausible for any given salesman to end up in either, then offer an incentive that makes him want to fall in the first rather than the second.
I think of society setting the targets for “good person” a lot like a CEO setting the targets for “good vacuum salesman”. If they’re attainable and linked to incentives – like praise, honor, and the right to feel proud of yourself – then they’ll make people put in an extra effort so they can end up in the “good person” category. If they’re totally unattainable and nobody can ever be a good person no matter how hard they try, then nobody will bother trying. This doesn’t mean nobody will be good – some people are naturally good without hope for reward, just like some people will slave away for the vacuum company even when they’re underpaid and underappreciated. It just means you’ll lose the extra effort you would get from having a good incentive structure.
So what is the right level at which to set the bar for “good person”? An economist might think of this question as a price-setting problem: society is selling the product “moral respectability” and trying to decide how many units effort to demand from potential buyers in order to maximize revenue. Set the price too low, and you lose out on money that people would have been willing to pay. Set the price too high, and you won’t get any customers. Solve for the situation where you have a monopoly on the good and the marginal cost of production is zero, and this is how you set the “good person” bar.
It mystifies me that as a Pro user my feature settings don't include the web search option, only the analysis tool. I wonder if it's a geographic location thing (I'm in Southeast Asia).
I like the optimal forager take, seems intuitively correct. I'd add that Dwarkesh struck gold by getting you on his podcast too. (Tangentially: this grand theory of intelligence video snippet reminds me of a page-ish-long writeup on that I stumbled upon deep in the bowels of https://gwern.net/ which I've annoyingly never been able to find again.)
Also thanks for the pointer to Werbos, his website Welcome to the Werbos World! funnily enough struck me as crackpot-y and I wouldn't have guessed just from the landing page that he's the discoverer of backprop, respected former program director at the NSF, etc.
Just to clarify, your post's bottomline is that AIs won't be omnipotent, and this matters for AI because a lot of common real-life problems are NP-hard, but also that this doesn't really matter (for us?) because there are ways around NP-hardness through cleverness and solving a different problem, or else by scaling hardware and writing programs more efficiently, or (referencing James) by just finding a good-enough solution instead of an optimal one?
I have 1050W of solar, ~10kWh of batteries, a 3kW hybrid inverter, and a 5.5kW gasoline generator. In spring and fall I can easily go a week without needing shore power or the generator. In summer and winter, I can't
Sorry naive question, I get that you can't do it in winter, but why not summer? Isn't that when solar peaks?
Aside from the normal cognitive benefits of being bilingual or multilingual, would learning some new language (or a conlang for this purpose) specifically to have conscious thought with be useful?
Not sure if this is exactly what you had in mind, since it's fictional transhumanist tech, but I was reminded of this passage from Richard Ngo's recent short story The Gentle Romance:
Almost everyone he talks to these days consults their assistant regularly. There are tell-tale signs: their eyes lose focus for a second or two before they come out with a new fact or a clever joke. He mostly sees it at work, since he doesn’t socialize much. But one day he catches up with a college friend he’d always had a bit of a crush on, who’s still just as beautiful as he remembers. He tries to make up for his nervousness by having his assistant feed him quips he can recite to her. But whenever he does, she hits back straight away with a pitch-perfect response, and he’s left scrambling.
“You’re good at this. Much faster than me,” he says abruptly.
“Oh, it’s not skill,” she says. “I’m using a new technique. Here.” With a flick of her eyes she shares her visual feed, and he flinches. Instead of words, the feed is a blur of incomprehensible images, flashes of abstract color and shapes, like a psychedelic Rorschach test.
“You can read those?”
“It’s a lot of work at first, but your brain adapts pretty quickly.”
He makes a face. “Not gonna lie, that sounds pretty weird. What if they’re sending you subliminal messages or something?”
Back home, he tries it, of course. The tutorial superimposes images and their text translations alongside his life, narrating everything he experiences. Having them constantly hovering on the side of his vision makes him dizzy. But he remembers his friend’s effortless mastery, and persists. Slowly the images become more comprehensible, until he can pick up the gist of a message from the colors and shapes next to it. For precise facts or statistics, text is still necessary, but it turns out that most of his queries are about stories: What’s in the news today? What happened in the latest episode of the show everyone’s watching? What did we talk about last time we met? He can get a summary of a narrative in half a dozen images: not just the bare facts but the whole arc of rising tension and emotional release. After a month he rarely needs to read any text.
That last link goes to Kurt Vonnegut on the 8 “shapes” of stories. The story is that Vonnegut wrote a master’s thesis on the shapes of stories that he submitted to the anthropology department at the University of Chicago, which rejected it. Here's a YouTube video of him talking about it; below is an infographic from that article:
That said, Richard's Vonnegut-inspired fictional tech is about communicating narratives efficiently, not precise facts or statistics. For that, Gwern's On the Existence of Powerful Natural Languages persuaded me that you can't really have powerful general-purpose conlangs that boost cognition across a wide variety of domains.
... big guests just don't really matter that much if you just look at what are the most popular episodes, or what in the long run helps a podcast grow. By far my most popular guest is Sarah Paine, and she, before I interviewed her, was just a scholar who was not publicly well-known at all, and I just found her books quite interesting—so my most popular guests are Sarah Paine and then Sarah Paine, Sarah Paine, Sarah Paine because I have electric chairs(?) a lecture series with her. And by the way, from a viewer-a-minute adjusted basis, I host the Sarah Paine podcast where I occasionally talk about AI.
(After Sarah Paine comes geneticist David Reich, then Satya Nadella and Mark Zuckerberg, "then [Sholto & Trenton] or Leopold (Aschenbrenner) or something, then you get to the lab CEOs or something")
In https://chatgpt.com/share/94152e76-7511-4943-9d99-1118267f4b2b I gave the new model a challenging complex analysis problem (which I had previously asked GPT4 to assist in writing up a proof of in https://chatgpt.com/share/63c5774a-d58a-47c2-9149-362b05e268b4 ). Here the results were better than previous models, but still slightly disappointing: the new model could work its way to a correct (and well-written) solution *if* provided a lot of hints and prodding, but did not generate the key conceptual ideas on its own, and did make some non-trivial mistakes. The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student. However, this was an improvement over previous models, whose capability was closer to an actually incompetent (static simulation of a) graduate student. It may only take one or two further iterations of improved capability (and integration with other tools, such as computer algebra packages and proof assistants) until the level of "(static simulation of a) competent graduate student" is reached, at which point I could see this tool being of significant use in research level tasks. (2/3)
I posed the identical question to my version of #o1 and it returned a perfect answer: https://chatgpt.com/share/66e7153c-b7b8-800e-bf7a-1689147ed21e . Admittedly, the above MathOverflow post could conceivably have been included in the training data of the model, so this may not necessarily be an accurate evaluation of its semantic search capabilities (in contrast with the first example I shared, which I had mentioned once previously on Mastodon but without fully revealing the answer). Nevertheless it demonstrates that this tool is on par with question and answer sites with respect to high quality answers for at least some semantic search queries. (1/2)
