Typo addressed in the latest patch!

Now addressed in the latest patch!

Now addressed in the latest patch!

Now addressed in the latest patch!

Thanks for the comment!

We have indeed gotten the feedback by multiple people that this part didn't feel detailed enough (although we got this much more from very technical readers than from non-technical ones), and are working at improving the arguments.

Thanks for the comment!

We'll correct the typo in the next patch/bug fix.

As for the more direct adversarial tone of the prologue, it is an explicit choice (and is contrasted by the rest of the document). For the moment, we're waiting to get more feedback on the doc to see if it really turns people off or not.

Yep, I think you're correct.

Will correct in the next minor update. Thanks!

Thanks for the comment!

We'll consider this point for future releases, but personally, I would say that this kind of hedging also has a lot of downsides: it makes you sound far more uncertain and defensive than you really want to.

This document tries to be both grounded and to the point, and so we by default don't want to put ourselves in a defensive position when arguing things that we think make sense and are supported by the evidence.

Thanks for the comment!

We have gotten this feedback by a handful of people, so we want to reread the links and the whole literature about o1 and its evaluation to check whether we've indeed gotten the right point, or if we mischaracterized the situation.

We will probably change the phrasing (either to make our criticism clearer or to correct it) in the next minor update.

Good catch, I think we are indeed mixing the sizes here.

As you say, the point still stands, but we will change it in the next minor update to either compare the same size or make the difference in size explicit.

Thanks for the comment!

We want to check the maths, but if you're indeed correct we will update the numbers (and reasoning) in the next minor version.

I guess it depends on if you’re pivoting based on things that you’ve learned, versus grass-is-greener.

Yeah, I didn't mean "iterative thoughtful processes", I meant "compulsion that unfold at the level of days". If you arbitrarily change your job every couple of days/weeks, not based on new significant information but because you feel this other one is the one, this is bad.

So there is a vibe here that I maybe didn't convey well, about the time frame and the auto-generated part of the loops I'm pointing at: it happens often enough that your friends and family can notice, and it happens in reaction to events that no one around you agree would lead to such a drastic change (highlighting that the events are not so much the cause as the post-hoc rationalization).

Recently found a new link: Annual Reviews

It sounds like a place that centralizes many different review articles across a lot of disciplines. Only checked a few for the moment, but definitely sounds worth a try!

@Elizabeth suggested that I share here the quick tips I gave her for finding cool history and philosophy of science books, so let's do it.

• I like using awards as starting points. They're not exhaustive, but often they point to particularly good references in a field that I don't know about.
  • For philosophy of science, often with a decent dose of history, there is the Lakatos Award.
  • For history of science, there is the Sarton Medal, which is given to individuals, not works
• Same with book reviews by journals focused on the topic
  • My favorite are from the British Journal for The Philosophy of Science reviews
• Knowing the terminology helps. I find that "History and Philosophy of X" is often a good google query
  • I recently discovered https://hiphilangsci.net/ on linguistics that way!
• Obviously, follow the citations: cool books tend to reference cool books. (And terrible ones, but let's not mention that)
• Also known, but just in case: going to https://scholar.google.com/ and searching for the most cited books that cite a book you liked often leads to great reading material.

Yeah, I agree with the general point (don't have strong opinion about chaos theory at the moment).

First, negative results are really, really important. Mostly because they let you not lose your time trying to do something impossible, and sometimes they actually point you toward an answer. In general, conservation laws in physics have this role.  And knowing what is undecidable is really important in formal methods, where the trick is generally to simplify what you want or the expressive power of your programs in order to sidestep it.

Then, they are indeed quite hard to prove, at least in non-trivial cases. Conservations laws are the results of literally centuries of reframing of classical mechanics and reduction, leading to seeing the importance of energy and potential in unifying everything in physics. Undecidability is the result of 60 years of metamathetical work trying to clean formalisms enough to be able to study these kind of properties.

Is there any empirical question the phlogiston theorists got right that compositional chemistry did not? AFAIK, no, but it's a real question and I'd like to know if I'm wrong here.

Although I haven't digged into the historical literature that much, I think there are two main candidates here: explaining the behavior of metals, and potential chemical energy.

On explaining the behavior of metal, this is Chang (Is Water H2O? p.43)

Phlogistonists explained the common properties of metals by saying that all metals were rich in phlogiston; this explanation was lost through the Chemical Revolution, as it does not work if we make the familiar substitution of phlogiston with the absence of oxygen (or, as Lavoisier had it, a strong affinity for oxygen). As Paul Hoyningen-Huene puts it (2008, 110): “Only after more than a 100 years could the explanatory potential of the phlogiston theory be regained in modern chemistry. One had to wait until the advent of the electron theory of metals”.

(Is Water H2O? p.21)

One salient case was the explanation of why metals (which were compounds for phlogistonists) had a set of common properties (Kuhn 1970 , 148). Actually by the onset of the Chemical Revolution this was no longer a research problem in the phlogiston paradigm, as it was accepted almost as common sense that metals had their common metallic properties (including shininess, malleability, ductility, electrical conductivity) because of the phlogiston they contained.  The oxygenist side seems to have rejected not so much this answer as the question itself; chemistry reclaimed this stretch of territory only in the twentieth century.

And on potential chemical energy, here are the quotes from Chang again

(Is Water H2O? p.46)

William Odling made the same point in a most interesting paper from 1871. Although not a household name today, Odling was one of the leading theoretical chemists of Victorian Britain, and at that time the Fullerian Professor of Chemistry at the Royal Institution. According to Odling (1871, 319), the major insight from the phlogistonists was that “combustible bodies possess in common a power or energy capable of being elicited and used”, and that “the energy pertaining to combustible bodies is the same in all of them, and capable of being transferred from the combustible body which has it to an incombustible body which has it not”. Lavoisier had got this wrong by locating the energy in the oxygen gas in the form of caloric, without a convincing account of why caloric contained in other gases would not have the ability to cause combustion.

(Is Water H2O? p.47)

Although phlogiston was clearly not exactly chemical potential energy as understood in 1871, Odling (p. 325) argued that “the phlogistians had, in their time, possession of a real truth in nature which,  altogether lost sight of in the intermediate period, has since crystallized out in a definite form.” He ended his discourse by quoting Becher: “I trust that I have got hold of my pitcher by the right handle.” And that pitcher (or Becher, cup?), the doctrine of energy, was of course “the grandest generalization in science that has ever yet been established.”

As a summary, let's quote Chang one last time. (Is Water H2O? p.47-48)

All in all, I think it is quite clear that killing phlogiston off had two adverse effects: one was to discard certain valuable scientific problems and solutions; the other was to close off certain theoretical and experimental avenues for future scientific work. Perhaps it’s all fine from where we sit, since I think the frustrated potential of the phlogistonist system was quite fully realized eventually, by some very circuitous routes. But it seems to me quite clear that the premature death of phlogiston retarded scientific progress in quite tangible ways. If it had been left to develop, I think the concept of phlogiston would have split into two. On the one hand, by the early nineteenth century someone might well have hit upon energy conservation, puzzling over this imponderable entity which seemed to have an elusive sort of reality which could be passed from one ponderable substance to another.

In that parallel universe, we would be talking about the conservation of phlogiston, and how phlogiston turned out to have all sorts of different forms, but all interconvertible with each other. This would be no more awkward than what we have in our actual universe, in which we still talk about the role of “oxygen” (acid-generator, Sauerstoff ) in supporting combustion, and the “oxidation” number of ions. On the other hand, the phlogiston concept could have led to a study of electrons without passing through such a categorical and over-simplified atomic theory as Dalton’s. Chemists might have skipped right over from phlogiston to elementary particles, or at least found an alternative path of development that did not pass through the false simplicity of the atom–molecule–bulk matter hierarchy. Keeping the phlogiston theory would have led chemists to pay more attention to the “fourth state of matter”, starting with flames, and served as a reminder that the durability of compositionist chemical building-blocks may only be an appearance. Keeping phlogiston alive could have challenged the easy Daltonian assumption that chemical atoms were physically unbreakable units. The survival of phlogiston into the nineteenth century would have sustained a vigorous alternative tradition in chemistry and physics, which would have allowed scientists to recognize with more ease the wonderful fluidity of matter, and to come to grips sooner with the nature of ions, solutions, metals, plasmas, cathode rays, and perhaps even radioactivity.

Apparently people want some clarification on what I mean by anti-library. It's a Nassim Taleb term which refers to books you own but haven't read, whose main value is to remind you and keep in mind what you don't know and where to find it if you want to expand that knowledge.

If the point you're trying to make is: "the way we go from preparadigmatic to paradigmatic is by solving some hard problems, not by communicating initial frames and idea", I think this points to an important point indeed.

Still, two caveats:

• First, Kuhn's concept of paradigm is quite an oversimplification of what actually happens in the history of science (and the history of most fields). More recent works that go through history in much more detail realize that at any point in fields there are often many different pieces of paradigms, or some strong paradigm for a key "solved" part of the field and then a lot of debated alternative for more concrete specific details.
  • Generally, I think the discourse on history and philosophy of science on LW would improve a lot if it didn't mostly rely on one (influential) book published in the 60s, before much of the strong effort to really understand history of science and practices.
• Second, to steelman John's point, I don't think he means that you should only communicate your frame. He's the first to actively try to apply his frames to some concrete problems, and to argue for their impressiveness. Instead, I read him as pointing to a bunch of different needs in a preparadigmatic field (which maybe he could separate better ¯\_(ツ)_/¯)
  • That in a preparadigmatic field, there is no accepted way of tackling the problems/phenomena. So if you want anyone else to understand you, you need to bridge a bigger inferential distance than in a paradigmatic field (or even a partially paradigmatic field), because you don't even see the problem in the same way, at a fundamental level.
  • That if your goal is to create a paradigm, almost by definition you need to explain and communicate your paradigm. There is a part of propaganda in defending any proposed paradigm, especially when the initial frame is alien to most people, and even the impressiveness require some level of interpretation.
  • That one way (not the only way) by which a paradigm emerges is by taking different insights from different clunky frames, and unifying them (for a classic example, Newton relied on many previous basic frames, from Kepler's laws to Galileo's interpretation of force as causing acceleration). But this requires that the clunky frames are at least communicated clearly. 

Curated. I've heard this book suggested a few times over the years, and feels like it's a sort of unofficial canon among people studying how preparadigmatic science happens. This review finally compelled me to get the book. 

There's something quite funny in that I discovered this book in January 2022, during the couple of days I spent at Lightcone offices. It was in someone's office, and I was curious about it. Now, we're back full circle. ^^

I do think this review would be a lot better if it actually distilled the messy-bits-that-you-need-to-experientially-stew-over into a something that was (probably) much longer than this post, but, much shorter than the book. But that does seem legitimately hard.

Agreed.

But as I said in the post, I think it's much more important to get the feel from this book than just the big ideas. I believe that there's a way to write a really good blog post that shares that feel and compresses it, but that was not what I had the intention or energy (or mastery) to write.

It sounds cool, though also intuitively temperature seems like one of the easiest attributes to measure because literally everything is kind of a thermometer in the sense that everything equillibrates in temperature.

Can't guarantee that you would benefit from it, but this sentence makes me think you have a much cleaner and simplified idea of how one develops even simple measuring device than what the history shows (especially when you don't have any good theory of temperature or thermodynamics).

So would say you might benefit from reading it. ;)

If you enjoyed Inventing Temperature, Is Water H2O? is pretty much the same genre from the same author.

Yeah, I am a big fan of Is Water H2O? (and the other Chang books). It's just that I find Is Water H2O? both less accessible (bit more focused on theory) and more controversial (notably in its treatement of phlogiston, which I agree with, but most people including here have only heard off phlogiston from fake histories written by scientists embellishing the histories of their fields (and Lavoisierian propaganda of course)). So that's why I find Inventing Temperature easier to recommend as a first book.

My another favorite is The Emergence of Probability by Ian Hacking. It gets you feeling of how unimaginably difficult for early pioneers of probability theory to make any advance whatsoever, as well as how powerful even small advances actually are, like by enabling annuity.

It's in my anti-library, but haven't read it yet.

It is my pet peeve that people don't (maybe can't) appreciate how great intellectual achievement first order logic really is, being the end result of so much frustrating effort. Because learning to use first order logic is kind of trivial, compared to inventing it.

I haven't read it in a while, but I remember The Great Formal Machinery Works being quite good on this topic.

It's rare that books describe such processes well, I suspect partly because it's so wildly harder to generate scientific ideas than to understand them, that they tend to strike people as almost blindingly obvious in retrospect.

Completely agreed!

I think this is also what makes great history of science so hard: you need to unlearn most of the modern insights and intuitions that didn't exist at the time, and see as close as possible to what the historical actors saw.

This makes me think of a great quote from World of Flows, a history of hydrodynamics:

There is, however, a puzzling contrast between the conciseness and ease of the modern treatment of [wave equations], and the long, difficult struggles of nineteenth-century physicists with them. For example, a modern reader of Poisson's old memoir on waves finds a bewildering accumulation of complex calculations where he would expect some rather elementary analysis. The reason for this difference is not any weakness of early nineteenth-century mathematicians, but our overestimation of the physico-mathematical tools that were available in their times. It would seem, for instance, that all that Poisson needed to solve his particular wave problem was Fourier analysis, which Joseph Fourier had introduced a few years earlier. In reality, Poisson only knew a raw, algebraic version of Fourier analysis, whereas modern physicists have unconsciously assimilated a physically 'dressed' Fourier analysis, replete with metaphors and intuitions borrowed from the concrete wave phenomena of optics, acoustics, and hydrodynamics. 

(Also, thanks for the recommendations, will look at them! The response to this post makes me want to write a post about my favorite books on epistemology and science beyond Inventing Temperature ^^)

Thanks for the links!

But yeah, I'm more interested in detailed descriptions of how things actually work, rather than models of ideal governance.

Thanks!

After checking them, it feels like most of your links are focused on an economic lens to politics and governance, or at least an economic bent. Does that seem correct?

And of course just reading the rule books for the various governments or parts of the government -- for the US that would be looking at the Constitution and the rules governing internal processes for both the House and Senate. Parlimentary systems  will have similar rules of governance.

Looking at the organizational charts likely also help -- what are the committee structures and how does legislation flow through.

Yeah, ideally I would prefer to read an overview and model of these, but I agree that if it doesn't exist, then reading the docs and charts is probably the simplest alternative.

That said I'm not sure I would view political governance as truely having any gears. I think all the rules tend to become more like the Pirate's Code in Piarates of the Caribbean: more like guidelines than hard and fast rule.

I would guess that there are probably gears level model of how the governments actually work. Whether these are exactly the models provided in rules and guidelines, I'm not sure, but assuming not.

The true deep philosophical answer was... I wanted to separate cakes from bread (in french we have patisserie and boulangerie), but couldn't find any obvious one in english (seems like indeed, english-speaking countries use baking for both). So I adapted the "patisser" verb in french, hoping that I would get away with a neologism given that english is so fit for constructing them.

My bad. Thanks for the correction, edited the post.

Unfortunately all the positives of these books come paired with a critical flaw: Caro only manages to cover two people, and hasn’t even finished the second one!

In my view, Caro is actually less guilty of this than most biographers.

Fundamentally, this is because he cares much more about power, its sources, and its effects on the wielders, beneficiaries, and victims. So even though the throughline are the lives of Moses and Johnson, he spends a considerable amount of time on other topics which provide additional mechanistic models with which to understand power. 

Among others, I can think of:

• The deep model of the geology and psychology of the trap of the hill country that I mention in the post
• What is considered the best description of what it was for women especially to do all their chores by hand in the hill country before Johnson brought them electricity
• Detailed models of various forms of political campaigning, the impact of the press and 
• Detailed models of various forms of election stealing
• What is considered the best history of the senate, what it was built for, with which mechanisms, how these became perverted, and how Johnson changed it and made it work.
• Detailed model of the process that led to the civil rights movements and passage of the civil rights bills
• Detailed model of the hidden power and control of the utilities
• In general, many of Moses' schemes mentioned to force the legistlature and the mayor to give him more funding and power

He even has one chapter in the last book that is considered on par with many of the best Kennedy biographies.

Still, you do have a point that even if we extend the range beyond the two men, Caro's books are quite bound in a highly specific period: mid 20th century america.

Have you found other biographers who’ve reached a similar level? Maybe the closest I’ve found was “The Last Lion” by William Manchester, but it doesn’t really compare giving how much the author fawns over Churchill.

I think it's kind of a general consensus that finding something of a similar level is really hard. But in terms of mechanistic models, I did find Waging A Good War quite good. It explores the civil rights movement successes and failures through the lens of military theory and strategy. (It does focus on the same period and locations as the Caro books though...)

I do find thinking on paper (a bit more intentional than freewriting, but the same vibe) to be particularly helpful, I agree. Just like walks.

The reasons I don't find them enough is that:

1. They generally happen after the fact, which means that some build up happened
2. Personally, I'm rarely able to release all the build up just through thinking on paper (happens, just rare)

Still, I find it's a good way to build emotional potential energy much slower, and to notice when you really need to have a full break/sabbaticl.

Oh, I like the neural annealing connection, I have read the post but didn't relate it to emotional potential energy, but it makes sense!

Hope you take some time to anneal away some of that potential energy soon. People consistently underestimate the negative ripples on the social web from being overstretched, as opposed to the obvious and tangible "but this thing right in front of me needs doing".

Thanks. That's the plan. ;)

No worries. ;)

However, when it comes to more inchoate domains like research skill, such writing does very little to help the inexperienced researcher. It is more likely that they'd simply miss out on the point you are trying to tell them, for they haven't failed both by, say, being too trusting (a common phenomenon) and being too wary of 'trusting' (a somewhat rare phenomenon for someone who gets to the big leagues as a researcher). What would actually help is either concrete case studies, or a tight feedback loop that involves a researcher trying to do something, and perhaps failing, and getting specific feedback from an experienced researcher mentoring them. The latter has an advantage that one doesn't need to explicitly try to elicit and make clear distinctions of the skills involved, and can still learn them. The former is useful because it is scalable (you write it once, and many people can read it), and the concreteness is extremely relevant to allowing people to evaluate the abstract claims you make, and pattern match it to their own past, current, or potential future experiences.

I wholeheartedly agree.

The reason why I didn't go for this more grounded and practical and teachable approach is that at the moment, I'm optimizing for consistently writing and publishing posts.

Historically the way I fail at that is by trying too hard to write really good posts and make all the arguments super clean and concrete and detailed -- this leads to me dropping the piece after like a week of attempts.

So instead, I'm going for "write what comes naturally, edit a bit to check typos and general coherence, and publish", which leads to much more abstract pieces (because that's how I naturally think).

But reexploring this topic in an in-depth and detailed piece in the future, along the lines of what you describe, feels like an interesting challenge. Will keep it in mind. Thanks for the thoughtful comment!

Just sharing some vibe I've got from your.. framing! 
Minimalism ~ path ~ inside-focused ~ the signal/reward 
Maximalist ~ destination ~ outside-focused ~ the world

These two opposing aesthetics is a well-known confusing bit within agent foundation style research. The classical way to model an agent is to think as it is maximizing outside world variables. Conversely, we can think about minimization ~ inside-focused (reward hacking type error) as a drug addict accomplishing "nothing"

Feels there is also something to say with dopamine vs serotonine/homeostasis, even with deontology vs consequentialism, and I guess these two clumsy clusters mirrors each other in some way (feels isomorph by reverse signe function). Will rethink about it for now.

I see what you're pointing out, but in my head, the minimalism and maximalism that I've discussed both allow you quick feedback loops, which is generally the way to go for complex stuff. The tradeoff lies more in some fuzzy notion of usability:

• With the minimalism approach, you can more easily iterate in your head, but you need to do more work to lift the basic concepts to the potentially more tricky abstactions you're trying to express
• With the maximalist approach, you get affordances that are eminently practical, so that many of your needs are solved almost instantly; but you need to spend much more expertise and mental effort to simulate what's going to happen in your head during edge-cases. 

As an aside note: I'm French too, and was surprised I'm supposed to yuck maximalist aesthetic, but indeed it's consistent with my reaction reading you about TypeScript, also with my K-type brain.. Anecdotally, not with my love for spicy/rich foods ^^'

I'm obviously memeing a bit, but the real pattern I'm point out is more for "french engineering school education", which you also have, rather than mere frenchness.

Interestingly, the Lean theorem prover is sometimes considered a bit of a mess type-theoretically. (an illustrative thread), but is perhaps the most popular theorem prover among mathematicians. I would say it's more on the "maximalist" side.

Didn't know this about Lean, but the fact that a maximalist option is most popular with mathematicians makes sense to me.  As someone who worked both with mathematicians and formal methods researchers (much more meta-mathematicians), the latter are much closer to programmers, in the sense that they want to build things and solve their own abstract problems, instead of necessarily wanting the most compositional machinery possible (although I still expect compositionality to be baked into the intuitions of many mathematicians).

Last I read about Rust's type system, it basically didn't have a theoretical basis, and seemed like it was just based around Graydon figuring out algorithms for getting the properties he wanted. Rust is much more popular than SML (or Haskell, though I'm not sure Haskell should really count as a 'minimalist' type system with all of its language extensions).

Rust is an interesting point in the design space. If I had to describe it quickly according to the framing above, it feels like a really pleasant fractal tradeoff between different type systems:

• It has basically affine type but with more practical usage through borrowing (see this survey for more details)
• It has an ML type system with algebraic datatypes (and even traits which are close to typeclasses in Haskell)

So it definitely feels more maximalist than some ML or some pure linear type system, but that's more from the combination and UX work than from a crazy "let's add this super advanced feature" rush à la TypeScript imo.

It is definitely one minimalist vs maximalist dimensions ^^.

Oh, I didn't see it actually mentioned your package. 😂

Units / dimensional analysis in physics is really a kind of type system. I was very big into using that for error checking when I used to do physics and engineering calculations professionally.

Definitely!

Dimensional analysis was the first place this analogy jumped to me when reading Fly By Night Physics, because it truly used dimensions not only to check results, but also to infer the general shape of the answer (which is also something you can do in type systems, for example a function with a generic type $a\to a$ can only be populated by the identity function, because it cannot not do anything else than return its input).

Although in physics you need more tricks and feels to do it correctly. Like the derivation of the law of the pendulum just from dimensional analysis requires you to have the understanding of forces as accelerations to know that you can us g here.

Dimensions are also a perennial candidate for things that should be added to type systems, with people working quite hard at implementing it (just found this survey from 2018).

I invented my own weird way to do it that would allow them to be used in places where actual proper types & type-checking systems weren’t supported—like most numerical calculation packages, or C, or Microsoft Excel, etc.

I looked at the repo, and was quite confused how you did it, until I read

A complete set of independent base units (meters, kilograms, seconds, coulombs, kelvins) are defined as randomly-chosen positive floating-point numbers. All other units and constants are defined in terms of those. In a dimensionally-correct calculation, the units all cancel out, so the final answer is deterministic, not random. In a dimensionally-incorrect calculations, there will be random factors causing a randomly-varying final answer.

That's a really smart trick! I'm sure there are some super advanced cases where the units might cancel out wrongly, but in practice they shouldn't, and this let's you interface with all the random software that exists! (Modulo the run it twice, as you said, because the two runs correspond to two different drawings of the constants)

Yeah a case where this came up for me is angles (radians, degrees, steradians, etc.). If you treat radians as a “unit” subjected to dimensional analysis, you wind up needing to manually insert and remove the radian unit in a bunch of places, which is somewhat confusing and annoying.

Another interesting thing with radiants is that when you write a literal expression, it will look quite different than in degrees (involving many more instances of $\pi$), so inspection can fix many errors without paying the full price of different types.

Thanks for the pointer!

Oh, didn't know him!

Thanks for the links!

Thanks for the comment!

I agree with you that there are situations where the issue comes from a cultural norm rather than psychological problems. That's one reason for the last part of this post, where we point out to generally positive and productive norms that try to avoid these cultural problems and make it possible to discuss them. (One of the issue I see in my own life with cultural norms is that they are way harder to discuss when in addition psychological problems compound them and make them feel sore and emotional). But you might be right that it's worth highlighting more.

In a more meta point, my model is that we have moved from societies where almost everything is considered ''people's fault" to societies where almost everything is considered "society's fault". And it strikes me that this is an overcorrection, and that actually many issues in day to day life and groups are just people's problem (here drawing from my experience of realizing in many situations that I was the problem, and in other — less common — that the norms were the problem.)

Oh, I definitely agree, this is a really good point. What I was highlighting was an epistemic issue (namely the confusion between ideal and necessary conditions) but there is also a different decision theoretic issue that you highlighted quite well.

It's completely possible that you're not powerful enough to work outside the ideal condition. But by doing the epistemic clarification, now we can consider the explicit decision of taking step to become more powerful and being better able to manage non-ideal conditions.

Good point! The difference is that the case explained in this post is one of the most sensible version of confusing the goal and the path, since there the path is actually a really good path. On the other version (like wanting to find a simple theory simply, the path is not even a good one!

In many ways, this post is frustrating to read. It isn't straigthforward, it needlessly insults people, and it mixes irrelevant details with the key ideas.

And yet, as with many of Eliezer's post, its key points are right.

What this post does is uncover the main epistemological mistakes made by almost everyone trying their hands at figuring out timelines. Among others, there is:

• Taking arbitrary guesses within a set of options that you don't have enough evidence to separate
• Piling on arbitrary assumption on arbitraty assumption, leading to completely uninformative outputs
• Comparing biological processes to human engineering in term of speed, without noticing that the optimization path is the key variable (and the big uncertainty)
• Forcing the prediction to fit within a massively limited set of distributions, biasing it towards easy to think about distributions rather than representative ones.

Before reading this post I was already dubious of most timeline work, but this crystallized many of my objections and issues with this line of work.

So I got a lot out of this post. And I expect that many people would if they spent the time I took to analyze it in detail. But I don't expect most people to do so, and so am ambivalent on whether this post should be included in the final selection.

I was mostly thinking of the efficiency assumption underlying almost all the scenarios. Critch assumes that a significant chunk of the economy always can and does make the most efficient change (everyone replacing the job, automated regulations replacing banks when they can't move fast enough). Which neglects many potential factors, like big economic actors not having to be efficient for a long time, backlash from customers, and in general all factors making economic actors and market less than efficient.

I expect that most of these factors could be addressed with more work on the scenarios.

I consider this post as one of the most important ever written on issues of timelines and AI doom scenario. Not because it's perfect (some of its assumptions are unconvincing), but because it highlights a key aspect of AI Risk and the alignment problem which is so easy to miss coming from a rationalist mindset: it doesn't require an agent to take over the whole world. It is not about agency.

What RAAPs show instead is that even in a purely structural setting, where agency doesn't matter, these problem still crop up!

This insight was already present in Drexler's work, but however insightful Eric is in person, CAIS is completely unreadable and so no one cared. But this post is well written. Not perfectly once again, but it gives short, somewhat minimal proofs of concept for this structural perspective on alignment. And it also managed to tie alignment with key ideas in sociology, opening ways for interdisciplinarity.

I have made every person I have ever mentored on alignment study this post. And I plan to continue doing so. Despite the fact that I'm unconvinced by most timeline and AI risk scenarios post. That's how good and important it is.

I agree that a lot of science relies on predictive hallucinations. But there are examples that come to mind, notably the sort of phenomenological compression pushed by Faraday and (early) Ampère in their initial exploration of electromagnetism. What they did amounted to vary a lot of the experimental condition and relate outcomes and phenomena to each other, without directly assuming any hidden entity. (see this book for more details)

More generally, I expect most phenomenological laws to not rely heavily on predictive hallucinations, even when they integrate theoretical terms in their formulation. That's because they are mostly strong experimental regularities (like the ideal gas law or the phenomenological laws of thermodynamics) which tend to be carried to the next paradigm with radically different hidden entities.

So reification means "the act of making real" in most english dictionaries (see here for example). That's the meaning we're trying to evoke here, where the reification bias amounts to first postulate some underlying entity that explain the phenomena (that's merely a modelling technique), and second to ascribe reality to this entity and manipulate it as if it was real.

You use the analogy with sports betting multiple time in this post. But science and sports are disanalogical in almost all the relevant ways!

Notably, sports are incredibly limited and well-defined, with explicit rules that literally anyone can learn, quick feedback signals, and unambiguous results. Completely the opposite of science!

The only way I see for the analogy to hold is by defining "science" in a completely impoverished way, that puts aside most of what science actually looks like. For example, replication is not that big a part of acience, it's just the visible "clean" one. And even then, I expect the clarification of replication issues and of the original meaning to be tricky.

So my reaction to this proposal, like my reaction to any prediction market for things other than sports and games, is that I expect it to be completely irrelevant to the progress of knowledge because of the weakness of such tools. But I would definitely be curious of attempts to explicitly address all the ambiguities of epistemology and science through betting mechanisms. Maybe you know of some posts/works on that?

Agreed! That's definitely and important point, and one reason why it's still interesting to try to prove P \neq NP. The point I was making here was only that when proofs are used for the "certainty" that they give, then strong evidence from other ways is also enough to rely on the proposition.

What are you particularly interested in? I expect I could probably write it with a bit of rereading.

Hot take: I would say that most optimization failures I've observed in myself and in others (in alignment and elsewhere) boil down to psychological problems.