LessWrong 2.0 Reader

Scenario: you have equity worth (say) $100 million in expectation, but of no realized value at the moment.

You are forced to pay unrealized gains tax on that amount, and so are now $25 million in the hole. Even if you avoid this crashing you immediately (such as by getting a loan), if your equity goes to $0 you’re still out for the $25 million you paid, with no assets to back it.

The fact that this could be counted as a prepayment for a hypothetical later unrealized gain doesn’t help you, you can’t actually get your money back.

I guess I'm confused by the assertion that phlogiston explains things about metal properties, that isn't equally explained by "metals are calxes with the oxygen removed." Both explanations are descriptive, not predictive, and yes that remains true until we figured out quantum mechanics. Neither will tell you how a metal will behave when burned, what color flame it'll produce, why you can reduce iron ore with charcoal but not aluminum, what alloys you can make under what conditions and what their behavior will be, and so on. 

I don't disagree with "you can't explain the properties of metals based on Lavoisier's chemistry paradigm without quantum mechanics." That's just straightforwardly true. I remember very well one quantum mechanics lecture where my professor said, after about a week of derivations, "and that's why metals are shiny." What I disagree with is the assertion that phlogiston does explain this, in any sense other than just postulating the existence of a substance that tautologically, exactly matches whatever is observed in all its complexity. Understanding oxygen's role better serves to highlight where the gaps in useful understanding already were, whether or not anyone had the tools yet to fill them.

Even if we do agree to identify phlogiston with electrons, then the phlogiston theorists were still mistaken to think of it as a substance separate from the other reactants. Electrons, and free energy too, are part of the reactant and product substances in question. "Atoms" aren't actually atomic, or unbreakable. Neither side of this disagreement had that truth in its toolbox, and that truth is the central one that allows quantum mechanics to improve on what came before.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

I don't have any experience with actual situations where this could be relevant, but it does feel like you're overly focusing on the failure case where everyone is borderline incompetent and doing arbitrary things (which of course happens on less wrong sometimes, since the variation here is quite large!). There's clearly a huge upside to being able to spot when you're trying to do something that's impossible for theoretical reasons, and being extra sceptical in these situations. (E.g. someone trying to construct a perpetual motion machine). I'm open to the argument that there's a lot to be wished for in the way people in practice apply these things.

Yeah, I think a question is whether I want to say "that kind of wireheading isn't mypoic" vs "that isn't wireheading". Probably fine eitherway if you're consistent / taboo adequately.

Is there any empirical question the phlogiston theorists got right that compositional chemistry did not? AFAIK, no, but it's a real question and I'd like to know if I'm wrong here.

Although I haven't digged into the historical literature that much, I think there are two main candidates here: explaining the behavior of metals, and potential chemical energy.

On explaining the behavior of metal, this is Chang (Is Water H2O? p.43)

Phlogistonists explained the common properties of metals by saying that all metals were rich in phlogiston; this explanation was lost through the Chemical Revolution, as it does not work if we make the familiar substitution of phlogiston with the absence of oxygen (or, as Lavoisier had it, a strong affinity for oxygen). As Paul Hoyningen-Huene puts it (2008, 110): “Only after more than a 100 years could the explanatory potential of the phlogiston theory be regained in modern chemistry. One had to wait until the advent of the electron theory of metals”.

(Is Water H2O? p.21)

One salient case was the explanation of why metals (which were compounds for phlogistonists) had a set of common properties (Kuhn 1970 , 148). Actually by the onset of the Chemical Revolution this was no longer a research problem in the phlogiston paradigm, as it was accepted almost as common sense that metals had their common metallic properties (including shininess, malleability, ductility, electrical conductivity) because of the phlogiston they contained.  The oxygenist side seems to have rejected not so much this answer as the question itself; chemistry reclaimed this stretch of territory only in the twentieth century.

And on potential chemical energy, here are the quotes from Chang again

(Is Water H2O? p.46)

William Odling made the same point in a most interesting paper from 1871. Although not a household name today, Odling was one of the leading theoretical chemists of Victorian Britain, and at that time the Fullerian Professor of Chemistry at the Royal Institution. According to Odling (1871, 319), the major insight from the phlogistonists was that “combustible bodies possess in common a power or energy capable of being elicited and used”, and that “the energy pertaining to combustible bodies is the same in all of them, and capable of being transferred from the combustible body which has it to an incombustible body which has it not”. Lavoisier had got this wrong by locating the energy in the oxygen gas in the form of caloric, without a convincing account of why caloric contained in other gases would not have the ability to cause combustion.

(Is Water H2O? p.47)

Although phlogiston was clearly not exactly chemical potential energy as understood in 1871, Odling (p. 325) argued that “the phlogistians had, in their time, possession of a real truth in nature which,  altogether lost sight of in the intermediate period, has since crystallized out in a definite form.” He ended his discourse by quoting Becher: “I trust that I have got hold of my pitcher by the right handle.” And that pitcher (or Becher, cup?), the doctrine of energy, was of course “the grandest generalization in science that has ever yet been established.”

As a summary, let's quote Chang one last time. (Is Water H2O? p.47-48)

All in all, I think it is quite clear that killing phlogiston off had two adverse effects: one was to discard certain valuable scientific problems and solutions; the other was to close off certain theoretical and experimental avenues for future scientific work. Perhaps it’s all fine from where we sit, since I think the frustrated potential of the phlogistonist system was quite fully realized eventually, by some very circuitous routes. But it seems to me quite clear that the premature death of phlogiston retarded scientific progress in quite tangible ways. If it had been left to develop, I think the concept of phlogiston would have split into two. On the one hand, by the early nineteenth century someone might well have hit upon energy conservation, puzzling over this imponderable entity which seemed to have an elusive sort of reality which could be passed from one ponderable substance to another.

In that parallel universe, we would be talking about the conservation of phlogiston, and how phlogiston turned out to have all sorts of different forms, but all interconvertible with each other. This would be no more awkward than what we have in our actual universe, in which we still talk about the role of “oxygen” (acid-generator, Sauerstoff ) in supporting combustion, and the “oxidation” number of ions. On the other hand, the phlogiston concept could have led to a study of electrons without passing through such a categorical and over-simplified atomic theory as Dalton’s. Chemists might have skipped right over from phlogiston to elementary particles, or at least found an alternative path of development that did not pass through the false simplicity of the atom–molecule–bulk matter hierarchy. Keeping the phlogiston theory would have led chemists to pay more attention to the “fourth state of matter”, starting with flames, and served as a reminder that the durability of compositionist chemical building-blocks may only be an appearance. Keeping phlogiston alive could have challenged the easy Daltonian assumption that chemical atoms were physically unbreakable units. The survival of phlogiston into the nineteenth century would have sustained a vigorous alternative tradition in chemistry and physics, which would have allowed scientists to recognize with more ease the wonderful fluidity of matter, and to come to grips sooner with the nature of ions, solutions, metals, plasmas, cathode rays, and perhaps even radioactivity.

Yeah I do think it is basically a reformulation of that idea, but tailored for a different cluster of problems. (I also think Leave a Line of Retreat [LW · GW] and some other sequences posts cover similar ground).

I completely agree, and we should just obviously build an organization around this. Automating alignment research while also getting a better grasp on maximum current capabilities (and a better picture of how we expect it to grow).

(This is my intention, and I have had conversations with Bogdan about this, but I figured I'd make it more public in case anyone has funding or ideas they would like to share.)

My short answer is that the argument would consist of human values are quite simple and are most likely a reasonably natural abstraction, and the felt complexity is due to adding both the complexity of the generators and the data, which people wouldn't do for AI capabilities, meaning the bitter lesson holds for human values and morals as well.

Also, the way AI is aligned depends far more on the data that is given and our control over synthetic data means we can get AIs that follow human values before it gets too capable to take over everything, and evolutionary psychology mispredicted this and the above point pretty hard, making it lose many Bayes points compared to the Universal Learning Machine/Blank Slate hypotheses.

Alignment generalizes further than capabilities for pretty deep reasons, contra Nate Soares but basically it's way easier to have an AI care about human values than it is to get it to be capable in real-world domains, combined with verification being easier than generation.

Finally, there is evidence that AIs are far more robust to errors than people thought 15-20 years ago.

In essence, it's a negation of the following:

1. Fragility and Complexity of Value

2. Pretty much all of evolutionary psychology literature.

3. Capabilities generalizing further than alignment.

4. The Sharp Left Turn.

It might be good to have a suggestion that people can't talk if it's not their turn

I notice I haven't really offered a way of governing table noise, the contract system is too formal, so there's an incentive to shout over others to get the most negotiation bandwidth. I don't think this will result in people shouting a lot, but it may result in them failing to apprehend the incentives (ie, the game).

Maybe the rule should be, the person whose turn it is decides who can talk.

It might be good to explain why the turn timer.

Wasn't it already explained?

manual.md:

Turns should be limited to 1 minute, as everything that's tricky about real world negotiation is about the way it strains under time constraints. After 1 minute, you must carry out your choice. You don't need to be strict about it, but it is very important. Perfection isn't always attainable. In life, the *efficiency* of your negotiation process matters a whole lot, you don't just want to be able to negotiate, you want to be able to negotiate fast.

Having an appropriate tolerance for error and capacity for forgiveness also matters.

Without the 1 minute rule, most of the thinking of the game will be crammed in before the first turn, which wont leave much for the rest of the game. [edited just now:] It's easier to digest if it's spread out. If you try to do all of your decisionmaking at once, well, that's a lot of decisions!

Describing how to normalize points maybe good

This seems incompatible with relaxed power balancing requirements. Tightening power balance increases the design load... although an argument needs to be explored as to whether power balance is just too similar to balanced access to strategic depth for game design to separate them.