Request for stories of when quantitative reasoning was practically useful for you. 2019-09-13T07:21:43.686Z · score: 7 (2 votes)
What are the merits of signing up for cryonics with Alcor vs. with the Cryonics Institute? 2019-09-11T19:06:53.802Z · score: 18 (6 votes)
Does anyone know of a good overview of what humans know about depression? 2019-08-30T23:22:05.405Z · score: 14 (6 votes)
What is the state of the ego depletion field? 2019-08-09T20:30:44.798Z · score: 28 (11 votes)
Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on? 2019-07-29T22:59:33.170Z · score: 85 (27 votes)
Are there easy, low cost, ways to freeze personal cell samples for future therapies? And is this a good idea? 2019-07-09T21:57:28.537Z · score: 21 (9 votes)
Does scientific productivity correlate with IQ? 2019-06-16T19:42:29.980Z · score: 28 (9 votes)
Does the _timing_ of practice, relative to sleep, make a difference for skill consolidation? 2019-06-16T19:12:48.358Z · score: 32 (11 votes)
Eli's shortform feed 2019-06-02T09:21:32.245Z · score: 31 (6 votes)
Historical mathematicians exhibit a birth order effect too 2018-08-21T01:52:33.807Z · score: 112 (36 votes)


Comment by elityre on Eli's shortform feed · 2019-09-14T10:02:26.387Z · score: 1 (1 votes) · LW · GW


: )

Comment by elityre on Reversal Tests in Argument and Debate · 2019-09-14T09:10:45.643Z · score: 4 (3 votes) · LW · GW

This post is pointing at a good tool for identifying bias and motivated reasoning, but l don’t think that the use of “reversal test”, here aligns with how the term was coined in the original Bostrom / Ord paper ( That use of the term makes the point that if you oppose some upward change in a scaler value, and you have no reason to think that that value is already precisely optimized, then you should want to change that value in the opposite direction.

Comment by elityre on Eli's shortform feed · 2019-09-14T08:01:22.943Z · score: 13 (3 votes) · LW · GW

In this case it seems fine to add the image, but I feel disconcerted that mods have the ability to edit my posts.

I guess it makes sense that the LessWrong team would have the technical ability to do that. But editing a users post, without their specifically asking, feels like a pretty big breach of... not exactly trust, but something like that. It means I don’t have fundamental control over what is written under my name.

That is to say, I personally request that you never edit my posts, without asking (which you did, in this case) and waiting for my response. I furthermore, I think that should be a universal policy on LessWrong, though maybe this is just an idiosyncratic neurosis of mine.

Comment by elityre on Eli's shortform feed · 2019-09-14T07:50:16.204Z · score: 1 (1 votes) · LW · GW

I knew that I could, and didn’t, because it didn’t seem worth it. (Thinking that I still have to upload it to a third party photo repository and link to it. It’s easier than that now?)

Comment by elityre on Alcor vs. Cryonics Institute · 2019-09-12T10:02:44.910Z · score: 1 (1 votes) · LW · GW
CI uses a less advanced (and cheaper) cryoprotectant but cryoprotects ONLY THE HEAD, allowing the rest of the body to be straight frozen with massive damage. That’s especially odd since (many of) CI members are insistent about being whole body patients rather than neuros.

I did not know this. Thanks.

Comment by elityre on What are the merits of signing up for cryonics with Alcor vs. with the Cryonics Institute? · 2019-09-12T09:54:13.418Z · score: 10 (2 votes) · LW · GW

Good idea!

Here is a simple 1.5 question survey.

Comment by elityre on G Gordon Worley III's Shortform · 2019-09-12T09:10:13.477Z · score: 12 (4 votes) · LW · GW

What's the difference?

Suppose I'm talking with a group of loose acquaintances, and one of them says (in full seriousness), "I'm not homophobic. It's not that I'm afraid of gays, I just think that they shouldn't exist."

It seem to me that it is appropriate for me to say, "Hey man, that's not ok to say." It might be that a number of other people in the conversation would back me up (or it might be that they they defend the first guy), but there wasn't common knowledge of that fact beforehand.

In some sense, this is a bid to establish a new norm, by pushing a the private opinions of a number of people into common knowledge. It also seems to me to be a virtuous thing to do in make situations.

(Noting that my response to the guy is not: "Hey, you can't do that, because I get to decide what people do around here." It's "You can't do that, because it's bad" and depending on the group to respond to that claim in one way or another.)

Comment by elityre on Eli's shortform feed · 2019-09-12T08:44:03.142Z · score: 6 (4 votes) · LW · GW

New (image) post: My strategic picture of the work that needs to be done

Comment by elityre on Hindsight Devalues Science · 2019-09-10T18:49:16.047Z · score: 3 (2 votes) · LW · GW

In which case, TraderJoe and Rixie, good job at being appropriately confused!

Comment by elityre on Eli's shortform feed · 2019-09-10T14:40:37.110Z · score: 3 (2 votes) · LW · GW

I very much agree.

Comment by elityre on Eli's shortform feed · 2019-09-10T14:33:35.013Z · score: 1 (1 votes) · LW · GW

This seems that it might be testable. If you force impulsive folk to wait and think, do they generate more ideas for how to proceed?

Comment by elityre on What happened to Leverage Research? · 2019-09-10T14:32:09.733Z · score: 1 (1 votes) · LW · GW

I belive there are 4 members of Leverage (including Geoff), and something like 7 members of Paradigm (including Geoff). Paradigm and Leverage are somewhat more distinct now, than they were over previous years, but are both still headed by Geoff, unlike the other groups, which are meaningfully spun-off.

Comment by elityre on Eli's shortform feed · 2019-09-09T04:30:20.271Z · score: 2 (2 votes) · LW · GW

I think that's what the book referenced here, is about.

Comment by elityre on Eli's shortform feed · 2019-09-07T06:34:39.335Z · score: 8 (4 votes) · LW · GW

New post: Capability testing as a pseudo fire alarm

[epistemic status: a thought I had]

It seems like it would be useful to have very fine-grained measures of how smart / capable a general reasoner is, because this would allow an AGI project to carefully avoid creating a system smart enough to pose an existential risk.

I’m imagining slowly feeding a system more training data (or, alternatively, iteratively training a system with slightly more compute), and regularly checking its capability. When the system reaches “chimpanzee level” (whatever that means), you stop training it (or giving it more compute resources).

This might even be a kind of fire-alarm. If you have a known predetermined battery of tests, then when some lab develops a system that scores “at the chimp level” at that battery, that might be a signal to everyone, that it’s time to pool our resources and figure out safety. (Of course, this event might alternatively precipitate a race, as everyone tries to get to human-level first.)

Probably the best way to do this would be for both training data, and compute / architecture. Start with a given architecture, then train it, slowly increasing the amount or quality of the training data, with regular tests (done on “spurs”, the agent should never have episodic memory of the . When increasing training data plateaus, iteratively improve the architecture in some way, either by giving the systems it more compute resources, or maybe making small adjustment. Again train the new version of the system, with regular tests. If you ever start to get very steep improvement, slow down and run tests more frequently.

Naively, it seems like a setup like this would prevent an AI team from overshooting and making a system that is much more capable that they think (which gives rise to all kinds of problems, like treacherous turns), regardless of how close “chimp” is to “human” on some absolute intelligence scale.

Some considerations:

  • This of course, depends on having metrics that work. Of course, it seems very likely that early general AIs will be minds with a very unusual “shape”, such that it has sophisticated reasoning abilities along dimensions that are not natural categories to humans, while being weak on many of the dimensions we’re testing for.
    • Although, this seems a bit surprising, since presumably we would be training it to hit those metrics? Somehow it would get better, faster, at a kind of reasoning we were not intending to train, than at the kind of reasoning that we were optimizing for?
    • Also, can we just design metrics that get around this problem, by measuring exactly what we care about, i.e. danger? Something like a metric for consequentialist reasoning?
  • This idea depends on the system not deceiving you, or intentionally underperforming on your metrics.
    • Naively, this doesn’t seem like that much of an issue, if you are doing tests regularly enough. I’m presuming (perhaps wrongly), that a chimpanzee is not smart enough to infer that it is an agent in a simulation that is overseen by human agents, and that it should try to decide those humans. That is, we stop our progression before the agent is at the point where deception is a consideration.
      • Which means we really need to be confident about how smart something needs to be before it tries to deceive us?
        • Really? Its first attempts at tricking humans will be pretty transparent. Just as the attempts of animals / babies to trick us are pretty transparent.
  • At least one “danger point”, is when the system is capable enough to realize the instrumental value of self improving by seizing more resources.
    • How smart is this?
      • My guess, is really smart. Animals come pre-loaded with all kinds of instincts that cause them to seek out food, water, etc. These AI systems would not have an instinct to seek more training data / computation. Most humans don’t reason their way into finding ways to improve their own reasoning. If there was a chimp, even loose in the internet (whatever that means), would it figure out to make itself smarter?
      • If the agent has experienced (and has memories of) rounds of getting smarter, as the humans give it more resources, and can identify that these improvements allow it to get more of what it wants, it might instrumentally reason that it should figure out how to get more compute / training data. But it seems easy to have a setup such that no system has episodic memories previous improvement rounds.
        • [Note: This makes a lot less sense for an agent of the active inference paradigm]
          • Could I salvage it somehow? Maybe by making some kind of principled distinction between learning in the sense of “getting better at reasoning” (procedural), and learning in the sense of “acquiring information about the environment” (episodic).
Comment by elityre on Eli's shortform feed · 2019-09-04T07:21:07.051Z · score: 11 (7 votes) · LW · GW

[Real short post. Random. Complete speculation.]

Childhood lead exposure reduces one’s IQ, and also causes one to be more impulsive and aggressive.

I always assumed that the impulsiveness was due, basically, to your executive function machinery working less well. So you have less self control.

But maybe the reason for the IQ-impulsiveness connection, is that if you have a lower IQ, all of your subagents/ subprocesses are less smart. Because they’re worse at planning and modeling the world, the only way they know how to get their needs met are very direct, very simple, action-plans/ strategies. It’s not so much that you’re better at controlling your anger, as the part of you that would be angry is less so, because it has other ways of getting its needs met.

Comment by elityre on What is the state of the ego depletion field? · 2019-09-02T05:49:08.446Z · score: 1 (1 votes) · LW · GW

This paper seems at least a little relevant.

Abstract: The brain’s reliance on glucose as a primary fuel source is well established, but psychological models of cognitive processing that take energy supply into account remain uncommon. One exception is research on self-control depletion, where debate continues over a limited-resource model. This model argues that a transient reduction in self-control after the exertion of prior self-control is caused by the depletion of brain glucose, and that self-control processes are special, perhaps unique, in this regard. This model has been argued to be physiologically implausible in several recent reviews. This paper attempts to correct some inaccuracies that have occurred during debate over the physiological plausibility of this model. We contend that not only is such limitation of cognition by constraints on glucose supply plausible, it is well established in the neuroscience literature across several cognitive domains. Conversely, we argue that there is no evidence that self-control is special in regard to its metabolic cost. Mental processes require physical energy, and the body is limited in its ability to supply the brain with sufficient energy to fuel mental processes. This article reviews current findings in brain metabolism and seeks to resolve the current conflict in the field regarding the physiological plausibility of the self-control glucose-depletion hypothesis.

Comment by elityre on Why are the people who should be doing safety research, but aren’t, doing something else? · 2019-09-02T03:42:10.225Z · score: 8 (5 votes) · LW · GW

I've been told, by people much smarter than me, and more connected to even smarter people, that the very elite, in terms of IQ have a sense of learned helplessness about the world.

According to this story, the smartest people in the world look around, and see stupidity all around them: the world is populated by, controlled by, such people who regularly make senseless decisions, and can't even tell that they're senseless. And it is obvious that trying to get people to understand is hopeless: aside from the fact that most of them basically can't understand, you are small, and the world is huge.

So these people go and do math, and make a good life for themselves, and don't worry about the world.

[I don't know if this story is true.]

Comment by elityre on Eli's shortform feed · 2019-09-01T04:57:12.787Z · score: 3 (2 votes) · LW · GW

Yeah. I think you're on to something here. My current read is that "mental energy" is at least 3 things.

Can you elaborate on the what "knowledge saturation" feels like for you?

Comment by elityre on What is the state of the ego depletion field? · 2019-08-30T01:43:56.290Z · score: 1 (1 votes) · LW · GW

I definitely consider this a relevant answer.

Comment by elityre on The Unfinished Mystery of the Shangri-La Diet · 2019-08-28T06:01:57.427Z · score: 3 (2 votes) · LW · GW

The "Only vodka" diet.


Comment by elityre on The Valley of Bad Theory · 2019-08-27T18:31:05.841Z · score: 1 (1 votes) · LW · GW

Well, mostly it promoted the question of if I should be running my life drastically differently.

Comment by elityre on The Valley of Bad Theory · 2019-08-25T21:35:41.571Z · score: 3 (2 votes) · LW · GW

Thank you for writing this. Since most of my effort involves developing and iterating on ad-hoc incomplete theories, this is extremely relevant.

Comment by elityre on Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on? · 2019-08-24T23:44:04.255Z · score: 10 (3 votes) · LW · GW

This is a really good example of a possible cultural/technological change that would alter the coordination landscape substantially. Thanks.

Comment by elityre on Eli's shortform feed · 2019-08-24T07:22:18.897Z · score: 3 (3 votes) · LW · GW

Can you say more about what you got from it?

Comment by elityre on Eli's shortform feed · 2019-08-24T02:44:31.028Z · score: 44 (13 votes) · LW · GW

Old post: RAND needed the "say oops" skill

[Epistemic status: a middling argument]

A few months ago, I wrote about how RAND, and the “Defense Intellectuals” of the cold war represent another precious datapoint of “very smart people, trying to prevent the destruction of the world, in a civilization that they acknowledge to be inadequate to dealing sanely with x-risk.”

Since then I spent some time doing additional research into what cognitive errors and mistakes  those consultants, military officials, and politicians made that endangered the world. The idea being that if we could diagnose which specific irrationalities they were subject to, that this would suggest errors that might also be relevant to contemporary x-risk mitigators, and might point out some specific areas where development of rationality training is needed.

However, this proved somewhat less fruitful than I was hoping, and I’ve put it aside for the time being. I might come back to it in the coming months.

It does seem worth sharing at least one relevant anecdote, from Daniel Ellsberg’s excellent book, the Doomsday Machine, and analysis, given that I’ve already written it up.

The missile gap

In the late nineteen-fifties it was widely understood that there was a “missile gap”: that the soviets had many more ICBM (“intercontinental ballistic missiles” armed with nuclear warheads) than the US.

Estimates varied widely on how many missiles the soviets had. The Army and the Navy gave estimates of about 40 missiles, which was about at parity with the the US’s strategic nuclear force. The Air Force and the Strategic Air Command, in contrast, gave estimates of as many as 1000 soviet missiles, 20 times more than the US’s count.

(The Air Force and SAC were incentivized to inflate their estimates of the Russian nuclear arsenal, because a large missile gap strongly necessitated the creation of more nuclear weapons, which would be under SAC control and entail increases in the Air Force budget. Similarly, the Army and Navy were incentivized to lowball their estimates, because a comparatively weaker soviet nuclear force made conventional military forces more relevant and implied allocating budget-resources to the Army and Navy.)

So there was some dispute about the size of the missile gap, including an unlikely possibility of nuclear parity with the Soviet Union. Nevertheless, the Soviet’s nuclear superiority was the basis for all planning and diplomacy at the time.

Kennedy campaigned on the basis of correcting the missile gap. Perhaps more critically, all of RAND’s planning and analysis was concerned with the possibility of the Russians launching a nearly-or-actually debilitating first or second strike.

The revelation

In 1961 it came to light, on the basis of new satellite photos, that all of these estimates were dead wrong. It turned out the the Soviets had only 4 nuclear ICBMs, one tenth as many as the US controlled.

The importance of this development should be emphasized. It meant that several of the fundamental assumptions of US nuclear planners were in error.

First of all, it meant that the Soviets were not bent on world domination (as had been assumed). Ellsberg says…

Since it seemed clear that the Soviets could have produced and deployed many, many more missiles in the three years since their first ICBM test, it put in question—it virtually demolished—the fundamental premise that the Soviets were pursuing a program of world conquest like Hitler’s.

That pursuit of world domination would have given them an enormous incentive to acquire at the earliest possible moment the capability to disarm their chief obstacle to this aim, the United States and its SAC. [That] assumption of Soviet aims was shared, as far as I knew, by all my RAND colleagues and with everyone I’d encountered in the Pentagon:
The Assistant Chief of Staff, Intelligence, USAF, believes that Soviet determination to achieve world domination has fostered recognition of the fact that the ultimate elimination of the US, as the chief obstacle to the achievement of their objective, cannot be accomplished without a clear preponderance of military capability.
If that was their intention, they really would have had to seek this capability before 1963. The 1959–62 period was their only opportunity to have such a disarming capability with missiles, either for blackmail purposes or an actual attack. After that, we were programmed to have increasing numbers of Atlas and Minuteman missiles in hard silos and Polaris sub-launched missiles. Even moderate confidence of disarming us so thoroughly as to escape catastrophic damage from our response would elude them indefinitely.
Four missiles in 1960–61 was strategically equivalent to zero, in terms of such an aim.

This revelation about soviet goals was not only of obvious strategic importance, it also took the wind out of the ideological motivation for this sort of nuclear planning. As Ellsberg relays early in his book, many, if not most, RAND employees were explicitly attempting to defend US and the world from what was presumed to be an aggressive communist state, bent on conquest. This just wasn’t true.

But it had even more practical consequences: this revelation meant that the Russians had no first strike (or for that matter, second strike) capability. They could launch their ICBMs at American cities or military bases, but such an attack had no chance of debilitating US second strike capacity. It would unquestionably trigger a nuclear counterattack from the US who, with their 40 missiles, would be able to utterly annihilate the Soviet Union. The only effect of a Russian nuclear attack would be to doom their own country.

[Eli’s research note: What about all the Russian planes and bombs? ICBMs aren’t the the only way of attacking the US, right?]

This means that the primary consideration in US nuclear war planning at RAND and elsewhere, was fallacious. The Soviet’s could not meaningfully destroy the US.

…the estimate contradicted and essentially invalidated the key RAND studies on SAC vulnerability since 1956. Those studies had explicitly assumed a range of uncertainty about the size of the Soviet ICBM force that might play a crucial role in combination with bomber attacks. Ever since the term “missile gap” had come into widespread use after 1957, Albert Wohlstetter had deprecated that description of his key findings. He emphasized that those were premised on the possibility of clever Soviet bomber and sub-launched attacks in combination with missiles or, earlier, even without them. He preferred the term “deterrent gap.” But there was no deterrent gap either. Never had been, never would be.
To recognize that was to face the conclusion that RAND had, in all good faith, been working obsessively and with a sense of frantic urgency on a wrong set of problems, an irrelevant pursuit in respect to national security.

This realization invalidated virtually all of RAND’s work to date. Virtually every, analysis, study, and strategy, had been useless, at best.

The reaction to the revelation

How did RAND employees respond to this reveal, that their work had been completely off base?

That is not a recognition that most humans in an institution are quick to accept. It was to take months, if not years, for RAND to accept it, if it ever did in those terms. To some degree, it’s my impression that it never recovered its former prestige or sense of mission, though both its building and its budget eventually became much larger. For some time most of my former colleagues continued their focus on the vulnerability of SAC, much the same as before, while questioning the reliability of the new estimate and its relevance to the years ahead. [Emphasis mine]

For years the specter of a “missile gap” had been haunting my colleagues at RAND and in the Defense Department. The revelation that this had been illusory cast a new perspective on everything. It might have occasioned a complete reassessment of our own plans for a massive buildup of strategic weapons, thus averting an otherwise inevitable and disastrous arms race. It did not; no one known to me considered that for a moment. [Emphasis mine]

According to Ellsberg, many at RAND were unable to adapt to the new reality and continued (fruitlessly) to continue with what they were doing, as if by inertia, when the thing that they needed to do (to use Eliezer’s turn of phrase) is “halt, melt, and catch fire.”

This suggests that one failure of this ecosystem, that was working in the domain of existential risk, was a failure to “say oops“: to notice a mistaken belief, concretely acknowledge that is was mistaken, and to reconstruct one’s plans and world views.

Relevance to people working on AI safety

This seems to be at least some evidence (though, only weak evidence, I think), that we should be cautious of this particular cognitive failure ourselves.

It may be worth rehearsing the motion in advance: how will you respond, when you discover that a foundational crux of your planning is actually mirage, and the world is actually different than it seems?

What if you discovered that your overall approach to making the world better was badly mistaken?

What if you received a strong argument against the orthogonality thesis?

What about a strong argument for negative utilitarianism?

I think that many of the people around me have effectively absorbed the impact of a major update at least once in their life, on a variety of issues (religion, x-risk, average vs. total utilitarianism, etc), so I’m not that worried about us. But it seems worth pointing out the importance of this error mode.

A note: Ellsberg relays later in the book that, durring the Cuban missile crisis, he perceived Kennedy as offering baffling terms to the soviets: terms that didn’t make sense in light of the actual strategic situation, but might have been sensible under the premiss of a soviet missile gap. Ellsberg wondered, at the time, if Kennedy had also failed to propagate the update regarding the actual strategic situation.

I believed it very unlikely that the Soviets would risk hitting our missiles in Turkey even if we attacked theirs in Cuba. We couldn’t understand why Kennedy thought otherwise. Why did he seem sure that the Soviets would respond to an attack on their missiles in Cuba by armed moves against Turkey or Berlin? We wondered if—after his campaigning in 1960 against a supposed “missile gap”—Kennedy had never really absorbed what the strategic balance actually was, or its implications.

I mention this because additional research suggests that this is implausible: that Kennedy and his staff were aware of the true strategic situation, and that their planning was based on that premise.

Comment by elityre on Two senses of “optimizer” · 2019-08-22T17:59:32.541Z · score: 1 (1 votes) · LW · GW
The implicit assumption seems to be that an optimizer_1 could turn into an optimizer_2 unexpectedly if it becomes sufficiently powerful.

I think the assumption is that a sufficiently capable optimizer_1 would need to be an optimizer_2.

Comment by elityre on Eli's shortform feed · 2019-08-19T22:36:06.928Z · score: 16 (6 votes) · LW · GW

Old post: A mechanistic description of status

[This is an essay that I’ve had bopping around in my head for a long time. I’m not sure if this says anything usefully new-but it might click with some folks. If you haven’t read Social Status: Down the Rabbit Hole on Kevin Simler’s excellent blog, Melting Asphalt read that first. I think this is pretty bad and needs to be rewritten and maybe expanded substantially, but this blog is called “musings and rough drafts.”]

In this post, I’m going to outline how I think about status. In particular, I want to give a mechanistic account of how status necessarily arises, given some set of axioms, in much the same way one can show that evolution by natural selection must necessarily occur given the axioms of 1) inheritance of traits 2) variance in reproductive success based on variance in traits and 3) mutation.

(I am not claiming any particular skill at navigating status relationships, any more than a student of sports-biology is necessarily a skilled basketball player.)

By “status” I mean prestige-status.

Axiom 1: People have goals.

That is, for any given human, there are some things that they want. This can include just about anything. You might want more money, more sex, a ninja-turtles lunchbox, a new car, to have interesting conversations, to become an expert tennis player, to move to New York etc.

Axiom 2: There are people who control resources relevant to other people achieving their goals.

The kinds of resources are as varied as the goals one can have.

Thinking about status dynamics and the like, people often focus on the particularly convergent resources, like money. But resources that are onlyrelevant to a specific goal are just as much a part of the dynamics I’m about to describe.

Knowing a bunch about late 16th century Swedish architecture is controlling a goal relevant-resource, if someone has the goal of learning more about 16th century Swedish architecture.

Just being a fun person to spend time with (due to being particularly attractive, or funny, or interesting to talk to, or whatever) is a resource relevant to other people’s goals.

Axiom 3: People are more willing to help (offer favors to) a person who can help them achieve their goals.

Simply stated, you’re apt to offer to help a person with their goals if it seems like they can help you with yours, because you hope they’ll reciprocate. You’re willing to make a trade with, or ally with such people, because it seems likely to be beneficial to you. At minimum, you don’t want to get on their bad side.

(Notably, there are two factors that go into one’s assessment of another person’s usefulness: if they control a resource relevant to one of your goals, and if you expect them to reciprocate.

This produces a dynamic where by A’s willingness to ally with B is determined by something like the product of

  • A’s assessment of B’s power (as relevant to A’s goals), and
  • A’s assessment of B’s probability of helping (which might translate into integrity, niceness, etc.)

If a person is a jerk, they need to be very powerful-relative-to-your-goals to make allying with them worthwhile.)

All of this seems good so far, but notice that we have up to this point only described individual pair-wise transactions and pair-wise relationships. People speak about “status” as a attribute that someone can possess or lack. How does the dynamic of a person being “high status” arise from the flux of individual transactions?

Lemma 1: One of the resources that a person can control is other people’s willingness to offer them favors

With this lemma, the system folds in on itself, and the individual transactions cohere into a mostly-stable status hierarchy.

Given lemma 1, a person doesn’t need to personally control resources relevant to your goals, they just need to be in a position such that someone who is relevant to your goals will privilege them.

As an example, suppose that you’re introduced to someone who is very well respected in your local social group: person-W. Your assessment might be that W, directly, doesn’t have anything that you need. But because person-W is well-respected by others in your social group are likely to offer favors to him/her. Therefore, it’s useful for person-W to like you, because then they are more apt to call on other people’s favors on your behalf.

(All the usual caveats about has this is subconscious, and humans are adaption-executors and don’t do explicit, verbal assessments of how useful a person will be to them, but rely on emotional heuristics that approximate explicit assessment.)

This causes the mess of status transactions to reinforce and stabilize into a mostly-static hierarchy. The mass of individual A-privileges-B-on-the-basis-of-A’s-goals flattens out, into each person having a single “score” which determines to what degree each other person privileges them.

(It’s a little more complicated than that because people who have access to their own resources have less need of help from other. So a person’s effective status (the status-level at which you treat them is closer to their status minus your status. But this is complicated again because people are motivated not to be dicks (that’s bad for business), and respecting other people’s status is important to not being a dick.)

[more stuff here.]

Comment by elityre on What is the state of the ego depletion field? · 2019-08-17T06:13:49.544Z · score: 3 (2 votes) · LW · GW

For those of you following along at home, this post has a section "What Ego-Depletion Studies Are Most Likely to Replicate?"

Comment by elityre on Eli's shortform feed · 2019-08-13T18:02:40.236Z · score: 38 (16 votes) · LW · GW

New post: What is mental energy?

[Note: I’ve started a research side project on this question, and it is already obvious to me that this ontology importantly wrong.]

There’s a common phenomenology of “mental energy”. For instance, if I spend a couple of hours thinking hard (maybe doing math), I find it harder to do more mental work afterwards. My thinking may be slower and less productive. And I feel tired, or drained, (mentally, instead of physically).

Mental energy is one of the primary resources that one has to allocate, in doing productive work. In almost all cases, humans have less mental energy than they have time, and therefore effective productivity is a matter of energy management, more than time management. If we want to maximize personal effectiveness, mental energy seems like an extremely important domain to understand. So what is it?

The naive story is that mental energy is an actual energy resource that one expends and then needs to recoup. That is, when one is doing cognitive work, they are burning calories, depleting their bodies energy stores. As they use energy, they have less fuel to burn.

My current understanding is that this story is not physiologically realistic. Thinking hard does consume more of the body’s energy than baseline, but not that much more. And we experience mental fatigue long before we even get close to depleting our calorie stores. It isn’t literal energy that is being consumed. [The Psychology of Fatigue pg.27]

So if not that, what is going on here?

A few hypotheses:

(The first few, are all of a cluster, so I labeled them 1a, 1b, 1c, etc.)

Hypothesis 1a: Mental fatigue is a natural control system that redirects our attention to our other goals.

The explanation that I’ve heard most frequently in recent years (since it became obvious that much of the literature on ego-depletion was off the mark), is the following:

A human mind is composed of a bunch of subsystems that are all pushing for different goals. For a period of time, one of these goal threads might be dominant. For instance, if I spend a few hours doing math, this means that my other goals are temporarily suppressed or on hold: I’m not spending that time seeking a mate, or practicing the piano, or hanging out with friends.

In order to prevent those goals from being neglected entirely, your mind has a natural control system that prevents you from focusing your attention on any one thing at a time: the longer you put your attention on something, the greater the build up of mental fatigue, causing you to do anything else.

Comments and model-predictions: This hypothesis, as stated, seems implausible to me. For one thing, it seems to suggest that that all actives would be equally mentally taxing, which is empirically false: spending several hours doing math is mentally fatiguing, but spending the same amount of time watching TV is not.

This might still be salvaged if we offer some currency other than energy that is being preserved: something like “forceful computations”. But again, it doesn’t seem obvious why the computations of doing math would be more costly than those for watching TV.

Similarly, this model suggests that “a change is as good as a break”: if you switch to a new task, you should be back to full mental energy, until you become fatigued for that task as well.

Hypothesis 1b: Mental fatigue is the phenomenological representation of the loss of support for the winning coalition.

A variation on this hypothesis would be to model the mind as a collection of subsystems. At any given time, there is only one action sequence active, but that action sequence is determined by continuous “voting” by various subsystems.

Overtime, these subsystems get fed up with their goals not being met, and “withdraw support” for the current activity. This manifests as increasing mental fatigue. (Perhaps your thoughts get progressively less effective, because they are interrupted, on the scale of micro-seconds, by bids to think something else).

Comments and model-predictions: This seems like it might suggest that if all of the subsystems have high trust that their goals will be met, that math (or any other cognitively demanding task) would cease to be mentally taxing. Is that the case? (Does doing math mentally exhaust Critch?)

This does have the nice virtue of explaining burnout: when some subset of needs are not satisfied for a long period, the relevant subsystems pull their support for all actions, until those needs are met.

[Is burnout a good paradigm case for studying mental energy in general?]

Hypothesis 1c: The same as 1a or 1b, but some mental operations are painful for some reason.

To answer my question above, one reason why math might be more mentally taxing than watching TV, is that doing math is painful.

If the process of doing math is painful on the micro-level, then even if all of the other needs are met, there is still a fundamental conflict between the subsystem that is aiming to acquire math knowledge, and the subsystem that is trying to avoid micro-pain on the micro-level.

As you keep doing math, the micro pain part votes more and more strongly against doing math, or the overall system biases away from the current activity, and you run out of mental energy.

Comments and model-predictions: This seems plausible for the activity of doing math, which involves many moments of frustration, which might be meaningfully micro-painful. But it seems less consistent with activities like writing, which phenomenologically feel non-painful. This leads to hypothesis 1d…

Hypothesis 1d: The same as 1c, but the key micro-pain is that of processing ambiguity second to second

Maybe the pain comes from many moments of processing ambiguity, which is definitely a thing that is happening in the context of writing. (I’ll sometimes notice myself try to flinch to something easier when I’m not sure which sentence to write.) It seems plausible that mentally taxing activities are taxing to the extent that they involve processing ambiguity, and doing a search for the best template to apply.

Hypothesis 1e: Mental fatigue is the penalty incurred for top down direction of attention.

Maybe consciously deciding to do things is importantly different from the “natural” allocation of cognitive resources. That is, your mind is set up such that the conscious, System 2, long term planning, metacognitive system, doesn’t have free rein. It has a limited budget of “mental energy”, which measures how long it is allowed to call the shots before the visceral, system 1, immediate gratification systems take over again.

Maybe this is an evolutionary adaption? For the monkeys that had “really good” plans for how to achieve their goals, never panned out for them. The monkeys that were impulsive some of the time, actually did better at the reproduction game?

(If this is the case, can the rest of the mind learn to trust S2 more, and thereby offer it a bigger mental energy budget?)

This hypothesis does seem consistent with my observation that rest days are rejuvenating, even when I spend my rest day working on cognitively demanding side projects.

Hypothesis 2: Mental fatigue is the result of the brain temporarily reaching knowledge saturation.

When learning a motor task, there are several phases in which skill improvement occurs. The first, unsurprisingly, is durring practice sessions. However, one also sees automatic improvements in skill in the hours after practice [actually this part is disputed] and following a sleep period (academic link1, 2, 3). That is, there is a period of consolidation following a practice session. This period of consolidation probably involves the literal strengthening of neural connections, and encoding other brain patterns that take more than a few seconds to set.

I speculate, that your brain may reach a saturation point: more practice, more information input, becomes increasingly less effective, because you need to dedicate cognitive resources to consolidation. [Note that this is supposing that there is some tradeoff between consolidation activity and input activity, as opposed to a setup where both can occur simultaneously (does anyone have evidence for such a tradeoff?)].

If so, maybe cognitive fatigue is the phenomenology of needing to extract one’s self from a practice / execution regime, so that your brain can do post-processing and consolidation on what you’ve already done and learned.

Comments and model-predictions: This seems to suggest that all cognitively taxing tasks are learning tasks, or at least tasks in which one is encoding new neural patterns. This seems plausible, at least.

It also seems to naively imply that an activity will become less mentally taxing as you gain expertise with it, and progress along the learning curve. There is (presumably) much more information to process and consolidate in your first hour of doing math than in your 500th.

Hypothesis 3: Mental fatigue is a control system that prevents some kind of damage to the mind or body.

One reason why physical fatigue is useful is that it prevents damage to your body. Getting tired after running for a bit, stops you for running all out for 30 hours at a time, and eroding your fascia.

By simple analogy to physical fatigue, we might guess that mental fatigue is a response to vigorous mental activity that is adaptive in that it prevents us from hurting ourselves.

I have no idea what kind of damage might be caused by thinking too hard.

I note that mania and hypomania involve apparently limitless mental energy reserves, and I think that theses states are bad for your brain.

Hypothesis 4: Mental fatigue is a buffer overflow of peripheral awareness.

Another speculative hypothesis: Human minds have a working memory: a limit of ~4 concepts, or chunks, that can be “activated”, or operated upon in focal attention, at one time. But meditators, at least, also talk a peripheral awareness: a sort of halo of concepts and sense impressions that are “loaded up”, or “near by”, or cognitively available, or “on the fringes of awareness”. These are all the ideas that are “at hand” to your thinking. [Note: is peripheral awareness, as the meditators talk about,  the same thing as “short term memory”?]

Perhaps if there is a functional limit to the amount of content that can be held in working memory, there is a similar, if larger, limit to how much content can be held in peripheral awareness. As you engage with a task, more and more mental content is loaded up, or added to peripheral awareness, where it both influences your focal thought process, and/or is available to be operated on directly in working memory. As you continue the task, and more and more content gets added to peripheral awareness, you begin to overflow its capacity. It gets harder and harder to think, because peripheral awareness is overflowing. Your mind needs space to re-ontologize: to chunk pieces together, so that it can all fit in the same mental space. Perhaps this is what mental fatigue is.

Comments and model-predictions: This does give a nice clear account of why sleep replenishes mental energy (it both causes re-ontologizing, and clears the cache), though perhaps this does not provide evidence over most of the other hypotheses listed here.

Other notes about mental energy:

  • In this post, I’m mostly talking about mental energy on the scale of hours. But there is also a similar phenomenon on the scale of days (the rejuvenation one feels after rest days) and on the scale of months (burnout and such). Are these the same basic phenomenon on different timescales?
  • On the scale of days, I find that my subjective rest-o-meter is charged up if I take a rest day, even if I spend that rest day working on fairly cognitively intensive side projects.
    • This might be because there’s a kind of new project energy, or new project optimism?
  • Mania and hypomania entail limitless mental energy.
  • People seem to be able to play video games for hours and hours without depleting mental energy. Does this include problem solving games, or puzzle games?
    • Also, just because they can play indefinitely does not mean that their performance doesn’t drop. Does performance drop, across hours of playing, say, snakebird?
  • For that matter, does performance decline on a task correlate with the phenomenological “running out of energy”? Maybe those are separate systems.
Comment by elityre on Could we solve this email mess if we all moved to paid emails? · 2019-08-13T03:24:29.507Z · score: 14 (7 votes) · LW · GW

The supermajority of the people that I interact with, in person and online, are people who were influenced by that book, and like me, make substantial life decisions on the bases of associated arguments. Many of them likewise interact largely with other people who were influenced by that book.

Even stronger than that, people of this category are densely socially connected. The fact that someone identifies as "a rationalist", is pretty strong evidence that I know them, or know of them. This is in contrast with "entrepreneurs", for instance. Even the most well-connected entrepreneurs don't know most of the people who identify as entrepreneurs. Dito for "people with IQs over 120", and biologists.

Why wouldn't I draw a boundary around that cluster of people, and attempt interventions on that cluster in particular?

It seems to me that the "rationality community" is both a natural category, and a useful category.

But perhaps you're claiming that I should use this category, but I shouldn't give it the label of "rationality", because then I'm I'm making the connotation (to myself) that this group is unusually rational?

Comment by elityre on Benito's Shortform Feed · 2019-08-08T05:44:04.828Z · score: 3 (2 votes) · LW · GW

It seems that AI safety has this issue less than every other problem in the world, by proportion of the people working on it.

Some double digit percentage of all of the people who are trying to improve the situation, are directly trying to solve the problem, I think? (Or maybe I just live in a bubble in a bubble.)

And I don’t know how well this analysis applies to non-AI safety fields.

Comment by elityre on Eli's shortform feed · 2019-08-07T19:19:27.478Z · score: 1 (1 votes) · LW · GW

Here you go.

New post: Napping Protocol

Comment by elityre on Eli's shortform feed · 2019-08-07T19:05:45.324Z · score: 1 (1 votes) · LW · GW

Older post: Initial Comparison between RAND and the Rationality Cluster

Comment by elityre on Benito's Shortform Feed · 2019-08-07T18:43:02.625Z · score: 1 (1 votes) · LW · GW

Do you have an example of a problem that gets approached this way?

Global warming? The need for prison reform? Factory Farming?

Comment by elityre on Eli's shortform feed · 2019-08-07T18:41:06.925Z · score: 7 (4 votes) · LW · GW

New post: Napping Protocol

Comment by elityre on My recommendations for gratitude exercises · 2019-08-05T20:39:35.947Z · score: 8 (6 votes) · LW · GW

This may be an asshole thing to say, but I wish this had a tl;dr. I want to skim some bullet points that would let me know if I already know this stuff or not.

Mostly this post was theoretical / an argument for gratitude noting, as opposed to a how-to guide, which is what I expected from the title.

The main thing I took away from this was:

To feel grateful for some of these things you might have to try to vividly imagine being without them for a time. If you are deprived of some of these things for a time (or temporarily believe you are) you can also try to remember what that feels like, so that you can recapture it later when you have them again.

Upvote for that pointer!

Comment by elityre on Eli's shortform feed · 2019-08-05T20:33:04.751Z · score: 9 (5 votes) · LW · GW

new post: Intro to and outline of a sequence on a productivity system

Comment by elityre on Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on? · 2019-07-31T23:04:55.153Z · score: 6 (4 votes) · LW · GW

Sure. 10,000 is an (almost) unobjectionable lower bound.

Comment by elityre on Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on? · 2019-07-31T19:39:35.670Z · score: 2 (2 votes) · LW · GW

Are there existing agreements constraining the deployment of applied genetic research? What are the keywords I should search for, if I want to know more?

The only thing I know about this area is that an unaffiliated researcher used CRISPR to modify human embryos, and that most of the field rebuked him for it. This suggests that there are general norms about which experiments are irresponisble to try, but not strong coordination that prevents those experiments from being done.

Comment by elityre on Eli's shortform feed · 2019-07-17T17:05:48.547Z · score: 1 (1 votes) · LW · GW

New post: my personal wellbeing support pillars

Comment by elityre on Inconvenience Is Qualitatively Bad · 2019-07-17T06:55:29.636Z · score: 6 (4 votes) · LW · GW

Wow. I really appreciate the curious spirit of this comment.

Comment by elityre on Eli's shortform feed · 2019-07-14T21:52:48.749Z · score: 5 (3 votes) · LW · GW

New (unedited) post: The bootstrapping attitude

Comment by elityre on Eli's shortform feed · 2019-07-14T21:51:54.416Z · score: 4 (3 votes) · LW · GW

New (unedited) post: Exercise and nap, then mope, if I still want to

Comment by elityre on 3 Levels of Rationality Verification · 2019-07-13T05:01:56.213Z · score: 4 (3 votes) · LW · GW

Let's see...

  • Prediction contests are an obvious one.
  • Also, perhaps, having people compete at newly designed games, so that everyone has the same amount of time to learn the rules and how to win, given the rules.
  • Perhaps we could design puzzles that intentionally have places where one would make a mistake, error, or wrong choice, and such errors are visible (to an observer who knows the puzzle) when made.
Comment by elityre on An Alien God · 2019-07-12T02:07:17.406Z · score: 24 (4 votes) · LW · GW
When I design a toaster oven, I don't design one part that tries to get electricity to the coils and a second part that tries to prevent electricity from getting to the coils. It would be a waste of effort. Who designed the ecosystem, with its predators and prey, viruses and bacteria? Even the cactus plant, which you might think well-designed to provide water fruit to desert animals, is covered with inconvenient spines.

Well, to be fair, if want to design an image classifier, I might very well make one part that tries hard to categorize photos and another part that tries hard to miscategorize them.

Comment by elityre on Schism Begets Schism · 2019-07-12T01:57:09.566Z · score: 2 (2 votes) · LW · GW
If the other group or community is, as you say, much worse than it could be, helping to improve it from the inside makes things better for the people already involved, while going and starting your own group might leave them in the lurch.

Sure. When everyone (or at least a majority) in the initial group are on board with your reform efforts, you should often try to reform the group. But very often there will be a conflict of visions or a conflict of interests.

In general I think you should probably at least initially try to reform things, though if it doesn't work well there's a point where you might have to say "sorry, the time has come, we're making our own group now".

I certainly agree with this, though it seems plausible that we have different views of the point at which you should switch to the "found a splinter group" strategy.

Comment by elityre on The AI Timelines Scam · 2019-07-12T01:32:52.279Z · score: 15 (5 votes) · LW · GW


...if you think both an urgent concern and a distant concern are possible, almost all of your effort goes into the urgent concern instead of the distant concern (as sensible critical-path project management would suggest).

This isn't obvious to me. And I would be interested in a post laying out the argument, in general or in relation to AI.

Comment by elityre on Schism Begets Schism · 2019-07-10T20:51:46.148Z · score: 12 (4 votes) · LW · GW
In point of fact, doing important things often requires coordination, teamwork, and agreeing to compromises. If you insist on everything being exactly your way, you'll have a harder time finding collaborators, and in many cases that will be fatal to a project -- I do not say all, but many.

This is true and important, and the same or a very similar point to the one made in Your Price for Joining.

But that post has a different standard than the one given by the OP:

If the issue isn't worth your personally fixing by however much effort it takes, and it doesn't arise from outright bad faith, it's not worth refusing to contribute your efforts to a cause you deem worthwhile. [emphasis mine]

Sometimes things are bad or (or much worse than they could be) in some group or community. When that's the case, one can 1) try and change the community from the inside, or 2) get a group of his/her friends together to do [thing] the way they think they should do it, or 3) give up and accept the current situation.

When you're willing to put in the work to make 2 happen, it sometimes results in a new healthier group. If (some) onlookers can distinguish between better and worse on the relevant axis, it will attract new members.

It seems to me that taking option 2, instead of option 1, is cooperative. You leave the other group doing it their way, in peace, and also create something good in the world in addition.

Granted, I think the situation may be importantly different in online communities, specifically because the activation energy for setting up a new online group is comparatively small. In that case, it is too easy to found a new group, and accordingly they splinter to regularly for any single group to be good.

Comment by elityre on Raised in Technophilia · 2019-07-08T00:06:28.555Z · score: 3 (2 votes) · LW · GW

Anyone have a citation for Drexler's motivations?

Comment by elityre on Being the (Pareto) Best in the World · 2019-06-25T03:34:12.808Z · score: 1 (5 votes) · LW · GW

This was great. Thank you!