I taught game theory at Princeton and wish I'd seen this explanation beforehand, excellent framing.

In the territory, bad event happens [husband hits wife, missile hits child, car hits pedestrian]. There is no confusion about the territory: everyone understands the trajectories of particles that led to the catastrophe. But somehow there is a long and tortuous debate about who is responsible/to blame ["She was wearing a dark hoodie that night," "He should have come to a complete stop at the stop sign", "Why did she jaywalk when the crosswalk was just 10 feet away!"].

The problem is that we mean a bunch of different things simultaneously by blame/responsibility:

1. Causality. The actual causal structure of the event. ["If she'd worn a reflective vest this wouldn't have happened," "If your left headlight wasn't broken you'd have seen her."]
2. Blame. Who should be punished/shamed in this situation. This question already branches into a bunch of cruxes about the purpose and effectiveness of punishment.
3. Responsibility. What is the most effective way of preventing such events in the future? ["If we passed a law that all pedestrians wear reflective vests it would halve incidents like this", "How about we institute mandatory pedestrian-sighting courses for drivers, and not blame the victim?"]

People argue about the same event with different causal models, different definitions of blame, and different notions of responsibility, and the conversation collapses. Fill in your own politically-charged example. 

Setting the zero point seems to be one "move" in this blame game [if the default is that all drivers take pedestrian-sighting courses, then you're to blame if you skipped it. if the default is that all pedestrians must wear reflective vests, then you're to blame if you didn't wear one.]

I don't have a complete reply to this yet, but wanted to clarify if it was not clear that the position in this dialogue was written with the audience (a particularly circumspect broad-map-building audience) in mind. I certainly think that the vast majority of young people outside this community would benefit from spending more time building broad maps of reality before committing to career/identity/community choices. So I certainly don't prescribe giving up entirely.

ETA: Maybe a useful analogy is that for Amazon shopping I have found doing serious research into products (past looking at purchase volume and average ratings) largely unhelpful. Usually if I read reviews carefully, I end up more confused than anything else as a large list of tail risks and second-order considerations are brought to my attention. Career choice I suspect is similar with much higher stakes.

Seeing patterns where there are none is also part of my writing process.

This paper of mine answers exactly this question (nonconstructively, using the minimax theorem).

I feel there is an important thing here but [setting the zero point] is either not the right frame, or a special case of the real thing, [blame and responsibility are often part of the map and not part of the territory] closely related to asymmetric justice and the copenhagen interpretation of ethics. 

Afaict, the first simple game is not the prisoner's dilemma, nor is it zero-sum, nor is the prisoner's dilemma zero-sum.

This is not intended as a criticism in any way, but this post seems to overlap largely with https://www.lesswrong.com/posts/k9dsbn8LZ6tTesDS3/sazen. 

[Edit: After looking at the timestamps it looks like that post actually came out after, anyway it might be an helpful alternative perspective on the same phenomenon.]

Is it just me or are alignment-related post titles getting longer and longer?

This post has a lot of particular charms, but also touches on a generally under-represented subject in LessWrong: the simple power of deliberate practice and competence. The community seems saturated with the kind of thinking that goes [let's reason about this endeavor from all angles and meta-angles and find the exact cheat code to game reality] at the expense of the simple [git gud scrub]. Of course, gitting gud at reason is one very important aspect of gitting gud in general, but only one aspect.

The fixation on calibration and correctness in this community trades off heavily against general competence. Being correct is just a very special case of being good at things in general. Part of Duncan's ethos is that it's possible to learn [the pattern of gitting gud], and furthermore this is more important and consistent than learning how to be good at one particular arbitrary skill.

It seems important to notice that we don't have control over when these "shimmying" strategies work, or how. I don't know the implication of that yet. But it seems awfully important.

A related move is when applying force to sort of push the adaptive entropy out of a certain subsystem so that that subsystem can untangle some of the entropy. Some kinds of meditation are like this: intentionally clearing the mind and settling the body so that there's a pocket of calmness in defiance of everything relying on non-calmness, precisely because that creates clarity from which you can meaningfully change things and net decrease adaptive entropy.

Two further comments: 
(a) The main distinction I wanted to get across is while many behaviors fall under the "addiction from" umbrella, there is a whole spectrum of how more or less productive they are, both on their own terms and with respect to the original root cause.
(b) I think, but am not sure, I understand what you mean by [let go of the outcome], and my interpretation is different from how the words are received by default. At least for me I cannot actually let go of the outcome psychologically, but what I can do is [expect direct efforts to fail miserably and indirect efforts to be surprisingly fruitful]. 

Yeah… for some reason, on this particular point, it always does, no matter how I present it. Then people go on to say things that seem related but importantly aren't. It's a detail of how this whole dimension works that I've never seen how to communicate without it somehow coming across like an attempt to hijack people. Maybe secretly to me some part of me is trying. But FWIW, hijacking is quite explicitly the opposite of what I want. Alas, spelling that out doesn't help and sometimes just causes people to say they flat-out don't believe me. So… here we are.

Sure, seems like the issue is not a substantive disagreement, but some combination of a rhetorical tic of yours and the topic itself being hard to talk about.

Very strongly agree with the part of this post outlining the problem, your definition of "addiction" captures how most people I know spend time (including myself). But I think you're missing an important piece of the picture. One path (and the path most likely to succeed in my experience) out of these traps is to shimmy towards addictive avoidance behaviors which optimize you out of the hole in a roundabout way. E.g. addictively work out to avoid dealing with relationship issues => accidentally improve energy levels, confidence, and mood, creating slack to solve relationship issues. E.g. obsessively work on proving theorems to procrastinate on grant applications => accidentally solve famous problem that renders grant applications trivial.

And included in this fact is that, best as I can tell, anyone who really groks what I'm talking about will want to prioritize peeling off adaptive entropy over any specific outcome. That using addiction or any other entropy-inducing structure to achieve a goal is the opposite of what they truly want.

This paragraph raised my alarm bells. There's a common and "pyramid-schemey" move on LW to say that my particular consideration is upstream and dominant over all other considerations: "AGI ruin is the only bit that matters, drop everything else," "If you can write Haskell, earning-to-give overwhelms your ability to do good in any other way, forget altruism," "Persuading other people of important things overwhelms your own ability to do anything, drop your career and learn rhetoric," and so on ad nauseum. 

To be fair, I agree to a limited extent with all of the statements above, but over the years I've acquired so many skills and perspectives (many from yourself Val) that are synergistic and force-multiplying that I'm suspicious any time anyone presents an argument "you must prioritize this particular force-multiplier to the exclusion of all else."

Sure, no big deal.

You can't fight fire with fire, getting out of a tightly wound x-risk trauma spiral involves grounding and building trust in yourself, not being scared into applying the same rigidity in the opposite direction. 

The comment is generally illuminating but this particular sentence seems too snappy and fake-wisdomy to be convincing. Would you mind elaborating?

Thanks for sharing this, it puts into relief a problem I've noticed about academic research: the real research happens behind closed doors and in private communications that the young people don't have access to. Young people end up only learning about the finished theorems much later on in polished form.

That's great to hear, I've been slowly working on this myself in recent years. E.g., it's greatly improved my gaming experience - from being a total lurker to engaging with Discords, posting bugs and suggestions, occasionally writing Steam guides - it's enriching for sure.

I don't know about skills plural, but the game definitely drilled in that particular skill of aiming to falsify one's hypotheses instead of just confirming them. That's a skill well worth a dozen hours of deliberate practice in my opinion.

Fantastic game, thanks for recommendation!

I reimplemented the game in vanilla Python and managed to simulate it several hundred times with ~10k random species for a total of hundreds of thousands of generations.

Unfortunately, I didn't read Hylang documentation carefully and thought foragers could simultaneously eat one of every food available, instead of just the most nutritious one...

Only my throwaway locust clone survived under the real rules. :'(

Nice!

Haven't played Osu! for many years now unfortunately. I only got into it briefly to practice mouse accuracy for FPS games, but that motivation has dried up. I suspect Osu! would still be damn good fun without it, so I'll let you know if it gets to the top of my gaming queue. :)

Here are two recentish papers I really enjoyed reading, which I think are fairly reasonable to approach. Some of the serious technical details might be out of reach.

https://arxiv.org/abs/1909.03562

http://www.cs.tau.ac.il/~nogaa/PDFS/induniv1.pdf

I tried Touhou Perfect Cherry Blossom at one point and never got past any difficulty, so I defer to your expertise here. There's a general skill of getting better at focusing one's attention in tandem with getting better at execution and this post is only a first approximation.

Yea, I think there's some general pattern of the form:

1. Research is weird and mysterious.
2. Instead of studying research, why don't we study the minds that do research?
3. But minds are equally weird and mysterious!
4. Ah yes, but you are yourself possessed of a mind, which, weirdly enough, can imitate other minds at a mysteriously deep level without consciously understanding what they're doing.
5. Profit.

I love the film study post, thanks for linking! This all reminds me of a "fishbowl exercise" they used to run at the MIRI Summer Fellows program, where everyone crowded around for half an hour and watched two researchers do research. I suppose the main worry about transporting such exercises to research is that you end up watching something like this.

But then he encounters the rigamarole of the whole process you describe in your post and it stops him from doing what he originally dreamed. He needs to get published. He needs to do original research. He needs to help his advisor and other professors do their research. He needs to do all of that because otherwise he won't be respected enough to actually have a career in physics research. But doing that kind of work isn't why he got into physics in the first place!

I'm confused about the claim that the academic process is at all misaligned with his original dream. Isn't doing original research and getting published the clearest path - though perhaps not the only one - on the way to the goal of restructuring quantum mechanics? Isn't helping his advisor and other professors do their research one of the best ways of learning the ropes in the meantime? Isn't acquiring the respect of your colleagues exactly the path to having a whole community and field at your back to effect those paradigm-shifting breakthroughs, instead of going it alone?

We are using the word "coast" differently - what I meant by coasting is that many of the professors I know would have to actively sabotage their own research groups and collaborators to not produce ~five nice papers a year (genuine though perhaps not newsworthy contributions to the state of knowledge). 

Of course, the state of affairs seriously varies with the quality of the institution.

Right, the structure is quite simple. The only thing that came to mind about finite factored sets as combinatorial objects was studying the L-function of the number of them, which surely has some nice Euler product. Maybe you can write it as a product of standard zeta functions or something? 

Are there any interesting pure combinatorics problems about finite factored sets that you're interested in?

This is great!

I'm interested in the educational side of this, particularly how to do one-on-one mentorship well. I've had effective mentors in the past who did anything from [blast me with charisma and then leave me to my own devices] to [put me under constant surveillance until I past the next test, rinse, repeat.] Can you say something about your educational philosophy/methods?

This is fascinating and I'd love to hear more depth on whatever you'd be willing to share.

Regarding the suggestion to start with something small, I think in hindsight it was kind of a manipulation on my part to make the tool seem safer and to try to get more people to try it. In my limited experience, internal conflicts that seem small rarely turn out to be. 

When I first tried IDC at CFAR, the initial "small starting point" of "Should I floss?" dredged up a whole complex about distrust of doctors in particular and authority in general. A typical experience with watching myself and others IDC is that regardless of the starting point, one ends up in a grand dramatic battle of angels and demons over one's soul.

Thanks for reminding me about this talk! I read it one more time just now and was struck by passages that I completely missed the first couple times:

Ed David was concerned about the general loss of nerve in our society. It does seem to me that we've gone through various periods. Coming out of the war, coming out of Los Alamos where we built the bomb, coming out of building the radars and so on, there came into the mathematics department, and the research area, a group of people with a lot of guts. They've just seen things done; they've just won a war which was fantastic. We had reasons for having courage and therefore we did a great deal. I can't arrange that situation to do it again. I cannot blame the present generation for not having it, but I agree with what you say; I just cannot attach blame to it. It doesn't seem to me they have the desire for greatness; they lack the courage to do it.

It seems an optimistic note, that some of what one lacks in ability or work ethic, one can make up for with courage, which one can train. And also:

For myself I find it desirable to talk to other people; but a session of brainstorming is seldom worthwhile. I do go in to strictly talk to somebody and say, ``Look, I think there has to be something here. Here's what I think I see ...'' and then begin talking back and forth. But you want to pick capable people. To use another analogy, you know the idea called the `critical mass.' If you have enough stuff you have critical mass. There is also the idea I used to call `sound absorbers'. When you get too many sound absorbers, you give out an idea and they merely say, ``Yes, yes, yes.'' What you want to do is get that critical mass in action; ``Yes, that reminds me of so and so,'' or, ``Have you thought about that or this?'' When you talk to other people, you want to get rid of those sound absorbers who are nice people but merely say, ``Oh yes,'' and to find those who will stimulate you right back. 

In other words, to be a good collaborator you have to contribute to the babble.

Is the following interpretation equivalent to the point?

It can be systematically incorrect to "update on evidence." What my brain experiences as "evidence" is actually "an approximation of the posterior." Thus, the actual dog is [1% scary], but my prior says dogs are [99% scary], I experience the dog as [98% scary] which my brain rounds back to [99% scary]. And so I get more evidence that I am right.

I'm not totally convinced this is the right way to think about it, any given useful mutation will depend on some constant number of coordinates flipping, so in this high-dimensional space you're talking about, useful mutations would look like affine subspaces of low codimension. When you project down to the relevant few dimensions, there's probably more copies of virus than points to fit in, and it takes a long time for them to spread out.

I guess it depends on the geometry of the problem, whether there are a small number of relevant mutations that make a difference, each with a reasonable chance of being reached, or a huge number of relevant mutations each of which is hard to reach.

Adding onto this a little, here's a toy model of viral genetic diversity based on my high-school level biology. 

Suppose the virus' DNA starts out as 000 (instead of ACTG for simplicity), and it needs to mutate into 111 to become stronger. Each individual reproduction event has some small probability p of flipping one of these bits. Some bit flips cause the virus to fail to function altogether, while others have no or negligible effect on the virus. As time goes on, the number of reproduction events starting from a given bitstring grows exponentially, so the likelihood of getting one more 1 grows exponentially as well. However, each time you jump from 000 to 100, it's not as if all other copies of 000 turn into 100, so making the next jump takes a while of waiting on lots of copies of 100 to happen. And then some 101 appears, and there's no jump for a while again as that strain populates.

The upshot is that you imagine the viral population to be "filling out the Hamming cube" one bitflip at a time and the weight of each bitstring is the total number of viruses with that code, and a genuinely new strain only appears when all 3 bits get flipped in some copy. But:

(a) The more total copies of the virus there is, the faster a bad mutation happens (speed scaling linearly).

(b) Assuming that some mutations require multiple independent errors to occur (which seems likely?), the virus population is "making incremental research progress" over time by spreading out across the genetic landscape towards different strains, even when no visibly different strains occur.

re: why are there more scary new strains now: 

Have people have already accounted for the fact that the more virus there is in the world, the more likely it is for one of these viruses to mutate? If there's 5x as many cases of covid floating around right now than in September, a strain as bad as the UK strain will emerge 5x as quickly in expectation.

This feels like an extremely important point. A huge number of arguments devolve into exactly this dynamic because each side only feels one of (the Rock|the Hard Place) as a viscerally real threat, while agreeing that the other is intellectually possible. 

Figuring out that many, if not most, life decisions are "damned if you do, damned if you don't" was an extremely important tool for me to let go of big, arbitrary psychological attachments which I initially developed out of fear of one nasty outcome.

I agree, but I was more asking about how you think your insight about the "distance to safety" can help with that.

Well, after a bounded number of initially difficult "far-out explorations" that cover the research landscape efficiently, the hope is that almost everything is reasonably close to safety henceforth.

Interesting. My own approach is usually to collaborate/ask someone who knows the subject you want to learn. But that does require being okay with asking stupid questions.

Yes, I think your approach is ideal for the efficiency of learning if anxiety was not a factor. Unfortunately the people who know the subjects I want to learn best are people I care about impressing and/or people so well-versed in the subject that they have difficulty bridging the inferential abyss between us. At least for me it is hard to treat them as a "psychologically nearby" companion who has my back.

Even after getting much better at asking stupid questions, it feels like the maximum I feel okay with asking in a meeting with someone who knows a subject already is ~3, and not ~40, which is the number I want to ask.

Very nice post! I would add that it is a useful and nontrivial skill to notice what you're paying attention to. It may not be helpful to try getting curious unless you know concretely what this means about how you move your eyes and attention.

To give a video game example, players new to a genre have no idea where to put their eyes on the screen. When I told a friend playing Hades to put their eyes on their own character, instead of on the enemies, they instantly started taking half as much damage. I got a lot better at Dark Souls, on the other hand, by staring at the enemies to catch their telegraphed movements and not on myself. Similarly, I had a friend who could not get into Path of Exile because they wanted to dive into playing the game mechanically and was frustrated about my claim that to properly enjoy the game you spend most of your energy staring at skill trees and item builds and wikis. I found that my natural state playing PoE was leaving my eyes half unfocused on the game, spamming my skill rotation while thinking about my next item or skill upgrade on my second monitor.

To listening properly and be curious, I think the main places one should focus one's attention (in addition to the words they're saying) are: (a) on the other person's face and body, (b) on the other person's tone of voice, and (c) on your own bodily sensations. In other words, everywhere but your own thoughts.

To be clear, the papers would almost certainly have gone through anyway, the helpful thing was being very comfortable with Bayes rule and immediately noticing, for example, that conditioning on an event with probability 1-o(1) doesn't influence anything by very much. 

Another trick I derived from this comfort is to almost never actually condition on small-probability events. Instead, the better thing to do is to modify the random variables you care about to fail catastrophically in the small probability scenario. 

For example, in graph theory I might care about controlling a random variable X which is the number of times a certain substructure appears in the random graph G(n,p), but to do so I need to condition away some tail event E like the appearance of a vertex of extremely high degree. Instead of working with conditional probability for the rest of the argument (which might go on to condition away 3 or 4 other tail events), the nicer thing to do is to modify X into a variable X' which is defined to be 0 when E occurs, and reason about X' instead. This is better for multiple reasons; the most important one being that the edge appearances in G(n,p) are no longer independent when you condition on E complement.

I think mostly what I got out of the Sequences was removing an air of mystery around Bayes rule. Here by mystery I mean "System 1 mystery," i.e. that before I read the Sequences, to figure out a conditional probability I would have to sit down and carefully multiply and divide. This post also helped.

How do you think this apply to intellectual pursuits? I have in mind research advising: in my experience, some people that I think could be great researchers are terrified of exploring some part of knowledge where there is no answer yet. And even we established researchers can easily be afraid of learning a new subject or a new technique that would help them tremendously. Maybe the comfort flags should be links with stuff that the graduate student/researcher knows well? Anecdotally, people seem more open to learning about what you want to say if you link it to their own field.

I don't pretend to be an established researcher, but here is what I had in mind. Most researchers at one point or other spend some amount of time white-knuckle learning things that are outside their comfort zones, but usually these things are just barely outside. My suggestion would be that all other things equal, some of that time should be spent learning things really far out instead.

Also I think learning in pairs is a very helpful tool. The active ingredient is to have someone you trust enough to freely share your ignorance and ask basic questions, and the easiest way to get this trust is to find someone who is also obviously ignorant of the same thing.

I wonder if the following are also examples of motive ambiguity:

• Mothers choosing to stay at home.
• Researchers choosing to be bad at teaching.
• Mathematicians choosing to work on problems with no applications.

Let me share some more gears/evidence. I believe something a little more interesting happens than what you're saying (which is definitely one piece of the puzzle).

(1) It's fun to look at how the audience organizes itself during math talks. The faculty almost always sit in the front row, point out mistakes more directly ("You mean this" instead of "Is this correct?"), ask questions more often (and with less hand-raising), and sometimes even feel comfortable to answer questions in the speaker's stead. I suspect this is a social role that everyone learns through attending enough seminars.

(2) Faculty have access to a lot more privileged information about other mathematicians than everyone else. They are on editorial boards, hiring committees, admissions committees, conference organization, awards panels, etc. I got a confidence boost after peer reviewing my first couple of papers, the transition to faculty is this x10 in terms of data to train on and notice you're being underconfident.

(3) Professors spend a lot of time with their research groups/PhD students/undergrads compared to in the company of other faculty, so they aren't doing as much comparing themselves with other faculty as you would think. At least in mathematics, it's generally preferred for faculty at the same university to have research interests as far as part as possible (to cover a breadth of fields), so each professor interacts a great deal on the day-to-day with their group of undergrads/grad students/postdocs. Meetings with other faculty are mostly logistical, with the possible exception of a handful of close collaborators. This is probably even more true in fields where a professor is literally a head of their own lab and the PI for all research that happens in the lab. I think status feelings tend to work on the level of "people you interact with most on a daily basis" instead of "people you intellectually compare yourself to."

Fascinating! Definitely plan to check this out, thanks for the recommendations and detailed introduction.

Thank you for writing this, it led me to reconsider this phenomenon from a different perspective and revisit Lsusr's post as well as competent elites, which seemed to really string things together for me.

• Lsusr is primarily talking about success "outside of the usual system", which generally frees someone up even more from the usual system. Start-ups are the primary example of this.
• Alkjash is primarily talking about success within the existing system. The stereotypical successful career is an example of this.

This definitely feels like part of the thing, but I would (as with many things) phrase it in the language of status. I claim that much of the "freedom" that Lsusr talks about and the "intelligence and aliveness" Eliezer talks about is consequences of feeling high status. In academia, the standard solution to all of the ennui and anxious underconfidence a grad student or postdoc feels is ... wait for it ... tenure. Your inhibitions magically disappear when you become faculty, and mathematicians often become confident to explore, gregarious, and willing to state beliefs even in dimensions orthogonal to their expertise (e.g. Terry Tao on Trump). This is explained by direct changes in the brain, as well as external changes in how the intelligent social web coopts your cognition, when a person gains status.

My guess is that the difference between what you call Lsusr's "outside of the usual system" and my "within the existing system" is the difference between systems with shorter and looser status hierarchies and those with longer and tighter ones. In the former it is easier for an exceptional individual to quickly gain competence and reputation and reap the benefits of status. This difference is in turn mostly explained by systems having different levels of play. Thus, one would find success more agency-limiting for a longer period of time in professional Go than in professional Starcraft, in mathematics than in AI/ML, in Google than in a startup, etc.

My interpretation of Lsusr's philosophy is that there is a magic sauce that rhymes with arrogance which allows one to turn on powerful high-status feelings and behaviors (confidence, agency, vision) regardless of circumstance. Unfortunately there are harsh cultural defenses against this kind of thing that one has to prepare for.

Very interesting! This thread is the first time I've heard of NLP (might have seen the acronym before but I thought it was ML people referring to Natural Language Processing), I will definitely check it out. I guess I just rounded off my observations to the nearest things I recognized. I'm not surprised that Robbins stuff is embedded in a larger technique but am kind of surprised that I've been ignorant of it for so long.

Is there a book or resource that you would most recommend to learn NLP?

The phenomenon I'm pointing to via "making couples fall in love with him" (which might be the wrong words) is that in relationship interventions he uses a combination of explicit models, personal charisma, and hard-to-transfer people-reading skills to make each person feel understood at a level that causes them to trust him deeply. This level of trust seems pretty extraordinary and hard to distinguish from love. After that he proceeds to use exercises to transfer this trust to between the couple in a way that requires very little agency on the couples' part. They sort of just go along with/paraphrase TR's statements to each other and then get this massive intimacy boost. I would guess that they come out of the experience feeling as positive or more so about TR than about each other.

I would love to hear which pieces of his written work you think of as "actually new or useful insights," the only thing that fits that description for me from his youtube videos is the Six Human Needs, which is a useful template for goal-factoring for me.

Like Asimov is back among the living.

No worries! Perhaps it's worth reminding everyone here that asymmetric justice incentivizes inaction. I hope I didn't do this just now, I very much appreciate the spirit of your experiment and encourage more people to try to state their beliefs and move fast and break things.

Your comment is interesting and helpful for me because I have only a small sample of people who don't follow the "pain is the unit of effort" heuristic to a pathological extent. Perhaps this is explained by my circle of friends being dominated by Asian-Americans who went to the top universities. I definitely didn't consider what possibly ill effects it might have on others for whom this is not true. So thanks for that information!

However, enough of my brain interpreted your comment as a status move/slapdown that I'd suggest you reconsider doing these reviews, at least in the tone you're currently doing them. I don't believe you intended it this way, but your comment comes off as claiming a position of authority and also encourages too much (imo) Goodharting on the LessWrong "top 15 posts" metric. Both of these feel icky to me. I predict you will at minimum annoy a lot of authors if you continue to write these.