First off, I'd like to apologise, I've only now read the OP, and I was talking past you. I still think we have some genuine disagreements however, so I'll try to clarify those.

I was surprised by how much I liked the post. It separates the following approaches for improving the state of the world:

1. Further our understanding of what matters
2. Improve governance
3. Improve prediction-making & foresight
4. Reduce existential risk
5. Increase the number of well-intentioned, highly capable people

My ordering of tractability on these is roughly as follows: 5 > 4 > 3 > 2 > 1. There is then a question of importance and neglectedness. I basically think they're all fairly neglected, and I don't have strong opinions on scope except that, probably, number 4 is slightly higher than the rest.

Understanding what matters seems really hard. For the other problems (2-5) I can see strong feedback loops based in math for people to learn (e.g. in governance, people can study microeconomic models, play with them, make predictions and learn). I don't see this for problem 1.

Sure, for all 1-5 there will be steps where you have to step sideways - notice a key variable that everyone has been avoiding even thinking about, due to various incentives on what thoughts you can think - but there's more scope for practice, and for a lot of good work to be done that isn't 80% deep philosophical insight / deep rationality abilities.

Luke Muehlhauser tried really hard and found the tractability surprisingly low (1, 2).

There are worlds consistent with what I've said above, where I would nonetheless want to devote significant resources to 1, if (say) we were making plans on a 100-200 year horizon. However, I believe that we're in a world where we're very soon going to have to score perfectly on number 4, and furthermore that scoring perfectly on number 4 will cause the other problems to get much easier - including problem number 1.

Summary: As I hadn't read the OP, I read your comment as claiming your approach to 1 was the only way to do good altruistic work. I then responded with reasons to think that other - more technical approaches - were just as good (especially for the other problems). I now pivot my response to reasons to think that working on things that aren't 1 are more important, which I think we may disagree on.

I can't tell if it's still necessary, but I wanted to try anyway. Here's my ITT for your position. This story isn't literally what I think you think, but I believe it's an accurate analogy.

---

You have recently done something new, like tried LSD, or gotten really addicted to the high of exercise, or just moved to a new place and been genuinley happy and excited every minute for a whole week.

And you learned something - you didn't know life could feel this good. It's a genuine insight and feels important to you, and does change how you will make life plans forevermore.

You talk to me. I talk about the importance of figuring out what really matters in life; I've just spent many weeks reading textbooks on the subject and philosophical treatise on experience. And now you're stuck trying to communicate to me the realisation that life can be really good - you think it's causing me to make mistakes regarding all sorts of things I discuss, from population ethics to the moral value of sleep.

You know that you don't know everything there is to know about the good life, but you're pretty sure that whatever other weird realisations are out there, I'm sure not gonna find them doing what I'm doing right now.

Regarding power laws, I am attempting to make a strong claim about how reality works - reality, not methodology. It is the case that the things we value are power law distributed - whether it's health interventions, successful startup ideas, or altruistic causes, it turns out that selecting the right one is where most of the variance is.

As a result, one's ability to do good will indeed be very noisy - this is why many good funders take hits-based approaches. For example, Peter Thiel is known for asking interesting people for their three weirdest ideas, and funding at least one of them. He funded MIRI early, though I expect he probably was quite unsure of it at the time, so I consider him to have picked up some strong hits.

That's also my feeling wrt the EA post on SSC. I'm generally not happy with the low variance approaches taken within EA and feel sad at how few new ideas are being tested, but I think that to say the number of orgs doing weird things wrt figuring out what matters, is to push the number in the wrong direction.

The usual formulation of the Mary's Room thought experiment makes a very strong assumption about what Mary knows while in the room: that she has a complete understanding of every physical and physiological process that happens when she is looking at things. This of course goes way beyond what any real human being has.

With this assumption, I claim that Mary would already know that there are such things as the qualia of seeing red; she wouldn't need convincing. And she would already know quite a lot about those qualia; e.g., if seeing red tends to increase physiological arousal and spark thoughts of love, anger and serious physical injury then she would know that.

(This argument is not original to me; I found it in a book by Daniel Dennett.)

Without that assumption, some arguments based on the thought experiment (e.g., ones claiming that qualia have to be something nonphysical) collapse. Probably not yours, though, because for your purposes it doesn't matter whether Mary understands anything at all about colour vision; I think you get a better example of experiences that might not be conveyable in text if we suppose she doesn't.

Still... it seems like one could convey at least some aspects -- though certainly not all -- of the experience of seeing red through text, even to someone who had never had that experience. Along the following lines. "At present, your eyes give you a picture of the world that could be described by saying for everything how bright it is. Other people -- perhaps including you, some day -- get more information from their eyes than that: for each location in their visual field, not just a brightness but something else called colour. Different objects have different colours, so this lets us distinguish more things about them than you can from brightness alone. One kind of colour is called "red", and it turns out that our brains respond in interesting ways to red things. Red is the colour of a liquid called blood that we have inside us. When your body is badly enough damaged, blood may flow out. That's bad news, and seeing red makes your brain prepare a little bit to fight or run away: it gives you a little of the feeling you might have if you imagine that there's a tiger just around the corner waiting to jump on you. Most people's skin is slightly reddish in colour, especially when they are embarrassed or excited, and perhaps for that reason red is also associated in most people's minds with love and sex. Seeing red may make your brain prepare a little bit for a romantic or sexual encounter: it gives you a little of the feeling you might have if you imagine that someone you find very attractive is just around the corner waiting to kiss you. When there's something bright red in your field of view, your eyes will tend to be attracted by it, and distract your attention from other things. In our culture, red is associated with fighting and celebration and love and danger and urgency. The colour red is a bit like the sound of a trumpet fanfare, and a bit like the feel of either a warm embrace or a slap in the face."

Hearing all that (and the much more that one could say on the same theme) isn't the same as actually having the experience of seeing red, for sure. But I think someone who had never seen the colour red would know more about what seeing red is like after reading a few paragraphs of that sort of thing. They'd have some information about specific effects it might have; some analogies to other things they might be familiar with; some understanding of why those effects might be there, which might help them predict other effects; some information about ideas and feelings and images that people associate with redness.

I tried this exercise and found it extremely interesting. My report is rot-13'ed to respect the blinding. I highly recommend taking five minutes to try it yourself before reading.

(One more line break for further encouragement to break your train of thought and try it.)

Ok, here:

V gevrq guvf fvggvat ng zl qrfx ng jbex. V'ir cerivbhfyl unq gebhoyr jvgu rkrepvfrf gung nfxrq zr gb zbir gur ybpngvba bs "zr"/zl rtb/zl crefcrpgvir bhg bs zl urnq naq vagb zl obql, fb V jnf fxrcgvpny V jbhyq fhpprrq. Ohg Dvnbpuh'f qrfpevcgvba va grezf bs "fubbgvat bhg ryrpgevpvgl gb gur erfg bs zl obql gb znxr vg zbir" znqr zr abgvpr n qvfgvapgvba orgjrra gur ybpngvba bs "gur jngpure" be zl bofreingvba/creprcgvba, naq "gur npgbe" be zl zbirzrag/npgvba. Guvf frpbaq guvat ghearq bhg gb or abg gung uneq gb zbir, juvyr gur svefg fgnlrq va cynpr.

V qba'g unir irel fgebat zragny vzntrel, fb V qvqa'g unir nalguvat yvxr n oyhr onyy bs raretl va zvaq. Zbgvba naq fcngvny eryngvbafuvc ner angheny gb zr, fb V sbphfrq ba gur vqrn bs na nofgenpg ybphf gung "chfurq bhg" gb pnhfr zbgvba.

Gb xrrc gur sbphf ba npgvba, V zbirq zl unaq va n ercrgvgvir hc-naq-qbja zbgvba juvyr V gevrq zbivat gur ybphf. Nf fbba nf vg tbg bhg bs zl urnq vagb zl hccre purfg, zl unaq zbirzragf tbg zhpu fybjre, yrff cerpvfr, naq yrff erthyne. Vg sryg n ovg yvxr jnyxvat onpxjneqf, be zbivat jvgu zl rlrf pybfrq. Gurer jnf nyfb n fbeg bs hcjneqf cerffher ba gur ybphf, yvxr jura lbh gel gb chfu fbzrguvat ohblnag qbja vagb jngre.

V ybfg genpx n pbhcyr gvzrf naq unq gb fgneg bire, ohg ba gur guveq gel V zbirq vg vagb zl thg, naq gura gb gur onfr bs zl fcvar. Nf V nccebnpurq gur onfr bs zl fcvar, na ryrpgevp, ohmmvat, fbzrjung birefgvzhyngvat raretl pragrerq gurer orpnzr fgebatre naq zber fnyvrag. Guvf raretl vf snzvyvne gb zr sebz zrqvgngvba, ohg V'ir cerivbhfyl sbhaq vg gbb vagrafr gb "ybbx ng" sbe ybat. Gb zl fhecevfr, gung jnfa'g gehr guvf gvzr, creuncf orpnhfr V jnf pbaarpgvat vg gb zbgvba naq npgvba engure guna whfg ybbxvat ng vg.

Gur raretl nebfr naq frggyrq dhvgr dhvpxyl, va n znggre bs n pbhcyr frpbaqf. Vg jnf (naq vf, nf V jevgr guvf) fgvyy cerfrag naq fnyvrag, ohg irel znantrnoyr. Nf fbba nf vg frggyrq, zl zbgvbaf jrag onpx gb orvat pbasvqrag, snfg, naq cerpvfr. Nsgre whfg n frpbaq, V (jvgubhg pbafpvbhfyl pubbfvat gb) fgbccrq gur ercrgvgvir unaq zbgvbaf naq ghearq gurz vagb n fgergpu, svefg sbe zl unaqf naq nezf, naq gura zl jubyr obql. Zl cbfgher jrag sebz fybhpurq gb fgenvtug-onpxrq (ohg ernfbanoyl erynkrq) naq V fcernq zl yrtf gb n zber pbzsbegnoyr cbfvgvba. V abgvprq gung zl cnagf jrer hapbzsbegnoyl gvtug naq erfgevpgvat, naq nyfb abgvprq gur cyrnfher bs tvivat zl obql n fngvfslvat fgergpu naq ernqwhfgzrag.

Guvf fgngr bs tbbq cbfgher naq obqvyl njnerarff vf snzvyvne gb fbzr qrterr, nf V bppnfvbanyyl trg vagb vg nsgre ubhe-ybat zrqvgngvba fvgf. Vg'f pbaarpgrq gb trareny tebhaqrqarff naq cerfrapr va fbpvny naq zragny ernyzf nf jryy, ohg vg'f n fgngr V unira'g orra noyr gb gevttre irel eryvnoyl.

V tbg hc gb tb gb nabgure ebbz, naq sbhaq zlfrys zbivat zber dhvpxyl naq pbasvqragyl guna V'z hfrq gb. Gurer'f n fjvatvat qbbe orgjrra gjb ebbzf, naq V fbeg bs yrnarq vagb vg jvgubhg oernxvat fgevqr. Zl zbgvbaf erzvaqrq zr n ovg bs ubj V'ir frra nguyrgrf va pnfhny fvghngvbaf zbir, jvgu n fbeg bs pbasvqrapr naq frafr gung gurve obql vf na rssrpgvir gbby. Vg nyfb orpnzr dhvgr boivbhf naq fnyvrag gung V'z uhatel, fvapr V unira'g rngra oernxsnfg.

V'ir xrcg gur ybphf bs npgvba ng gur onfr bs zl fcvar juvyr jevgvat guvf zrffntr. Vg frrzf gb pheeragyl erdhver n fznyy nzbhag bs onpxtebhaq nggragvba gb xrrc vg gurer (fvzvyne gb, yvxr, crepuvat ba n oenapu juvyr ernqvat n obbx, be fbzrguvat yvxr gung), ohg qbrfa'g frrz gb abgnoyl vagresrer jvgu guvaxvat be jevgvat. V srry trarenyyl sbphfrq naq ratntrq, gubhtu vg'f uneq gb gryy jurgure gung'f na rssrpg be whfg n znggre bs orvat vagrerfgrq va guvf rkcrevzrag.

Thank you very much for the detailed reply!

Consider the example of Mary’s room. …

I’m quite familiar with Mary’s Room, yes, so this was indeed a helpful analogy.

I can also demonstrate, in real time, the difference between me being in my body and not being in my body; there are things I can do in my body that are harder to do when I’m not in my body, and I have no idea how to convince you that I can do these things via text.

Video!

(Also, convincing me aside, do you mind saying what these things are?)

Oh, actually, here’s an exercise you can do right now … It goes like this: try to get a clear sense of where the “locus of control” of your body is. Probably it’s where your brain is by default. I visualize it as something like a blue ball of energy shooting out electricity to the rest of my body to make it move.

I don’t think I have one of these things… (I don’t visualize any blue balls of energy, at any rate…) I suppose this is one of those “fundamental experiences that some people are missing”, etc.

Anyway, I do understand what you mean, now, and if you manage to get the phenomenon into words I will certainly read it!