Some blindspots in rationality and effective altruism

post by Remmelt Ellen (remmelt-ellen) · 2021-03-19T11:40:05.618Z · LW · GW · 32 comments

Contents

  Common brightspots: We often focus on analysing how an abstract thing will function.
  Meta-considerations
None
32 comments

Last month, Julia Galef interviewed Vitalik Buterin. Their responses to Glen Weyl’s critiques of the EA community struck me as missing perspectives he had tried to raise.

So I emailed Julia to share my thoughts. Shortened version:

Personally, I thought Vitalik’s and your commentary on Glen Weyl’s characterisation of the EA and rationality community missed something important. 

Glen spent a lot of time interacting with people from the black community and other cultural niches and asking for their perspectives. He said that he learned more from that than from the theoretical work he did before. 

To me, Glen’s criticism came across as unnuanced (eg. EAs also donate to GiveDirectly, and it’s not like we force people to take what we give them). I also resonate with that critiques of rationality and EA often seem unfair and devoid of reason. They lack specific examples and arguments relating to what the community actually does, and come across as a priori judgements of our community being cold, reductionist and weird. It’s also frustrating that such critiques can constrain EA efforts to improve the lives of others.

But Glen’s criticism hit an important point: our community wields usefully biased styles of thinking to comprehend the world and impact the lives of beneficiaries far removed from us. But we overlook the perspectives held by persons we affect, perspectives which are adapted to the contexts they live in (with ‘adapted’ I roughly mean that their perspectives are useful for navigating their surrounding environment in ways that allow them to reach opportunities and avoid hazards).

It’s hard though to discern these perspectives without hanging out and talking. 
Most rationalists, me included, have spent little time travelling overseas and immersing themselves in local cultures different from theirs. It’s also hard for Glen (or Tyler [LW · GW]) to convey perspectives unfamiliar to listeners in x minutes.

If it’s helpful, I could try and share a summary with you of views and styles of thinking that the rationality community is not much in touch with. I read about 150 psychology papers in my spare time to try and form a better understanding of our blindspots (and complementary 'brightspots').

Julia graciously replied that she was interested in a summary of what I think EAs might be missing, or how in particular our views on philanthropy might be biased due to lack of exposure to other cultures/communities.

I emailed her a summary of my upcoming sequence – about a tool I’m developing to map group blindspots. It was tough to condense 75 pages of technical explanation clearly into 7 pages, so bear with me! I refined the text further below:

Here are brightspots (vs. blindspots) that EAs and rationalists might hold in common, ie. areas we notice (vs. miss) that map to relevant aspects of reality.

Common brightspots: 
We often focus on analysing how an abstract thing will function.

This focus is narrow (compared to the portion of hypothetical space that humans are able to perceive and meta-learn from). This is because we are mapping the territory at the intersection of several views.

I think we especially focus on viewing...

  1. future (vs. past),
  2. far (vs. near vs. centrally present),
  3. precisely (vs. imprecisely) sorted
  4. structures (vs. processes) of 
  5. independent individuals (vs. the interdependent collective) that are 
  6. externally (vs. internally) present.
     
A group lights up a brightspot, near vs. far.
(kudos to my mum for this sketch)

Let me clarify each point of view, with examples of different-minded groups:

  1. We reference future possibilities
    We seem to neglect past case studies or historical accounts somewhat in predicting the effects our philanthropic actions have on beneficiaries. Some conservative cultures attach more value to passing on and studying past accounts and practices. Our community has a progressive leaning. We seem more open to letting go of past learnings and to reimagine the future. As a result, we also reinvent the wheel more. 

    (Complicating factors:
     - Over the years, EA orgs fixated more on upholding established paradigms. Community building practices are have especially become more conservative. Unconventional entrepreneurs receive less support [EA · GW].
     - I'm confusing different meanings of conservatism. Is this about being closed to unfamiliar people or things, a need to maintain purity and order, deferring to authority or tradition, preventing the loss of what you acquired in the past, updating less on novel inferences, referencing past experiences in planning..?)

     
  2. We represent far across the distances we perceive
    I. Far across more far-fetched scenarios, sequenced in time.
        eg. existential risk ends society’s long-term trajectory (vs. current civic issue)
    II. Far across places stitched in space.
        eg. global poverty (vs. local homelessness)
    III. Separated from bounded entities which neighbour us:
        inanimate objects, goal-directed beings, social partners.
        eg. pandemic containment (vs. on-the-ground fieldwork)
    IV. Decoupled from the context of individual identities we sort entities into:
        things, agents, persons.
        eg. LW on game-theoretic agents (vs. activists on rescuing a caged animal)

    Since we aim to impact the lives of persons far removed from us, we get loose or no feedback on how our actions affected our supposed beneficiaries. Instead, we rely on feedback from our social circle to correct our beliefs. We talk with trusted collaborators who understand what we aim to do. But people close to us share similar backgrounds and use similar mental styles to map and navigate their environment. 

    Worryingly, contexts to which EAs were exposed in the past (W.E.I.R.D. academia, coding, engineering, etc), and later generalised arguments from, are very dissimilar to the contexts in which their supposed beneficiaries reside (villages in low-income countries, non-human animals in factory farms, cultural and ethnic groups who will be affected by technology developments).

    Many of us seem motivated to focus and act on beliefs for doing good by a Calvinistic sense of responsibility that was impressed on us through social interactions since our childhood. We later generalised it to other hypothetical humans, out of a need to have a coherent ontology, as well as to assess value and pursue our derived goals consistently. Such underlying motivations are different from actually caring, and therefore drive a subtle wedge between the information we seek and act on, and what’s actually true, relevant, and helpful for the persons we claim to be trying to improve the lives of.

    Going by projects I've coordinated, EAs often push for removing paper conflicts of interest over attaining actual skin in the game.


     
  3. We sort entities precisely into identities
    Rationalists who detachedly and coolly model the external world (vs. say nature-loving hippies who feel warm and close to their surroundings), are guided by an aesthetic preference, is my sense:

    We distill individual identities into abstract types that are elegant and ordered, based on primary features held in common (your interview on beauty in physics highlighted these aesthetics). Distillation allows us to block out context-specific features that appear messy and noisy

    Perhaps, we perceive contexts as more messy because we sort more precisely. Rationalists have, on average, more pronounced autistic traits. People diagnosed with autism tend to precisely sort entities they notice around them within a narrower confidence band of identity. They need tighter coherence amongst instances to feel certain about them being the same. So their threshold for perceiving ambiguity is lower: a smaller deviation in an entity’s features (superficial or essentialised) will trigger ambiguity as to which identity that entity belongs to. 

    Upon entering a new context, they fixate on sorting out all these special instances. They get overwhelmed, missing the proverbial forest for the trees. But if they’re able to select primary features individuals hold in common, they can extract a signal from the noise. By distilling individuals as general elements of an ordered system, they can draw neat lines of inference between them.

    When we describe that representation of the world to a non-STEM group, it may come across as clinical, segregated, barren, and impoverished of meaning (look into how this post or Brave New World is worded). They struggle with ambiguity of a different kind – about which real-life concrete features embody such an abstract concept (what the hell does a ‘game-theoretic agent’ mean?). 

    Decoupling conflicts with their motivation to read into concrete nuances. They want to get close to, and be part of, a context that is rich, alive and interwoven
    Glen's personal quote: ❝I work to imagine, build and communicate a pluralistic future for social technology truer to the richness of our diversely shared lives.

    EAs and rationalists tend to be context-blind. We are more likely to miss subtle social cues. We are also more naively confident about our models fitting uniformly across various contexts (vs. say the peasants Luria interviewed who were hesitant to speculate about places they hadn't seen before).

    One pattern: early EA thinkers proposing explicit arguments that were elegant and ordered. Their analysis of causes and interventions was idealised – only refined later by new entrants who considered concrete applications.

    In hindsight, judgements read as simplistic and naive in similar repeating ways (relying on one metric, study, or paradigm and failing to factor in mean reversion or model error there; fixating on the individual and ignoring societal interactions; assuming validity across contexts):
     - Eliezer Yudkowsky's portrayal [LW · GW] of a single self-recursively improving AGI (later overturned disputed by some applied ML researchers)
     - Will MacAskill's claim that you can do 100x more good by giving to low-income countries
     - Toby Ord’s analysis of DCP2: 'the best of these interventions is estimated to be 1,400 times as cost-effectiveness as the least good'
     - ACE researchers’ estimate of 1.4 animals saved per vegan leaflet 
     - CEA staff recommending community building models derived from Oxford settings to all local organisers (fortunately opened up to criticism and adapted [EA · GW])

    Current EA arguments still tend to build on mutually exclusive categorisations (unlike this email :), generalise across large physical spaces and timespans, and assume underlying structures of causation that are static. Authors figure out general scenarios and assess the relative likelihood of each, yet often don't disentangle the concrete meanings and implications of their statements nor scope out the external validity of the models they use in their writing (granted, the latter are much harder to convey). Posts usually don't cover variations across concrete contexts, the relations and overlap between various plausible perspectives, or the changes in underlying dynamics much.

    RadicalxChange, on the other hand, emphasises combining relevant perspectives in their modelling, and co-creating solutions with stakeholders who are working from different contexts. I could make a case say for GiveWell doing the former (eg. cluster thinking), but not much of the latter.

     
  4. Our views are built around structures
    We tend to perceive the world as consisting of parts:
    - we represent + recognise an observation to be a fixed cluster (a structure)
       eg. an observation’s recurrence, a place, a body, a stable identity
    - we predict + update on the possible location of this structure
       eg. how likely it ends up present within some linear sequence, geographic surface, physical boundaries, or levels of feature abstraction

    Our structure-based view is a reflection of our Westernised culture. English sentence descriptions centre around the subject, object, and adjectives. Westerners also often perceive active causation as the inherent causal feature of something or someone (eg. a mechanical function, a growth gene, or an aggressive personality).

    Conversely, some traditional cultures foster a process-based view. Things are perceived as impermanent and ever-changing (see (action) verb-based Native American languages, or this wacky interpretation of Dzogchen philosophy).
    - they represent + recognise an observed change to be a trajectory in presence (a process)
     eg. a transition, movement, interaction, relation
    - they predict + update on whenever, wherever, and so forth, this process may initiate again

    We tend to perceive the world as consisting of parts:
    A. we represent + recognise an observation to be a fixed cluster (a structure)
         eg. an observation’s recurrence, a place, a body, a stable identity
    B. we predict + update on the possible location of this structure
         eg. how likely it ends up present within some linear sequence, geographic surface, physical boundaries, or levels of feature abstraction

    Our structure-based view is a reflection of our Westernised culture. English sentence descriptions centre around the subject, object, and adjectives. Westerners also often perceive active causation as the inherent causal feature of something or someone (eg. a mechanical function, a growth gene, or an aggressive personality).

    Conversely, some traditional cultures foster a process-based view. Things are perceived as impermanent and ever-changing (see (action) verb-based Native American languages, or this wacky interpretation of Dzogchen philosophy).
    A. they represent + recognise an observed change to be a trajectory in presence    (a process)
         eg. a transition, movement, interaction, relation
    B. they predict + update on whenever, wherever, and so forth, this process may initiate again
    __________________________________________________________
    Edits interlude: 
    People asked for examples of focussing on (interpreting & forecasting) processes vs. structures.
    See here.

    This links to more speculative brightspot-blindspot distinctions:
    7. Trading off sensory groundedness vs. representational stability of 
    believed aspects:
    - in learning structure: a direct observation's recurrence vs. sorted identity
    - in learning process: a transition of observed presence vs. analogised relation 

    8. Trading off updating your interpretations vs. forecasts:
    A. Interpret (recognise + represent) aspects
         eg. classical archaeologists focus on differentiated recognition of artefacts, 
         linguistic anthropologists on representing differentiated social contexts 
    B. Forecast (sample + predict) aspects
         eg. development economists focus more on calibrated sampling of metrics, 
         global prio scholars on calibrating their predictions of distilled scenarios 

    9. Gain vs. loss motivated focus [LW · GW
        (attain more positive vs. remove more negative valence)
    __________________________________________________________
     
  5. We view individuals as independent
    Since we prefer sorting different things into ordered and mutually exclusive categories, we aren't as aware of the relations between them (besides maybe endorsing this as a fact of reality upon reflection). In our attempts to carve nature at its joints, we neglect the ways persons and things are actually interdependent. That is, we do this even more strongly than broader Western individualist society (vs. say Asian collectivist cultures). 

    Quoting from the paper Culture and the Self:
    ❝Western view of the individual as an independent, self-contained, autonomous entity who
     (a) comprises a unique configuration of internal attributes (eg., traits, abilities, motives, and values) and
     (b) behaves primarily as a consequence of these internal attributes
    ❝Experiencing interdependence entails seeing oneself as part of an encompassing social relationship and recognizing that one's behavior is determined, contingent on, and, to a large extent organized by what the actor perceives to be the thoughts, feelings, and actions of others in the relationship.

    The RadicalxChange community recognises the interdependence of people as part of a collective whole. They attract members who often take up a more interdependent culture or mindset (social scientists, artists, African-Americans, women). To be fair, this might also be because RxC engages minority groups more, who in turn feel empowered to contribute to mechanisms that can overcome systemic exploitation.

    Revisiting Glen's critiques, I interpret one basis to be our neglect of social interconnectedness: 
    ❝We’re not going to pay that much attention to getting feedback from the people whose lives that affects or being in conversation with them.
    ❝cloistering themselves into a room
    ❝You have to think of the things you’re doing as speech acts and not purely intellectual acts... They condition a certain sort of a society.
    ❝a lot of the overly field experiment driven, effective ways of charitable giving that didn’t think about broader social structures and effects of that sort of stuff
    ❝the actual power that people derive in a decentralized way almost always comes from their ability to act collectively. It’s almost never possible on your own to exercise much power.


     
  6. We view things as being external to us
    In conversations, EAs and rationalists often attempt to convey a more impartial or objective view of the external world. This leads us to disregard personal interpretations when those are actually relevant (eg. for supporting a collaborator, or considering a friend's needs and constraints when advising them on their career options).

    In terms of individual persons, we could do worse though. Though people can be socially awkward, they do check in and consider how their conversation partner is feeling. Some of us are also really into introspection techniques and self-awareness (eg. rationalists writing about meditation experiences, Qualia Research Institute).  

    But since we focus more on the individual than the interdependent collective, we're particularly unaware of the cultural feeling of our community, as well as any broader social repercussions of our actions.  

    Examples:
    - cases where 80,000 Hours staff didn’t catch on to the effects that their general career recommendations were having on the broader EA community (they've now more clearly specified the scope of their goals and whom they can serve)
    - criticism around EAs neglecting the effects of what they say and do on societal norms (eg. Rob Reich's criticism of GiveWell)
    - LessWrong members who felt unwelcome and lonely, prompting Project Hufflepuff's start [LW · GW] (effective animal advocacy meetups though are more warm and cozy in my experience; one organiser shared more or less with me that outsiders see them as cutesy Hufflepuffs)
     

Meta-considerations

  • Personal motivations matter because it guides how people frame their focus, pursue strategies, and assess aims. For example, x-risk leaders and s-risk managers may be predisposed to and socially reinforced to set goals differently, going by my stereotyped impressions:
     - Eliezer and Nick: to eagerly leap towards attaining their ideal for a more positive world state (powering over obstacles, towards the possible presence of a gain). Each originally led the start-up phase of a techno-optimistic institute. But they realised they needed to go pioneer research to prevent AGI misalignment in order to not sadly miss out on utopia
     - Center on Long-Term Risk: to vigilantly maintain their responsibility to prevent a more negative world state (shut off any intrusion of a potential loss, to control its absence). Founders originally met through discussions of moral philosophy, infused with German Weltschmerz. Managing secure, incremental charity operations is their core competence. But they're prepared to promote innovative start-up practices in order to relieve the universe from pessimistic scenarios of suffering.
  • Non-duality: both the internal (now, here, this, my) and the external (then, there, that, them) are located within present awareness.



If you got this far, I’m interested to hear your thoughts!  Do grab a moment to call so we can chat about and clarify the concepts. Takes some back and forth.

32 comments

Comments sorted by top scores.

comment by Rafael Harth (sil-ver) · 2021-03-19T21:06:26.825Z · LW(p) · GW(p)
  • Eliezer Yudkowsky's portrayal of a single self-recursively improving AGI (later overturned by some applied ML researchers)

I've found myself doubting this claim, so I've read the post in question. As far as I can tell, it's a reasonable summary of the fast takeoff position that many people still hold today. If all you meant to say was that there was disagreement, then fine -- but saying 'later overturned' makes it sound like there is consensus, not that people still have the same disagreement they've had 13 years ago. (And your characterization in the paragraph I'll quote below also gives that impression.)

In hindsight, judgements read as simplistic and naive in similar repeating ways (relying on one metric, study, or paradigm and failing to factor in mean reversion or model error there; fixating on the individual and ignoring societal interactions; assuming validity across contexts):

Replies from: remmelt-ellen, remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-19T21:29:54.952Z · LW(p) · GW(p)

Sorry, I get how the bullet point example gave that impression. I'm keeping the summary brief, so let me see what I can do. 

I think the culprit is 'overturned'. That makes it sound like their counterarguments were a done deal or something. I'll reword that to 'rebutted and reframed in finer detail'. 

Note though that 'some applied ML researchers' hardly sounds like consensus. I did not mean to convey that, but I'm glad you picked it up.

As far as I can tell, it's a reasonable summary of the fast takeoff position that many people still hold today.

Perhaps, your impression from your circle is different from mine in terms of what proportion of AIS researchers prioritise work on the fast takeoff scenario?

Replies from: sil-ver
comment by Rafael Harth (sil-ver) · 2021-03-20T17:39:31.099Z · LW(p) · GW(p)

I think the culprit is 'overturned'. That makes it sound like their counterarguments were a done deal or something. I'll reword that to 'rebutted and reframed in finer detail'.

Yeah, I think overturned is the word I took issue with. How about 'disputed'? That seems to be the term that remains agnostic about whether there is something wrong with the original argument or not.

Perhaps, your impression from your circle is different from mine in terms of what proportion of AIS researchers prioritise work on the fast takeoff scenario?

My impression is that gradual takeoff has gone from a minority to a majority position on LessWrong, primarily due to Paul Christiano, but not an overwhelming majority. (I don't know how it differs among Alignment Researchers.)

I believe the only data I've seen on this was in a thread where people were asked to make predictions about AI stuff, including takeoff speed and timelines, using the new interactive prediction feature. (I can't find this post -- maybe someone else remembers what it was called?) I believe that was roughly compatible with the sizeable minority summary, but I could be wrong.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-20T18:12:13.554Z · LW(p) · GW(p)

How about 'disputed'?

Seems good. Let me adjust!

My impression is that gradual takeoff has gone from a minority to a majority position on LessWrong, primarily due to Paul Christiano, but not an overwhelming majority

This roughly corresponds with my impression actually. 
I know a group that has surveyed researchers that have permission to post on the AI Alignment Forum, but they haven't posted an analysis of the survey's answers yet.

comment by Remmelt Ellen (remmelt-ellen) · 2021-03-20T09:47:20.635Z · LW(p) · GW(p)

To disentangle what I had in mind when I wrote ‘later overturned by some applied ML researchers’:

Some applied ML researchers in the AI x-safety research community like Paul Christiano, Andrew Critch, David Krueger, and Ben Garfinkel have made solid arguments towards the conclusion that Eliezer’s past portrayal of a single self-recursively improving AGI had serious flaws.

In the post though, I was sloppy in writing about this particular example, in a way that served to support the broader claims I was making.

comment by romeostevensit · 2021-03-19T21:15:03.153Z · LW(p) · GW(p)

On 3, I'd like to see EA take sensitivity analysis more seriously.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-19T21:48:42.529Z · LW(p) · GW(p)

This resonates, based on my very limited grasp of statistics. 

My impression is that sensitivity analysis aims more at reliably uncovering epistemic uncertainty (whereas Guesstimate as a tool seems to be designed more for working out aleatory uncertainty). 

Quote from interesting data science article on Silver-Taleb debate:

Predictions have two types of uncertainty; aleatory and epistemic.
Aleatory uncertainty is concerned with the fundamental system (probability of rolling a six on a standard die). Epistemic uncertainty is concerned with the uncertainty of the system (how many sides does a die have? And what is the probability of rolling a six?). With the latter, you have to guess the game and the outcome; like an election!

Replies from: viktor.rehnberg
comment by viktor.rehnberg · 2021-03-22T15:09:46.394Z · LW(p) · GW(p)

This is my go-to figure when thinking about aleatoric vs epistemic uncertainty.

Source: Michel Kana through Olof Mogren
Edit: Explaining two types of uncertainty through a simple example. x are inputs and y are labels to a model to be trained. 

Edit: In the context of the figure. The aleatoric uncertainty is high in the left cluster because the uncertainty of where a new data point will be is high and is not reduced by the number of training examples. The epistemic uncertainty is high in regions where there is insufficient data or knowledge to produce an accurate estimate of the output, this would go down with more training data in these regions.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-24T14:13:39.309Z · LW(p) · GW(p)

Looks cool, thanks! Checking if I understood it correctly:
- is x like the input data?
- could y correspond to something like the supervised (continuous) labels of a neural network, which inputs are matched too?
- does epistemic uncertainty here refer to that inputs for x could be much different from the current training dataset if sampled again (where new samples could turn out be outside of the current distribution)?
 

Replies from: viktor.rehnberg
comment by viktor.rehnberg · 2021-04-12T09:11:44.069Z · LW(p) · GW(p)

Thanks, I realised that I provided zero context for the figure. I added some.

- is x like the input data?
- could y correspond to something like the supervised (continuous) labels of a neural network, which inputs are matched too?

Yes. The example is about estimating y given x where x is assumed to be known.

- does epistemic uncertainty here refer to that inputs for x could be much different from the current training dataset if sampled again (where new samples could turn out be outside of the current distribution)?

Not quite, we are still thinking of uncertainty only as applied to y. Epistemic uncertainty here refers to regions where the knowledge and data is insufficient to give a good estimate y given x from these regions.

To compare it with your dice example, consider x to be some quality of the die such that you think dies with similar x will give similar rolls y. Then aleatoric uncertainty is high for dies where you are uncertain for values of new rolls even after having rolled several similar dies and rolling more similar dies will not help. While epistemic uncertainty is high for dies with qualities you haven't seen enough of.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-04-13T11:04:29.334Z · LW(p) · GW(p)

Thank you! That was clarifying especially the explanation of epistemic uncertainty for y. 

1. I've been thinking about epistemic uncertainty more in terms of 'possible alternative qualities present', where 

  • you don't know the probability of a certain quality being present for x (e.g. what's the chance of the die having an extended three-sided base?).
  • or might not even be aware of some of the possible qualities that x might have (e.g. you don't know a triangular prism die can exist).

2. Your take on epistemic uncertainty for that figure seems to be

  • you know of x's possible quality dimensions (e.g. relative lengths and angles of sides at corners).
  • but given a set configuration of x (e.g. triangular prism with equilateral triangle sides = 1, rectangular lengths = 2 ), you don't know yet the probabilities of outcomes for y (what's the probability of landing face up for base1, base2, rect1, rect2, rect3?).
     

Both seem to fit the definition of epistemic uncertainty. Do correct me here!
 


Edit: Rough difference in focus: 
 1. Recognition and Representation
      vs. 
 2. Sampling and Prediction

Replies from: viktor.rehnberg
comment by viktor.rehnberg · 2021-04-13T15:19:45.748Z · LW(p) · GW(p)

Good point, my example with the figure is lacking in regards to 1 simply because we are assuming that x is known completely and that the observed y are true instances of what we want to measure. And from this I realize that I am confused about when some uncertainties should be called aleagoric or epistemic.
 

When I think I can correctly point out epistemic uncertainty:

  • If the y that are observed are not the ones that we actually want then I'd call this uncertainty epistemic. This could be if we are using tired undergrads to count the number of pips of each rolled die and they miscount for some fraction of the dice.
  • If you haven't seen similar x before then you have epistemic uncertainty because you have uncertainty about which model or model parameters to use when estimating y. (This is the one I wrote about previously and the one shown in the figure)

My confusion from 1:

  • If the conditions of the experiment changes. Our undergrads start to pull dice from another bag with an entirely different distribution p(y|x), then we have insufficient knowledge to estimate y and I would call this epistemic uncertainty. 
  • If x is lacking in some information to do good estimates of y. x is the color of the die and when we have thrown enough dice from our experimental distribution we get a good estimate of p(y|x) and our uncertainty doesn't increase with more rolls, which makes me think that it is aleatoric uncertainty. But on the other hand x is not sufficient to spot when we have a new type of die (see previous point) and if we knew more about the dice we could do better estimates which makes me think that it is epistemic uncertainty.

 

You bring up a good point in 1 and I agree that this feels like it should be epistemic uncertainty, but at some point the boundary between inherent uncertainty in the process and uncertainty from knowing too little about the process becomes vague to me and I can't really tell when a process is aleatoric or epistemic.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-04-13T15:51:22.579Z · LW(p) · GW(p)

I also noticed I was confused. Feels like we're at least disentangling cases and making better distinctions here.
BTW, just realised that a problem with my triangular prism example is that theoretically no will rectangular side can face up parallel to the floor at the same time, just two at 60º angles).

But on the other hand x is not sufficient to spot when we have a new type of die (see previous point) and if we knew more about the dice we could do better estimates which makes me think that it is epistemic uncertainty.

This is interesting. This seems to ask the question 'Is a change in the quality of x like colour actually causal to outcomes y?' Difficulty here is that you can never fully be certain empirically, just get closer to [change in roll probability] for [limit number of rolls -> infinity] = 0.

Replies from: viktor.rehnberg, viktor.rehnberg
comment by viktor.rehnberg · 2021-04-14T11:25:10.609Z · LW(p) · GW(p)

To disentangle the confusion I took a look around about a few different definitions [LW · GW] of the concepts. The definitions were mostly the same kind of vague statement of the type:

  • Aleatoric uncertainty is from inherent stochasticity and does not reduce with more data.
  • Epistemic uncertainty is from lack of knowledge and/or data and can be further reduced by improving the model with more knowledge and/or data.

However, I found some useful tidbits

Uncertainties are characterized as epistemic, if the modeler sees a possibility to reduce them by gathering more data or by refining models. Uncertainties are categorized as aleatory if the modeler does not foresee the possibility of reducing them. [Aleatory or epistemic? Does it matter?]

 

Which sources of uncertainty, variables, or probabilities are labelled epistemic and which are labelled aleatory depends upon the mission of the study. [...] One cannot make the distinction between aleatory and epistemic uncertainties purely through physical properties or the experts' judgments. The same quantity in one study may be treated as having aleatory uncertainty while in another study the uncertainty maybe treated as epistemic. [Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management]

 

In the context of machine learning, aleatoric randomness can be thought of as irreducible under the modelling assumptions you've made. [The role of epistemic vs. aleatory uncertainty in quantifying AI-Xrisk [LW · GW]]

 

[E]pistemic uncertainty means not being certain what the relevant probability distribution is, and aleatoric uncertainty means not being certain what a random sample drawn from a probability distribution will be. [Uncertainty quantification]

With this my updated view is that our confusion is probably because there is a free parameter in where to draw the line between aleatoric and epistemic uncertainty.

This seems reasonable as more information can always lead to better estimates (at least down to considering wavefunctions I suppose) but in most cases having this kind of information and using it is infeasible and thus having the distinction between aleatoric and epistemic depend on the problem at hand seems reasonable.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-04-15T23:06:54.367Z · LW(p) · GW(p)

This is clarifying, thank you!

comment by viktor.rehnberg · 2021-04-14T11:14:04.529Z · LW(p) · GW(p)

Good catch

This seems to ask the question 'Is a change in the quality of x like colour actually causal to outcomes y?'

Yes, I think you are right. Usually when modeling you can learn correlations that are useful for predictions but if the correlations are spurious they might disappear when the distributions changes. As such to know if p(y|x) changes from only observing x, then we would probably need that all causal relationships to y are captured in x?

comment by Dan Weinand (dan-weinand) · 2021-03-19T20:31:13.876Z · LW(p) · GW(p)

I found it immensely refreshing to see valid criticisms of EA. I very much related to the note that many criticisms of EA come off as vague or misinformed. I really appreciated that this post called out specific instances of what you saw as significant issues, and also engaged with the areas where particular EA aligned groups have already taken steps to address the criticisms you mention.

I think I disagree on the degree to which EA folks expect results to be universal and generalizable (this is in response to your note at the end of point 3). As a concrete example, I think GiveWell/EAers in general would be unlikely to agree that GiveDirectly style charity would have similarly sized benefits in the developed world (even if scaled to equivalent size given normal incomes in the nation) without RCTs suggesting as much. I expect that the evidence from other nations would be taken into account, but the consensus would be that experiments should be conducted before immediately concluding large benefits would result.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-19T21:11:42.209Z · LW(p) · GW(p)

I appreciate your thoughtful comment too,  Dan.

You're right I think that I overstated EA's tendency to assume generalisability, particularly when it comes to testing interventions in global health and poverty (though much less so when it comes to research in other cause areas). Eva Vivalt's interview with 80K, and more recent EA Global sessions discussing the limitations of the randomista approach are examples. Some incubated charity interventions by GiveWell also seemed to take a targeted regional approach (e.g. No Lean Season [EA · GW]). Also, Ben Kuhn's 'local context plus high standards theory' for Wave. So point taken!

I still worry about EA-driven field experiments relying too much, too quickly on filtering experimental observations through quantitive metrics exported from Western academia. In their local implementation, these metrics may either not track the aspects we had in mind, or just not reflect what actually exists and/or is relevant to people's local context there. I haven't heard yet about EA founders who started out by doing open qualitative fieldwork on the ground (but happy to hear examples!). 

I assume generalisability of metrics would be less of a problem for medical interventions like anti-malaria nets and deworming tablets. But here's an interesting claim I just came across:  

One-size-fits-all doesn’t work and the ways medicine affects people varies dramatically. 

With schistosomiasis we found that fisherfolk, who are the most likely to be infected, were almost entirely absent from the disease programme and they’re the ones defecating and urinating in the water, spreading the disease.

Replies from: dan-weinand
comment by Dan Weinand (dan-weinand) · 2021-03-19T21:21:30.248Z · LW(p) · GW(p)

Fair points!

I don't know if I'd consider JPAL directly EA, but they at least claim to conduct regular qualitative fieldwork before/after/during their formal interventions (source from Poor Economics, I've sadly forgotten the exact point but they mention it several times). Similarly, GiveDirectly regularly meets with program participants for both structured polls and unstructured focus groups if I recall correctly. Regardless, I agree with the concrete point that this is an important thing to do and EA/rationality folks are less inclined to collect unstructured qualitative feedback than its importance deserves.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-19T21:36:10.041Z · LW(p) · GW(p)

Interesting, I didn't know GiveDirectly ran unstructured focus groups, nor that JPAL does qualitative interviews at various stages of testing interventions.  Adds a bit more nuance to my thoughts, thanks! 

Replies from: dan-weinand
comment by Dan Weinand (dan-weinand) · 2021-03-19T22:04:26.112Z · LW(p) · GW(p)

One of GiveDirectly's blog posts on survey and focus group results, by the way.
https://www.givedirectly.org/what-its-like-to-receive-a-basic-income/

comment by mike_hawke · 2021-03-22T03:47:55.922Z · LW(p) · GW(p)

we also reinvent the wheel more.

Could you elaborate on this? Which wheels are you thinking of?

Replies from: remmelt-ellen, RayTaylor
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-24T14:58:30.362Z · LW(p) · GW(p)

This is a good question hmm. Now I’m trying to come up with specific concrete cases, I actually feel less confident of this claim.

Examples that did come to mind:

  1. I recall reading somewhere about early LessWrong authors reinventing concepts that were already worked out before in philosophic disciplines (particularly in decision theory?). Can't find any post on this though.
     
  2. More subtly, we use a lot of jargon. Some terms were basically imported from academic research (say into cognitive biases) and given a shiny new nerdy name that appeals to our incrowd.  In the case of CFAR, I think they were very deliberate about renaming some concepts, also to make them more intuitive for workshops participants (eg. implementation intentions -> trigger action plans/patterns, pre-mortem -> murphijitsu). 

    (After thinking about this, I called with someone who is doing academic research on Buddhist religion. They independently mentioned LW posts on 'noticing', which basically is a new name for a mediation technique that has been practiced for millennia.)

    Renaming is not reinventing of course, but the new terms do make it harder to refer back to sources from established research literature.  Further, some smart amateur blog authors like to synthesise and intellectually innovate upon existing research (eg. see Scott Alexander's speculative posts, or my post above ^^). 

    The lack of referencing while building up innovations can cause us to misinterpret and write stuff that poorly reflects previous specialist research. We're building up our own separated literature database.

    A particular example is Robin Hanson 'near-far mode', from a concise and well-articulated review paper about psychological distance to the community, which spawned a lot of subsequent posts [? · GW] about implications for thinking in the community (but with little referencing to other academic studies or analyses). 
    E.g. Hanson's idea that people are hypocritical when they signal high-construal values but are more honest when they think concretely – a psychology researcher who seems rigorously minded said to me that he dug into Hanson's claim but that conclusions from other studies don’t support this.
     
  3. My impression from local/regional/national EA community building is that a many organisers (including me [EA · GW]) either tried to work out how to run their group from first principles, or consulted with other more experienced organisers.  We could also have checked for good practices from and consulted with other established youth movements. I have seen plenty of write-ups that go through the former, but little or none of the other.
     

Definitely give me counter-examples!

Replies from: Wei_Dai
comment by Wei_Dai · 2021-06-12T17:21:39.454Z · LW(p) · GW(p)

I recall reading somewhere about early LessWrong authors reinventing concepts that were already worked out before in philosophic disciplines (particularly in decision theory?). Can't find any post on this though.

See Eliezer’s Sequences and Mainstream Academia [LW · GW] and scroll down for my comment there. Also https://www.greaterwrong.com/posts/XkNXsi6bsxxFaL5FL/ai-cooperation-is-already-studied-in-academia-as-program [LW · GW] (According to these sources, AFAWK, at least some of the decision theory ideas developed on LW were not worked out already in academia.)

comment by RayTaylor · 2021-06-11T22:23:15.979Z · LW(p) · GW(p)

EAs invented neither effectiveness nor altruism, as Buddha, Quakers, Gandhi, Elizabeth Fry, Equiano and many others can attest!

EAs tend to be slow/behind the curve on coms and behavioural science and Implementation Science, and social science/realpolitik in general. But they do learn over time.

comment by Timothy Underwood (timothy-underwood-1) · 2021-03-20T13:24:49.398Z · LW(p) · GW(p)

The way I've tended to think about these sorts of questions is to see a difference between the global portfolio of approaches, and our personal portfolio of approaches.

A lot of the criticisms of EA as being too narrow, and neglecting certain types of evidence or ways of thinking make far more sense if we see EA as hoping to become the single dominant approach to charitable giving (and perhaps everything else), rather than as a particular community which consists of particular (fairly similar) individuals who are pushing particular approaches to doing good that they see as being ignored by other people.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-20T17:57:56.868Z · LW(p) · GW(p)

Yeah, seems awesome for us to figure out where we fit within that global portfolio! Especially in policy efforts, that could enable us to build a more accurate and broadly reflective consensus to help centralised institutions improve on larger-scale decisions they make (see a general case [EA · GW] for not channeling our current efforts towards making EA the dominant approach to decision-making). 

To clarify, I hope this post helps readers become more aware of their brightspots (vs. blindspots) that they might hold in common with like-minded collaborators – ie. areas they notice (vs. miss) that map to relevant aspects of the underlying territory. 

I'm trying to encourage myself and the friends I collaborate with to build up an understanding of alternative approaches that outside groups take up (ie. to map and navigate their surrounding environment), and where those approaches might complement ours. Not necessarily for us to take up more simultaneous mental styles or to widen our mental focus or areas of specialisation. But to be able to hold outside groups' views so we get roughly where they are coming from, can communicate from their perspective, and form mutually beneficial partnerships.

More fundamentally, as human apes, our senses are exposed to an environment that is much more complex than just us. So we don't have the capacity to process our surroundings fully, nor to perceive all the relevant underlying aspects at once. To map the environment we are embedded in, we need robust constraints for encoding moment-to-moment observations, through layers of inductive biases, into stable representations.

Different-minded groups end up with different maps. But in order to learn from outside critics of EA, we need to be able to line up our map better with theirs. 

 



Let me throw an excerpt from an intro draft on the tool I'm developing. Curious for your thoughts!

Take two principles for a collaborative conversation in LessWrong and Effective Altruism:

  1. Your map is not the territory: 
    Your interlocutor may have surveyed a part of the bigger environment that you haven’t seen yet. Selfishly ask for their map, line up the pieces of their map with your map, and combine them to more accurately reflect the underlying territory.
  2. Seek alignment:
    Rewards can be hacked. Find a collaborator whose values align with your values so you can rely on them to make progress on the problems you care about.

When your interlocutor happens to have a compatible map and aligned values, such principles will guide you to learn missing information and collaborate smoothly. 

On the flipside, you will hit a dead end in your new conversation when:

  1. you can’t line up their map with yours to form a shared understanding of the territory. 
    Eg. you find their arguments inscrutable.
  2. you don’t converge on shared overarching aims for navigating the territory.
    Eg. double cruxes tend to bottom out at value disagreements.
     

You can resolve that tension with a mental shortcut: 
When you get confused about what they mean and fundamentally disagree on what they find important, just get out of their way. Why sink more of your time into a conversation that doesn’t reveal any new insights to you? Why risk fuelling a conflict?

This makes sense, and also omits a deeper question: why can’t you grasp their perspective? 
Maybe they don’t think the same things through as rigorously as you, and you pick up on that. Maybe they dishonestly express their beliefs or preferences, and you pick up on that. Maybe they honestly shared insights that you failed to pick up on.

Underlying each word you exchange is your perception of the surrounding territory ...
A word’s common definition masks our perceptual divide. Say you and I both look at the same thing and agree which defined term describes it. Then, we can mention this term as a pointer to what we both saw.  Yet, the environment I perceive that I point the term to may be very different from the environment you perceive.

Different-minded people can illuminate our blindspots.  Across the areas they chart and the paths they navigate lie nuggets – aspects of reality we don’t even know yet that we will come to care about.

Replies from: timothy-underwood-1
comment by Timothy Underwood (timothy-underwood-1) · 2021-03-22T19:19:00.151Z · LW(p) · GW(p)

Yeah, I really like this idea -- at least in principle. The idea of looking for value agreement and where do our maps (that likely are verbally extremely different) match is something that I think we don't do nearly enough.

To get at what worries me about some of the 'EA needs to consider other viewpoints discourse' (and not at all about what you just wrote, let me describe two positions:

  1. EA needs to get better at communicating with non EA people, and seeing the ways that they have important information, and often know things we do not, even if they speak in ways that we find hard to match up with concepts like 'bayesian updates' or 'expected value' or even 'cost effectiveness'. 
  2. EA needs to become less elitist, nerdy, jargon laden and weird so that it can have a bigger impact on the broader world.

I fully embrace 1, subject to constraints about how sometimes it is too expensive to translate an idea into a discourse we are good at understanding, how sometimes we have weird infohazard type edge cases and the like.

2 though strikes me as extremely dangerous.

To make a metaphor: Coffee is not the only type of good drink, it is bitter and filled with psychoactive substances that give some people heart palpitations. That does not mean it would be a good idea to dilute coffee with apple juice so that it can appeal to people who don't like the taste of coffee and are caffeine sensitive.

The EA community is the EA community, and it currently works (to some extent), and it currently is doing important and influential work. Part of what makes it work as a community is the unifying effect of having our own weird cultural touchstones and documents. The barrier of excluisivity created by the jargon and the elitism, and the fact that it is one of the few spaces where the majority of people are explicit utilitarians is part of what makes it able to succeed (to the extent it does). 

My intuition is that an EA without all of these features wouldn't be a more accessible and open community that is able to do more good in the world. My intuition is an EA without those features would be a dead community where everyone has gone on to other interests and that therefore does no good at all.

Obviously there is a middle ground -- shifts in the culture of the community that improve our pareto frontier of openness and accessibility while maintaing community cohesion and appeal.

However, I don't think this worry is what you actually were talking about. I think you really were focusing on us having cognitive blindspots, which is obviously true, and important. 

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-03-24T15:26:00.711Z · LW(p) · GW(p)

Well-written! Most of this definitely resonates for me.

Quick thoughts:

  • Some of the jargon I've heard sounded plain silly from a making-intellectual-progress-perspective (not just implicit aggrandising). Makes it harder to share our reasoning, even to each other, in a comprehensible, high-fidelity way. I like Rob Wiblin's guide on jargon.
  • Perhaps we put too much emphasis on making explicit communication comprehensible. Might be more fruitful to find ways to recognise how particular communities are set up to be good at understanding or making progress in particular problem niches, even if we struggle to comprehend what they're specifically saying or doing.

(I was skeptical about the claim 'majority of people are explicit utilitarians' – i.e. utilitarian not just consequentialist or some pluralistic mix of moral views  – but EA Survey responses [EA(p) · GW(p)]seems to back it up: ~70% utilitarian)

comment by Remmelt Ellen (remmelt-ellen) · 2021-05-15T09:14:46.419Z · LW(p) · GW(p)

Two people asked me to clarify this claim:

Going by projects I've coordinated, EAs often push for removing paper conflicts of interest over attaining actual skin in the game.

Copying over my responses:

re: Conflicts of interest:

My impression has been that a few people appraising my project work looked for ways to e.g. reduce Goodharting, or the risk that I might pay myself too much from the project budget. Also EA initiators sometimes post a fundraiser write-up for an official project with an official plan, that somewhat hides that they're actually seeking funding for their own salaries to do that work (the former looks less like a personal conflict of interest *on paper*).

re: Skin in the game:

Bigger picture, the effects of our interventions aren't going to affect us in a visceral and directly noticeable way (silly example: we're not going to slip and fall from some defect in the malaria nets we fund). That seems hard to overcome in terms of loose feedback from far-away interventions, but I think it's problematic that EAs also seem to underemphasise skin in the game for in-between steps where direct feedback is available. For example, EAs seem sometimes too ready to pontificate (me included) about how particular projects should be run or what a particular position involves, rather than rely on the opinions/directions of an experienced practician who would actually suffer the consequences of failing (or even be filtered out of their role) if they took actions that had negative practical effects for them. Or they might dissuade someone from initiating an EA project/service that seems risky to them in theory, rather than guide the initiator to test it out locally to constrain or cap the damage.

Replies from: remmelt-ellen
comment by Remmelt Ellen (remmelt-ellen) · 2021-05-15T09:22:21.534Z · LW(p) · GW(p)

This interview with Jacqueline Novogratz from Acumen Fund covers some practical approaches to attain skin in the game.

comment by Remmelt Ellen (remmelt-ellen) · 2021-05-01T13:42:55.229Z · LW(p) · GW(p)

Some further clarification and speculation:

Edits interlude: 
People asked for examples of focussing on (interpreting & forecasting) 
processes vs. structures. 
See here.

This links to more speculative brightspot-blindspot distinctions:
7. Trading off sensory groundedness vs. representational stability of 
believed aspects
- in learning structure: a direct observation's recurrence vs. sorted identity
- in learning process: a transition of observed presence vs. analogised relation 

8. Trading off updating your interpretations vs. forecasts:
A. Interpret (recognise + represent) aspects
     eg. classical archaeologists focus on differentiated recognition of artefacts, 
     linguistic anthropologists on representation of differentiated social contexts 
B. Forecast (sample + predict) aspects
     eg. development economists focus more on calibrated sampling of metrics, 
     global prio scholars on calibrating their predictions of distilled scenarios 

9. Gain vs. loss motivated focus [LW · GW
    (attain more positive vs. remove more negative valence)