AidGrade - GiveWell finally has some competition

I'm also glad to see competition, and I would not be surprised at all if their overviews of the evidence were stronger than GiveWell's. It would be nice to see what they are doing, and I don't trust their results too much without that. I guess there is a book where they describe the meta-analyses they've done, which I have not had a chance to see.

The comparison with GiveWell seems mostly unreasonable, and I think reflects somewhat badly. Most of the points are either mistaken or misleading, and I would be surprised if they could be made by someone writing in good faith. [Edit: apologies for the snarky tone and inaccurate claim!]

• They suggest GiveWell doesn't understand causal attribution because Holden lists instrumental variables at the top of an inventory of methods of causal attribution. However, the list is in alphabetical order, and Holden says specifically that they rarely see compelling studies of this form.
• They criticize GiveWell for vote counting and provide a detailed description of what vote counting is. They link to GiveWell's analysis of microfinance as their only example, and suggest it makes elementary statistical errors. The procedure GiveWell is following is to look at a variety of studies with methodological failures, point out that the positive results are in the studies with the largest methodological failures, and that there is strong evidence that those failures will exaggerate the positive impact of microfinance. It is hard to mistake this for vote counting in good faith. A study which shows that eliminating methodological flaws reduces measured positive impacts does constitute evidence that undermines other methodologically flawed studies which find a positive effect. This is not the same as "accepting the null."
• AidGrade posts the comparison on their blog but doesn't take responsibility for it (disclaiming: these are the views of the author) or allow responses there.

Eva Vivalt thinks that making comparisons between outcome measures is not the place of a charity evaluator, and faults GiveWell for being willing to do so. No argument is provided for this, nor any response to the arguments GiveWell has given in favor of making such judgments.

This seems like a common and defensible position, but as an altruist concerned with aggregate welfare it doesn't make too much sense for me. Yes, there is value in producing the raw estimates of effectiveness on respective outcome measures (which GiveWell does as well), but encouraging discussion about what outcome measures are important is also a valuable public good, and certainly not an active disservice.

@Raemon: saying this is better for "people with choice paralysis or who don't have any idea how to evaluate different types of outcomes" seems to be missing the point. It is a significant, largely empirical challenge to determine which intermediate outcome measures most matter for the things we ultimately care about. Whether or not GiveWell does that passably, it is clearly something which needs to be done and which individual donors are not well-equipped to do.

The two valid points of criticism Eva raises:

• GiveWell is willing to accept data of the form "these two graphs look pretty similar" when common-sense reasoning suggests that similarity reflects a causal influence. At best, GiveWell's willingness to use such data requires donors to have more trust in them. At worst, it causes GiveWell to make mistakes. Such data was misleading once in the past, indeed in the example GiveWell cites of this approach. That said, a formal statistical analysis wouldn't have helped at all if GiveWell was willing to accept difference-of-difference estimators or matching [see the next point]. Overall this seems like a valid criticism of GiveWell, although I think GiveWell's position is defensible and has been defended, while this criticism includes no argument other than an appeal to incredulity. Indeed, even the incredulity is incorrectly described, as it applies verbatim to difference-of-difference estimators and matching as well, which economists do accept and which the author chastises GiveWell for not explicitly listing in their inventory.
• GiveWell fails to note difference-of-differences or matching in their list of methods of causal attribution. This is technically valid but a bit petty, given that GiveWell does in fact use such estimators when available. Making an error of omission in an expository blog post probably does not license the conclusion "[GiveWell is] not in a good position to evaluate studies that did not use randomization." [Edit: as Eva points out, GiveWell is in fact in a worse position to evaluate studies that don't use randomization, though I don't think the evidence presented is very relevant to this claim and I think Eva overstates it significantly.]

This is excellent! I haven't judged the quality of their evaluations or how much overlap there is between them and Givewell, but all that aside, this changes the naive-donor pitch from

"Before donating, you should find out which is the most effective charity by checking a charity evaluator such as GiveWell"

to

"Before donating, you should find out which is the most effective charity by checking a charity evaluator such as GiveWell or AidGrade"

That is, having two credible charity effectiveness evaluators makes it possible to pitch the idea of charity effectiveness evaluation without having to also pitch for a specific organization, which implicitly validates the idea and makes it less political.

I knew this would happen. Now I need a charity to tell me which of GiveWell and AidGrade is most effective!

It is good that there are more organizations in this important area. However, it seems very strange which outcomes they list.

Let's put it this way: I don't care about how many people are dying of malaria. I just don't. What I do care about is people dying, or suffering, of anything. That is why I find the attitude of AidGrade to be (almost) completely useless. The outcome I care about is maximizing QALYs, or maybe some other similar measure, and I actually don't care about the listed outcomes at all, except for as much as optimizing on them may help people not suffer and die. Basically, AidGrade tries to help with our instrumental goals, and that is well and fine, but in the end what we are trying to optimize are our terminal goals, and AidGrade doesn't help at that at all.

There's also Giving What We Can.

I'm trying to figure out how to use this site. It seems like I pick some metrics I care about - maybe height-for-age for starters. So it shows me that deworming did better than school meals at increasing height-for-age. But I don't know what either of those interventions cost. So if I have $100 that I want to use to increase height-for-age, I'm still not sure which intervention gets me more improvement for my money.

I'm not sure how to pick which metrics to care about - is height-for-age more important than weight-for-age? Neonatal deaths? Labor force participation? I guess one answer is that nobody is all that sure about which metrics to care about, and this is an approach that gives donors more data on specific interventions if they have different priorities than, say, GiveWell's.

The "portfolio" part isn't up yet, but I assume it will eventually connect me to some organizations that do those various interventions. I'll be interested to see which organizations they recommend.