No standard metric for CFAR workshops?

(This is Dan from CFAR)

CFAR's 2015 Longitudinal Study measured the Big 5 and some other standard psychology metrics. It did find changes including decreased neuroticism and increased conscientiousness.

Here’s a sketch of my thinking about the usefulness of metrics like the Big 5 for what CFAR is trying to do.

It would be convenient if there was a definitive measure of a person’s rationality which closely matched what we mean by the term and was highly sensitive to changes. But as far as I can tell there isn’t one, and there isn’t likely to be one anytime soon. So we rely on a mix of indicators, including some that are more like systematic metrics, some that are more like individuals’ subjective impressions, and some that are in between.

I think of the established psychology metrics (Big 5, life satisfaction, general self-efficacy, etc.) as primarily providing a sanity check on whether the workshop is doing something, along with a very very rough picture of some of what it is doing. They are quantitative measures that don’t rely on staff members’ subjective impressions of participants, they have been validated (at least to some extent) in existing psychology research, and they seem at least loosely related to the effects that CFAR hopes to have. And, compared to other ways of evaluating CFAR’s impact on individuals, they’re relatively easy for an outsider to make sense of.

A major limitation of these established psychology metrics is that they haven’t been that helpful as feedback loops. One of the main purposes of a metric is to provide input into CFAR’s day-to-day and workshop-to-workshop efforts to develop better techniques and refine the workshop. That is hard to do with metrics like the ones in the longitudinal study, because of a combination of a few factors:

1. The results aren’t available until several months after the workshop, which would make for very slow feedback loops and iteration.
2. The results are too noisy to tell if changes from one workshop to the next are just random variation. It takes several workshops worth of data to get a clear signal on most of the metrics.
3. These metrics are only loosely related to what we care about. If a change to the workshop leads to larger increases in conscientiousness that does not necessarily mean that we want to do it, and when a curriculum developer is working on a class they are generally not that interested in these particular metrics.
4. These metrics are relatively general/coarse indicators of the effect of the workshop as a whole, not tied to particular inputs. So (for example) if we make some changes to the TAPs class and want to see if the new version of the class works better or worse, there isn’t a metric that isolates the effects of the TAPs class from the rest of the workshop.

I suppose of late I’ve updated that pre-paradigmatic research is the very sort where you’re trying to come up with a metric to validate, not the type of research where you start with one. Note that if a project claims to be doing this kind of work, it means you need a way larger amount of evidence to determine that the project is doing useful things, and your prior will always be very low.

In the particular case, I expect that if CFAR did put in the work to measure e.g. big five changes, even if they found a notable effect size they’d say “Well, this doesn’t confirm for me that we’re on the right track, because this isn’t at all what I’m trying to measure” or something.

Added: Though I too would be really interested to know if CFAR workshops lead to any substantial changes in ravens matrices, and to a lesser-but-still-large extent any changes in Big 5.

It's no excuse (for not testing), but worth mentioning. The last time I looked at this (a few years ago), I noticed that it's really hard to test.

I spent some time (hours) trying to design a test and I struggled to get anything useful.

Do you have suggestions for where to acquire either of the tests mentioned? Are there expected differences in results if the tests are administered by laypeople rather than experienced practitioners?

Just want to note that I'm sorta disappointed by CFAR's response here.

Dan responded very quickly to my initial post (very quick = within 30 minutes), pointing to CFAR's 2015 impact assessment [LW(p) · GW(p)]. (I didn't know about the 2015 assessment and was grateful to be pointed to it.)

But as far as I can tell, no one affiliated with CFAR engaged with the discussion that followed. A bunch of follow-up questions [LW(p) · GW(p)] fell out of that discussion; I'm sad that no one from CFAR fielded them.

I'm parsing the very quick initial response + lack of follow-on engagement as CFAR acting defensively, rather than engaging in open discourse. This is disappointing.

Also this is the internet – I'm open to the possibility that I'm misinterpreting things here.