Meetup : Melbourne, practical rationality

Notes from discussion about finding better metrics to measure CfAR's effectiveness

[Discussion about what CfAR is trying to teach.]

Have to avoid CfAR just measuring how much people are paying attention in their courses

Beforehand: write down goals

Six months later: measure them against their pre-camp goals

Control group

Just attending a camp might make it feel like you're making more progress towards your goals.

Some goals hard to measure

How much throughput are they expecting? 3 camps/month.

Measuring income: a lot of people might decide during the camp to change career, or devote their life to effective charity, so income may well drop when your rationality goes up.

Also income has a long lag before interventions take effect

epicwin: gamify your life.

We're going to stop proposing solutions for a while

Suggestion to generate ideas individually.

Desirable qualities: low stat noise, objective, quantitative etc.

What sorts of measures: e.g. based on other's opinions, performance at a task, other classes

Could talk about what goals people have? Then we could think about how to measure those particular goals. Money, happier, healthier, etc.

Whose perspective are we taking? The consumer of the minicamps.

In self-help field, typically sell books or lecture tours based on how good your stuff sounds. Most of it doesn't come with metrics. If there's some of that works (e.g. GTD), then CfAR should be teaching it.

[Tangent about quality of scientific literature.] Problem with finding material to teach by looking for stuff whose effectiveness has already been well measured in the literature is that you're setting the bar too high. We don't need to published, so can make do with messier non-laboratory non-ethics-committee-approved measures.

Come up with a test you like. Try to teach to that.

Break up to discuss:

1. Desirable qualities of metrics

2. Types of test

3. Goals that people might have

[We break into groups]

I'm with yurifury (in hammock) + patrick (in bean bag)

1. Types of tests

Maybe have an interactive test that heaps you make decisions (like it includes a checklist of things to think about and biases to avoid) and it would generate metrics along the way. An iterative process, so you could track how people's goals change. And if it's a process to help people, people more likely to use it. Blurring line between self-help tool and pure metric. Or not a tool just for significant decisions, but a weekly thing. People's goals might change from month to month and a tool that let you say "i'm doing well on this one" "i don't care about this one anymore" would be helpful and would generate metrics.

An agile version opposed to the current waterfall 1-year-apart surveys pre- and post-minicamp.

Difference between quantified selfers, who measure things to detect correlations, and people who just measure things they're trying to improve.

They should run a course specifically for quantified selfers. Gwern could be the Matt of the course. Then the could observe what changes about them and they might get less obsessive people to measure those specific things.

1. Goals

Big problem with motivation: how to get people to take the tests. 1. Have measurements be side-effects of something otherwise useful. 2. Social pressure.

X, what would you like to change about yourself? Answer: spend more time reading. Akrasia.

People who talk regularly following minicamp (already) could evaluate each other. People at meetups could evaluate each other. 5 minute quiz at meetups.

Have an award for the person who changes the most. <- hard to make honest

We seem to be generating more ideas/unit time in our small group, than the group as a whole. Same for other groups? Yes.

We'll focus on goals that are easiest to measure. Health. Measures like resting heart rate not BMI Facebook friends/klout. Happiness Grades <- not so good Memory/IQish tests

[We decide that the notes above are a really good example of how useless a 12-person group discussion can be. AVOID AVOID AVOID]

The table was observed to be red-shifted.

Good suggestions from other groups:

Possible tests:

Idealogical Turing test as a measure. People learning to model each other. Tests things including: are you able to model other people well, do you understand other people's ideas. rather labour intensive

Search skill: able to use google. measure how well people are able to find information (with time constraints). Apparently google has a daily challenge.

Get people to do a three minute video sales pitch.

Game playing. Useful metrics in how well you play poker, Wits and Wagers.

Might be worth having preprepared list of goals that are easy to achieve and/or measure and have people select goals from that list

Look at course material and try to come up with tests inspired by course material, but being sure to ask "Why are we delivering this material?" too

[Tangent/derailment]

[Splitting up again, given what we learned. Other group is dancing.]

Summary of what we want to tell CfAR:

1. Summarize desirable characteristics of goals. Need as common knowledge.

2. Directions to go in towards gathering metrics

3. What we learned about how to have this discussion. Talking about goals seemed too general and not useful; discussion about transferable skills seemed more useful. Things that people wanted that came up frequently: Calibrated, appraise evidence, socially competent, ability to persuade others, motivation esp. across different time scales.

4. Some suggestions of metrics (filter above)

Also we want to post in the meetup organizers group what we learned about group discussions. Benefit of small groups is it's okay to interrupt.

[At this point my battery about to run out.]