Minicamps on Rationality and Awesomeness: May 11-13, June 22-24, and July 21-28

I posted earlier that the surveys were confidential, but actually, I just reread them, and there doesn't seem to be anything personal in the "Which parts of the camp didn't work particularly well for you, and what do you think we could do to improve?" column, which was basically the "what negative comments do you have?" column. So I pasted those answers into a new spreadsheet and posted them to the web; you can read participants' collected complaints here.

If you or anyone wants to do a survey, I can give you the email addresses of the minicampers, and you can email them and see what you get and post it online (assuming you write in your email that you are looking for publishable comments, etc.). Let me know if you/anyone seriously wish to do this.

Many of the minicampers are on LW, also; the folks with testimonials above have linked LW accounts; but there is of course selection bias there.

$115/person/night

Wait, what? Can I just stay in a hostel and eat from the gorcery store?

I have friends and relatives who live in the area. How central to the camp is the communal living aspect? What would you charge to commute to it, if that is possible?

That not necessarily a bad thing. Lots of people pay for those. And such people are in need of rationality help!

Good to know.

I mean, it kind of is a standard workshop (like ones on public speaking, or italian cooking, or, yes, self-help)... except that the content is about Bayesian math, and microeconomics, and the cognitive science of standard human error patterns and so on. And the people you get to network with are other LW-ers who are interested in actually applying this content to practical problems, and coming to embed Bayesian patterns into the details of one's day-to-day thoughts instead of just into the way you answer pen-and-paper questions.

But, yes, similar workshop format, different content. Maybe we should make the ad different too in some way. I wonder if my inclusion of the testimonials and survey data, in particular, may have been misleading -- I was trying to say "look, past participants (who were smart LW-ers a lot like you) liked it, so maybe you will too", but it may have come across as a stronger claim. I'd say come check it out if you're interested, or else wait for a subsequent year if you want to have seen "proof that this will definitively change your life" first or something (which we may or may not ever manage, though we're certainly working on it), and, meanwhile, whether you come or not, do keep contributing on LW, trying exercises yourself in your own life, and generally helping to figure out what rationality can be.

The difficulty stems from how much good stuff is mixed in with all the scams, making outside evaluation much harder. Most self help programs include a lot of the same basic items by necessity (fixing the big problems first). We also seem to understand implicitly that people's self judgement of the effects of these types of things is terrible, especially when those judgements are very close time-wise to the event itself (rationalization of expense and effort, unwillingness to signal disloyalty to new ingroup, even internally).

To address your second point first, the -attendees- were not a group who strongly shared common beliefs. Some attended due to lots of prior exposure to LW, a very small number were strong x-risk types, several were there only because of recent exposure to things like Harry Potter and were curious, many were strongly skeptical of x-risks. There were no discussions that struck me as cheering for the team -- and I was actively looking for them!

Some counter evidence, though: there was definitely a higher occurrence of cryonicists and people interested in cryonics than you'd find in any random sample of 30 people. I.e.: some amount >2 vs some amount close to 0. So we weren't a wildly heterogeneous group.

As for the instructors - Anna and Luke were both very open about the fact that the rationality-education process is in its infancy and among the various SIAI members there is discussion about how to proceed. I could be wrong, I interpreted Eliezer as being somewhat skeptical of the minicamp process. When he visited, he said he had almost no involvement related to the minicamp. I believe he said he was mainly a sounding board for some of the ideas. I'm interpreting his involvement in this thread now and related threads/topics as a belief shift on his part toward the minicamp being valuable.

I think your order of magnitude increases well describes a bad conceivable scenario, but poorly describes the scenario I actually witnessed.

Now, for cost, I don't know. I'm attending a guitar camp in August that will be 7 days and cost me $2000. I would put the value of minicamp a fair amount above the value of the guitar camp, but I wouldn't necessarily pay $3000 to attend minicamp. To answer the price question I would ask:

1) What else do I plan to spend the $1500 on? What plans or goals suffer setbacks? What would I otherwise buy?

2) What do I value the information from attending at? I can see how it would be easier to measure the value of information from a guitar camp versus one about something that feels more abstract. So maybe the first step is to find the concrete value you've already gotten out of LW. If you've read the sequences and you think there are useful tools there, you might start with 'what would be the estimated value from being able to clarify the things I'm unsure about." So you take some measurement of value you've already gotten from LW and do some back of the napkin math with that.

3) Consider your level of risk aversion versus the value of minicamp now vs later. If these new minicamps are successful, more people will post about them. Attendees will validate or negate past attendee experiences. It may be that if $1500 is too much for you when measured against your estimation of the pay-off discounted by risks, that you simply wait. Either the camps will be shown to be valuable or they will be shown to be low value.

4) Consider some of the broad possible future worlds that follow from attending minicamp. In A you attend and things go great, you come out with new rationality tools. In B you attend and your reaction is neutral and you don't gain anything useful. In C you attend and have poor experiences or worse suffer some kind of self-damage (ex: your beliefs shift in measurably harmful ways that your prior self would have not agreed to submit to ahead of time). Most attendees are suggesting you'll find yourself in worlds like A. We could be lying because we all exist in worlds like C or we're in B but feel an obligation to justify attending the camp or whatever. Weigh your estimate of our veracity with your risk aversion. Update the connected values.

I would suggest it unlikely that the SIAI be so skilled at manipulation that they've succeeded in subverting an entire group of people from diverse backgrounds and with some predisposition to be skeptical. Look for evidence that some people exist in B or C (probably from direct posts stating as much -- people would probably want to prevent other people from being harmed).

There are other things to put into a set of considerations around whether to spend the money, but these are some.

Jasen Murray suggested specific techniques and specific resources, which I unfortunately cannot remember (I was not that interested in that part of RBC).

I'd suggest measuring the Net Promoter Score (NPS) (link). It's used in business as a better measure of customer satisfaction than more traditional measures. See here for evidence, sorry for the not-free link.

1. "On a scale of 0-10, how likely would you be to recommend the minicamp to a friend or colleague?"
2. "What is the most important reason for your recommendation?

To interpret, split the responses into 3 groups:

• 9-10: Promoter - people who will be active advocates.
• 7-8: Passive - people who are generally positive, but aren't going to do anything about it.
• 0-6: Detractor - people who are lukewarm (which will turn others off) or will actively advocate against you

NPS = [% who are Promoters] - [% who are Detractors]. Good vs. bad NPS varies by context, but +20-30% is generally very good. The followup question is a good way to identify key strengths and high priority areas to improve.

One idea (which you might be doing already) is making the people collecting the data DIFFERENT from the people organizing/running the sessions.

For example, if Bob organizes and runs a session, and everyone likes Bob, but thinks that the session was so-so, they may be less willing to write negative things down if they know Bob is the one collecting and analyzing data.

If Bob runs the sessions, then SALLY should come in at the end and say something like "Well we want to make these better, so I'M gathering information of ways to improve, etc"

Even if Bob eventually gets the negative information, I think people might be more likely to provide it to Sally (one step removed) than to Bob directly.

(Even better: Nameless Guy organizes a session. Bob teaches session (making sure everyone knows this is NAMELESS' session, and Bob is just the mouthpiece.)

Back when I did training for a living, my experience was that those forms were primarily useful for keeping my boss happy. The one question that was sometimes useful was asking people what they enjoyed most and least about the class, and what they would change about it. Even more useful was asking that question of people to their faces. Most useful was testing to determine what they had actually learned, if anything.

I've seen "rate from 1 to 5, with 3 excluded", which should be equivalent to "rate from 1 to 4" but feels substantially different. But there are probably better ones.

Another thing you could do is measure in a more granular way - ask for NPS about particular sessions. You could do this after each session or at the end of each day. This would help you narrow down what sessions are and are not working, and why.

You do have to be careful not to overburden people by asking them for too much detailed feedback too frequently, otherwise they'll get survey fatigue and the quality of responses will markedly decline. Hence, I would resist the temptation to ask more than 1-2 questions about any particular session. If there are any that are markedly well/poorly received, you can follow up on those later.

You could have a rubric without any numbers, just 10 sentences or so where participants could circle those that apply. E.g. "I learned techniques in this session that I will apply at least once a week in my everyday life", "Some aspects of this session were kind of boring", "This session was better presented than a typical college lecture", etc.

You could try a variant of this (give someone a d10 and a d6, hide roll from surveyor, if the d6 comes up 1 they give you a 1-10 rating based on the d10 and are otherwise honest) but this may not be useful in cases where people aren't deliberately lying to you, and is probably only worth it if you have enough sample size to wipe out random anomalies and can afford to throw out a sixth of your data.

Or weight the die.

I'm not a pro, but you probably want to turn the data into a z-score (this class is ranked 3 standard deviations above the ranking for other self-help classes). If you can't turn it into a z-score, the data is probably meaningless.

Also, maybe use some other ranking system. I imagine that people have a mindless cached procedure for doing these rankings that you might want to interupt to force acually evaluating it (rank is a random variable with mean = 7 and stddev = 1).

Some of the effect could be accounted for by a substitution principle; instead of asking oneself for each option whether one's used it in the last year, it's easier to ask which of them one recalls using most recently (or just which of them seems most salient to memory), check that, and move on. If people do actually switch between products often enough, this would create that dynamic.

I tried to take that into account when reading.

I know, I did too, but that is really the sort of calculation that should be done by a large-scale study that documents a control distribution for 0-10 ratings that such ratings can be calibrated against.

treating the indexes as utilities

Please explain.

In my engineering school, we had some project planning classes where we would attempt to calculate what was the best design based on the strength of our preference for performance in a variety of criteria (aesthetics, wieght, strength, cost, etc). Looking back I recognize what we were doing as coming up with a utility function to compute the utilities of the different designs.

Unfortunately, none of us (including the people who had designed the procedure) knew anything about utility functions or decision theory, so they would do things like rank the different criteria, and the strength of each design in each criteria, and then use those directly as utility wieghts and partial utilities.

(so for example strength might be most important (10), then cost (9) then wieght (8) and so on. and then maybe design A would be best (10) in wieght, worst (1) in strength, etc)

I didn't know any decision theory or anything, but I have a strong sense for noticing errors in mathematical models, and this thing set off alarm bells like crazy. We should have been giving a lot of thought to calibration of our wieghts and utilities to make sure arbitraryness of rankings can't sneak through and change the answer, but no one gave a shit. I raised a fuss and tried to rederive the whole thing from first principles. I don't think I got anything, tho, it was only one assignment so I might have given up because of low value (it's a hard problem). Don't remember.

Moral:

With this sort of thing, or anything really, you either use bulletproof mathematical models derived from first principles (or empirically) with calibrated real quantities, or you wing it intuitively using your built-in hardware. You do not use "math" on uncalibrated pseudo-quantities; that just tricks you into overriding your intuition for something with no correct basis.

This is why you never use explicit probabilities that aren't either empirically determined or calculated theoretically.

but I do not think I will be able to afford or attend the minicamp

If you want to attend but can't afford the fees, please do apply anyhow, and check the "need scholarship" box. Even if it turns out that we can't admit you this year, we'll at least know there are people out there who want to attend but can't afford it, and we can possibly take this information to potential donors as the Center for Modern Rationality gets on its own non-profit feet.

The particular changes I've made (like changing my advisor) have been very personalized for me, by me... but they have been fueled by a few root adjustments:

1) More curiosity about my life choices. Caused in part by being surrounded by a group of smart similar people doing very different things with their lives.

2) More willingness and desire to make up my mind more quickly and effectively about Big Life Decisions. Caused in part by Anna Salamon generating, on the spot, a steady stream of helpful questions for me that I could ask and answer to myself about career choices. I never came to any conclusions that she suggested (which I consider a good sign; I wouldn't expect someone else to know what I should do with my life from a few conversations), but she gave me a sense that more is possible in terms of how quickly a person can generate important, answerable questions.

3) More curiosity / motivation to experiment with productivity hacks, until I found some that work for me (Getting Things Done system + Pomodoro technique). Caused by being surrounded by productivity-obsessed people for a week with lots of cool ideas that helped me internalize a belief in the existence of popular productivity hacks that would work for me.

4) More desire to Be Successful (which I'd had very little of throughout most of my life), caused by feeling like I was part of a community that I cared about and who might benefit in some small way from my success.

I suppose this highlights risks from a general increase in credibility-giving by close temporal association with other new ideas you're giving credibility to?

Yes, this.

Usually this risk is low, but here it was actually quite high. This particular instance was an Ugly example, because the category - ideas with close temporal association - was false. But there were many scary examples based on good categories. The most outlandish was meditation. Remember that other people's brains are part of evidence, now witness quite a few people who have just spent the last few days on activities that convinced you they are pretty decent (compared to baseline, damn good) at doing their research, discarding bullshit, not strongly espousing ideas they don't strongly hold, examining the ideas they do hold, etc. etc... witness them say with a straight face that meditation, which you (I) assumed was a crock of mystic religion that just took a different turn than the Western religions you're familiar with... witness them say that meditation is super-useful. Then watch your brain say "Bull! Wait, they're good at things. Maybe not bull? Hey, argument from authority, bull after all! Wait, argument from authority is evidence... :S I... have to take this seriously..."

IFS, NVC, nootropics? Guess I have to take them seriously too.

(I exaggerate slightly, but my feelings were stronger than I think they should have been, so that story is in line with how I felt, if not precisely what my beliefs were)

I feel like most of the value I got out of the minicamp in terms of techniques came early. This is probably due a combination of effects:

1) I reached a limit on my ability to internalize what I was learning without some time spent putting things to use. 2) I was not well mentally organized -- my rationality concepts were all individual floating bits not well sewn together -- so I reached a point where new concepts didn't fit into my map very easily. Did you attend the 3-day version or the week-long version? I would be curious to know after what length of time you saw significantly diminishing returns.

Relatedly, I wonder what minimum consecutive length of time you need to get a lot out of this. How would the returns from three spaced-apart day-long workshops compare to those from a single three-day workshop? (This would of course work better with a group of people who don't need to travel a significant distance.) Is the New York meetup group what happens if you take this sort of thing, break it into small chunks and spread it out over time?

People who attended minicamp can probably provide more informed speculation on these matters than I can.

I realize these will not be very useful out of the box, but considering how a number of Stanford classes were successfully ported to long-distance format (with interactive exercises, quizzes, etc), this might be a good first step in the refinement process.

I think analyzing your performance via video is underrated outside of sports.

Well, it's potentially one vector for folks to learn how to do the practice and exercises.

Sort of. I meant to say that I decided to make explicit long term, medium term, and short term goals, regularly check their progress, estimate their difficulty and likelihood, and also had a better sense of the space of other opportunities, all as a direct result of minicamp (there was a session or two on goals, sessions on estimation and prediction calibration, and in general while there I realized that I sucked at seeing opportunity costs).

After I did all those things, it effectively resulted in a systematic decision and planning process, since I had a much better sense about what tasks had the highest expected payoffs for my goals, and I simply work on those first.

The main problem is that a test tests ability to take the test, independently of what its makers intended. The more similar tests are to each other, the more taking the first is training for the second, and the easier it is to teach directly to the test rather than to the skill that inspired the test. The less similar the before and after tests are, the less comparable they are.

Rationality training is particularly tricky because one is to learn formal models of both straight and twisted thinking, recognize when real-life situations resemble those patterns, and then decide how much formal treatment to give the situation, as well as how much weight to give to one's formal model as against one's feelings, reflexive thoughts, and so on.

Traditional classroom tests are set up to best test the first bit, knowledge of the formal models, if one did solve the problems inherent in testing. Even to the extent one can ask people about how one ought to react in the field, e.g. when to use which sort of calculation, that is still a question with a correct answer according to a formal model and one is still not testing the ability to apply it!

These problems resemble those the military has faced in its training and testing. They use indoctrination, simulations, and field tests. Decision making is tested under uncomfortable conditions, ensuring probable good decision making under most circumstances. In general, knowing what they do is likely to be helpful.

The problems with tests are not intractable. One can limit the gain on the second test from having taken the first test by saturating the test taker with knowledge of the test before it is taken the first time, though few would be motivated. One can try to make a test similar to the skill tested, so ability at the test is well correlated with the skill one intends to test. One can try to devise very different sorts of tests that measure the same thing (I doubt that will work here).

One component of a useful classroom test might resemble the classic research on correspondence bias. In it, people judge individuals' support for positions based off an essay they supposedly wrote. Some subjects are told that the writer chose the thesis, others that the writer had it assigned. (The theses were either pro- or anti-Castro.) People inferred that the essay's author significantly agreed with the thesis even when they were told it was assigned to them. The quality of an essay a person produces is some evidence of what they believe, as is their willingness to write it at all, etc., but in general people overly infer others' dispositions from actions they take under social constraint, even when they know of the constraint.

Here is how the framework could translate into a useful rationality test: the test would give people some evidence for something they are biased to overly believe, and the quantity and quality of legitimate evidence in the test would vary widely. One would not be able to pass the test by simply detecting the bias and then declare oneself unmoved in that wrong direction, as one might be able to do for, say, sunk costs. Instead, the valid evidence and invalid inclination would be along the same vector such that one would have to distinguish the bias from the rest of the evidence in the environment.

This solves the problem of having a classroom test be an easy exercise of spotting the biased thought pattern and quashing it. Videos or essays of various people with known beliefs arguing for or against those beliefs could be used to train and test people in this. It's actually probably a skill one could learn without any idea of how one was doing it.

Expressed abstractly, the idea is to test for ability to quantify wrong thinking by mixing it with legitimate evidence, all of which increases confidence in a particular conclusion. This is hard to game because the hard part isn't recognizing the bias. The material's being media from real life prevents testers from imposing an unrealistic model that ignores actual evidence (e.g., a strongly pro-Castro person really might refuse to write an anti-Castro essay).

Ah, yes, that is indeed the first thing one should work on, otherwise the MW (Must Win) interpretation of Rationality is little better than the MW (Many Worlds) interpretation of Quantum Mechanics. I didn't realize that, after all this time, there are still no objective metrics to measure the success of the course. I wish I had good ideas as to how to experimentally measure rationality, but alas. Hopefully other forum regulars do. Or maybe EY can spend some time thinking about it.

I guess an obvious way to start is to score a particular behavior based on some objective criteria, like the pass/fail on those sunk cost situations Anna (?) linked here some time ago. It's not nearly as good as actually putting people into the circumstances where they have to apply their newly learned skills (such as detecting confusion, recognizing cognitive dissonance, what have you), but it's a start.

As a next step, my guess is that if you look through the standard psychological experiments (maybe something less drastic and notorious than the Stanford prison experiment), you will find quite a number of them that can be cheaply replicated in a controlled setting like a mini-camp. I'm sure that gwern can dig up a whole whack of them in no time flat. Or maybe you are already doing this, for all I know. The important thing is that the participants should be inside the situations, not outside of them, and hopefully unaware that they are being tested. I guess it is sort of similar to giving two sets of CRTs, before and after.

The week-long camp will not have filler. At the minicamp, there were exercises that I suspected weren't useful -- the "rationalizing game" stands out in my mind -- but probably a bigger problem was trying to pack so many things into a week's span. I definitely had the feeling that I needed to be taking good notes, because I couldn't possibly absorb everything in the various sessions during those sessions.

I'd like to second the second question. Should I be worried about the 3 day camp attempting to cram in too many useful techniques or the week long camp having filler?

They ran our camp for 10 weeks or so. Sure, some of that was filler. But they have enough there for at least a week.

Thanks. I doubt I will go this year for the reasons I listed above. Next year when I have more vacation time built up I'd consider doing it.

Although if you'd like to include "read advance chapters of HPMOR" into the benefits, I'm in.

FWIW, it was not clear from a skim.

Yes, it was clear to me.

I would prefer the weeklong one but am considering the weekends. The cost of airfare is the same.

Another question: Is there any particular reason why you are including the hotel costs into the fee? I can see the marketing value from "single number", but for those already in the Bay Area (or with friends/family there), reducing that cost by a bit and staying elsewhere would be helpful.

If activities/teambuilding are going late, that makes sense, but that was not clear on a single read through (I could read through again to find out, but figure the feedback on this not being clear might be helpful).

It is clear now.

However I'd like to mildly dispute "Hopefully a 3-day camp wouldn't be that expensive in terms of vacation time not spent elsewhere".

...some of us come from another country (quite a few actually) - and the travel time alone is equal to the 3 days we'd spend on the actual camp. Plus a day or so adjusting to jetlag enough that we could even stay awake for the first day.... I'd estimate that for me to attend even the three day camp would be, at minimum, a seven day vacation.

Factor in the significant cost of the airfares, and we're starting to talk a lot of vacation-opportunity-cost.

Obviously, attending the longer camp would be a better signal-to-noise ratio... but would still be a serious undertaking.

PS: that's not to say, of course, that's it's not still worth the effort. Personally I'm looking into my options to see if I can pull it together to get to the week-long one this time...

The main point is just not to forget that a significant number of LW readers are in a quite different situation to your own and thus the difficulty of attending should not be inadvertently trivialised.

Hey, I'm glad to hear that :)

Again, forgetting all the rationality training - there were moments at mini-camp when everyone was hanging out and I would literally have trouble deciding where to stand in a room because every conversation going around me was so ridiculously interesting that I couldn't stand choosing where to place myself. I felt like a wealth of knowledge was being spilt around me, and if I didn't scramble to consume as much as possible I'd miss some lifechanging insight and regret it forever. It was so beautiful it hurt.

Wow. That's like the opposite of most parties.

Can you describe the difference between a typical conversation at the mini-camp, and a typical conversation on LW? (Would it be accurate to say that you're more impressed with the former than the latter? I'm curious to find out why if that's the case.)

I'm concerned, however, that the camp can't produce evidence of the kind, "Before the minicamp, Mary Sue was in rehab for crack. A year later, she's clean and has a successful web consultancy." (Exaggerating the expected magnitude of change, of course.) Religious retreats don't produce this, and tend to produce results more like, "Immediately after the retreat I felt really good, and a year later I do awesome on unobservable metrics!"

Replace "glad I went" with a better criterion- that question deserves a good response.

The cooperation has actually been happening; it's just that it was achieved by ostracizing the guy who asked if you were adhering to the principles expected of that kind.

If you don't care whether the cooperation is doing useful work, then sure. Otherwise, criticism seems to be a necessary evil.

If not then I'm still curious, are there any plans to attempt to study rationality of attendees and non-attendees to establish such evidence?

Yes, that's an oft-repeated goal, and as Eliezer mentions in a sibling, there's a one-year follow-up planned but it has not yet been a year.

Right, it's been nearly a year since the last one. The long-term evidence is out there. How are attendees doing in their lives now vs how they were doing before?

I'm pretty sure there's been enough time to find this information out by now.

It's easy to imagine a Christian brainwashing retreat run by someone similar to Luke that would also have that property.

I'm not sure you answered the question. I think SilasBarta is looking for evidence, that someone can provide for him right now, that he will be glad he went.

ETA: and for purposes of this discussion, I think a bald assertion does not fall in the cluster "evidence".

Oops. I wasn't thinking along the lines of "the people we most want to come to minicamp are the people who are most easily brainwashed into giving us money". Sorry if I gave that impression.

I was more thinking along the lines of "when should I buy utilons, and when should I buy self-improvement in order that I can acquire more resources and/or more effectively turn resources into utilons?"

Minor point: can you clarify whether Rationality Group is the same thing as the Center for Modern Rationality?

Thanks! She's reminded me of some practical reasons she can't go this time. Hopefully we'll be bringing this to London before long.

When do you hope to answer applications? The sooner I can book flights the better :)

I'd love to know this - heck, I'd teach a diet course if anyone is interested (those at the last NYC mega-meetup listened to an hour lecture/qa of my nutritional philosophies :)

I see now that there's a "report" button in my inbox - I knew I'd seen it somewhere on this site.

Cheers, I'll PM in future.

Thanks for responding. I'm trying not to underestimate the value offered here, but the degree of non-monetary cost of going varies fairly widely. For those of us who would have to disrupt things fairly severely to go, would getting meetup groups, etc. trained on the curriculum be a means to train more people, sooner, even if removing the intensive camp setting means that the participants take more time overall?

Thank you.