Reinforcement and Short-Term Rewards as Anti-Akratic

This is awesome! I'm really excited because I've been playing around with related things for a while, and by sharing techniques we can all become stronger!

So: A couple months ago I had a massive problem with my Anki reviews. I use Anki for a lot of things that aren't standard question-answer training, and this other work tends to be really mentally effortful. So, naturally, the reviews piled up until I had a massive backlog, around 10,000 reviews.

I tried a bunch of tricks; from this post ("Applying Behavioral Psychology on Myself") and elsewhere I got ideas for two Anki plugins intended to reinforce reviewing, by manipulating music volume or by showing reinforcing pictures. This was somewhat effective, but not very. Then I moved on to using candy, which worked better but still wasn't very effective.

But the latest thing I've been trying works a lot better. Look at my anki review stats:

The technique that did that was this:

• Reinforce reviews by giving a 20% chance of reward upon the completion of each.
• A reward is a small piece of food or a sip of a drink.
• Never eat or drink anything except when you've earned a reinforcer.

As you procrastinate, you become hungrier and hungrier until your desire for rewards exceeds your desire for non-work. By keeping rewards small, you remain perpetually hungry and work remains reinforcing. The brain was built to extrapolate from "I'm less hungry" to "I should do whatever I just did more often".

This is especially good for something that needs to be done regularly, like anki reviews. If rewards and reward-probabilities are small enough, it also functions as caloric restriction. This system is also good for very granular tasks, like question-answer Anki reviews.

My non-question-answer reviews have irregular lengths and so aren't as granular, so for them I use another reinforcement system in addition to the old one: every 10 seconds, with probability ~20%, show a message in the background. When that appears, I examine my thoughts just prior to it appearing and reward if they were about work or other productive things. This also seems to work well, and works better for tasks where you don't want to incentivize rushing through things. (For question-answer pairs, success is measurable in correct responses, so generally you should rush through them, as long as you still get the correct responses.)

I also have a thing that periodically asks whether I'm in a correct posture, and standing instead of sitting, and not procrastinating sleep. If I'm in the right state, those give additional medium-sized rewards. I implemented this only 2 days ago, so I don't know if it works yet.

I strongly encourage you to try variable reinforcement, because my impression from reading things is that it's a lot better; I haven't tried non-variable reinforcement.

A similar thing I've been experimenting with is punishing unwanted behaviors, mostly by using the rubber band technique; mixed results so far.

I'm very interested in your automatic dog-feeder setup. I've been thinking about further automating my reinforcement things, and about automating punishment by buying an electroshock collar (normally used to train dogs).

How to implement some of the stuff above:

See the two anki plugins I posted. I just put up a more basic version that shows popups instead of pictu res.

On Ubuntu Linux, to show a background popup every 10 seconds (sorta), with some probability, do crontab -e and add this:

* * * * * bash -c "export DISPLAY=:0 && for i in 1 2 3 4 5; do if [ \$(( \$RANDOM / 327 )) -lt 20 ] ; then notify-send 'REINFORCE?'; (sleep 3; killall notify-osd ) & fi; sleep 10; done"

That's what I currently use; I used to use this, which shows foreground popups and should thus probably be kept commented-out most of the time:

* * * * * bash -c "export DISPLAY=:0 && for i in 1 2 3 4 5; do if [ \$(( \$RANDOM / 327 )) -lt 10 ] ; then zenity --info --text='\n\nREINFORCE?' --timeout=5 --width=1000 --height=800 & fi; sleep 10; done; if [ \$(( \$RANDOM / 327 )) -lt 10 ] ; then zenity --info --text='\n\nREINFORCE?' --timeout=5 --width=1000 --height=800 & fi"

If anyone is planning to use any of this, please tell me. Also please share any ideas you have, even if they don't seem useful.

Also, does anyone want to help me think of a good name for the system? Right now it's called the "extrinsic motivator." While descriptive, this name isn't snappy at all.

How about "Sir Rob the Rewarding"?

+1 Awesome!

I get sick of candy more quickly than I expected. The portion my machine emits (about a small handful) tends to stop motivating me after about 4 in a day.

Could you try using smaller candy?

if the reward is in the system, I tend not to wait very long before using it.

This seems OK to me. The reward should be instant, to create a better connection with the behavior you want to reward.

aside from my task list, what else can I connect to? Can I track note-taking in class? Can I set it up to reward continuing effort towards a task, like writing a few hundred words a day? Can I use it to create new, more rational habits?

I use my reward system for: exercising, getting to work early, avoiding sugar, avoiding web procrastination, meeting with people, taking an afternoon nap, writing blog articles, learning foreign languages, etc. And there is also an umbrella category for "other important stuff", such as making an appointment with a dentist or fixing something at my home.

To prevent myself from replacing a difficult task with many easier tasks, some of these rewards have a limit of 1 per day. For example, I can only take 1 afternoon nap each day. Or I get 1 point for not eating sugar all day (rewards from the system are not included in this rule). For writing N blog articles I get N points, because honestly "writing too many blog articles at the same day" never happened to me. Somewhere in between is the exercise: I can get only 1 point per day for doing a "small exercise", and then another 1 point in the same day for doing a "big exercise" too (5 times more difficult than the small one).

Note: My system does not include priorities and deadlines, so I don't know how easy would it be to include this stuff in your system. And with more rewards you really need to use smaller candies. Or perhaps giving 1 candy per N points? Or giving a candy with probability 1/N? -- Instead of truly random numbers I would recommend an algorithm that would increase the probability; for the first point, you would get the candy with a probability 1/N, but if you don't get it, the next point has a probability 1/(N-1), etc.; and if you get it, the probability is reset to 1/N again. (Truly random numbers don't feel random to humans; we emotionally expect that after a few failures the probability of a success should increase.)

Meta: It is great that you describe what works and what doesn't work. With enough articles like this, someone could make a review of motivational systems, describe the good and bad parts confirmed by many different systems, and make recommendations for people who want to create their own system.

To provide the data, I linked the device to an earlier productivity hack that I already had around, a custom webapp integrating a task list with a Pomodoro timer.

I've been looking for something like that for a while, can you share it?

Given that in fact, you can eat or drink whatever you want whenever you want, how do you enforce the conditionality on yourself? Isn't this an exercise in role-playing, and if so, how do you sustain the pretence?

I get sick of candy more quickly than I expected. The portion my machine emits (about a small handful) tends to stop motivating me after about 4 in a day. Additionally, I seem to be entirely incapable of pacing myself; if the reward is in the system, I tend not to wait very long before using it. This has crippled all of the rules about involving taking away rewards - unless the rewards are blocked, they don't stick around in the system long enough to be taken away.

All of these problems could probably be fixed by denominating rewards in Quirrel points instead of house points. ;-) (HPMOR reference)

That is, have the device display (you said it had a display) a count of points, which has to reach a certain level before the reward is dispensed. Ideally, make it flash and ring a bell or make a sound for each point as well.

Since you will be using smaller point denominations, this means you'll also want to break some of your larger tasks into smaller pieces, so you can then fix this problem:

if I think a task will take me more than a day to finish, that's more than a day of work which earns me no short-term rewards

For example, you could break that task down into pomodoro-estimable units, then give yourself a point per pomodoro or something. Ideally, calibrated so that it takes you the whole day to get all four dispensings.

If you work a 16-pomodoro day, for example, then four pomodoros could be worth one dispensing. This would certainly increase motivation to not interrupt those later pomodoros in each 2-hour work/rest/eat cycle. (The first one in each cycle you'd not have much to lose, but that should also be when you're generally freshest due to having rested or just starting for the day.)

I like the name "Extrinsic motivator".

I'd be worried about eating candy constantly for the sake of your teeth. You might want to try Trident gum, which is also tasty and should actually be slightly good for your teeth. Gum chewing is supposed to improve alertness and focus, although I find that if I spend too much of the day chewing it, I get this weird sense of burnout.

I wonder if it's possible to hack an electronic air freshener in this way. Make it make the room smell nice when you're working especially quickly. Sound might also work, or maybe both together. I suspect it would be a weaker reinforcer than candy, but it could be immediate without distracting you, and you might tire of it less quickly than candy. (I should try this myself at some point, but I think I've already been classically conditioned to associate typical perfume smells with my eyes burning (classmates at a school I went to liked to spray one I was allergic to all over the locker room), so I'd have to find one that has an unusual scent, or one that can spray things like spirit of peppermint instead.)

I find that just marking when I finish things is a bit reinforcing for me, especially if there's a chart, and especially if it shows completion time relative to the due date. (Right now, I'm using Tom's Planner, a Gantt chart app, for one of my class projects. It's working really well.)

I also tried adding "meetings" to Google Calendar for everything I did for about a week, and I think it reduced the time I spent not working on things, but my "work" time might have been a little less intensely productive than normal. I stopped because having to record everything I did got really annoying after a while. I suspect part of the problem was that the user interface wasn't optimized for using it that way, so I'm planning to try building something that makes it easier to record, while still showing a chart of how the time was spent.

I wonder if involving another person in this system would make it more effective, since social approval tends to be really motivating for humans. Though, that would require there to be another person who is usually around when you're working and wants to participate.

aside from my task list, what else can I connect to?

• RescueTime (records what websites and applications you use and for how long)
• word count of a text editor or word processor
• a keylogger

If your typing speed has a strong correlation to how quickly you're getting your work done, I wonder if you could combine RescueTime to automatically only count typing in certain applications. Though you'd also have to have some way to avoid accidentally rewarding for non-work use of word processors and such.

Precommitting to never take candy out of the machine seems like it'll be a lot easier than precommitting to only sometimes take candy out of the bag.

I've been having some issues being clear about when I'm actually rewarding myself; this sheds some light on the subject. I need to find a machine to put my rewards in. Thanks.

Be aware that there's significant research that extrinsic motivators crowd out intrinsic ones. Essentially, they increase total motivation at the cost of sometimes reducing intrinsic motivation, which in turn creates a reliance on the extrinsic motivators. A good book on the subject is Punished by Rewards: The Trouble With Gold Stars, but if you do decide to read the book, be aware that the author is biased towards self-determination theory.

Some of my recent forays into reinforcement learning have been very helpful. I should point out that my life is made a whole lot easier by having a very co-operative OH who is willing to reward me or withhold reward as appropriate, so I've not needed to resort to building a robot!

Things that have been successful:

• Every time I think about {thing I enjoy obsessing about}, I go and do the washing up. I used to have a massive ugh field around washing up, but this has quickly diminished (within days!) via association with the nice thoughts. We're thinking of applying this method to other things I have ugh fields around, since it was so quick and effective.
• I've been doing a similar thing to D_Malik with regards to Anki cards. However, it was impractical for me to withhold a reward I would be having on a daily basis, so my OH is implementing "withhold {nice thing} unless I have reviewed my Anki cards for the previous 5 days". It's not as immediate as not eating, but seems to be sufficiently encouraging thus far.

But yeah, having a person help me do it means I avoid any sort of precommitment failure, and generally makes things much easier!

(Side note: Curly brackets clearly denote euphemisms, but I didn't want to be too crude.)

CFAR describes reinforcement in a very striking way in some of their course materials: they call it "training your inner pigeon." Not only is that a nice, snappy turn of phrase, it illustrates the problem with attempting to self-administer rewards very nicely. Did Skinner's pigeons self-administer their rewards? No, of course they didn't. I shouldn't expect my inner pigeon to, either. So, my next step is to build a robot that gives me candy when I get stuff done.

The idea is that your conscious mind is to a degree separate from your lizard brain, so "you" train your "inner pigeon". You aren't your inner pigeon, you're Skinner.

Of course, the reinforced parts of the brain overriding the smart bits is what akrasia is all about, so I'm not sure how well this neat demarcation works in practice.