The Power of Reinforcement

Excellent insight. Downvoted.

So you are saying that, to change your mode of behavior, all one has to do is create a judging context? That would actually make you very easy to manipulate..

Explicitly manipulative socially mediated praise/M&Ms would strike my brain as outright evil and would stand some chance of being inverted entirely. That said I don't get the impression this sort of defense mechanism is very common.

I experience this as common, but I suspect it's because of a small number of exceptionally vocal "manipulation is evil!" types in my life, rather than a larger number of typically vocal ones.

[...] On a meta level, for someone even the idea of "learning" or "improving" or "changing" may be already associated with pain, so they will resist any such process if they notice it. A human mind can be messed up rather easily.

This becomes painfully common (and obvious to any observant third-party that knows these concepts) for subjects that students "are just not made for", such as large amounts of students that "just don't get" maths. They've been trained in so many ways to associate actual learning (especially the actions taken when attempting to learn a concept) with negativity that it becomes obviously so much more rewarding to just guess the teacher's password, and so they are positively reinforced into doing everything they can to avoid mental modeling and seek password-guessing through aggregation and correlation of symbol-data. In most cases I've observed, they become experts at the skill of subconsciously forming "truth-tables" of teachers' passwords through brute-force trial-and-error tactics. What's more, this tactic, which they've been trained to do and learned so well and associate so much with positive feedback, often feeds itself into a vicious circle through several possible methods, which makes getting out (or, for that matter, even realizing that it's there and you need to get out of it) so much more difficult than if that behavior had been blocked immediately when it first appeared.

When I realized that, I've started to feel sad for every student I see showing signs of spending hours upon hours studying and memorizing and headsmashing against the same math problems "until they finally understand them", when in truth they haven't really gained anything worthwhile (IMO) from the experience.

Most people want to be sincerely praised. Someone who reads this post and applies it poorly is going to be saying praise while their body language says something else entirely. Or acting out of character for themselves, leading the reinforcee to suspect that the praise is insincere. Or they may go around praising seemingly everything, causing the reinforcee to interpret the praise as meaningless noise.

There are lots of ways for using praise as reinforcement to go wrong, and if someone is in one of those environments for long enough they will end up being conditioned to interpret praise as neutral or negative.

I suspect it is common enough that when you observe that praising someone doesn't reinforce their behavior or makes them uncomfortable, you should consider that they might have an unusual aversion to praise.

I would expect the rate at which people run counter to "usual" reinforcers to be far less than the rate at which people claim to run counter to "usual" reinforcers.

Yes, which is why I said 'someone for whom this doesn't seem to work', not 'someone who claims that this doesn't work on them' - though of course in the latter case it's at least polite to humor them.

I also didn't say that reinforcing techniques don't work on me - I've never run into anyone for whom that was even remotely plausible, in fact. Just, you have to use things that don't squick me out as positive reinforcers, and overt praise and rewards aren't in that category.

Yay recursive reinforcement!

(I didn't until EY pointed that out.)

That strikes me as goofy, not phygish.

what's "phygish"?

What is LRS?

"Woman Training Man" is generally presented as funny with no negative ramifications. "Husband training wife" is presented in the context of either physical abuse, emotional abuse, or as part of a widespread societal trend of women being "domesticated" which is now generally considered distasteful. If this had been phrased "husband training wife", it wouldn't pattern match to "funny, harmless joke", it'd pattern-match to either abuse or societal oppression. (The abuse angle wouldn't necessarily be accurate, but for many people it would come to mind before the "mirror-image-of-the-woman-training-man" concept did).

So whether it actually makes sense, the example would produce negative affect in many people.

No, it sounds like you're aware of the relevant cultural meme.

lower-status husband

Interesting I have seen research that suggests a major difference in perceptions between men and women. Men tend to assess the average woman as, well, average in overall attractiveness. Women tend to assess about 80% of men as below average. So in a monogamous society women tend to think they have settled too low.

Such a gap in perceptions would make sense in a polygamous society where a few men at the top have most of the women - so the women marry up, and end up perceiving this as normal. From my reading, most hunter gatherer societies were polygamous.

You should reread it. Compare to the original. Consider what emotions you or the average person reading these two two experiences.

Then consider why I would point that difference out.

Given the many asymmetries between men and women, it seems at least plausible to me that the above would be much more problematic than the original.

It also seems plausible that the reverse is true. Or neither.

Sounds like standard PUA to me.

This explanation of why the complementary comment on the article was effective is itself effective, because it gives specific reasons why the complement is unlikely to evoke the embarrassment sometimes associated with more vague complements.

I suspect it's very difficult to design the right reinforcement strategy. It's easy to reward something that seems related to the goal, but can gradually become a replacement for the goal.

For example rewarding success and punishing failure reinforces choosing only trivial tasks, which prevents learning new things. Rewarding starting new things reinforces starting new tasks without finishing them, also choosing tasks for being new, not being useful. Etc.

Rational thinking about consequences, and changing the strategy when necessary, cannot be avoided. So perhaps this should be reinforced. But how do we distinguish between genuine rationality and signalling? Yeah, rationalists should win, but by rewarding success and punishing failure... see the previous paragraph.

Anyway, many people do worse than random, so some reinforcement can be used to improve the situation.

EDIT: Another problem: I suspect that any reinforcement inevitably goes meta. When I get a reward for doing X, I will do X more, but I will also like the reinforcement mechanism more. When I get punished for doing Y, I will do Y less, but I will also hate the reinforcement mechanism and rationalize why I must get rid of it.

I suspect that people prefer wireheading, except in cases when it becomes too obvious that it is wireheading. If I am allowed to choose my reinforcement mechanisms, I will probably unknowingly slowly optimize them towards wireheading. If someone else chooses my reinforcement mechanisms, I suspect they will choose it to optimize their utility function instead of mine.

Yoiks! This may well be why my procrastination at work has increased and increased over the decades. I almost always (habitually?) feel like my efforts are not good enough, will be criticized negatively.

If that's the case (and it seems like it is), then reinforcing yourself is going to be almost impossible, because you will by definition know the reinforcement script.

In my case I'm not terribly perceptive, but there's a lot of repetition of the same situation to give you a clue.

I'm curious about the anecdotes. I feel like I'm reading travellers' tales of the weird customs of a distant tribe.

What about self punishment?

meh. downvoted.

(just joking)

(demonstrating to observers / demonstrating to self / demonstrating to punished; status / power / resources / justification / need / etc; person / cognitive subsystem / institution / problem representation / etc)

Here's a fixed, less passive-agressive version of the deleted comment:

This article implicitly reinforces reinforcement and punishes punishment. But there are situations in which punishment should be reinforced, e.g. if this article is in fact correct to punish punishment. I hope someday someone writes out a list of ways to efficiently torture oneself into having at least some hope of ultimately not being seen as obviously stupid in retrospect, to complement this article and perhaps adjust for any optimistic-biased selection that might have generated it.

You get the option to delete if you retract and no one's commented. Which is perhaps not good, because I made a rather embarrassing terminological error in that comment that I probably deserve to be punished for.

I'd think a habit is something that just goes on as long as nothing happens to disrupt it. You no longer need to reinforce it.

Yup. When reading through the comments about the new website, I could feel my effort being punished.

Perhaps you could have somebody read them for you and summarize them in a non-critical way, thus creating a reinforcement shield.

Alternately, you could adapt what internet marketing "personalities" do, and promote doing: practice celebrating criticism. One marketer (I forget which one) described making a practice of throwing his hands in the air and shouting "Woo!" when he received a criticism via email.

(Background: "personality" marketers promote by writing emotionally charged material that's intended to divide their audience into people who either love or hate them. Thus, the presence of hate mail is evidence that their strategy is working. They will then often publicize the hate mail, in order to stir up the emotions of the people on the opposite side of the debate. Talk radio hosts, bloggers, political commentators, etc. also use these strategies, even if they're not always considered "marketers" in a traditional sense. Whether you consider this "dark arts" is largely a political question, since the LW sequences use these tactics also. Whether he knows it or not, Eliezer is a personality marketer in this sense, it's just that he's not as efficiently monetizing the results. ;-) )

Sorry about that.

It seems to me that if humans were emotionless utility maximizers, we would prefer hearing criticism over praise, the same way programmers purchase more utility by fixing bugs in their programs than polishing features that already work. I suspect criticism is generally more valuable from a pure decision theoretic perspective.

I wonder if there is an effective way to buy encouragement and criticism separately. Also, it's hard to know exactly how best to encourage folks. In theory it's possible that making a new website is not the best use of SI's resources, which suggests reinforcement would not be optimal. But we still may want to reinforce you towards the more general behavior of taking steps to achieve your organizational goals. So what's the best response?

Maybe someone can develop some general guidelines for reinforcing/criticizing people, similar to what the nonviolent communication people came up with. (When {observable event} happened, I felt {feeling} because I need/value {underlying need that felt unmet or value that felt jeopardized}. Would you be willing to {specific request that person could do} in the future?) E.g. check to see if the person was acting with good intentions and reinforce them for those if they existed, check for super goals you endorse and reinforce them for working to accomplish those, check to see if the person could just have easily have sat around doing nothing and reinforce them for expending effort if this was the case, etc.

I think optimally criticism would have lots more reinforcers associated with it: people should be reinforced for requesting, giving, and receiving criticism because these are all activities that are naturally aversive but actually have high expected value.

So, I wholeheartedly endorse the following actions of yours: attempting to maximize humanity's collective utility function, working on the super goal of AGI safety, actually doing stuff, and deliberately gathering critical feedback. Go Luke!

Yup. When reading through the comments about the new website, I could feel my effort being punished.

I am slightly surprised to hear this. I perhaps expected slightly less emotional involvement with the effort and more of a ", Go! Fix!" feeling.

I don't have that response, which probably accounts for me not being sufficiently mindful of expressing criticism to others... Do you think there may be a way to train positive or neutral response to criticism? Are there effective methods for making criticism less painful to a typical person?

matt had a similarly negative experience when LW was redesigned a little while ago.

That circumstance is somewhat different in nature. While as far as I know nobody wanted matt to experience negative affect the discouragement of 'effort' was actually a perceived instrumental good, given an expectation that more effort would produce undesired outcomes.

I note that this relies on beliefs at the time. In that context users had to make the prediction "If a website administrator implements detrimental changes when previous discussion had already explained why such a thing was not desired and a prediction had been made that a change to the website would probably be bad, what is the probability that future 'effort' will be beneficial?" The answer is very, very low. The emotional distress matt experience was his social instincts warning him that interfering with the tribe when they do not want you to is a dangerous act - especially when that interference is to (in effect) institute a prohibition against something they could previously do.

It turns out, however, that matt is a superior human being to the typical person in his role. While his ego did cause him to act more defensive than optimal and seemed to cause him to experience emotional distress it did not cripple his ability to respond to user feedback or cause him to lash out with actions against the users as many would. The undesired change was eventually fixed, as were the few bugs that were introduced.

I expect users to drastically update how they would respond to matt if he made future website upgrades due to having more information about matt. He definitely deserves a lot of rewarding for going ahead and doing the bugfixes and implementing 'retraction then deletion' despite having received discouragement. He lives (here) in Melbourne. Perhaps I should give him a packet of M&Ms if I run in to him at one of our meetups!

Well, a lot of non-specific praise would water down the value of non-specific praise as a reinforcer, but taking the time to pick out more specific elements that are good/improving would probably reduce discouragement.

I think one of the things I forget most as an instructor is how easy it is to get discouraged, especially when you're being taught by someone who seems to be able to do all of it effortlessly. There's also the element of "I already know I'm doing it wrong! I just can't get my body to listen to my brain!" Instructors who don't acknowledge this and give praise for trying or noticing that I'm doing it wrong are a major source of discouragement for any new physical skill I try to learn.

Hmm, I wonder if providing a lot of negative reinforcement on some attribute of them you don't care about would make the positive reinforcements more effective on the things you do care about.

Example: trying to teach someone math, and praising them at everything they do right with the math, including trying, but complain abut their physique, fashion choices, hygiene, etc. Especially timing those unrelated complaints to when they seem less focused on the math but subtly enough they don't consciously notice the correlation.

Not that this isn't a bad idea for other unrelated reasons...

Would it? There would be greater contrast between the reinforcement and the ignoring of poor performance.

True. But I bet if coffeespoons makes this a routine thing, they'll eventually find themselves enjoying work more.

Perhaps this is why I like Autofocus better than GTD. "It is fine to have incomplete tasks in your task list".

Also, non-punishment for failures may be one of the distinctions between play-like work and work-like work.

I want to design a reinforcement schedule in one of our apps. Can anyone link me to some specific guidelines on how to optimise this?

(Reinforce exactly what % of successes (30%? 26%? 8%?)? Reinforce performances in the top 10% of past performances (or the top 12%, or the top 8%?)? How does time factor (if the user hasn't used the app for a week, should I push a reinforcer forward?)?)

Everyone who's looked at the data thinks that punishment can change behavior. The question is whether punishment makes the changes you want- and people dramatically overestimate the usefulness of punishment and dramatically underestimate the usefulness of positive reinforcement.

Yes, I think so and apparently so does Kahneman. I don't think this is particularly controversial. Kahneman does say that positive reinforcement is more efficient (both in animals and humans).

Some people react quite viscerally to the awareness that another party is trying intentionally to steer their behavior in any way. It seems to just be a massive squick button for some (indeed, I notice that most randomly-selected people who are made aware of explicit attempts to condition behavior react with discomfort at minimum); for others, there seems to be a correlation with triggers gained from abusive interactions earlier in life; a few I knew who reacted strongly showed strong indications of sociopathy and seemed to instinctively feel violated if someone else successfully, or even just obviously, tried to affect their behavior in a deliberate manner toward some end (a normal part of cognition and social interaction for them directed at others).

Seriously? You'd shun your wife because she said thank you?

(No, I said I would shun kisses delivered under those circumstances. No cutting and pasting of my keywords for the sake of hyperbole thanks.)

If people use their affection in a way that is obviously intended to systematically manipulate me to do things that I do not, in fact, wish to do then yes, of course those instances of affection I will shun. While I know some people are more tolerant to that kind of blatant disrespect I would expect you to at least be able to comprehend the subset of people that will not.

I'm afraid that all women who want kisses to serve the role of doggy treats within our relationship are out of luck. I have yet to experience a problem with having that policy. My model of myself predicts that rewarding hostile-to-my-interests-reward-training with increased compliance or acceptance would leave me with relationships that were far less satisfying and in particular far less enjoyment of displays of affection.

I do accept this kind of reinforcement from my significant other, assuming that:

Note that I consider the reinforcement you are describing to be entirely different in kind (not "this kind"). The boundaries around the kind I accept are approximately the same as yours:

• it is for a goal I agree with (extrapolated volition)
• I am free to say "stop doing this" if I don't feel like to be reinforced; and my wish is respected
• I do get the same signs of affection in other situations too.

I go by what my intuition tells me but when formalizing those intuitions something similar is generated.

Actually I consider it very useful, and for me it would be a waste not to use this kind of cheap "external willpower".

I make a point of rewarding desired reinforcement (while attempting 'extinction' on less desirable influence tactics like nagging or punishment.)

It may be worth sharing, anecdotally, that years ago my husband expressed annoyance with me over the fact that I only ever rubbed his back while he was doing dishes, and it made him feel much like how wedrifid describes.

This utterly bewildered me, so I agreed to pay attention to the behavior and see what was going on. Pretty quickly it became clear to me that this was absolutely true, for reasons I wasn't entirely clear on myself, although my working theory was it was the only time that I'd regularly walk past him while he was hunched over in that particular posture, which apparently served as a "give me a backrub" signal for me, for whatever reason.

My response to this was to start giving him random backrubs at other times, which solved the problem.

My point being that (a) being annoyed by this sort of behavior is not at all unique to wedrifid, and (b) whether the behavior pattern is intentional doesn't necessarily matter very much. (I don't mean to suggest that it doesn't matter to wedrifid; actually, they have made it somewhat clear that it's part of what they're objecting to.)

Is this because of the "damn it, I know I made a mistake, you telling me I did doesn't help!" effect?

No, I react the same way whether I was previously aware of my mistake or not. I only experience that effect when I'm told to do something I am already doing.

A good thought experiment is that if I was making a type of mistake that I couldn't automatically tell I was making on my own, I would prefer it to be pointed out, even if not in a concise detailed fashion–the idea of not knowing that I'm making a mistake is kind of scary. What would your reaction be in that situation?

Pragmatically, we as humans, just barely over the threshold into sapient intelligence, make mistakes we're not aware of constantly. If we didn't, we wouldn't need a superintelligence to fix the world; we'd have already done it ourselves. So finding the concept scary seems kind of pointless.(Sort of like being hydrophobic about the water in one's own body.) However, I would, of course, rather be aware of my mistakes than not.

But none of this is really on the topic, which was that the listed reinforcements don't seem even remotely applicable to humans in a universal way.

I think it's still hard to privilege if that kind of effect exists in the first place.

Too infrequent. They need to start by giving him an M&M every time he thinks about writing more HPMoR.

It worked with similar effectiveness as other techniques I implement - that means only until I have done enough to feel good about myself (2-5 productive days)...

When a dolphin does something wrong, the trainer doesn't respond in any way.

(from the link)

I think this should be "never downvote".

Seems to me that a downvote would associate negative valance with both the act of posting on LW and with whatever their specific mistake is, with the latter being stronger. So no vote and a comment with a mixture of praise and criticism is probably the stronger play if you're looking to improve someone's writing or fix some technical mistake while keeping them as a contributor, but a downvote is still effective if all you care about is seeing fewer posts of that kind.

never comment on bad posts or explain my downvote on LW

I think this should be "never downvote".

That would be true if the point was actually about implementing the reinforcement ideal rather than using it to validate a premeditated ideal.

More generally, a positive motivation often contains an implicit negative motivation -- a threat of not receiving the same reward next time. ("What pushes you forward, holds you back.")

Telling someone they are smart implies that the teacher has ability to judge smart and stupid students based on their work. So if tomorrow the work is not good enough, the same student could be judged as stupid. This could also happen if the student tries something new, where they obviously cannot have as good results as when they stick with what they already know well.

Telling someone they work hard avoids this danger somehow. Maybe because it contains an actionable advice what to next time to achieve the reward -- so it feels more under control, less threatening.

Maybe the secret is in finding a motivation that feels under control, but not too much to allow cheating. Maybe it's a moving target; I suspect that given enough time, some children in the experiment would find ways to appear working hard without doing the hard work.

Now that is exciting as hell. I've always felt that rationality was a potential competitive advantage for organizations that wasn't being utilized. Way to go.

Perhaps it's just the unexpected generosity.

I remember a few years ago I was reading about positive psychology's various findings, such as that spending money on experiences with friends or family made one happier than spending money on an object for oneself; so I tried out buying donuts or bagels for a few university clubs. Everyone seemed much happier, for what was a relatively trivial amount of money invested (<$10 in every case).

(I don't remember noticing that the donut people were happier than the bagel people.)

Hm, I think I wasn't clear. I'm interested in comment upvotes per comment, not top-level post upvotes per comment. My assumption is that everyone here is primed to upvote due to the content of the post that most of the comments here are highly upvoted.

Or maybe you really did measure that that? You are probably right that some long-lasting classic posts have gotten high numbers of upvotes on their comments over the years just since so many people have read them.

Love bombing is just a tool -- its morality depends on how it is used. In a typical situation it is used to ruin the person's natural resistance towards groups that exploit them; that is obviously evil.

A different thing would be to use love bombing with the person's explicit consent, as a reinforcement for things the person values, and for nothing else. Preferably for a limited time specified in advance. It could be a great tool to overcome akrasia.

LWers do many cultish things

How many?

That's a fair point: I may have been treating a conditional like a bi-conditional. I think my sense of the matter is this: if a friend told me that he spent a lot of our time together thinking through ways to positively reinforce some of my behaviors, even to my benefit, I would become very suspicious of him. I would feel that I'd been treated as a child or a dog. His behavior would seem to me to be manipulative and dishonest, and I think I would feel this way even if I agreed that the results of his actions were on the whole good and good for me.

Do you think this sort of reaction on my part would be misguided? Or am I on to something?

.... And here begins the debate.

What do we do? What do we think about this piece of freaking powerful magic-science?

I vote we keep it a secret. Some secrets are too dangerous and powerful to be shared.

Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M."

That story doesn't trouble you at all?

For most people, there's lots of low hanging fruit from trying to recognize when they are reinforcing and punishing behaviors of others. Also, positive reinforcement is more effective at changing behavior than positive punishment.

But that doesn't mean that we should embrace conditioning-type behavior-modification wholesale. I'm highly doubtful that conditioning responses are entirely justifiable by decision-theoretic reasons. And "not justifiable by decision theoretic reasons" is a reasonable definition of non-rational. Which implies that relying on those types of processes to change others behaviors might be unethical.

This statement without context is clearly incorrect...

You seem to know what I mean, so I won't go into a buch of unnecessary qualifications.

is A an act of convincing me?

Not necessarily. Is the meaning of 'convince' really unclear? Threatening someone with a gun seems to satisfy your description, but it's obviously not a case of convincing. I'm not sure what you're unclear about.

I disagree with it strongly.

If you care to explain why, please do so.

Leave sleeping mind killers lie.

'My Little SIAI: Positive Reinforcement is Magic'?

No.

How long has it been since you had a post that stabilized at net negative votes?

I hope you realize that LWers would give you M&Ms for this particular post regardless if you where right or not.

The fact that reinforcement can be very effective in changing frequency of behavior doesn't say that punishment should never be used to change the frequency of behavior.

Reinforcement is useful for increasing frequency of behavior. When decreased frequency of behavior is desired, punishment is the type of intervention to use. (For applied behavior analysis, those are the definitions of reinforcement and punishment).

Be aware that some people upvote comments "back to zero" that they wouldn't otherwise upvote. (Some other people consider this bad practice.)