Fixing Moral Hazards In Business Science

I wonder if there should be a group where they still get Beeminder's graph, but they don't pay anything for going off their road. (In order to test whether the pledge system is actually necessary.)

Not cool to screw them if they make a mistake

This is exactly the moral hazard companies face with the normal procedure too.

The main advantage I see is that the webapp approach is much cheeper allowing companies t do it early thus reducing the moral hazard.

The privacy issue here is interesting.

It makes sense to guarantee anonymity. Participants recruited personally by company founders may be otherwise unwilling to report honestly (for example). For health related studies, privacy is an issue for insurance reasons, etc.

However, for follow-up studies, it seems important to keep earlier records including personally identifiable information so as to prevent repeatedly sampling from the same population.

That would imply that your organization/system needs to have a data management system for securely storing the personal data while making it available in an anonymized form.

However, there are privacy risks associated with 'anonymized' data as well, since this data can sometimes be linked with other data sources to make inferences about participants. (For example, if participants provide a zip code and certain demographic information, that may be enough to narrow it down to a very few people.) You may want to consider differential privacy solutions or other kinds of data perturbation.

http://en.wikipedia.org/wiki/Differential_privacy

I was thinking something like the karma score here. People could comment on the data and the math that leads to the conclusions, and debunk the ones that are misleading. A problem would be that, If you allow endorsers, rather than just debunkers, you could get in a situation where a sponsor pays people to publicly accept the conclusions. Here are my thoughts on how to avoid this.

First, we have to simplify the issue down to a binary question: does the data fairly support the conclusion that the sponsor claims? The sponsor would offer $x for each of the first Y reviewers with a reputation score of at least Z. They have to pay regardless of what the reviewer's answer to the question is. If the reviewers are unanimous, then they all get small bumps to their reputation. If they are not unanimous, then they see each others' reviews (anonymously and non-publicly at this point) and can change their positions one time. After that, those who are in the final majority and did not change their position get a bump up in reputation, but only based on the number of reviewers who switched to be in the final majority. (I.e. we reward reviewers who persuade others to change their position.) The reviews are then opened to a broader number of people with positive reputations, who can simply vote yes or no, which again affects the reputations of the reviewers. Again, voting is private until complete, then people who vote with the majority get small reputation bumps. At the conclusion of the process, everyone's work is made public.

I'm sure that there are people who have thought about reputation systems more than I have. But I have mostly seen reputation systems as a mechanism for creating a community where certain standards are upheld in the absence of monetary incentives. A reputation system that is robust against gaming seems difficult.

Max L.

StackOverflow et al are also free and offer no compensation except for points and awards and reputation. Maybe it can be combined. Points for regular participation, prominent mention somewhere and awards being real rewards. The downside is that this may pose moral hazards of some kind.

One option is simply to report it to the Federal Trade Commission for investigation, along with a negative publicity statement. That externalizes the cost.

If you would like assistance drafting the agreements, I am a lawyer and would be happy to help. I have deep knowledge about technology businesses, intellectual property licensing, and contracting, mid-level knowledge about data privacy, light knowledge about HIPAA, and no knowledge about medical testing or these types of protocols. I'm also more than fully employed, so you'd have the constraint of taking the time I could afford to donate.

Max L.

I think I get your meaning. You mean that the webapp itself would carry out the testing protocol. I was thinking that it would be designed by the sponsor using standardized components. I think what you are saying is that it would be more rigid than that. This would allow much more certainty in the meaning of the result. Your example of "using X resulted in average weight loss of Y compared to a control group" would be a case that could be standardized, where "average weight loss" is a configurable data element.

Max L.

There are people in my department who do work in this area. I can reach out and ask them.

I think Mechanical Turk gets used a lot for survey experiments because it has a built-in compensation mechanism and there are ways to ask questions in ways that filter people into precisely what you want.

I wouldn't dismiss Facebook ads so quickly. I bet there is a way to target mobile app developers on that.

My hunch is that like survey questions, sampling methods are going to need to be tuned case-by-case and patterns extracted inductively from that. Good social scientific experiment design is very hard. Standardizing it is a noble but difficult task.

Roughly speaking: "Act normal" vs "Use Beeminder" vs "some alternative intervention". Basically, I expect to see "do something different" produce results, at least for a little while, for almost any value of "something different". Literally anything at all that didn't make it clear to the placebo group that they were the placebo group. Maybe some non-Beeminder exercise and intake tracking. Maybe a prescribed simple exercise routine + non-Beeminder tracking.

This kind of research lends itself extremely well to protocol templates

This is not at all self-evident to me. How, for example, would you demonstrate product safety (for a diverse variety of products) via a standard template?

Use a standard template then get back to work.

I don't see how a "template" frees one from

significant time costs in setting up the data collection, and performing the analysis.

Research costs money and requires competent people. If it were possible to do meaningful research on the cheap just by reusing the same template, don't you think it would be a very popular opinion already?

I have thought a bit more about the blinding issue.

One question that comes to mind: How do we trust that the placebo is a real placebo and substantially different then the drug? The company producing the product wants to show a difference. Therefore they have no incentive to give both parties the same product.

On the other hand the company could mix some slight poison into the placebo. Even an ineffective drug beats a poison.

Therefore the placebo has to be produced or purchased by a trusted organisation and that organisation has to package the placebo in the same box that it's packaging the drug.

You ship both a placebo package and a non-placebo package to the participant, and have them flip a coin to decide which one to use. They either throw away or disregard the other package for the duration of the study.

That's possible but it means that you double your product costs. The advantage would be that you can do crossover trials.

In the case of your probiotic a crossover trial might to worthwhile even without this reason.

In this case you could encrypt a list with products ID for placebos and non-placebos before you ship your product. The participant has no opportunity to know which of the two products are placebos.

When he starts the study the participant puts the product ID of the product he decides to use into the webapp. He also puts the ID of the product he doesn't want to use in the webapp.

The participant has no way to decide between the two packages or know which one is the placebo and which one is the real thing so he doesn't need to go through the process of flipping a real coin.

Once the study is finished you releases the decryption key that can be used to distinguish placebos from real products. You can give the decryption key to a trusted third-party organisation so that your company can't prevent the results of the study from being published.

You ship N packages to Total/N participants. The participants which receive N packages then randomly assigns himself a package, and randomly distributes the remaining (N-1) packages to other participants.

That means the participants has to do the work of going to the post office and remailing packages. Some of them will require additional time to remail and it might produce complications.

About the shipping of products and placebos to people, I see a physical way of doing it, but it is definitely not scalable.

Maybe you can win a company such as Amazon as a partner for distribution. Amazon warehouses can store both products and placebos and ship randomly.

For Amazon it should be relatively little work and it could be good PR.

Labels don't have to be printed on the bottle. Amazon has the capabilities of adding a paper with study instructions to a shipment and that paper can contain a unique ID for the product. Amazon could again publish an encrypted version of the ID at the beginning of the study and release the decryption key at the end of the study.