Trial and error in policy making
post by DavidAgain · 2011-06-15T10:39:10.812Z · LW · GW · Legacy · 8 commentsContents
8 comments
I’m working in government in the UK, and interested in rationality in policy making: at the individual level, but also how we can build systems (in terms of requirements during policy creation, creation of independent bodies, incentive structures for officials/ministers…) that encourage more rational policies. This is the context for the stuff below.
I recently went to a talk on the use of trial and error in policy making. It was hosted by the Institute for Government (a non-political think tank advising ministers and civil servants on good government), and included the Undercover Economist, Tim Harford. Details including a recording can be found here: http://www.instituteforgovernment.org.uk/our-events/105/government-by-trial-and-error-a-discussion-with-tim-harford
Ben Goldacre was also attending in the audience, and has written about this: http://www.badscience.net/2011/05/we-should-so-blatantly-do-more-randomised-trials-on-policy/
The basic message of the talk was that much of policy could benefit from being trialled more effectively, whether in the form of formalised, controlled trials: following the medical example inasmuch as that is practically possible. This approach seems to be being seriously considered by this government in several areas: one example of how it might work in theory can be found here: http://www.straightstatistics.org/article/what-works-criminal-justice-time-find-out
Of course, the reality is often messier, and ministers are looking for both swifter results and more security. Other forms of ‘experimentation’ involve devolving power to local councils (Localism Bill) or schools (Free Schools) etc. In these cases there aren’t the same formal comparison methods but you at least get lots of ideas and some ability to judge which ones have gone well. An underlying issue is the quality of assessments: a policy that has clear criteria for appraisal built into its design will be much easier to judge, whereas one that is simply acted upon and studied later might suffer from poor evidence and be more open to being interpreted in line with preconcieved or desired outcomes.
A final approach is ‘payment by results’, where tasks such as preventing reoffending or getting people off drugs is paid for on the basis of successes achieved rather than services delivered, the idea being that this provides a fiscal incentive to get it right. A critical view of one such scheme is here
http://www.straightstatistics.org/article/flaws-mojs-reoffending-study
I was wondering what people here felt about the idea of trial-and-error in policy. I think it seems like a good idea, with the following limitations
- a) some policies probably can’t be trialled on a national scale (can you have half the country making cuts while the other half keeps spending?)
- b) some cases will have ethical issues as with medicine
- c) some cases might not allow clear-cut results, eliminating the ability to learn and simply creating a chaotic mess of different approaches
- d) the biggy: I don’t know if the political culture is open to it.
Any thoughts?
8 comments
Comments sorted by top scores.
comment by D_Alex · 2011-06-16T09:26:57.253Z · LW(p) · GW(p)
FWIW, I think the impediments to rational policy making do not generally stem from lack of data (and hence will not be solved by using a trial-and-error approach to evaluate policy effectiveness). They mostly stem from objectives other than "rational policy" - e.g. re-election, which can be better achieved by making policies irrational (in certain ways).
Replies from: DavidAgain↑ comment by DavidAgain · 2011-06-17T07:25:23.404Z · LW(p) · GW(p)
For some big ticket policies, this is probably true: but at the very least, there are lots of little sub-policies which are attempting to deliver the big aims, and the precise details of how these are done is not likely to effect re-election. Except that there's a motive to do them well so they can push the bigger outcomes (higher employment, economic growth, lower rates of illness...) in the right direction.
comment by Jay_Schweikert · 2011-06-16T06:25:38.149Z · LW(p) · GW(p)
David, if you haven't already seen it, you might want to check out Randomizing Law, a bold article written by several prominent law professors, recently published in the University of Pennsylvania Law Review.
More generally, the value of trial-and-error seems like one of the major benefits of a federal system, rather than a wholly national system -- the classic "laboratories of democracy" argument. Of course, composite states themselves probably wouldn't be any more open to direct "testing" than nation states themselves, so you wouldn't exactly be able to run experiments. But just being able to compare natural results across states could likely achieve at least some of the benefits of true trial and error.
Replies from: DavidAgain↑ comment by DavidAgain · 2011-06-17T07:26:54.187Z · LW(p) · GW(p)
Cheers, I'll check out the article. The talk I went to explored both formal and informal trials. I think laboratories of democracy are great, if the uncontrolled tests are nevertheless properly and clearly assessed. The problem is that when governments have policies inspired by other countries, there is often confused data on whether it's worked in those cultures!
comment by asr · 2011-06-16T06:23:13.259Z · LW(p) · GW(p)
Some notes about limits to governmental experiments:
As I understand, it would be a grave violation of medical and scientific ethics to experiment on somebody, in possibly dangerous ways, without their consent. Government actions routinely lack the consent of some of the people being acted upon. The cases are therefore not parallel. Since consent to a proposed policy measure is likely highly correlated with political affiliations, a randomlzed trial is likely to exacerbate the problem.
Moreover, I think experimenting on citizens is likely to dangerously reduce the legitimacy of the government. Much of government's claim to authority is that it's good for the citizens. Being experimented upon without consent might very easily NOT be good for the citizens concerned. "You are using us as guinea pigs for your harebrained social engineering experiments that resulted in negative consequence X" seems like a powerful election slogan.
Some experiments are probably in violation of national constitutions. It's a very basic principle of liberal democracy to treat all citizens equally and under the same laws. So you'd have a problem if you tried to experiment, e.g., with giving some people different rights or duties than others.
Replies from: DavidAgain↑ comment by DavidAgain · 2011-06-17T07:23:00.056Z · LW(p) · GW(p)
I must say, my assumption was very much this. But as far as I can make out, we are currently planning to trial certain things such as experimenting with different jail times: which I'm surprised got past an ethics committee, so to speak.
In Britain, the only likely risk is if it breaks human rights: we don't have a constitution in quite the same way.
Of course, you could let people opt-in to the trial, but to make it meaningful you'd have to only provide the new service/approach to some people who opted in.
It's odd that in some ways the idea of a deliberate trial with the aim of getting the best policy seems less shocking than devolving power and having a host of different approches spring up naturally.
comment by sixes_and_sevens · 2011-06-15T16:00:06.620Z · LW(p) · GW(p)
Do you have any more specific questions? I suspect the majority of LWers would have a very favourable view of trial-based policy, but there's an awful lot that can be said about it.