Decision-Making and Accountability

post by Ian David Moss · 2020-12-14T21:30:18.493Z · LW · GW · 1 comments

Contents

  What the research says
  A better way to foster accountability
None
1 comment

Quick: what’s the best decision you’ve ever made?

(I’m serious! Take a second to think about it.)

.

.

.

Now that you have an answer, think about what it was about that decision that made it a good one. Was it the impressive array of interlocking spreadsheets you amassed in support of your analysis? Was it the masterful way in which you facilitated a stakeholder discussion to promote shared learning while respecting the diversity of perspectives in the room?

No, I’m guessing you think the decision was good because it worked out well! And this is totally normal. Judging decisions and actions by their outcomes is deeply embedded into our language and idioms. I mean, it just seems obvious.

It’s also deeply embedded in management psychology. The idea has become so widely accepted that you can find it in the names of popular frameworks like results-based management, performance-based budgeting, and the Objectives and Key Results (OKR) monitoring methodology.

There’s just one problem: judging decisions by their outcomes is a really tricky proposition. Most of us would agree that getting behind the wheel while sporting a blood alcohol level above the legal limit would be a bad decision, even if we reach our destination safe and sound. Same goes for holding a maskless house party during a pandemic, even if no one gets infected. To give a professional example, what if a team takes on a project that is unlikely to succeed but could provide massive benefits if it did, and meets with the expected result of total failure. Can we say that it was nevertheless a good decision to try? On the basis of the outcome alone, we would have to conclude that it was not.

So we have a bit of a puzzle to resolve. On the one hand, social-sector decisions can carry immense consequences. On the other, it’s not clear that we can reliably improve those consequences by holding managers, teams, and whole organizations accountable to them. So how can we design accountability systems that do lead to better outcomes?

What the research says

It turns out that there is a fairly robust scientific literature on both personal and organizational accountability. It says that in general, accountability is a good thing — a review of five decades of experimental research in the behavioral sciences found that accountability mechanisms “consistently” cause decision-makers to search for more relevant information, engage in more analytical decision-making strategies, and put more effort into the decision-making process, among other benefits. But that same literature largely agrees that letting decision-makers know they will be judged by their process, rather than the outcomes, “often yields more empirically accurate and logically defensible judgments.” Outcome-oriented paradigms, like the examples discussed in the previous section, are quite vulnerable to hindsight bias (wherein we assume that considerations that seem obvious after the fact should have been anticipated at the time when the decision needed to be made). In reality, low-probability events and unconventional causal chains often shape outcomes in unexpected ways, especially in complex environments. Looking to outcomes in such situations can thus hinder our learning efforts, as the success or failure of an initiative may cause us to see brilliance where there is only luck and overlook strong strategies that might well have worked in different circumstances.

So should we simply throw outcome-based accountability to the wind? Alas, it’s not quite that simple. A deeper look at the science of accountability paradigms shows that outcome- and process-based accountability each have distinct advantages, and an ideal management system will incorporate elements of both.

Researchers in both the behavioral sciences and public administration have drawn a distinction between outcome and process accountability since at least the 1980s. Oddly, the two disciplines have come to starkly different conclusions about which approach is more favorable.

In contrast to the experimental research in the behavioral sciences, which emphasizes the superior decision quality that can come from process accountability, public administration research often emphasizes the downsides of process accountability. For one thing, almost everyone agrees that process accountability slows the decision-making process down — which may be perfectly appropriate in some situations, but in others could be deeply frustrating for stakeholders and beneficiaries. Moreover, poor implementations of process accountability can also act as a drag on innovation if organizational culture comes to be focused on box-checking compliance exercises. Outcome accountability, by contrast, can incentivize faster action and more creative approaches to achieving the desired outcomes.

This divergence may reflect choices about frames and research methods as much as underlying realities. The bulk of behavioral accountability experiments have taken place in a lab setting, while the public administration literature tends to be more theory- and case-study-driven and based on experience with real-life organizations. A reasonable synthesis of the two perspectives could be that while the upsides of process accountability are real, the harms from poor implementation of process accountability are more salient than the benefits of thoughtful implementation, which may not be evident to decision-makers for months or even years.

It’s worth noting as well that the advantages of outcome accountability can get diluted if an emphasis on short- or medium-term success indicators makes it more difficult for an organization to think creatively about its theory of change for achieving its longer-term objectives. True outcome accountability for a complex and long-term mission, or what I might call “impact accountability,” should not treat the theory of change as set in stone and, in the context of reporting, might allow managers to cite whatever evidence seems appropriate to show that progress toward the grander vision is taking place.

A better way to foster accountability

Since both process- and outcome-based accountability systems have something important to offer, organizations should seek to integrate them in ways that match the types of challenges they face and environments they work in. Relevant factors to consider include the complexity of the underlying system(s), the heterogeneity of the decisions to be made, the time horizon of the strategy and the importance of speed, and the presence or lack of a smooth continuum between success and failure.

In general, outcome accountability works best when goal(s) are stable and clearly defined, progress toward each goal is readily apparent, and decision-makers have a limited window of opportunity to get it right. This can hold true even when the underlying system is highly complex, for example in a political campaign. By contrast, process accountability is a better fit for environments in which decision-makers are trying to solve persistent problems and still learning about what levers in the system can be most effectively pulled (e.g., reforming public education in the United States).

This breakdown would seem to suggest that most philanthropic and public sector programming is better suited to process accountability than outcome accountability. Still, there are ways we can improve process accountability by tying it to “outcomes” within the decision-making process that directly address its potential weaknesses:

• Accountability to strategy demands regular checkpoints to ensure a tight connection between an organization’s mission/vision/values, understanding of the landscape, theory of change, and day-to-day decisions. If any of these elements shifts, the others may need to evolve with it. As part of any accountability system, leaders should set up regular checkpoints to confirm this integration remains as tight as possible.
• Accountability to innovation ensures that decision-makers consider a diverse enough range of potential actions when considering next steps. Does your organization make a habit of brainstorming additional options to put on the table, especially when a decision is being framed as a “whether or not” choice?
• Accountability to judgment recognizes that decisions are inevitably predictions about the results of future actions. Fortunately, recent research has demonstrated that it is possible to track and improve one’s accuracy in judging such risks and opportunities if we are willing to make our forecasts explicit as part of the decision-making process.
• Accountability to context acknowledges that appropriate practices look different for different situations, and considers whether leaders and managers effectively matched the right kinds of decision-making approaches (including quick-and-dirty ones, where appropriate) to the situations and challenges at hand.

Ironically, the mix of process and outcome accountability mechanisms described above should, in the long run, yield better results than a supposedly pure emphasis on outcomes. With the right design elements, a hybrid accountability system can foster better-quality judgments and learning without snuffing out an organization’s imagination or saddling employees and constituents with burdensome red tape. Effective management requires creativity and focus, structure and flexibility.

This article grew out of a memo created for the Omidyar Network as part of a consulting engagement in 2020. My thanks to Jessica Kiessel and Clara Bennett for their support and feedback.

1 comments

Comments sorted by top scores.