Friday, September 23, 2016

Asymmetric Scrutiny, Superforecasting, and Public Policy


by Nick Charney RSS / cpsrenewalFacebook / cpsrenewalLinkedIn / Nick Charneytwitter / nickcharneygovloop / nickcharneyGoogle+ / nickcharney

A few weeks ago I wrote about the problem of asymmetric scrutiny as it relates to the change agenda and public sector innovation culture more broadly speaking (See: Asymmetric Warfare: Agents of Changes vs Agents of the Status Quo). In short I think the problem of asymmetric scrutiny significantly impacts our organizations and the innovation agenda writ large. What I didn't explore last week was how the cultural practice applies to the development of public policy options and public opinion. However, before we can make that connection more explicit, there's an important bridging concept that's worth introducing: superforecasting.

Superforecasting: The Art and Science of Prediction is a book written by Philip E. Tetlock and Dan Gardner (who was later hired as an advisor to the current Prime Minister) released in 2015; the book details findings from The Good Judgment Project and generally explains the art and science of prediction. Its worth noting that the book itself was rumoured to be making the rounds politically in Ottawa (though this was largely overshadowed by talk of deliverology) and the concept was the subject of one of presentations at the last Policy Ignite.

I won't walk you through the whole thesis but essentially the books makes the case that people are generally pretty bad at making predictions about the future (i.e. most people are bad forecastors). Superforecastors are different in that they represent a very small subset of people who can assign a numeric probability (i.e. make odds) to the likelihood of particularly complex global event occurring (e.g. Brexit) with a high degree of accuracy. If you are looking for a good introduction to the topic I would suggest listening to an episode of Freakonomics entitled: "How to be less terrible at predicting the future" as it provides a great introduction to the concept. The podcast included an interview with Superforecasting co-author Phillip Tetlock where he summarized some of the most important characteristics of superforecastors. While all of these characteristics may be important for superforecasting some of them are more important than others when it comes to improving how we understand the problem of asymmetric scrutiny; more specifically:


  • Starting with an outside view rather an inside view
  • Willingness to change your view in the face of new information


My basic premise being that asymmetric scrutiny is prevalent precise because we are generally terrible at these two things; and since we can't accurately predict the future we measure its worth, or hold it to account, with the yardstick of the past. In other words, the two phenomena go hand in hand. Let's explore each of these characteristics in turn.

Starting with an outside view rather an inside view

First, 'starting with an outside rather than an inside view' means looking at the broader trends (rather than the specifics of the particular situation) and using the broader context as an anchor for prediction (rather than the specifics of the immediate and narrow circumstances). This isn't generally something that we do from either a change or public policy perspective. In my experience the downward pressure within the bureaucracy typically comes to bare on the specifics of a given change initiative rather than the broader context from within which it is being advanced. It doesn't meet the specifics of guidelines X, or it fails to align with corporate initiative Y, or Z dollars is too costly in today's figures. The pressures are seldom about how a particular initiative is out of sync with the generalities of zeitgeist, flies in the face of the workplace culture we are espousing, or might not generate the anticipated value over the lifespan of the project. Take Blueprint 2020 as a concrete example, many people have taken issues with its specifics but few can argue that it was not a step in the right general direction.

The same thinking applies to the formation of public policy. Look at all the concern about the implications of self-driving cars -- epitomized by the discussions about what algorithms should decide to do in a 'who to kill' situation where loss of life is inevitable. Public discourse on this issue tends to over emphasize the issue of deaths due to autonomous vehicles in absolute terms (i.e. taking an inside view) rather than as a percentage of the overall mortality rate for traffic accidents (there were 1.25 million road traffic deaths globally in 2013). While concerns about autonomous vehicles causing accidents is real, perhaps it ought not to factor so heavily into how we understand the issue. Taking an outside view rather than an inside view on this issue might alter the balance of the discussion and reshape the public discourse. The inside view generates asymmetric scrutiny on autonomous vehicles, shapes public opinion and thus limits the government's ability to make 'progressive' (outside view driven) policy.

Willingness to change your view in the face of new information

Second, the 'willingness to change your view in the face of new information' is conceptually very straightforward but occurs rarely in practice in large permission-based cultures. These cultures tend to make sense of new information by contextualizing it within the current frame or rule set, they do not easily re-frame or change the rule set in the face of new information. This is precisely why the Treasury Board Policy Suite is something that needed to be reset -- it finally became apparent that so much of it was stale -- rather than something that adapted and changed over time as the context changed. The problem here is well known and again directly connected to the notion of asymmetric scrutiny. How many people are responsible for ensuring compliance with the rule set across the organization, all of whom are unable to change the rules in the face of new information and forced to apply them as they are written. This dogmatism is the very definition of the problem asymmetric scrutiny and illustrates why the problem is systemic and slows innovative forces within our institutions.

Again the same thinking applies to the formation of policy options. Being slow to move from wherever public discourse is currently anchored slows down our policy response and often means we end up behind the eight-ball, bringing policy solutions that are largely about mitigation rather than prevention. This can especially be the case when highly specialized knowledge (technical or scientific) is slow to move into the mainstream. The impact of tobacco on public health in the 1990s was a good example of this as is (perhaps) the impact of sugar today. All in all this makes 'progressive' policy incredibly difficult to pursue.

How do we do culture better?

First, we can change hiring practices so that it better privileges candidates who approach problems from an outside rather than an insider view; candidates who are willing to change their view in light of new information.

Second, we can set mandatory review dates for all of the guidelines and directives in the Treasury Board Policy Suite to ensure that the policy framework is evergreen and invest the necessary resources to prune it and keep it healthy so that it never needs to be 'reset' again.

How do we do policy better?

First, we can introduce more specific and purposeful policy making techniques (e.g. foresight) that privilege the outside view by design.

Second, we can popularize highly technical and/or scientific knowledge by using plain language to introduce complex ideas and raise issues with the general public (e.g. public education) to move public opinion.


Wednesday, September 21, 2016

Public engagement and hard policy evidence


by Kent Aitken RSS / cpsrenewalFacebook / cpsrenewalLinkedIn / Kent Aitkentwitter / kentdaitkengovloop / KentAitken

Last week I pressed send on my dissertation, on which I'll blame my lack of posting. The topic was public engagement in Canada, and particularly, the role of economic analysis. My plan is to reduce the interesting parts of that research into a readable length, but I thought I might share six points that fell out of my conclusion.

1. Public engagement on policy, program, and service development is a thing. There's always a slate of ongoing public consultations in Canada, but the pace has picked up in the last year and the major difference is that there are far more that are intended for a broad public audience, rather than niche stakeholder groups. There are pros and cons to this.

2. As a general rule, government consultations are designed to understand what citizens value, but in a qualitative, rather than quantitative, way. That is, public input is viewed as a source of ideas and general feedback, not as empirically rigourous data. As currently practiced, public engagement is better suited for generating general insights, achieving social licence for policies, and avoiding major pitfalls than it is for systematically adding to the evidence base for policy options.

3. Each public engagement activity is important. Each represents an experience through which citizens' trust in government, and their willingness to participate in future engagement, can rise or fall. Public perception of the value of these engagements is crucial. The major variables here are the extent to which public input can theoretically influence policy, and the extent to which governments can prove that input was meaningfully considered. 

4. While it can be appropriate for governments to seek public input for general ideas and feedback, there's a massive downside. The greater the extent to which public input can be considered hard evidence, the easier it is for governments to incorporate that input in policy decisions, and the easier it is for governments to demonstrate how it influenced policy. There are many goals to engagement, including education, consensus-building, and legitimacy, but insofar as better policy is a central goal, engagement should be designed to produce data.

5. Public engagement is complex. There are hundreds of studied formats, each requiring a set of detailed design decisions, to align governments' needs with citizen satisfaction while producing the required data and insights. However, the way public engagement is governed, most of these design decisions are lost. If I may be blunt, it's essentially like a first-time homeowner overruling an architect on how their plumbing and electrical will work.

6. Governments need to build capacity for public engagement, particularly digital, but as per the above they may already have more capacity than they realize. So they must also develop governance that prioritizes expertise and good practices over ad hoc goals. The strongest version of this would be governance that includes public input and oversight on how engagement is designed and evaluated.






Friday, September 9, 2016

Asymmetric Warfare: Agents of Change vs Agents of the Status Quo


by Nick Charney RSS / cpsrenewalFacebook / cpsrenewalLinkedIn / Nick Charneytwitter / nickcharneygovloop / nickcharneyGoogle+ / nickcharney

Last week I shared a short reflection on the problem of asymmetric scrutiny (See: The Problem of Asymmetric Scrutiny) – the cultural norm of applying a higher approval threshold to new things than we apply to the status quo. If this observation is indeed correct – and many have told me recently that it is – than perhaps there are lessons to be gleaned from the idea of asymmetric warfare.

Now – to be clear – I’m not necessarily of the view that there is a culture war between innovation and the status quo (though perhaps there is) and I usually don’t like using combative metaphors; that said, I think it’s worth dusting off my undergraduate PoliSci text book and exploring the idea a little further.

What is asymmetric warfare?

Asymmetric warfare is simply a conflict between actors whose relative power, strategy or tactics differ significantly. It’s typically it’s a conflict between a traditional force and resistance movement. This power asymmetry creates situations where each party attempts to exploit each other’s characteristic weaknesses. This makes it incredibly difficult to anticipate who will win the conflict because there is no way to effectively measure the resources available to either side.

But why do weaker actors fight stronger ones? 

Academics have a number of hypotheses as to why this happens, including:
  • Weaker actors may have secret weapons
  • Weaker actors may have powerful allies
  • Weaker actors must consider other weak rivals when responding to threats from powerful actors
  • Stronger actors are unable to make threats credible
  • Stronger actors make extreme demands
And how do weaker actors win?

And a number of theories as to how weak actors can defeat strong actors even when they lack traditional sources of power:
  • Weaker actors can more reliably implement a coherent strategy
  • Weaker actors are willing to suffer or bear higher costs
  • Weaker actors are supported by external actors
  • Stronger actors are unwilling to escalate the severity of their response when their demands are not met
  • Changing attitudes of the rivals over time

How asymmetric warfare apply to innovation? 

Asymmetric warfare (as explained above) is not a dissimilar characterization of what “innovators” face when they run up against the “no-machine” (protectors of the status quo).

Power asymmetry between agents of change and agents of the status quo creates leads both parties to to exploit each other’s weaknesses to gain the advantage. For example, change agents (the resistence force) try to outmanoeuvre the system by moving faster, ignoring established rules, and cutting hierarchy when it suits them; whereas, agents of the status (the traditional force) counter by clogging the system to slow it down, deploying checks and balances that favour them and invoking hierarchy wherever it is to their advantage. The net result here is similar: it's hard to pick winners in the innovation ecosystem.

Why do agents of change fight?

They could fight because:
  • Agents of change may be better at anticipating important shifts in the terrain ("secret weapons")
  • Agents of change may have strong influence networks ("powerful allies")
  • Agents of change must consider other competing interests moving through the system ("weak rivals") 
  • Agents of the status quo may be unable to make the consequences meaningful to agents of change ("credible threats")
  • Agents of the status quo may require proof beyond that which is reasonable to agents of change ("extreme demands")

How do agents of change win?

Change is likely when:
  • Agents of change can implement and articulate a coherent strategy
  • Agents of change are willing to suffer or bear higher costs (e.g. personal reputation or career progression)
  • Agents of change are supported by external actors (e.g. politicians, think tanks, scientists, etc.)
  • Agents of the status quo are unwilling to escalate the severity of their response when their demands are not met 
  • Changing attitudes of the rivals over time

How does all this help us understand the problem of asymmetric scrutiny?

Within this context I think we better can understand that is likely more purposeful than accidental. it is a tool deployed tool by agents of the status quo against agents of change. The former control where scrutiny is applied and to what degree and so they apply it asymmetrically against those who would undermine their authority (caveat: this is likely more frequently and more simply referred to under the rubric of 'accountability').

The lesson then is that agents of change must not only be champions for the thing they want to see changed or implemented but must also be engaging in active and frequent discussions about accountability and scrutiny and how a better balance ought to be struck between that which is new and that which is not.





Friday, September 2, 2016

The Problem of Asymmetric Scrutiny


by Nick Charney RSS / cpsrenewalFacebook / cpsrenewalLinkedIn / Nick Charneytwitter / nickcharneygovloop / nickcharneyGoogle+ / nickcharney

One of the largest barriers to innovation is simply the problem of asymmetric scrutiny.

In short, the prevailing cultural norm is to apply more scrutiny to the new than to the status quo.

I think if we applied scrutiny more evenly -- even if say we only split the current difference -- we'd probably be doing far less of what we currently do and a lot more of what we don't (yet) do.

Archive

Disclaimer


Note that while we work as public servants this is entirely our own initiative and what we post here does not necessarily reflect the view of the government, our offices or our positions there in.

Notez bien que nous travaillons commes functionnaires, ceci est entièrement notre propre initiative et ce que nous publions sur ce site ne reflète pas nécessairement le point de vue du gouvernement, de nos organisations ou de nos postes.

Subscribe