Incentives Matter – The Misuse of Metrics

The lyrics of the rapper Notorious B.I.G chronicled the routine violence on the streets of the neighborhood known as Bedford-Stuyvesant.  Unfortunately, his songs reflected reality, and this Brooklyn neighborhood continues to be one of the highest crime areas in New York City.  The 81st precinct which covers the eastern part of  “Bed-Stuy”, has recently been the setting for an eye opening series of investigations involving police corruption.

The genesis of these investigations was the secret recording of police roll calls by Adrian Schoolcraft, then an officer at the 81st. Roll calls are a key part of precinct life.  They’re an opportunity for the station’s “brass” to send a daily message to patrol officers about priorities and focus.  In the case of the 81st, Officer Schoolcraft recorded some disturbing messages from commanders.

Officers were repeatedly exhorted to write specific numbers of summonses for such violations as not using a seat belt or talking on a cell phone while driving.  They were also strongly encouraged to perform a high number of “stop and frisks”, a controversial tactic that has led to complaints around racial profiling.  At the same time, the officers were pressured to be skeptical of crime victims and refuse to take crime reports under certain conditions.  Additionally, supervisors would get involved in reclassifying crime reports to make them less serious.

The roll calls were not just “pep talks”, but high pressure sessions in which officers were threatened with serious disciplinary action if they did not meet their quotas.  To understand the back story behind this culture, one must step back in time to 1995 when the NYPD instituted a system for analyzing crime data called CompStat.  CompStat was credited with significant reductions in crime.  A key feature of CompStat is a weekly report for each precinct showing crimes, arrests and summonses activities.  The roll calls that Officer Schoolcraft recorded showed the enormous political pressure on commanding officers to show favorable CompStat numbers.  It’s easy to see how this culture of statistical reporting could take hold, with the police commissioner and mayor both wanting to show continuously improving numbers.

After Officer Schoolcraft reported his concerns to an NYPD Quality Assurance Division, an investigation ensued, resulting in charges against the precinct’s commanding officer, Steven Mauriello.  Interestingly, after new a new team was installed to supervise the precinct, there were sharp increases in reported crimes.  While not proven, it is suspected that this was the result of more accurate reporting measures.

In the corporate world, we can witness similar types of unfortunate leadership behaviors.  When you create incentives and punishments around the achievement of particular statistical goals you will create unintended consequences.  People will strive hard for those numbers, without concern for anything else.  Examples in the enterprise are common, from project ontime percentages to sales quotas to departmental budgets.  In each case, a blind pursuit of a target number in a vacuum is a recipe for poor overall performance.

Let’s examine one specific measure in the context of a typical IT organization.  Many organizations track ontime performance of projects.  A traditional process for tracking these projects involves a meeting where project owners meet with management to track weekly progress.  In organizations that are focused religiously on specific metrics, the process can become narrowly focused and rather unpleasant.  An individual who is running late on a project can be singled out for intense “cross examination”.  The message becomes clear; under no circumstances should you show up at the meeting with a late effort.

If this pressure simply resulted in 100% on-time performance, it could be excused as a rational management practice.  Unfortunately, as in the case of the 81st Precinct it results in counterproductive behaviors.  Those individuals who want 100% on-time performance will do their best to play the system to make their numbers.  They will “pad” their project plans with additional time.  If they are running behind, they’ll cut corners in areas like documentation and testing that are hard to initially uncover.  They will also cease being of help to any fellow team members or customers.  All of these negative behaviors will be difficult to observe or measure.  The organization, however, will foolishly believe it has achieved success by meeting it desired metric goals.

The problems in the 81st Precinct demonstrate what happens when statistics are confused with results.  The goal of a police department should be to improve the quality of life of citizens by reducing actual crime.  Focusing officers on ticket writing simply diverts resources away from more important work, especially in high crime areas.  Dissuading citizens from reporting serious crimes, or downgrading those reports does nothing to reduce actual offenses.  It simply alienates the citizens and demoralizes the officers.

Enterprises can learn from the experience of the 81st.  Here are some best practices for avoiding metrics with poor incentives:

  • Make sure your operational metrics are consistent with your overriding goals around customer satisfaction, growth and long term profitability
  • Never make metrics a “do-or-die” proposition
  • Utilize rank and file teams to help create their own metrics
  • Get active and open feedback from team members on the metrics process

In an age of evidence based management and easy access to data, metrics will continue to be an important method for managing teams and firms.  Progressive minded enterprises can incorporate metrics in a smart way that minimizes unintended, negative behaviors.

This entry was posted in Organization, Probability/Statistics, Project Management. Bookmark the permalink.