The NFL Draft – Steals, Busts and The Science of Recruiting

Football PlayersThis past week, an important rite of spring took place; the annual NFL draft. This highly followed process allows pro football teams to replenish their rosters with fresh talent from the college ranks. It has become a media spectacle, with fans and pundits alike, ranking the prospective draftees and grading teams after they have made their picks. Arguably, there is no recruiting process in America in which an institution invests more time and resources. Which brings up an interesting question: Does the NFL draft provide useful lessons for other institutions, such as corporations, regarding the recruitment of personnel?

Before answering that question, let’s look at some important aspects of the draft process. First, the draft is an essential element for staffing. The signing of free agents represents the only other viable way for a team to improve their roster. Teams are limited by league rules to having 53 players on their active roster. In a typical year, teams are able to draft somewhere between 6 and 10 new players. Selecting the best possible players is an important element in a team’s future success.

Because of how critical the draft process is, teams invest huge amounts of resources in an attempt to make the best possible recruiting decisions. They have full time scouts that attend college games, watching and grading every play of the top prospects. These scouts review endless amounts of game film, re-watching countless plays in slow motion, from multiple angles. They attend the NFL Combine, where players are put through a series of standardized drills and tests, rating everything from speed to strength to intelligence. Teams will then bring their top prospects to their headquarters for personal workouts and interviews.

During the draft itself, each team sets up a sophisticated war room. This NASA-like environment has owners, general managers, coaches and scouts huddled over laptops, video screens and posters – chock full of key player data. As the draft unfolds, the team is constantly reevaluating the remaining talent, considering trades with other teams, and ultimately landing on their choice for each round.

Upon completion of the draft, it’s become a ritual to grade teams on the effectiveness of their picks. This starts instantly with tweets, blog posts and articles and frequently becomes a part of team folklore, many years later.┬áThose players that ultimately outperform expectations are labeled steals, while those who significantly underperform are called busts. The classic example of the former is Tom Brady, a sixth round pick (199th player selected) in the 2000 draft, widely regarded as one of the greatest quarterbacks of all time. The epitome of busts is Ryan Leaf, the 2nd pick of the 1998 draft, who had a short, unproductive career, highlighted by poor play, injuries and behavioral incidents.

There has been significant debate amongst journalists, academics and fans regarding the ability of teams to effectively judge the talent of prospective draftees. The conventional wisdom seems to be that teams differ in their ability to distinguish talent, with some organizations consistently making the better selections. However, some folks, including noted author Malcolm Gladwell, have made the alternative claim that forecasting the success of prospective NFL players, especially quarterbacks, is not really possible. This school of thought believes that there is a huge element of luck involved in making effective draft selections.

The economist Cade Massey has done some of the most significant research on this topic. His findings seem to reflect an amalgam of the man-on-the-street viewpoint blended with the skeptical view of Gladwell et al. He recently presented his findings in a talk entitled Flipping Coins in the War Room: Skill and Chance in the NFL Draft that he delivered at the Sports Analytic Conference at MIT Sloan. Based on extensive research of previous NFL drafts, Massey arrived at the following conclusions:

  • Teams are able to do a pretty good job of identifying talent. Those players selected earlier in the draft, on average, have more successful pro careers than those selected in later rounds.
  • Teams are roughly equal in their draft selection prowess. While there may be significant variance in a single year, over the long term, the differences are small.
  • While teams are generally able to identify the better players, there is still a large component of luck within the selection process. It is not unusual for later picks to have more successful careers than earlier selections.
  • Since the selection process is prone to error, and personal bias, utilizing a broad set of independent evaluators provides the greatest level of assessment accuracy (wisdom of the crowds).
  • Since there is an element of chance, a smart strategy is to amass a greater number of picks (usually done by “trading down”).

Now let’s compare the NFL draft to the recruiting process typically utilized in corporate America. In traditional institutional settings recruiting tools have historically been limited to a classic triad: resumes, interviews and references. These three devices have some significant limitations as selection tools. First, resumes are documents created by the candidate, designed to offer a favorable, rather than critical view. They can weed out candidates that are completely “off base” for the advertised position or serve as a discussion document for the interview. Interviews have historically been conducted in unstructured formats. That is, each interviewer utilizes their own personal style, conducting the interview in an ad hoc fashion. There is a significant body of research that has found that unstructured interviews are poor predictors of future job success. The last tool, references, much like resumes, are supplied by the candidate. It is rare to have an individual that can’t find someone willing to speak highly of their accomplishments and potential. As such, self-supplied references, like unstructured interviews, are a weak predictive tool.

In addition to utilizing recruiting tools that are inferior to those used by NFL teams, corporations have an even greater challenge of mapping prospective employees to new job roles. While there are definite differences between college and pro football, and team to team differences, a draftee’s college role maps fairly well to their pro role. A quarterback is still a quarterback, with roughly equivalent responsibilities. In the corporate world, the actual duties and success criteria associated with jobs of equivalent title can be quite varied. As opposed to the NFL, where each team is playing the same game, with the same positions, and consistent rules and goals, the corporate world has a much greater degree of variability. Companies can have vastly different sizes, cultures, mandates, and objectives. A job in one firm, (e.g. Financial Analyst), can have a wildly different set of responsibilities in another firm.

So, what should corporations do to maximize the effectiveness of their recruiting process? Let’s start by acknowledging some of the findings outlined above:

  • Predicting the future performance of recruits is a difficult business, fraught with errors, biases and randomness
  • Many corporate recruiting efforts use a limited set of ineffective tools
  • Improving corporate recruiting effectiveness should focus on creating a more robust, comprehensive and standardized process

With these findings in mind, the following practices would seem to be sensible adds to corporate recruiting practices:

  • Structured Interviews – Utilizing predetermined, standardized questions allows for more accurate comparison of candidates. Standardized scoring processes allow for consistent rating practices by interviewers.
  • Assessments – A large body of research indicates that assessment tests are a more reliable predictor of future performance than interviews. Consider the NFL draftees. At the combine, and in team workouts, they are given what amounts to a comprehensive and standardized assessment test. In the corporate world, similar tests are available for skills, aptitude and personality for different job types. The best way to design and validate assessment tests is to utilize existing personnel. You can compare their results on particular tests to their historical success at your firm. This can allow for the customization of tests that are optimized for your particular environment.
  • Independent Evaluation – Ensure that interviewers complete their grading and assessment of candidates prior to discussing the situation with other interviewers. This prevents each interviewer from influencing the assessment of other evaluators. By including a broad set of raters, you can create a “wisdom of the crowds” effect.
  • Independent Reference Checking – In today’s age of social media, it’s not difficult to locate independent references. The best references are those located by the recruiter, who are intimately familiar with the candidate’s capabilities and history, familiar with the hiring firm’s culture, and our trustworthy themselves.
  • Try/Buy – Even with the comprehensive of the NFL draft process, the busts, and even moderate disappointments still occur. One would expect no greater capability from even a world class, best practices corporate recruiting program. A simple way to get greater insights into a candidates capabilities and suitability for a position and culture is through a try/buy program. In this case, the candidate is brought on as a temporary or provisional employee through some probationary period. After observing the candidate’s performance “in real life” a decision can be made about hiring them on a permanent basis.

While each of these practices can improve the reliability of your recruiting program, they all have limitations and drawbacks. In some cases they add to the expense of the effort. In other cases they lengthen the recruiting process. Other practices, such as try/buy may be unacceptable to particular candidates. The tightness of the employment market and the desirability of your firm as an employer will both factor into the creation of an optimal recruiting program. Your goal is finding that sweet spot that provides the right amount of predictive power without excessive “overhead”.

This entry was posted in Economics, Future of Work, General Management, Organization, Process Improvment. Bookmark the permalink.