Statistics That Lead Us Astray – Part II

A recent article on the website silicon.com highlighted a Forrester Research report on attitudes of boomers and millenials. The conclusion of the article was that the conventional wisdom typically expressed in the media was incorrect. Specifically, it indicated that boomers and millenials have similar attitudes towards technology in the workplace. “Man bites dog” news stories are definitely a way to generate reader interest. Rehashing the same story of boomer/millenial divide doesn’t create catchy headlines. However, upon reading the article, I immediately began to question the conclusions. Could it really be that all of the conventional wisdom and intuition that people share about boomers and millennials is dead wrong?

The article mentioned several items from the survey as supporting data. One survey question showed similar numbers of millennials/boomers (approx. 55%) were satisfied with the technologies that they use to do their jobs. Another item showed that 31% of millennials thought the technology they had at home was better. However “they were no more likely to bring it into the office than baby boomers.” Another report item showed that only 10% of millennials felt that their IT department was either “clueless” or “significantly hindered” their ability to work. There was no mention in the article of how boomers responded to this last question.

As a quick disclaimer, I don’t have access to the original Forrester Report. So, in fairness, there may be supporting data and a description of the survey methodology that would give it greater credibility. My immediate instinct, however, is to question the validity of the findings. Could they represent an instance of Simpson’s Paradox, a concept I covered in a previous post?

The potential problem that I see with the survey revolves around the likely differences in employers and roles between the two generational groups. Boomers are more likely to work for older, traditional firms that are more conservative in their use of workplace technology. Millennials are more typically found in startups and newer firms who place an emphasis on cutting edge use of technology. Additionally, even within the same firms, boomers and millennials gravitate to different roles and departments. The boomer is more often found in a more traditional area such as Finance. The millennial will be over represented in areas like marketing and web development that make use of newer technology.

By not comparing the groups in similar settings, one could draw incorrect conclusions from the aggregated survey data. Let me provide a hypothetical example, drawing from data in the survey. Let’s use the example of boomers and millennials both expressing a 55% satisfaction rate with the technology needed to do their job. Let’s break the survey data down by imagined department. For simplicity, let’s say that there are two department types. “Conservative” departments will include Finance, Operations and Legal. “Progressive” departments will include Marketing, Web Development and R&D. Let’s show a hypothetical table reflecting the survey data:

As you can see, the Millenials are significantly less satisfied with their workplace technology when compared to counterparts in the same department. However, because they are over represented in “Progressive” areas, their “total” satisfaction appears to be equal to Boomers.

I’ll reiterate that without having access to the original survey data I’m not able to conclude that the article’s contentions are incorrect. However, I believe that rational professionals should be aware of statistical thought traps and learn how to evaluate data with a critical eye. Statistics are frequently used to confer a sense of authority and legitimacy to viewpoints and opinions. Accepting them uncritically can lead to incorrect beliefs and counterproductive decisions.

This entry was posted in Probability/Statistics. Bookmark the permalink.