I recently came across a series of blog posts that were widely shared but suffered some deep flaws. The purpose of this post is not to shame the author so I will not name them or link to their blog. Instead, I want to discuss the faulty assumptions that were made using a fictional example. I think it’s illustrative of how we may draw incorrect conclusions when we do not critically examine our biases.Fictional Example: Why does well-being increase among older adults?
Tom notices something that catches their attention: Self-reported well-being decreases in middle age but rapidly increases among older adults.
Based on this graph, Tom thinks that there’s clearly something that older adults are doing right but what is it?
Tom begins conducting research and finds that the CDC’s website states “well-being encompasses physical, mental, and social domains.” Tom then finds a graph that shows Americans have been exercising more over time so that can’t be what’s dragging well-being down.
Tom then looks at mental health disorders, particularly depression, and because the graph doesn’t closely overlap with well-being, Tom concludes that clearly can’t be the reason either.
When it comes to the social factor, average number of friends doesn’t appear to match the well-being trend either.
So, “What could it be?” Tom asks. “Aha!” Tom realizes it’s not about our physical activity, depression levels, or the number of friends, it’s about how much time we spend with others.
Now, Tom sees that it’s the amount of time that we’re spending alone. That’s what’s driving higher levels of self-reported well-being at older ages.
Okay, let’s quickly recap what just happened. The fictional subject, Tom, is clearly a bright and curious person. They noticed a phenomenon and independently explored potential explanations for what they observed. Tom even went as far as diving into some of the factors that drive well-being, as defined by the CDC. Tom then went and looked at empirical research and trusted sources to find a measure for each one of the three factors identified by the CDC and found that they didn’t appear to closely correlate with well-being. In other words, the individual trends in physical activity, mental health, and number of friends, didn’t appear to overlap with the trend in well-being. Undeterred, Tom realized that it must be something related to one of the factors rather than those specific factors themselves. Tom specifically looks into how much time we spend with others. That’s when Tom concludes that spending more time alone must be the reason for the rise in well-being since the two trends appear to overlap as people age.
While Tom was clever, in many ways, there are some issues with the way that Tom conducted research.
- Tom jumped immediately into the definition of well-being defined by the CDC. The original graph that inspired his research was from Deloitte. Deloitte may not use the same definition of well-being as the CDC. In fact, Tom linked to an article that explained that researchers use many different definitions of well-being. So, the problem is that Deloitte may be measuring well-being differently than what is suggested in the article by the CDC.
- Tom also researched a very narrow set of measures for the three factors that were singled out in the CDC article. Tom decided that a graph of physical exercise over time was a sufficient measure for the physical domain, five mental health disorders by age of onset was sufficient for the mental domain, and average number of friends by age was sufficient for the social domain. However, each one of these factors is complex and involve multiple different measures—some are unidimensional scales while others are bidimensional or even multidimensional. In other words, Tom cherry picked some variables that appeared to be logical representations of those factors rather critically examining whether these variables were supported by empirical studies.
- Another issue is that one of Tom’s graphs is a measure over time (in years) while the other three are reported by age. Given that the original graph that sparked Tom’s interest was a pattern by age, age is the appropriate x-axis or time variable. Additionally, if Tom had dug a little deeper, they may have noticed that each of these graphs use cross-sectional data. This means that the data offer a snapshot from one particular point in time rather than following the same respondents over time as they age. This matters because research has shown that these cross-sectional snapshots may provide an inaccurate picture of the relationship between age and self-reported well-being. This is the sort of tricky stuff that researchers deal with. But to Tom’s credit, experienced researchers have been fooled these data as well.
- Lastly, Tom’s decision to examine how much time people spend with others as they age is what I found the most fascinating but unsettling. When Tom couldn’t find an explanation for the observed association between age and well-being, Tom decided to look more critically at only one aspect of well-being. Tom could have done this with any variable. For example, how we exercise (e.g., running, biking, gymnastics) or which activities we do with our friends (e.g., drinking at bars; hiking; dancing). Tom also could have looked at something else entirely, like whether we say “hi” versus “hello” or the number of hours we sleep at night. Instead, Tom somehow decided that the amount of time we spend with others was the key factor that mattered over everything else. This is also how Tom ultimately concludes that spending more time alone is good for well-being despite experts finding the opposite is true, especially among older adults.
Now, (I sincerely hope) most of us know that Tom’s conclusion is incorrect and you may even find this fictional example to be humorous. But the real set of blog posts that inspired me to write this post is kind of similar to what I’ve reconstructed here. The original writer took a subject that many people think they know and understand well and flipped that logic on top of its head. Notably, people may have been more willing to question their own knowledge about the subject because this person is fairly successful and well-known and, therefore, influential. I should also note that I did not find any evidence of malicious intent. They seem 100% genuinely concerned by the phenomenon they observed and is writing about the subject as a type of public awareness campaign. If their research had turned out to be correct, they may be successfully steering people who read their blog posts (or watched YouTube videos that were inspired by these posts) into healthier behaviors. Unfortunately, the opposite is likely occurring. As well-intentioned as this person may be, faulty assumptions led them and some of their followers to flawed conclusions.
I hope that my post helps others begin to poke holes in the way that arguments like these are constructed. Whether it’s related to your work or someone else’s, take the time to critically examine the logic and conclusions of the research that you consume. You never know how much influence you may have on others. It’s worth the extra time and reflection to try and ensure, to the best of your ability, that your impact is positive.