On November 28, this Dilbert comic came out:
This comic addresses one of my pet peeves with the media when they report results from research studies. A lot of the studies reported in the media are studies where a large group of people are followed and they answer questionnaires and provide health information at various intervals. Then, researchers crunch the numbers and see what relationships appear. Often, these studies show correlations – or relationships – between things. For example, let’s say that a researchers have been following the population of Small Town, Texas for 20 years. Every 5 years, the residents fill out surveys and provide their health information. At each 5 year interval, researchers also collect information about the businesses in Small Town, Texas. The researchers put all of the data into their statistical software and out pops a correlation showing that as the number of doughnut stores in Small Town, Texas goes up, weight of the residents of the town has dropped so they are all normal weight – they are NOT overweight or obese. The local news station picks up the story and reports “Towns with more doughnut shops have thinner residents!”
Really? Do more doughnut shops cause residents to be normal weight? This is the danger with studies like this and drawing conclusions about causality (A causes B). From the initial analysis, it looks like we need to start building doughnut shops on every corner to get people to be normal weight. However, in studies that monitor people over time, statements about causality cannot and should not be made. This is because statistically, statements about causality should only be made when something is changed (for example, if the researchers built 10 doughnut shops every year, then they changed the number of doughnut shops then they might be able to make statements about the number of doughnut shops and the residents’ weight). The only thing researchers can say is that it looks like the number of doughnut shops and the weights of people in Small Town, Texas are related and in what way (positive – as one goes up so does the other; or negative – as one goes up the other goes down).
What does this have to do with the Dilbert comic? The comic shows that Dilbert’s boss (the one with pointy hair) thinks Dilbert (they guy with the curly red striped tie) is sending him emails about world’s worst bosses because he gets one each time the boss leaves Dilbert’s cubicle. However, the boss doesn’t know that Wally (the guy with the green tie) is watching Dilbert’s cube and sending the emails to the boss when he leaves. Using the sample study talked about previously, the media reports that as the number of doughnut shops rise, weights fall in Small Town, Texas. However, what the study did not show is that Small Town, Texas is on I-10 at the border with another state where doughnuts are illegal. People are coming across the border to buy doughnuts and when people go visit friends and family in Doughnuts are Illegal State, they stop in Small Town, Texas and buy dozens of doughnuts to take to their doughnut deprived friends and family. This has spurred the growth of the doughnut industry in Small Town, Texas. The study didn’t take Small Town’s location into account which is the cause for the high number of doughnut shops and is not related in any way to the weights of the residents.
The moral of the story – be wary and question, question, question when the media reports the findings from the latest and greatest study.