Statistics: the hidden disease

It seems like every week there’s another medical breakthrough announced in the press – only later to fizzle when additional studies show it didn’t really hold up. Why are there so many false starts?

Dr. Peter Austin of the Institute for Clinical Evaluative Sciences in Toronto says it has to do with the way researchers use statistics. All statistical studies rely on “confidence intervals” – if an event has only a 5% chance of happening at random, then doctors can be 95% confident that it isn’t a random fluke. They assume they’ve discovered a real phenomenon, and start looking for a cause.

(For instance, a coin has about a 3% chance of landing heads five times in a row. If you had a coin that did that the first time you tried it, you’d have good reason to suspect something funny was going on, and conduct more tests.)

But Dr. Austin notes that many studies run multiple tests simultaneously. When you do that, the odds of at least one test giving an unusual result, just by chance, is very high. In our coin-flipping example, if you tested 100 identical coins by tossing each 5 times, it would be perfectly normal for at least one, and probably a few, to land all heads, without anything “funny” about them at all.

The point is, you have to look at the whole test, not just selected parts of it. And doctors – and journalists – need to be more careful when presenting the results of studies, so they don’t report false relationships.

Your Comments, Thoughts, Questions, Ideas

James Satter's picture

Thanks for this excellent reminder about dangers of selective data, which people often use unintentionally. As you mention, these points apply to many areas of research, not just medicine.

posted on Tue, 03/06/2007 - 11:37am
sarina lee's picture
sarina lee says:

this is interesting to learn about because i am learning the same stuff in school

posted on Sat, 03/10/2007 - 1:28pm
Big John's picture
Big John says:

For decades I have been frustrated with reports that say "something" increases your risk of "such and such" cancer or disease. Never do they mention what my chance is of contracting it in the first place.

Is there a web site that accurately reports what my chances are of getting --colon cancer, liver disease, prostate cancer,ect.???? Thanks..John

posted on Fri, 06/15/2007 - 2:27pm
Liza's picture
Liza says:

I've never seen such a site, and I don't believe that an accurate one exists.

Even with a detailed family history and an accurate assessment of diet, physical activity, and other lifestyle risk factors, doctors have a tough time telling patients what their chances are of getting any particular type of cancer.

Risk factors are a complicated jumble of genetics and environment, and, after all that, they're still just probabilities. Even if you knew exactly your personal odds of developing a particular cancer, there would still be a chance that you wouldn't, and you'd still be looking at ways to try to reduce that risk.

You might read a fascinating article, first published in Discover Magazine, by Stephen Jay Gould about statistics and risk: The Median Isn't the Message.

posted on Fri, 06/15/2007 - 3:08pm
Gene's picture
Gene says:

The problem with applying statistics to individual diagnoses is that the sample size is just too small. ;-) A study might show that, out of 100 people who get a disease, 50% of them die. What does that mean to you? Almost nothing. You can't be 50% dead. You either die of the disease (100%) or you don't (0%).The 50% figure describes the results for a large group, but says little about any individual.

posted on Wed, 06/20/2007 - 1:44pm

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <h3> <h4> <em> <i> <strong> <b> <span> <ul> <ol> <li> <blockquote> <object> <embed> <param> <sub> <sup>
  • Lines and paragraphs break automatically.
  • You may embed videos from the following providers vimeo, youtube. Just add the video URL to your textarea in the place where you would like the video to appear, i.e.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Images can be added to this post.

More information about formatting options