Skip to content
May 11, 2012 / ashfonty

Statistics Gone Wrong – But Where?

It’s the ugly duckling of the scientific world. Very few scientists find it attractive or appealing to begin, however with a bit of time and attention (not to mention a few gallons of coffee) the beauty of it shines through. What am I talking about you ask? STATISTICS of course!

Lets face it, when we signed up for our degree very few of us realised exactly how much time we would spend sitting at a computer. We pictured ourselves in the lab discovering a cure for cancer, saving the world from global warming and (if you’re me) soaking up the sun on the beach while tagging sea turtles! But alas, we have come to the realisation that any data we collect will need to be analysed and presented a results.

The harsh reality of science: What we thought we’d do VS what we do.

But whose responsibility is it to communicate the results to a greater audience in an accurate manner? When a journalist picks up a scientific paper, can we really expect them to understand the statistics? After all they can be incredibly complicated with so much room for error and misinterpretation.

Let’s look at some examples where statistics have been taken out of context from the Australia Science Media Centre (2011) :

Facts

Conclusion

The issue

4 in every 1,000 women will die of breast cancer. This new drug will reduce a women’s risk of breast cancer by 25% This new drug will reduce a women’s risk of breast cancer by 25% For every 1,000 women that take the drug, 3 (instead of 4) will die of breast cancer. In terms of a women’s overall risk – the drug gives a 0.1% reduction

In this example we can see that while the conclusion is correct, it is very misleading to the general population – especially if it were to be published without the initial statistic that 4 in 1,000 women will die from breast cancer. If you were to explain that the absolute risk would be reduced by 0.1% as opposed to the relative risk of 25%, I’d hazard a guess that fewer women would be willing to take the drug.

Another classic issue in reporting statistics, whether its a scientist or a journalist is framing of messages. Again, it isn’t incorrect or inaccurate, it just puts a spin on the statistics to make it look better than it is. For example the classic scenario from our friends at Grey’s Anatomy:

Young mother: ‘Doctor, if I have this surgery to remove my brain tumour, what are the chances I’ll survive?’

Surgeon: ‘there is a 50% chance of dying on the operating table.’ Umm, no thanks. However, if the surgeon says, ‘there is a 50% chance of you surviving ‘… well now you’re talking!

Giselle Bunchden famously put her foot in it when she made this statement.

I know your logical and scientific brains are really angry at these statements and so they should be. It’s the same statistic. She would have a 50% chance of dying or surviving, but how much better does ‘surviving’ sound?

Goldin (2011) highlights the issues in media reporting about breastfeeding as a solution to the obesity crisis amongst children (don’t worry I didn’t realise there was a crisis in kids this young either). Despite a study clearly concluding that babies were gaining weight due to eating bad foods such as sugar drinks and French fries, the media reported that mothers should be breastfeeding their babies instead of using formula (Giselle will be happy). The fact that they came to this conclusion is staggering as the study did not look at breastfeeding at all! The journalists totally missed quoted the science as they picked up on a sentence in the article which stated there was a correlation between formula use & obesity, however this was related to the lifestyle of the child as opposed to the lack of breast milk.

When/where have you seen some seriously misconstrued statistics or results? Do you think that it is more the responsibility of the scientist to make their findings clear or do the journalists need to work on their statistical understanding?

References

Australian Science Media Centre (2011) Available from: <http://www.aussmc.org/for-media/tips-on-reporting-science/#Communicating%20statistics%20and%20risk%20responsibly&gt; [19 March 2012].

Goldin, R. (2011) Can breastfeeding halt obesity – or is the media misreading the research? Statistical Assessment Service. Available from: <http://www.stats.org/stories/2011/breastfeeding_halt_obesity_jan21_11.html&gt; [27 April 2012].

Images

Getty (2012) A man stressed at his computer.About.com. Available from: <http://learningdisabilities.about.com/od/parentsandfamilyissues/tp/Stress_Management_Tips.htm&gt; [10 May 2012].

Giselle Bundchen (2011) Available from: <http://www.bio27.com/tag/giselle-bundchen-age> %5B10 May 2012].

The Orangutan Project (2012) VIDA Australian Volunteers for International Development Program. Available from <://www.orangutan.org.au/newsletter-articles/vida-australian-volunteers-for-international-development-program> [10 May 2012].

Advertisements

13 Comments

Leave a Comment
  1. elenav90 / May 12 2012 4:12 am

    Journalists will always tend to give a one-sided story unfortunately, won’t they? To do this they need good arguments, and nothing speaks louder than scientific statistics. Of course they’ll be selective with which they will include, or exactly how they will paraphrase the data. I don’t think this means they misunderstand the scientific results or statistics which derive from them! They are making a conscious choice as to what to communicate and how, to have a large impact on readers, ultimately to sell. Sad, uh?

    At the other hand we have the scientists who, I believe, do make their results clear. You can’t get much more straightforward than the journal publications they publish after their experiments/studies. However they’re usually too busy or don’t have access to means of ‘communicating’ these to the general public – translating them efficiently for a wider audience.

    And so, this leads me to the brilliant conclusion that it is up to us a science communicators to mediate between the role of the scientist and the journalist, to interest and inform the general public with precision and clarity!

  2. noelynn / May 13 2012 2:21 am

    I guess both the journalist and the scientist need to be responsible for what and how they communicate statistics to the audience. As elenav90 highlights, the journalist would be conscious of what and how to express the findings to the readers in the aim of selling the publication. And in doing so he/she is selective of the content.

    However, on the other hand, the source (scientist’s results) should be communicated clearly with precision and accuracy. So to be understandable by a non-scientist (perhaps a journalist), scientists should be responsible to simplify the statistics.

    And so if there is a gap between the journalist and the scientist simplification of statistics, I believe a science communicator MIT help to minimize or fill in the gap.

    Good post Ash.

    • ashfonty / May 16 2012 12:27 pm

      I agree with what you & elenav90 have said, but aren’t all scientist’s science communicators. And journalists that choose to communicate science, would also be science communicators. In an ideal world expert sicence communicators should be employed to brdige the gap. However how likely do you think it is that publication and media companies like newspapers, magazines and TV stations, would actually employ their services?

  3. keikok / May 13 2012 5:48 am

    Interesting topic Ashfonty!
    I believe the responsibility of the scientists is to work on their statistics and write the paper on their work. On the other hand, the responsibility of the journalists is to make their findings clear.
    However, as you mentioned, it is very easy to cause misleading between scientists and journalists as they often do not know how to communicate each other. In the way, I agree to noelynn saying it is the responsibility of the science communicator to minimize the misunderstanding between them.
    It seems that we have tons of things to do but also, it should be hard to work on because we need to think both of our target audience as well as their target audience!

  4. baileymoser / May 14 2012 1:09 pm

    Great post! I completely agree with @elenav90 ‘s comment.

    Scientists absolutely have a responsibility to report their results, statistics in all, in a complete, accurate, and thorough way. This is especially important in papers that will be reviewed and used by others in the field. They should be clear in a scientific sense.

    I think science communicators are responsible for communicating the results to people with no background in science–the Average Joe. Scientists need to be clear enough with their results to make this possible for science communicators and eliminate ambiguity.

    Perhaps journalists are responsible for the next step? Maybe they make the science popular knowledge?

    Another interesting question about this is: When a miscommunication occurs and statistics are misinterpreted, who is responsible for ‘debunking’? Is it the science communicators’ or journalists’ responsibility when the statistics were presented as just too complicated in the first place? To what extent should the scientist be on the lookout for the correct spreading of their results?

    • ashfonty / May 16 2012 12:42 pm

      Such an important question!

      In the breastfeeding example mentioned above, the offending article stated, ‘…exclusive breastfeeding — breastfeeding alone, not breastfeeding combined with bottle-feeding — prevents obesity, says McCormick’. However, the doctor that they quote (Dr McCormick) is from the University of Texas Medical Brach and had nothing to do with the study they were discussing (which was from a group called LiveScience).

      The journalist had done a lot of researching around the idea and spoken to some other experts in the field of infant obesity, but unfortunately they missed that the study never even looked at whether the children were breast- or bottle-fed.

      How can we avoid this ‘confusion’ when journalists read scientific results?

  5. muza2009 / May 16 2012 12:48 am

    Very informative post which uses three different examples to illustrate how statistics are reported. I feel the blog could have started at paragraph 3. I also would have like to see the actual sentence the journalists misquoted in the obesity study and maybe a summary paragraph or sentence tieing all the good examples you discuss.

  6. JamieAlexandraGraves / May 16 2012 4:39 pm

    I agree with what elenav90 said – there is nothing more definitive and conclusive than cold, hard statistical evidence! And you are very right in saying that it is “ugly duckling of the scientific world” however it’s necessity for the verification of scientific publications far outweighs it’s dry and extremely boring nature. I think you have summarised this point well with your choice of images!

    I think you have also successfully highlighted the issue of miscommunication between journalists and scientists in your example of breast cancer from the Australia Science Media Centre.

    However I do not believe that scientists would survive without journalists, or that journalists would not survive without scientists either. There has to be a middle ground between the two professions and I think that science communicators play a major role in this relationship.

  7. kellyfitzsimons1 / May 18 2012 2:01 am

    Firstly, congratulations on an engaging and interesting blog, I especially loved the images that you included. I also agree with elenav90 in that rarely do you come across a well balanced story, just think of the headlines in The West Australian. They are often one-sided and controversial – but let’s face it- it makes an interesting story. Often, the drama of a story is taken away when it is balanced. In my opinion drama has a large impact on the readers and usually increases sales.
    I think it is the responsibility of both the scientist and the journalist to work together to ensure that statistics are not miscommunicated. The scientist needs to make them readily available and easy to understand to the journalist but the journalist should be confirming all statistics before the article is printed. But, I understand that in a breaking new story, the journalist wants to be the first one reporting on it. It could be argued that at times our desire to know knowledge first is driving miscommunication.

    Hopefully as science communicators we can all help to prevent the miscommunication of science! 🙂
    Good luck to you all! You have a challenging task ahead of you.

  8. caitiedunlap / May 18 2012 2:03 am

    I think statisticians can only do so much. They can present the data in the best way possible for the public but they cannot control what someone will do with that information. Journalists make comments on issues which can be portrayed as controversial even if there is little or no scientific background between them. Theyre just focused on selling papers or magazines so will write about a topic guaranteed to cause the biggest reaction. I do agree with the other comments that scientists need to present their own findings and communicate their science rather than letting journalists do it otherwise it can really get out of hand.

    Good job though but a conclusion would have been good too.

  9. maria93 / May 18 2012 7:18 am

    I agree that journalists usually present one-sided stories, it makes for a more dramatic and controversial story although it has the possibility of being misleading.
    Presenting just plain scientific facts may not have as much of an impact on their readers.

    However scientists have a responsibility to accurately present their findings and so their information may not interest or be easily understood by the general public.

    I also agree that this is where the role of science communicators come in, they should help to find the middle ground, so information can be interesting and understandable while still being accurate.

    Great post!

  10. markforeman92 / May 19 2012 3:43 am

    I think everyone is pretty spot on. The main point is that Scientists will present the facts and present the issue in a (usually) non-biased way whereas journalists will often take a certain side. Journalists need to sell their papers or magazines, and shock value/bias gets this done. I think another good point that everyone has raised is that scientists need to be able to present their findings in a more interesting manner, hence be good science communicators! We’ve all read countless “scientific papers” which send us to sleep. They are important, particularly for those who understand them, but if they could also present a public friendly version, this personal bias would probably be eliminated. Good job with the post!

  11. tobiasgrey / Jun 8 2012 11:48 am

    I think people have already clued on in this post – most journalists (in magazines, newspapers not related to science) have a tendency of overstating results. Especially those which have a tendency to report psuedoscience. Manipulating results and erroneous conclusions. Unfortunately, it’s mostly up to the reader to sift through the nonsense, or to get their information straight from source.
    As a bit of a stats nerd, I’m always intrigued by the ways people manipulate results. One of the most misleading is when a drug trial will state ‘This treatment is 10-20% better than placebo’. While that may be a good start, it’s not of much use for practitioners who need to know how a treatment stacks up against the best in practice. As you can imagine, a new treatment that is effective in 20% of cases won’t by any better than a current treatment that works at 30% in the first place.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: