Bad Surveys make Bad ‘Research’

10Jul10

NB The title of this has altered but the permalink remains unchanged so people can still find it.

Printable: fishy research as pdf

Adverts lie. This is not a big surprise. A hint of the truth, of course, makes an advert much more believable. Advertising is about what they don’t say, much more than what is explicitly stated. Now, as much as I can accept this (being allegedly grown up and everything – adult, if not mature) it doesn’t mean I should accept it when they use or abuse science to help them mislead the audience.

A recent post on Ben Goldacre’s Bad Science site – to be exact, a brief entry on the delicious miniblog which appears to demonstrate his uncensored stream of consciousness – caught my attention. Although I’m not on Mumsnet myself I had heard about the site and Ben’s comment suggested that some dubious research had used the brand to get attention. The weblink didn’t work, possibly because once Ben was on to them the company decided to pull the press release, but I found another one [EDIT, which they also pulled - copy now found here, and if you go here you'll find the text of it in case they get it removed it again]. The ‘research’ was into the taste and health benefits of an Omega-3 (fish-oil) supplement for kids. A little more work found two posts on Mumsnet, one asking for participants and another listing their feedback. Comparing the data (I’m assuming that the feedback posts comprise the total of the data collected) to the press release, a few things caught my eye.

  1. The participants had to already use omega-3 supplements or have tried them in the past; this means any ‘evidence’ collected in the second (health effects) stage is even more worthless than the average survey.
  2. Because earlier survey answers are visible, surely this means that people are more likely to follow previous trends? I remembered reading about experiments showing an extreme case of this by Solomon Asch, a social psychologist in the 1950s .
  3. The comments in the feedback did not, on first glance, seem to be as positive as the press release had suggested.

A little time spent tallying responses confirmed this last impression. The press release claims that 93% of parents had said that the product didn’t taste of fish. Of the 42 responses I found, seven said it did taste at least slightly fishy while 35 said it didn’t. The only way I can get that to be 93% is by taking that as 7 Yes answers out of 100 responses – even though the question was about the kid and/or parent. This is either sloppy or deliberately deceptive.

They ask the question very carefully – they ask if it tastes of fish. By quoting this (slightly mangled) statistic, they can ignore the large number who said it tasted bad (the word ‘vile’ came up more than once). This seems to me to be a good example of a carefully selected proxy outcome (explained nicely in the fantastic How To Read Health News article, found on the NHS Behind The Headlines site).

They also claim that over half of the parents would recommend the product to a friend, while I counted only 14 Yes answers of the 42 who responded – exactly a third.

Well, what else should a science teacher and long term reader of BadScienceBlogs dowhen faced with something like this? Produce a lesson activity and post it on his blog, of course! The printable activity (downloadable as fishy research pdf) has several possible approaches.

The ideal in some ways would be to give your students the tally sheets and weblinks, asking them to total up the answers to each question. There’s a page you can give them access to with the links they need. Alternatively, there’s a page with an extract from the press release, a sample answer and my totals. I’ve only gone through this once myself, so please let me know if my counting is off. I am confident that although I may be off by one or two either way there’s no way the data says what they claim in the press release. The only other possiblity, of course, is that they collected data directly as well as through the forum. Of course that must be it. Silly of me to suspect anything else.

Either way, the last page is a (write-on) worksheet, with questions which will lead them through the ideas I have covered here and a few more. Students will have to compare the data to the press release and comment on possible reasons for the differences. They are invited to consider the phrasing of the questions (it specifies a fishy taste rather than a bad taste) and speculate on how the process could have been rather more rigorous. Finally, they will be asked to consider a brief summary of the evidence for fish oil for ‘average’ children and suggest how the popular ‘brain boosting’ hypothesis could be best tested.

As always, I’d be very grateful for any feedback on the activity. In this case I’d be especially grateful if you can let me know if my arithmetic isn’t what it should be! I know I haven’t especially focused on the evidence, or lack thereof, for the brainboosting effects of fish oil. I figured I’d leave that to the professionals. I’m a teacher – I’ll stick with teaching. If you like this activity, you might like to check out my previous post (and associated scheme plus resources) on homeopathy. I will leave you with one last quote.

“Advertising is about making whole lies out of half truths.”

 

About these ads


2 Responses to “Bad Surveys make Bad ‘Research’”

  1. 1 Jon Hall

    Your second link to the press release was also removed – how surprising.

    • Link now replaced, and I’ve also saved a local copy. Just in case. Hope you found it useful despite ad agency shenanigans (check out Memory4Teachers post for similar kind of stuff…)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 132 other followers

%d bloggers like this: