Summer Holiday

I have to admit, I feel a little guilty. With all the usual end-of-term rush, I hadn’t managed to post in several weeks. Now it’s the summer, I’ve taken a few days to try not to do anything even vaguely related to teaching. So no blog posts. And tomorrow, I’m off on holiday so won’t be posting again for a week!

I’m not dead. I’m resting.

I do have several ideas for more regular posting after I get back, hopefully continuing once term starts again in September. In the meantime, last night’s TV inspired the following musical link. Tim Minchin’s comedy is great (I also like Bill Hicks for what it’s worth) although you may possibly find it offensive. Oh well.

Happy holidays and remember, more feedback will mean more chance I take the time to write more posts…

Advertisements

Bad Surveys make Bad ‘Research’

NB The title of this has altered but the permalink remains unchanged so people can still find it.

Printable: fishy research as pdf

Adverts lie. This is not a big surprise. A hint of the truth, of course, makes an advert much more believable. Advertising is about what they don’t say, much more than what is explicitly stated. Now, as much as I can accept this (being allegedly grown up and everything – adult, if not mature) it doesn’t mean I should accept it when they use or abuse science to help them mislead the audience.

A recent post on Ben Goldacre’s Bad Science site – to be exact, a brief entry on the delicious miniblog which appears to demonstrate his uncensored stream of consciousness – caught my attention. Although I’m not on Mumsnet myself I had heard about the site and Ben’s comment suggested that some dubious research had used the brand to get attention. The weblink didn’t work, possibly because once Ben was on to them the company decided to pull the press release, but I found another one [EDIT, which they also pulled – copy now found here, and if you go here you’ll find the text of it in case they get it removed it again]. The ‘research’ was into the taste and health benefits of an Omega-3 (fish-oil) supplement for kids. A little more work found two posts on Mumsnet, one asking for participants and another listing their feedback. Comparing the data (I’m assuming that the feedback posts comprise the total of the data collected) to the press release, a few things caught my eye.

  1. The participants had to already use omega-3 supplements or have tried them in the past; this means any ‘evidence’ collected in the second (health effects) stage is even more worthless than the average survey.
  2. Because earlier survey answers are visible, surely this means that people are more likely to follow previous trends? I remembered reading about experiments showing an extreme case of this by Solomon Asch, a social psychologist in the 1950s .
  3. The comments in the feedback did not, on first glance, seem to be as positive as the press release had suggested.

A little time spent tallying responses confirmed this last impression. The press release claims that 93% of parents had said that the product didn’t taste of fish. Of the 42 responses I found, seven said it did taste at least slightly fishy while 35 said it didn’t. The only way I can get that to be 93% is by taking that as 7 Yes answers out of 100 responses – even though the question was about the kid and/or parent. This is either sloppy or deliberately deceptive.

They ask the question very carefully – they ask if it tastes of fish. By quoting this (slightly mangled) statistic, they can ignore the large number who said it tasted bad (the word ‘vile’ came up more than once). This seems to me to be a good example of a carefully selected proxy outcome (explained nicely in the fantastic How To Read Health News article, found on the NHS Behind The Headlines site).

They also claim that over half of the parents would recommend the product to a friend, while I counted only 14 Yes answers of the 42 who responded – exactly a third.

Well, what else should a science teacher and long term reader of BadScienceBlogs dowhen faced with something like this? Produce a lesson activity and post it on his blog, of course! The printable activity (downloadable as fishy research pdf) has several possible approaches.

The ideal in some ways would be to give your students the tally sheets and weblinks, asking them to total up the answers to each question. There’s a page you can give them access to with the links they need. Alternatively, there’s a page with an extract from the press release, a sample answer and my totals. I’ve only gone through this once myself, so please let me know if my counting is off. I am confident that although I may be off by one or two either way there’s no way the data says what they claim in the press release. The only other possiblity, of course, is that they collected data directly as well as through the forum. Of course that must be it. Silly of me to suspect anything else.

Either way, the last page is a (write-on) worksheet, with questions which will lead them through the ideas I have covered here and a few more. Students will have to compare the data to the press release and comment on possible reasons for the differences. They are invited to consider the phrasing of the questions (it specifies a fishy taste rather than a bad taste) and speculate on how the process could have been rather more rigorous. Finally, they will be asked to consider a brief summary of the evidence for fish oil for ‘average’ children and suggest how the popular ‘brain boosting’ hypothesis could be best tested.

As always, I’d be very grateful for any feedback on the activity. In this case I’d be especially grateful if you can let me know if my arithmetic isn’t what it should be! I know I haven’t especially focused on the evidence, or lack thereof, for the brainboosting effects of fish oil. I figured I’d leave that to the professionals. I’m a teacher – I’ll stick with teaching. If you like this activity, you might like to check out my previous post (and associated scheme plus resources) on homeopathy. I will leave you with one last quote.

“Advertising is about making whole lies out of half truths.”

 

Adapting Question Papers

 The problem with past exam papers is that they cover everything. Or should. This is great at the end of a course but not so useful part way through. I’ve recently produced several more targeted resources for my students using past papers in a slightly different way and thought it might be worth sharing.

I always try to get my students checking their own understanding, using various ticklists and audits. This means that those who take it seriously end up with a list of priorities. They pointed out that finding practice questions for those specific areas is fairly time-consuming, especially if they want realistic exam-style material. My solution was to look back at the oldest past papers and extract relevant questions for each area, copying them from the pdfs with the photo tool and pasting them into a Publisher document. On average I was able to fit three higher level questions or four foundation on to a double-sided document, with the markscheme included so they could check their answers. It was even possible to include the answers upside-down, like on a crossword puzzle, so they could ignore them while working.

Printable: c1a pracQs atoms as pdf

These targeted worksheets were made available through the school VLE, meaning that students could access them from home when it suited them rather than having to remember to collect them. What I’m planning to do next time is to offer rewards for pupils who produce their own versions and make them available to their classmates. I already try to get them to share revision material with friends, but there’s no reason the files they produce couldn’t be uploaded to the VLE, anything from mind maps (produced through MindMeister or Bubbl.us) to revision cards. 

We currently teach the AQA Science A syllabus which is tested by multiple choice module papers in Year 10. I was looking for material to produce a homework sheet but wanted to include something other than multiple choice questions. The other resources on our network didn’t have anything appropriate, and I wanted a document I could make available to the kids through the VLE. I was on the point of starting to write my own questions when I remembered the alternative AQA Science B course. It covers the same material (as far as I can tell) but instead of assessing Year 10 biology in two multiple choice module exams, B1a and B1b, it’s a single mixed question exam, B1. Now, leaving aside the issue about this being a much more effective way of testing students, I much prefer more open questions for homework. It saves kids guessing in form time and then claiming they struggled for ages before getting them all wrong. It’s also much more revealing about their areas of weakness (or strength, to be fair).

So my students’ homework this week is a double-sided pdf, saved from Publisher, made up of two questions from an exam they’re not doing. This one doesn’t have the answers, but I have copied the markschemes to a separate sheet. They’ll be using this to mark each other’s questions next lesson, then we’ll summarise the lessons to be learned. I suspect that I could write the feedback ahead of time – more detail needed and particularly better use of key terms.

Printable: P1b pracQs red shift as pdf : P1b pracQs red shift ms as pdf