Data, Bias and Poisoning the Well

Dear Reader, I did it again.

I could say that I’m blogging this because it could be used in the classroom. (It could, as a discussion about using data in context.) I could justify it with the fact that I’ve recommended books by the scientist-communicator in question. (And will again, because they’re ace.) I could talk about the challenges of the inevitable religious questions in a science lab, which we’ve all faced. (Like the year 10 who honestly believed, as he’d been told, that human bodies were literally made of clay like his holy book said.)

But the truth is I got annoyed on Twitter, got into a bit of a discussion about it, and don’t want to give up without making the points more clearly. So if you’re not up for a bit of a rant, come back when I’ve finally sorted out the write-up from the #ASEConf in Sheffield.

(I should point out that family stuff is a bit tricky at the moment, due to my Dad breaking his brand-new, freshly-installed hip. Before he’d even left the ward. So it’s possible that I’m procrastinating before lots of difficult decisions and a long journey to the Wild South.)

Appropriate Context?

A PR write-up of an academic study has been shared by several people online. The tweet I saw was from @oldandrewuk, who I presume shared direct from the page or RSS as it used the headline from there.

I responded pointing out the source of the research funding, the Templeton Foundation, which was founded to promote and investigate religious viewpoints. He suggested I was ‘poisoning the well’, a phrase I vaguely recognised but to my shame couldn’t pin down.

a fallacy where irrelevant adverse information about a target is preemptively presented to an audience, with the intention of discrediting or ridiculing everything that the target person is about to say. (Wikipedia)

I agree that this was preemptive, but would challenge the judgment that the information is irrelevant. The Templeton Foundation has a history of selectively funding and reporting research to fulfil their aim of promoting religious viewpoints. I thought of this information as providing valuable context; the analogy I used later in discussion was that of tobacco companies funding research showing limited effects of plain packaging. This was fresh in my mind due to recent discussions with another tweeter, outside of education circles. So when does providing context become a form of introducing bias? An interesting question.

Correlation and Causation?

Another point I made was that the data shared in the press release (although not in the abstract) seemed to hint at a correlation between the respondents’ religious views and their criticism of Richard Dawkins. It’s not unreasonable to suggest that this might be causative. The numbers, extracted:

  • 1581 UK scientists responded to the survey (if answers here mentioned Dawkins it’s not referenced annywhere I can see)
  • 137 had in-depth interviews
  • Of these, 48 mentioned RD during answers to more general questions*
  • Of these 48, 10 were positive and 38 negative

*Before I look at those numbers in a little more detail, I’d like to point out: at no time were the scientists asked directly their view on Richard Dawkins. The 89 who didn’t mention him might have been huge fans or his archenemies. They might never have heard of him. To be fair, in the paper some follow-up work about ‘celebrity scientists’ is suggested. But I’d love to have seen data from a questionnaire on this specific point addressed to all of the scientists.

Of the 48 who mentioned him:

rd-numbers

I suggested that the apparent link had been glossed over in the press release. That not a single scientist identified as positive had been positive about his work stood out for me. I wasn’t surprised that even non-religious scientists had identified problems; he is, let’s face it, a polarising character! But the balance was interesting, particularly as a ratio of one third of respondents being religious seeming a higher proportion that I remembered for UK scientists. But the makeup of the 137, in terms of religious belief vs non, wasn’t in the available information.

The Bigger Picture

I wanted more information, but the paper wasn’t available. Thankfully, #icanhazpdf came to my rescue. I had a hypothesis I wanted to test.

And so more information magically made its way into my inbox. I have those numbers, and it turns out I was right. It’s not made perfectly clear, perhaps because the religious orientation or lack thereof is the focus of other papers by the authors. But the numbers are there.

According to the paper, 27% of UK scientists surveyed are religious (from ‘slightly’ to ‘very’). It doesn’t make clear whether this is based on the questionnaire or applies specifically to the 137 interviewed. (EDIT: I’ve reached out to the authors and they weren’t able to clarify.) 27% of the 137 gives 37 who are religious, and therefore exactly 100 who are not. These numbers are used as I’ve nothing better, but I’ve labelled them ‘inferred’ below.

Now, there are loads of ways to interpret these numbers. I’m sure I’ve not done it in the best way. But I’ve had a go.

rd-data-2

What stands out for me is that religious scientists make up just over a quarter of those in the sample, but well over a third of those critical of Dawkins’ approach to public engagement. What’s clearer from this table is that the religious scientists were more likely to mention him in the first place, and as pointed out earlier these mentions were all negative. Is the difference significant?

  • 15 of 37 religious respondents were negative: 41%
  • 23 of 100 non-religious respondents were negative: 23%

I can’t help but think that’s a significant – although perhaps unsurprising – difference. Religious respondents were nearly twice as likely to be negative. So my hypothesis is supported by this data; the religious are over-represented in those who mentioned Dawkins during their answers. I’m surprised that this correlation escaped the Templeton-funded researchers. An equally correct headline would have been:

Scientists identifying as religious twice as likely to criticise Richard Dawkins’ approach to engagement unprompted.

Conclusions

I think in a lot of ways the numbers here aren’t the big story. I don’t think any of them are particularly surprising. I don’t have any answers for myself about the difference between providing necessary and important context, and ‘poisoning the well’ as @oldandrewuk put it. But I do have two insights that are important to me, if nobody else.

  1. The headline used in the article press-release is subtly misleading. “Most British scientists cited in study feel Richard Dawkins’ work misrepresents science.” My italics highlight the problem; 38 who were negative is not a majority of the 137 interviewed.
  2. The data used was selected to show one particular aspect of the results, and arguably some links were not explored despite being of interest. This can never be as good as a study designed to test one particular question. Only by closely reading the information was it clear how the judgments were made by the researchers.

I’d like to highlight that, as seemed fair to me, I invited @oldandrew to comment here following our discussion on twitter. He has so far chosen not to do so.

Conflicts of Interest

To be transparent, I should point out for anyone who doesn’t realise that I’m an atheist (and humanist, and secular). I often also disagree with Dawkins’ communications work – in fact, if they’d asked me the same questions there’s a fair chance I would have made the point about him causing difficulties for the representation of science to non-scientists – but that’s why I recommend his science books specifically!

Links

The wonderful @evolutionistrue posted about this research too. As a contrast, have a look at how EvangelismFocus wrote it up.

Herd Immunity

A very quick post as I’ve loads of other things I should be doing. Like ironing shirts for the first day back tomorrow. But I got into a brief discussion on Twitter about measles data being a topical way to get kids thinking about patterns, and it was pointd out to me that I never finished off my mini-scheme about MMR from ages ago. Obviously this is topical (and tragic) at the moment. I don’t have time to do this properly, but this is the easiest way to share the ‘herd immunity’ activity I put together then.

Herd immunity is a simple idea; if most of the population are protected against a disease, then it is much less likely to spread and so even those unvaccinated are effectively protected. Scientific views about the precise percentage that need to be immune vary, but it’s certainly well above the current proportions, especially for MMR. This can be blamed on the media, who gave headlines and airtime to Wakefield (struck off for poor ethics, not to mention falsifying data) and many of the anti-vaccination groups. Some organisations, like the BBC, say they need to provide ‘balance’. Others know better but seem to like the increased sales the scare stories bring in.

Anyway, enough ranting. Back to herd immunity.

Some people aren’t vaccinated yet or their vaccinations, for many reasons, have resulted in them being less than 100% protected. This group includes young babies, the immune-suppressed and so on. This is less than ideal but unavoidable. Others remain unprotected because they or their parents believed the media intead of the science. There’s some interesting research to show that people feel more guilt when they have acted to cause a problem, than when their inaction causes an equivalent problem. I just can’t find the link – anyone?

If you never meet anyone who can pass the pathogens on to you, then you won’t get the disease. It’s simple. So more people protected means less chance of bumping into someone who gives you horrible germs. Yes, I’m simplifying, partly because I’m a physicist and partly to put this in classroom-suitable terms. This is herd immunity, where the whole population is effectively protected because enough of them are actually protected. (It has the additional benefit that by removing possible reservoirs there’s less mutation and outbreaks tend to be less severe.)

Herd Immunity as a pdf

The above worksheet gives students a chance to see why even those who are unvaccinated get protection. You’ll need to give them the background, or send them off to research it. Hopefully not just on Wikipedia.

Finally, some extension questions:

  • should immunisation be compulsory when medically possible?
  • should vaccination be required before starting school (true in some parts of the USA)?
  • is it unethical to rely on herd immunity if you are not prepared to risk the small but measurable (millions to one) possibility of adverse effects of a vaccination? (I mentioned this in a guest post for NoodleMaz in January)

Getting Spammed By Mabus

I’m told that getting Twitter-spammed by the troll Mabus is recognition that I’ve annoyed somebody by commenting about alternative medicine. Is it sad that I feel proud? I ignored him and intend to continue to do so.

It happened in the process of a discussion on Twitter with a chiropractor, @Liam_Mulvany – who I am sure is sincere, for what that’s worth – about the use and value of his chosen ‘therapy’. One thing I commented on during the discussion was that his website testimonials only mentioned back pain and asking if he treated other conditions with chiropractic; he said that he did not. (“I only deal with musculo skeletal probs.” was his tweet, to be exact.) This has resulted in him commenting on an obscure page on my blog. I’ve no problem with that – I’m here to engage, although I expected it to be more with teachers than chiropractors. I’ve replied to it, here. I did however feel that as I’d spent a little while checking my references and laying out my reasoning, just like I tell my kids to do at work, I should put it up as a blog post too.

So here it is.

His comment:

I like this.

We examine a situation(somebody with back pain), perhaps one we’ve set up ourselves called an experiment(maybe used chiropractic methods on back pain), and collect data(asked the individual “whats changed?”). When we analyse this data,(how many people report an improvement) it tells us if our hypothesis truly describes the real world or not.(does chiropractic work) Either way we can now give a better hypothesis,(a reason as to why chiropractic works) a description of the world that is, in some way,(the best method of treatment) a better match to reality. This process, simple and elegant, has turned into a separate assessment target where we need to teach our kids ‘how science works’, as if it’s separate from all the facts we’ve discovered using it.

My reply:

For a medical treatment – or anything which claims to treat a health issue – there are two aspects to consider.

1) Does it work? (is it effective, which is not quite the same as does the patient say it helps.)
2) Is it safe?

Together, these two aspects will help qualified medical staff to make a judgement based on the benefit versus the risk. For chiropractic, There is a fair bit of evidence that it is about as effective as other manipulative therapies, such as physiotherapy or, for that matter, osteopathy, for back pain and related conditions. There is, as far as I’m aware, very little evidence to suggest it is at all effective for conditions such as asthma, ear infection or colic. You could argue that claiming – as many chiropractors have done, and for all I know you may have done – to treat these conditions with chiropractic is ‘bogus‘. And it is a matter of record that chiropractors over-use X-rays to ‘diagnose’ mythical subluxations, in direct opposition to NICE Guidelines. Of course, chiropractors generally (and their ‘professional’ association) have shown a rather poor understanding of research to date.

Which brings me nicely to the risk. There is evidence that a small number of ‘adjustments’, as performed by chiropractors, can lead to rather severe consequences, including stroke and death. Now, in medicine, if there is a risk of these kind of irreversible side-effects, we manage the risk. We make sure to try alternative treatments first. We check for sensitivity to drugs, and where necessary have crash-carts and trained personnel on hand. Chiropractors appear to lack the insight into possible consequences, which means they cannot adequately balance a non-zero risk against a benefit which appears minimal.

Talking about looking for evidence, or investigating ‘how’ it works, is an irrelevant distraction. In summary, the researchhas already been done, and like for so many other forms of alternative medicine, the result is the same. It doesn’t work much (if at all), and the risks – which are so rarely made clear to patients – mean it should be avoided if possible.

Are we done now?

I would welcome any useful links, clarifications or suggestions. Please note I feel that data is more inportant that anecdotes. The one point in his comment that I agree with is that it is odd in science teaching to divorce the scientific method from the evidence collected with it and the theories created by it. But then, like most science teachers I know, I don’t separate them. And if Liam’s teachers had made this clearer, perhaps he would find it easier to consider the data rather than the anecdotes and realise that the therapy he practises has significant issues.

Immunisation 4/5 Choices

This is the fourth of five posts designed as a teaching mini-scheme about the controversy surrounding the MMR vaccination; it is partly inspired by the recently published work by Brian Deer. Please note I feel, quite strongly, that MMR is safe and highly desirable (albeit underused in the UK right now). This is my effort to provide colleagues with the tools (and printable resources etc) to provide good information. I suppose you could see it as immunising them against bad science such as that recently published in the Sunday Express. At the end of the five posts I will put the ideas and resources together into one downloadable scheme if there is enough feedback to make it worthwhile.

Starter: Making a Choice

Ask students to spend a couple of minutes considering the choices their parents have made for them – school, part of the country, hobbies etc. Were these choices always right? Were they well-informed?

Main: Roleplay

Discussion

Put students into groups – aim to mix them up in terms of gender and ability. Each group will be assigned a role and asked to discuss arguments for or against MMR vaccination. They need to be able to justify their arguments as well as quantify them (perhaps using an opinion line?). If there is time you might want to give them a chance to research their position, perhaps in their own time. The powerpoint includes printable slides which will give them a starting point, as well as information they may choose not to share with the rest of the class. These slides can be used in the plenary, after they have chosen from the point of view of a parent. Some ideas about producing a roleplay can be found here (and I’m sure at many other places too).

Persuasion

There are many ways the students could share the arguments they have considered. If they produce a group poster or display then the decision could be run as a marketplace activity, with one ‘stall-holder’ left to explain the ideas and the remainder considering all the opinions before making their choice. Alternatively each group could present their ideas for a couple of minutes before answering questions, or make a video explaining their thoughts – perhaps as interviews for a TV show? The most challenging would be an open discussion, hard to manage and time consuming. In some ways the ideal would be brief presentations first, then ask them to speak to each other and challenge ideas one-to-one before reforming in groups for any final questions.

Decision

Finally, all students should record their choice, perhaps using anonymous votes or personal whiteboards. It is worth pointing out that although scientific questions can’t be settled democratically, people’s choices – such as whether or not to vaccinate – are much more likely to be based on persuasion.

Plenary

Tally the total score, perhaps asking them to predict the result first. How does this compare with vaccination rates, nationally and locally? (Useful figures are summarised in this report from the House of Commons Library.)

As before, I’ve put this together as immunisation4 saved as ppt. The last six slides can be printed as briefing cards for the role play, although you may have other/better ideas. If so, please share them below!

Immunisation 3/5

This is the third of five posts designed as a teaching mini-scheme about the controversy surrounding the MMR vaccination; it is partly inspired by the recently published work by Brian Deer. Please note I feel, quite strongly, that MMR is safe and highly desirable (albeit underused in the UK right now). This is my effort to provide colleagues with the tools (and printable resources etc) to provide good information. I suppose you could see it as immunising them against bad science such as that recently published in the Sunday Express. At the end of the five posts I will put the ideas and resources together, informed by any and all comments and feedback, into one downloadable scheme. Sorry this post is a little late but real life got in the way (more about this in a later blog post).

3 Data in Context

Starter

School A gets 25 A* grades in GCSE Science. School B gets 44 A*s. Which is better? (Not enough info as don’t know how many students in total, how they have chosen courses, exam specification <cough>21st Century.)

Main Activities

Numbers published during news stories about Implanon can be used to practice calculations. Ask students to consider view from each ‘end’ of the spectrum – those who suffer ill effects or for whom a product fails, compared with those who are satisfied with result. All cases have odd factors/coincidences – this is why we look at large numbers. You could point students towards the news stories (or have them look themselves using Google) or use the downloadable pdfs listed below. These include Ben Goldacre’s Guardian column and an excerpt from Dr Petra Boynton’s blog post about the media coverage. These should provide an interesting balance to the newspaper’s approach. The powerpoint asks students to read and discuss their assigned article in groups before comparing to other contraceptives. (Warning: check compatibility with age and PSHE in your setting.) They then compare the coverage between groups.

To bring the focus back to MMR, students could do with examining figures of incidence and mortality for each of the three infections. Numbers are available at HPA if you’d like them to graph it themselves, electronically or by hand. Getting hold of autism numbers is trickier, but most reputable sites agree that it’s hard to tell whether it is more common or just more diagnoses. WikidScience has an activity comparing absolute numbers in California with numbers per 100000, which might be useful. Ask students how we could explain increasing numbers.

Important to give a few facts – or guide the students to finding them – showing the consequences of measles, mumps and rubella. This is the route I’ve taken in the powerpoint below.

Plenary

Challenge students to explain the difference between anecdotes and data. Realising the power of personal experience will set students up nicely for the role play in lesson 4 of the sequence (coming soon, I hope).

Printable media articles: www-bbc-co-uk : www-dailymail-co-uk : www-drpetra-co-uk : www-guardian-co-uk : www-mirror-co-uk : www-telegraph-co-uk all converted to pdf in one way or another.

immunisation3 as ppt

Any and all feedback and suggestions welcome – many thanks.

Immunisation 2/5 Correlation and Causation

This is the second of five posts designed as a teaching mini-scheme about the controversy surrounding the MMR vaccination; it is partly inspired by the recently published work by Brian Deer. Please note I feel, quite strongly, that MMR is safe and highly desirable (albeit underused in the UK right now). This is my effort to provide colleagues with the tools (and printable resources etc) to provide good information. I suppose you could see it as immunising them against bad science such as that recently published in the Sunday Express. At the end of the five posts I will put the ideas and resources together, informed by any and all comments and feedback, into one downloadable scheme.

2 Correlation and Causation

Starter

Ask students to explain ‘links’ between things that happen at the same time:

  • boys growing facial hair and starting to produce sperm
  • height and shoe size
  • final exams and sunny weather (in the UK, more or less)
  • ice cream sales and deaths by drowning at the beach
  • star sign and academic success

Alternatively use the card sort and ask them to find the pairs of linked variables. You could perhaps add in one or two pairs that are causatively linked.

Printable: correlation cardsort as pdf.

Main Activities

Start by defining correlation and causation. It might be worth going back to tricky examples from the starter, pointing out when two variables are controlled by a third. (Flow charts show this nicely). Explain why observational studies can’t easily distinguish between correlation and causation, but that a link is often interesting and may help to suggest a hypothesis. Examples such as the British Doctors Study (more detail at the MRC) might be useful. Individual facts about this study could be used to produce a ‘murder mystery’ activity.

Recap definitions of variables (independant, dependant and control) and elicit ideal experimental design. Discuss limitations, especially with health-related research (compliance, ethics, comparison to placebo etc). We try to look for a pattern once other factors are excluded, then investigate links without endangering patients. Cue debate about value/ethics of animal trials!

A few weeks back Matt Parker produced a fantastic bad science/stats trap – explained here in the Guardian – suggesting a link between mobile phone masts and fertility. It is of course simple – we put mobile phone masts where there is demand. Most students should be able to suggest that this is because that’s where the people are. I suspect the correlation is enhanced by the fact that demand is linked even more closely to the section of the population aged 15-45, who are both more likely to have mobiles and have babies. He’s archived the data as an Excel file, available through his website StandupMaths. Students could plot (some of) this data and then try to explain the pattern.

Give students a simplified version of the original claims about MMR, according to Wakefield and authors. Challenge students to suggest (1)possible other factors/explanations (2)best ways to investigate this possible link. What would they have done if they had been involved in medical science at the time? What would they have recommended to parents/GPs? Instead of a simple ‘yes’ or ‘no’, it can be useful to have students show their feelings on an ‘opinion line’. This allows more subtlety, for example by letting them position themselves between ‘Cancel all vaccines’ and ‘Don’t even bother researching a possible link’ – both unhelpful extremes.

Extension/Plenary

Why is this funny?

Above cartoon is of course from the wonderful xkcd.

Some of these ideas and activities are introduced in immunisation2 saved as powerpoint.

As before, please let me know of any ideas, suggestions, improvements – through the comments here, by email or via Twitter.

Immunisation 1/5 Infection

This will be the first of five posts designed as a teaching mini-scheme about the controversy surrounding the MMR vaccination; it is partly inspired by the recently published work by Brian Deer. Please note I feel, quite strongly, that MMR is safe and highly desirable (albeit underused in the UK right now). This is my effort to provide colleagues with the tools (and printable resources etc) to provide good information. I suppose you could see it as immunising them against bad science such as that recently published in the Sunday Express. At the end of the five posts I will put the ideas and resources together, informed by any and all comments and feedback, into one downloadable scheme.

1 Infection

Starter

Distinguish between conditions e.g. asthma and infections.

Main 1: Pathogens and Immunity

  • Recap types of pathogen, basics of how the immune system works.
  • Blog is refusing to let me upload a .swf animation which is a shame as the Brainpop one on disease is quite good.
  • Ask students about their understanding of immunisation and clarify if necessary. (Depending on age they may have recently had HPV to protect against cervical cancer, which is a tricky example).
  • Might want to give examples of difficulties e.g. polio if they are up to it.

Main 2: MMR

  • Have students investigate the symptoms, mortality rates etc of a range of diseases, including MMR. How many you use will depend on class and group size.
  • Give a range of websites – I found NetDoctor, WHO, HPA (follow link to epidiomological data, fairly technical but gives good data for graphs) but I’m sure there are others – and ask them to extract basic facts for comparison.
  • Using a comparison table, or having them use something like Venn diagram to show similarities and differences, will stop them simply copying and pasting.
  • Each group could add basic facts to a wiki (if you’re feeling technical) or to the whiteboard (if you’re not).

Extension

Why is it sometimes called vaccination? Finding out why it is named for cows (vaccus) will lead students to cowpox/smallpox and Jenner’s work.

Plenary

MMR – use students’ figures to show how the MMR immunisation saves lives in the UK – compare total mortality rates for these three diseases now with those from previous decades. This graph from Wikipedia shows the dramatic change in cases (but not mortality).

One version of this teaching plan is immunisation1 saved as a powerpoint file. Please comment, both on the ideas above and the powerpoint. A full version will be produced once I have some useful feedback.