Data, Bias and Poisoning the Well

Dear Reader, I did it again.

I could say that I’m blogging this because it could be used in the classroom. (It could, as a discussion about using data in context.) I could justify it with the fact that I’ve recommended books by the scientist-communicator in question. (And will again, because they’re ace.) I could talk about the challenges of the inevitable religious questions in a science lab, which we’ve all faced. (Like the year 10 who honestly believed, as he’d been told, that human bodies were literally made of clay like his holy book said.)

But the truth is I got annoyed on Twitter, got into a bit of a discussion about it, and don’t want to give up without making the points more clearly. So if you’re not up for a bit of a rant, come back when I’ve finally sorted out the write-up from the #ASEConf in Sheffield.

(I should point out that family stuff is a bit tricky at the moment, due to my Dad breaking his brand-new, freshly-installed hip. Before he’d even left the ward. So it’s possible that I’m procrastinating before lots of difficult decisions and a long journey to the Wild South.)

Appropriate Context?

A PR write-up of an academic study has been shared by several people online. The tweet I saw was from @oldandrewuk, who I presume shared direct from the page or RSS as it used the headline from there.

I responded pointing out the source of the research funding, the Templeton Foundation, which was founded to promote and investigate religious viewpoints. He suggested I was ‘poisoning the well’, a phrase I vaguely recognised but to my shame couldn’t pin down.

a fallacy where irrelevant adverse information about a target is preemptively presented to an audience, with the intention of discrediting or ridiculing everything that the target person is about to say. (Wikipedia)

I agree that this was preemptive, but would challenge the judgment that the information is irrelevant. The Templeton Foundation has a history of selectively funding and reporting research to fulfil their aim of promoting religious viewpoints. I thought of this information as providing valuable context; the analogy I used later in discussion was that of tobacco companies funding research showing limited effects of plain packaging. This was fresh in my mind due to recent discussions with another tweeter, outside of education circles. So when does providing context become a form of introducing bias? An interesting question.

Correlation and Causation?

Another point I made was that the data shared in the press release (although not in the abstract) seemed to hint at a correlation between the respondents’ religious views and their criticism of Richard Dawkins. It’s not unreasonable to suggest that this might be causative. The numbers, extracted:

  • 1581 UK scientists responded to the survey (if answers here mentioned Dawkins it’s not referenced annywhere I can see)
  • 137 had in-depth interviews
  • Of these, 48 mentioned RD during answers to more general questions*
  • Of these 48, 10 were positive and 38 negative

*Before I look at those numbers in a little more detail, I’d like to point out: at no time were the scientists asked directly their view on Richard Dawkins. The 89 who didn’t mention him might have been huge fans or his archenemies. They might never have heard of him. To be fair, in the paper some follow-up work about ‘celebrity scientists’ is suggested. But I’d love to have seen data from a questionnaire on this specific point addressed to all of the scientists.

Of the 48 who mentioned him:

rd-numbers

I suggested that the apparent link had been glossed over in the press release. That not a single scientist identified as positive had been positive about his work stood out for me. I wasn’t surprised that even non-religious scientists had identified problems; he is, let’s face it, a polarising character! But the balance was interesting, particularly as a ratio of one third of respondents being religious seeming a higher proportion that I remembered for UK scientists. But the makeup of the 137, in terms of religious belief vs non, wasn’t in the available information.

The Bigger Picture

I wanted more information, but the paper wasn’t available. Thankfully, #icanhazpdf came to my rescue. I had a hypothesis I wanted to test.

And so more information magically made its way into my inbox. I have those numbers, and it turns out I was right. It’s not made perfectly clear, perhaps because the religious orientation or lack thereof is the focus of other papers by the authors. But the numbers are there.

According to the paper, 27% of UK scientists surveyed are religious (from ‘slightly’ to ‘very’). It doesn’t make clear whether this is based on the questionnaire or applies specifically to the 137 interviewed. (EDIT: I’ve reached out to the authors and they weren’t able to clarify.) 27% of the 137 gives 37 who are religious, and therefore exactly 100 who are not. These numbers are used as I’ve nothing better, but I’ve labelled them ‘inferred’ below.

Now, there are loads of ways to interpret these numbers. I’m sure I’ve not done it in the best way. But I’ve had a go.

rd-data-2

What stands out for me is that religious scientists make up just over a quarter of those in the sample, but well over a third of those critical of Dawkins’ approach to public engagement. What’s clearer from this table is that the religious scientists were more likely to mention him in the first place, and as pointed out earlier these mentions were all negative. Is the difference significant?

  • 15 of 37 religious respondents were negative: 41%
  • 23 of 100 non-religious respondents were negative: 23%

I can’t help but think that’s a significant – although perhaps unsurprising – difference. Religious respondents were nearly twice as likely to be negative. So my hypothesis is supported by this data; the religious are over-represented in those who mentioned Dawkins during their answers. I’m surprised that this correlation escaped the Templeton-funded researchers. An equally correct headline would have been:

Scientists identifying as religious twice as likely to criticise Richard Dawkins’ approach to engagement unprompted.

Conclusions

I think in a lot of ways the numbers here aren’t the big story. I don’t think any of them are particularly surprising. I don’t have any answers for myself about the difference between providing necessary and important context, and ‘poisoning the well’ as @oldandrewuk put it. But I do have two insights that are important to me, if nobody else.

  1. The headline used in the article press-release is subtly misleading. “Most British scientists cited in study feel Richard Dawkins’ work misrepresents science.” My italics highlight the problem; 38 who were negative is not a majority of the 137 interviewed.
  2. The data used was selected to show one particular aspect of the results, and arguably some links were not explored despite being of interest. This can never be as good as a study designed to test one particular question. Only by closely reading the information was it clear how the judgments were made by the researchers.

I’d like to highlight that, as seemed fair to me, I invited @oldandrew to comment here following our discussion on twitter. He has so far chosen not to do so.

Conflicts of Interest

To be transparent, I should point out for anyone who doesn’t realise that I’m an atheist (and humanist, and secular). I often also disagree with Dawkins’ communications work – in fact, if they’d asked me the same questions there’s a fair chance I would have made the point about him causing difficulties for the representation of science to non-scientists – but that’s why I recommend his science books specifically!

Links

The wonderful @evolutionistrue posted about this research too. As a contrast, have a look at how EvangelismFocus wrote it up.

Lies, Damned Lies and Christian Statistics

I’m a science teacher. When talking about the characteristics of sound in my lessons, I encourage students to give detail. It’s not enough to say that a change causes ‘more vibrations’. If the sound is a higher pitch, the vibrations of the ear drum will be faster, or more frequent. If the sound is louder, the displacement of the ear drum is bigger; we say the vibrations have greater amplitude or more energy. So it’s not that the ‘more vibrations’ answer is wrong – just incomplete. If we don’t give a full answer it can be misunderstood.

So I was catching up with news and read an article on the BBC about the continued arguments about institutionalized discrimination and hate speech in the Anglican church. Now, this isn’t about Welby being sorry for the discrimination – just not sorry enough to stand against it – or the hypocrisy of them sending out advice to schools on homophobic bullying. Instead, it’s simply about a number in the report.

_87734930_topanglicancountries

I teach my students to do a ‘common sense check’ as part of any calculation and I was bemused that the BBC didn’t appear to have thought this through. Since when was a third of the UK Anglican? Now, I understand that calculating exactly how many (Anglican) Christians in the UK might be tricky, but 26 million seemed too far off to be reasonable. So I did some digging myself, and asked the organisation behind the ‘World Christian Database’ for the source of this number. It’s important to note that on Twitter they were very definite it was an aggregate figure and they used many sources of data.

 

So how should we find out how many (Anglican) Christians there are in the UK?

Simple, isn’t it? Pop into your local church on Sunday morning and count heads. But which Sunday? What about parishioners who are too ill to make it in, or are shift-workers? Would a Christmas or Easter service be more meaningful? And surely some believers prefer to worship in other ways. So church attendance figures, although useful, can probably be considered a lower limit. The Statistics for Mission 2014 (pdf) figures are just under a million for average Sunday attendance during October, with significantly higher numbers for Easter and Christmas services.

Church Attendance: 0.98m (980000)
Christmas Services: 2.4m

There’s been lots of arguments about the census question, starting with the fact that it assumes the respondent will have a religion in the first place. The cultural identity part of this is recognized within the Census analysis, as the quote below demonstrates:
The question (‘What is your religion?’) asks about religious affiliation, that is how we connect or identify with a religion, irrespective of actual practice or belief.

According to the last Census figures, England and Wales has 33m Christians, but this isn’t broken down into denominations. Most data I’ve found suggests around half of UK Christians consider themselves Anglican, so we can get a reasonable estimate.

Census Anglicans: 17m (approx)

Many surveys call this number into question, for example this report discussing data that only 30% of Britons consider themselves religious at all. As a contrast, the British Social Attitudes Survey asks a range of questions of a randomly selected sample (around 3000 people), including their religion and religious upbringing. The last dataset suggests 17% of the population describes themselves as Anglican, a significant drop.
Self-described: 8.5m (from BSAS)
Of course, if we wanted to simply collect data on the number of people who had been baptized, this would be easier. The agreed estimate – which send to have been used for not just years but decades – is 26m. I’d be very interested to know how this value hasn’t changed; surely infant baptisms and deaths of those baptized can’t have coincidentally been in balance for all this time?
Baptized Anglicans: 26m
Most of these are, naturally, infant baptisms – which brings me to an important and obvious point. I was baptized. But like many others, the fact of my baptism is completely irrelevant to my (lack of) belief. This number includes me – and if you were baptized, it includes you too. (Some non-believers, starting with John Hunt in 2009, are trying to do something about this.) So using this figure, while ignoring all the other values, seems disingenuous to say the least and knowingly dishonest at the most. It’s like the TK maxx adverts, ‘always up to 60% off’. It could mean 59% off. It could mean 1% off. That there are apparently 26 million people baptized as Anglican in the UK is a meaningless figure without the context – which significantly undermines any argument based upon it.

Might it be reasonable, I wonder, to suggest that claiming 26m Anglicans in the UK is bearing false witness?

 

2016: Looking Ahead

2015 was a pretty busy year. Lots going on – not that life with three kids is ever any different – and it didn’t always feel like there was a chance to breathe. So now I’m taking a moment to write what I’m hoping to get out of 2016.
Professionally, I’m hopeful. The day job is going well, I think, and I’m looking forward to sharing more ideas with colleagues. I’m hoping to collect some data on the classroom impact of what I’ve been doing with ‘my’ schools, which will possibly lead to an article with my name on it. I need to start searching more systematically for freelance work, delivering CPD or getting involved in the textbook and resource side of things. I think my blog can be a good way to showcase what I do, so I’m planning to schedule days to build it up each month. I’d like to get some more classroom time, with real students, but supply work is both hard to fit in and really badly paid, so it will probably be in small chunks in my partner schools.
As ‘payment’ for a recent review piece I scored several education texts, so the reading will form a fair bit of my own CPD over the coming months. I plan to share highlights and insights on here as ever. Plus I got What If? just before Christmas, which will get a post all of its own. (Summary: it’s great.) I was hoping to make at least one day of #ASEconf but my wife and I are both shattered as smallest person is doing a lot more teething than sleeping. I’ll have to look out for something later on in the year. Suggestions?
Due to circumstances beyond our control – persistent bullying by pupils, and a lack of willingness to deal with it coupled with deeply unprofessional behaviour on the part of the school management – we’ve taken our eldest out of school. He’ll be finishing his year 6 being educated at home, which has its own set of challenges but also opportunities. To start with, he’s missing the huge ordeal of SATs. They had already started doing practice papers, in November, showing how disproportionate an effect external testing can have. So instead he’s learning to cook, improving his photography and gymnastics skills, and developing his computing by designing and building a website for our child-minder. Plus regular maths practice, book reviews and comprehension exercises based on articles from chicken care to architecture. The good news is that by September he will also have all the skills I’ve so often seen lacking in Year 7 science students, if it kills me. Which it might.
Illness on my side of the family – Dad’s very much past his best-before date, and it shows – means we’ll be making the trip south on a fairly regular basis. Several hours stuck in a car with a crying infant now constitute a hot date for my wife and I, sadly. This is probably one of many issues that have encouraged the return of my personal ‘black dog‘, so apologies for frequent silences and grumpiness on Twitter. More running might help. I need to sign up for some events – I won’t call them races because I’m only competitive with board games – to motivate me to get out and train. I got out climbing with the boys last week, which reminded me of just how much I ennjoy it and just how out of shape I’ve become. Less chocolate. More climbing. Repeat.
I’d like to get into the habit of baking bread. I want to get at least one more tattoo, and I’m still playing with some science-inspired ink designs. Drink good coffee. Read good books.
And to all my loyal readers… all the best for the coming year.

A Scientific Summer

My eldest son already identifies as geeky. Maybe it’ll change, and that’s fine. But right now he’s got a Raspberry Pi, several electronics kits and a burning desire to make a robot that will allow him to take over the world. He likes coding (secret ciphers as well as Scratch and Python) and is getting into astronomy, microscopes and wildlife. He wants to be a zoologist (this week) and is learning to touch type so he can blog his discoveries. In the interests of fairness: he also reads continuously, rock climbs, draws really badly and has taught himself to turn cartwheels and do backflips on a trampoline. He’s as well rounded as the average nine year old, for which his mother deserves a lot more credit than I.
The thing is, supporting his interest in science always came easily to me. I get science. It makes sense to me. Although I mainly teach physics, I’ve enough of a grounding across the board that I know where to look for good explanations if I’m stumped myself. (Recently: what’s the difference between moths and butterflies?) Sport, drama, music – these are mysteries to me. But science I get.
Lots of people don’t. Which means lots of parents don’t.
The ExpeRimental project – the middle capital letter is because it’s being run out of the Ri, the Royal Institution – is about giving parents the tools to play scientifically with their kids this summer. Instructions for simple, kitchen science demonstrations that help kids explain basic scientific ideas. It’s free, it’s supported via YouTube and it’s led not by famous faces but by real kids and parents. Alom Shaha is part of the project which means both the science and the videos are top-notch. The focus isn’t on recreating something they could see online, but on thinking scientifically. This is about questions, not answers, and it’s something everyone can learn, young or old.
By this point you’re probably nodding enthusiastically, but you’re also realising, as I did, that we’re the wrong audience. This site isn’t intended for me, although I’ll use it. As science teachers, we don’t need this. Parents need this project.
I don’t need this site
Because I’m a geek.

There, I said it. Am I stepping out of some intellectual closet here? (Bonus points to anyone who gets the reference, answer in the comments.)

This doesn’t make me a character in The Big Bang Theory, which I see as undemanding (rather sexist) comedy rather than a life blueprint. It doesn’t make me a genius. I’m neither a scientist nor an engineer. It makes me a person who recognises a fact about the world that I’d probably take for granted, if I didn’t have to make it explicit in a daily basis for my students.

Science is cool.

Not just because it leads to great stuff. Although it does. Not just because it involves big explosions. Ditto. And not just because it’s useful (candidate for this year’s understatement award) when facing crises like climate change, water scarcity and antibiotic-resistant bacteria. Science is cool because it’s arguably the best way to answer one of the first abstract questions human beings ask.

Why?

Yes, I know that often the explanation of cause and effect involves people and their motives as well as scientific principles. Yes, I know that sometimes we’re answering the question in terms of justifying a choice rather than explaining how something works. But in its simplest possible incarnation, science is about describing how the world works and suggesting a reason for it. Magic and religion may have offered the first explanations, but science offered the first that actually worked. And still does. So science is cool, and thinking scientifically is useful to everyone, and lots of people think they can’t do it.

Who To Tell
share button 2
Is everyone too simplistic? It would be better to tell people who care. Parents who have six weeks of summer holiday ahead of them and know the summer reading challenge, great as it is, won’t be enough. ExpeRimental will be adding a new video each week over the summer. Kids and their parents will be encouraged to share their own contributions – videos, surprises, results, questions – via the project Facebook page.
Pass it on via your preferred social media, ideally a few times over the next few weeks. Email those in your family who are zoo keeping child minding this summer. Better yet, send it to older brothers and sisters and challenge them to help. Bribery is good – Horrible Science books are easy to find in charity shops and discounters, or the Klutz range provide great materials and ideas.
I’d love to see promotional materials in local libraries and their noticeboards – I might add a few myself. And perhaps bookshops might add something in their Science for Kids shelves? The point is to get the ideas into the hands of desperate parents who want to get through the summer without spending a fortune or shouting too much.
And next?
One approach that seems to have been missed; as well as primary teachers and those running science clubs, the resources would be great for youth groups. The project would be a great foundation for Rainbows or BeaversBrownies and Cubs. Pass on the link to youth leaders for a ready-made activity in the autumn.
Something I’m using with my own kids this summer is an American site, diy.org. Kids can share projects, from Actor to Zoologist, and earn badges to show what they’ve done. The ideas would link really nicely with the PhilosopherData Visionary and Film Maker skills as well as the more obvious scientific ones.
Next stop would be visiting somewhere scientific. I’m sure there must be lists of science centres, museums with dinosaur exhibits and wildlife centres suitable for kids. This could be as easy as printing off a checklist and heading for the park to look at bugs. Or as involved as staying overnight in the Science Museum. Just like science, this project is about starting to ask questions, not about giving final answers. And this post isn’t really about science teaching, any more than visiting the Roald Dahl museum is about teaching literacy. Instead, it’s about being a parent.
Which is what I’ll be up to for the next six weeks or so. Probably armed with a Pringles cannon…

Human Rights for Children

So, I had this idea.

If you read this blog at all regularly, you’ll know that I consider @alomshaha a friend. As well as writing, making films and teaching science, he should be credited with getting me on to Twitter two years ago. Thank him later. Right now, I’ve something more important for you to do.

I read Alom’s excellent book, The Young Atheist’s Handbook, when it first came out. Despite the title it’s easy to read it as a personal story of how he came to consider himself n atheist, despite his early upbringing in a Bangladeshi Muslim community. The references and explanations of ideas supporting his lack of religious belief are a comfortable part of an honest and evocative story. I’d recommend it to anyone, and have done.

The problem, as I saw it, is that the very kids who would benefit most from reading it were those least likely to have the chance. If your parents are invoking freedom of religion (actually an example of religious privilege) to teach you from a young age to follow their faith, without question or deviation, then they are unlikely to be pleased at you putting this on your birthday list. I’m sure some young people will read it discreetly as an eBook of whatever format. But, I reasoned, there had to be a better way.

School is for learning. School is where kids learn the things their parents don’t or can’t teach them. Like swear words and how to think for yourself. So, I reasoned, if Michael Gove can send bibles into schools, and the Gideons can visit, and the Church of England can explicitly plan to use their schools to indoctrinate kids, why not provide a different viewpoint?

And so the #YAH4schools campaign was born. The admin is complicated, and is being supported by the British Humanist Association. This idea is simple (that was my bit).

We want to raise enough money to send copies of the book to every UK secondary school library.

Young people have the right to choose to be free from religion. It is not just their parents’ freedom of religion we should respect. We want young people to feel supported, not isolated, if they choose to exercise that right. Freedom of religion shouldn’t mean that parents have the choice to force their children into one particular faith. If you think this project is a good idea, there are two things you can do:

  • Donate to the campaign via YAH4schools.org.uk
  • If you feel able, tell your friends and your family; share the link on Facebook or via Twitter (hashtag #YAH4schools).

Whether you agree or disagree with the project, then of course I’d be interested to hear your views in the comments below.

Book Busking

Where possible, I buy local. We have organic meat and eggs, for welfare reasons rather than concerns about pesticides or whatever. Chocolate, coffee and bananas are fairtrade. And yet, for a long time, I bought an awful lot of books through Amazon. I really shouldn’t have needed the recent revelation that Amazon are evading tax to encourage me to do my buying elsewhere. After all, it was in WeAreWhatWeDo’s great Save the World For a Fiver book years ago (and is still online).
 
Now I have a local bookshop. And it’s on a boat. Unfortunately right now I can’t see the review I added to the Guardian, as the map doesn’t seem to work very well, but maybe you can. I was very complimentary, and it was well-deserved. Sarah’s found books for me with only the vaguest idea of the title, refused to let me pay full price far too often, made tea while we chatted and said nice things about my baking. We’ve swapped recommendations and so far have both been delighted by the other’s suggested titles. She’s patient with the kids, and enthusiastically stocks the books they’ve enjoyed. She’s even promised to look over my book proposal and sample chapter, when I finally get around to writing it.
 
I now browse for books online or in Waterstones, then buy them through the Book Barge. As a business, she’s having trouble – and apologies for the pun – staying afloat. Not enough people buy books through ‘real’ bookshops, especially places that can’t afford to subsidize bestsellers with ‘3 for 2’ offers, or where you have to wait a day or so before they get in what you’re after.
 
So why should you care?
 
Well, if you’re local (South Staffordshire-ish), you could visit. I recommend it. If you’re not local, you could visit your own local independant bookshop, while it’s still there. If there’s not one handy, you can use The Hive Network, which means an independent gets a percentage (small, I suspect). In each of these cases, the difference to the Amazon price is likely to be noticeably less than a coffee out, which most of us wouldn’t hesitate to have.
 
Finally, you could support Sarah’s newest idea, which I would describe as ‘book busking’. She explains it better than I could here, but in summary it will mean a weekly chapter of a book explaining the volumes that have inspired her. These will be online for free, but with the electronic equivalent to an ‘honesty box’ – a PayPal button. I’m already subscribed via RSS, and look forward to the first installment. My aim is to either buy the book she recommends each time, or contribute a small amount in the hope that others are doing the same. To pay for this I’m giving up buying cakes, which is probably good for me too.
 
Thank you for taking the time to read a non-teaching post – especially one that is so selfish on my part. Not financially, I hasten to add – I just like having such a great shop nearby. Call it ‘enlightened self-interest’. Please let me know in the comments what you think of the idea. (And to make it slightly relevant, this would be a fantastic case study for both English and Business teachers…)

A New Exam Board?

We’ve seen a lot of problems with exams recently – just look at the problems last summer with mistakes in a wide range of exam papers. Today I’ve found that AQA have spent so little time checking that suitable research sources are online that the only good Google results are their own teacher notes, and a primary science investigative cartoon. On top of this, a new specification inevitably means a lack of practice material which means students and teachers don’t really know what to expect.  If you have to explain why this is unfair to non-teachers, perhaps this analogy might help; we wouldn’t expect to have a driving test on the road having only practised in car parks, would we?

I have an idea.

In fact, I have two ideas, neither of which is mine. If we take the ‘backward design’ principle (originated by Wiggins and McTighe, introduced to me by Robin Millar’s work) and combine it with a ‘curated crowdsourced’ model, maybe there’s a way to do a better job. 

Backward Design

My apologies to Robin and other experts if I miss the subtleties – I’m just a classroom teacher with delusions of writing grandeur. Instead of beginning a syllabus with the content that we want to teach, backward design asks what we want students to be able to do at the end – how will they be tested? How will we know if the course was successful or not (or more precisely, how successfully the student has completed it)? If we create assessment tasks that will allow us to differentiate between students – ideally including, but not limited to written exams – then we can develop a list of what students should learn, which gives us a list of possible learning/teaching activities. As Robin and others point out, ‘teaching to the test’ is only a problem if the test is not fit for purpose. If we produce a realistic, useful test then being prepared for it is a positive thing. 

Crowdsourcing

So who better to contribute possible questions than teachers? Imagine a Google form set up by a new exam board; let’s call it CCEB. Exemplar material, based on accepted good practice, shows how to lay out mathematical working. Questions are entered, with a markscheme. Dropdown boxes allow those entering the question to define marks available, and from key words describing the area(s) of science being tested. Active teachers, retired staff, academics – even students – all can contribute. The contributions are freely given on the basis that the results will be freely available as far as practical, probably via Creative Commons licensing.

Curated

When a certain threshold is reached – which if every science teacher in the UK supplies a single question, won’t take long – the submissions are sorted by category and checked by CCEB staff. Because they are being proofread rather than written, it will be quicker and easier. If you have some of the original contributors – determined by random allocation – paid for a day’s work, they can be pre-moderated as well. Mathematical questions can be kept in the same form but with different numbers substituted. A large pool of questions is now complete, ready for the exam, which can be balanced between topics. There will be enough questions, all produced at the same time, for several specimen papers to be made available. With a large enough pool, you could even make all the questions open source, like those for the theory element of the UK driving test.

One Day 

It’s feasible that in the future, with enough questions available, every student could get a different but equivalent exam, as described in John Barnes’ book Orbital Resonance.

In the meantime, maybe we as science educators can get involved with setting better exams than the ones we complain about. The exam boards could ask for submissions in this way now. The cynic in me thinks that this would make it much harder for them to justify their existence. Maybe they would like to prove me wrong.