Calling BS

You have to be wary of newspaper articles published on 1 April, but I think this one is genuine. The Guardian on Monday contained a report about scientific research into bullshit. Or more specifically, a scientific/statistical study into the demographics of bullshitting.

Now, to make any sense of this, it’s important first to understand what bullshit is.  Bullshit is different from lying. The standard treatise in this field is ‘On Bullshit‘ by Harry Frankfurt. I’m not kidding. He writes:

It is impossible for someone to lie unless he thinks he knows the truth. Producing bullshit requires no such conviction

In other words, bullshitting is providing a version of events that gives the impression you know what you are talking about, when in fact you don’t.

Unfortunately, standard dictionaries tend to define bullshitting as something like ‘talking nonsense’, though this is – irony alert – bullshit. This article explains why and includes the following example. Consider the phrase

Hidden meaning transforms unparalleled abstract beauty.

It argues that since the sentence is grammatically correct, but intellectually meaningless, it is an example of bullshit. On the other hand, the same set of words in a different order, for example

Unparalleled transforms meaning beauty hidden abstract.

are simply nonsense. Since they lack grammatical structure, the author isn’t bullshitting. He’s just talking garbage.

So, bullshit is different from lying in that the bullshitter will generally not know the truth; and it’s different from nonsense in that it has specific intent to deceive or misdirect.

But back to the Guardian article. The statistical study it refers to reveals a number of interesting outcomes:

  • Boys bullshit more than girls;
  • Children from higher socioeconomic backgrounds tend to bullshit more than those from poorer backgrounds;
  • North Americans bullshit the most (among the countries studied);
  • Bullshitters tend to perceive themselves as self-confident and high achievers.

If only I could think of an example of a self-confident, North American male from a wealthy background with a strong tendency to disseminate bullshit in order to illustrate these points.

But what’s all this got to do with Statistics? Well, it cuts both ways. First, the cool logic of Statistics can be used to identify and correct bullshit. Indeed, if you happen to study at the University of Washington, you can enrol for the course ‘Calling Bullshit: Data Reasoning in a Digital World‘, which is dedicated to the subject. The objectives for this course, as listed in its syllabus, are that after the course you should be able to:

  • Remain vigilant for bullshit contaminating your information diet.
  • Recognize said bullshit whenever and wherever you encounter it.
  • Figure out for yourself precisely why a particular bit of bullshit is bullshit.
  • Provide a statistician or fellow scientist with a technical explanation of why a claim is bullshit.
  • Provide your crystals-and-homeopathy aunt or casually racist uncle with an accessible and persuasive explanation of why a claim is bullshit.

I especially like the fact that after following this course you’ll be well-equipped to take on both the renegade hippy and racist wings of your family.

So that’s the good side of things. On the bad side, it’s extremely easy to use Statistics to disseminate bullshit. Partly because not everyone is sufficiently clued-up to really understand statistical concepts and to be critical when confronted with them; and partly because, even if you have a good statistical knowledge and are appropriately sceptical, you’re still likely to have to rely on the accuracy of the analysis, without access to the data on which they were based.

For example, this article, which is an interesting read on the subject of Statistics and bullshit, discusses a widely circulated fact, attributed to the Crime Statistics Bureau of San Francisco, that:

81% of white homicide victims were killed by blacks

Except, it turns out, that the Crime Statistics Bureau of San Francisco doesn’t exist and FBI figures actually suggest that 80% of white murder victims were killed by other white people. So, it’s a bullshit statement attributed to  a bullshit organisation. But with social media, the dissemination of these mis-truths becomes viral, and it becomes impossible to enable corrections with actual facts. Indeed, the above statement was included in an image posted to twitter by Donald Trump during his election campaign: full story here. And that tweet alone got almost 7000 retweets. So though, using reliable statistics, the claim is easily disproved, the message is already spread and the damage done.

So, welcome to Statistics: helping, and helping fight, bullshit.




Mr. Wrong


As a footnote to last week’s post ‘How to be wrong‘, I mentioned that Daniel Kahneman had been shown to be wrong by using unreliable research in his book ‘Thinking, Fast and Slow’. I also suggested that he had tried to deflect blame for this oversight, essentially putting all of the blame on the authors of the work which he cited.

I was wrong.. pointed me to a post by Kahneman in the comments section of the blog post I referred to in which Kahneman clearly takes responsibility for the unreliable interpretations he included in his book, and explaining in some detail why they were made. In other words, he’s being entirely consistent with the handy guide for being wrong that I included in my original post.


But while we’re here, let me just explain in slightly more detail what the issue was with Kahneman’s analysis…

As I’ve mentioned in other settings, if we get a result based on a very small sample size, then that result has to be considered not very reliable. But if you get similar results from several different studies, all based on small sample sizes, then the combined strength of evidence is increased. There are formal ways of combining results in this way, and it often goes under the name of ‘meta-analysis‘. This is a very important technique, especially as time and money constraints often mean the sample sizes in individual studies are small, and Kahneman used this approach – at least informally – to combine the strength of evidence from several small-sample studies. But there’s a potential problem. Not all studies into a phenomenon get published. Moreover, there’s a tendency for those having ‘interesting results’ to be more likely to be published than others. But a valid combination of information should include results from all studies, not just those with results in a particular direction.

Let’s consider a simple made-up example. Suppose I’m concerned that coins are being produced that have a propensity to come up Heads when tossed. I set up studies all around the country where people are asked to toss a coin 10 times and report whether they got 8 or more heads in their experiments. In quite a few of the studies the results turn out to be positive – 8 or more heads – and I encourage the researchers in those studies to publish the results. Now, 8 or more heads in any one study is not especially unusual: 10 is a very small sample size. So nobody gets very excited about any one of these results. But then, perhaps because they are researching for a book, someone notices that there are many independent studies all suggesting the same thing. They know that individually the results don’t say much, but in aggregate form the results are overwhelming that coins are being produced with a tendency towards Heads. And they conclude that there is very strong evidence that coins are being produced with a tendency to come up Heads. But this was a false conclusion, due to the fact that the overwhelming number of studies where 8 or more Heads weren’t obtained didn’t get published.

And that’s exactly what happened to Kahneman. The uninteresting results don’t get published, while the interesting ones do, even if they are not statistically reliable due to small sample sizes. Then someone combines via meta-analysis the published results, and gets a totally biased picture.

That’s how easy it is to be wrong.

How to be wrong

When I’m not feeling too fragile to be able to handle it, I sometimes listen to James O’Brien on LBC. As you probably know, he hosts a talk show in which he invites listeners to discuss their views on a wide range of topics, that often begin and end with Brexit. His usual approach is simply to ask people who call in to defend or support their views with hard facts – as opposed to opinion or hearsay – and inevitably they can’t. James himself is well-armed with facts and knowledge, and is consequently able to forensically dissect arguments that are dressed up as factual, but turn out to be anything but. It’s simultaneously inspiring and incredibly depressing.

He’s also just published a book, which is a great read:


This is the description on Amazon:

Every day, James O’Brien listens to people blaming benefits scroungers, the EU, Muslims, feminists and immigrants. But what makes James’s daily LBC show such essential listening – and has made James a standout social media star – is the careful way he punctures their assumptions and dismantles their arguments live on air, every single morning.

In the bestselling How To Be Right, James provides a hilarious and invigorating guide to talking to people with faulty opinions. With chapters on every lightning-rod issue, James shows how people have been fooled into thinking the way they do, and in each case outlines the key questions to ask to reveal fallacies, inconsistencies and double standards.

If you ever get cornered by ardent Brexiteers, Daily Mail disciples or little England patriots, this book is your conversation survival guide.

And this is the Sun review on the cover:

James O’Brien is the epitome of a smug, sanctimonious, condescending, obsessively politically-correct, champagne-socialist public schoolboy Remoaner.

Obviously, both these opinions should give you the encouragement you need to read the book. Admittedly, it’s only tenuously related to Statistics, but the emphasis on the importance of fact and evidence is a common theme.

But I don’t want to talk about being right. I want to talk about being wrong.

One of my first tasks when I joined Smartodds around 14 years ago was to develop an alternative model to the standard goals model for football. I made a fairly simple suggestion, and we coded it up to run live in parallel to the goals model. We kept it going for a year or so, but rather than being an improvement on the goals model, it tended to give poorer results. This was disappointing, so I looked into things and came up with a ‘proof’ of how, in idealised circumstances, it was impossible for the new model to improve on the goals model. Admittedly, our goals model didn’t quite have the idealised form, so it wasn’t a complete surprise that the numbers were a bit different. But the argument seemed to suggest anyway that we shouldn’t really expect any improvement, and since we weren’t getting very good results anyway, we were happy to bury the new model on the strength of this slightly idealised theoretical argument.

Fast-forward 14 years… Some bright sparks in the RnD team have been experimenting with models that have similar structure to the one which I’d proved couldn’t really work and which we’d previously abandoned. And they’ve been getting quite good results, that seem to be an improvement on the performance of the original goals model. At first I thought it might just be that the new models were so different to the one I’d previously suggested, that my arguments about the model not being able to improve on the goals model might not be valid. But when I looked at things more closely, I realised that there was a flaw in my original argument. It wasn’t wrong exactly, but it didn’t apply to the versions of the model we were likely to use in practice.

Of course, this is good and bad news. It’s good news that there’s no reason why the new versions of the model shouldn’t improve on the goals model. It’s bad news that if we’d understood that 14 years ago, we might have explored this avenue of research sooner. I should emphasise, it might be that this type of model still ends up not improving on our original goals model; it’s just that whereas I thought there was a theoretical argument which suggested that was unlikely, this argument actually doesn’t hold true.

So what’s the point of this post?

Well, all of us are wrong sometimes. And in the world of Statistics, we’re probably wrong more often than most people, and sometimes for good reasons. It might be:

  • We were unlucky in the data we used. They suggested something, but it turned out to be just due to chance.
  • Something changed. We correctly spotted something in some data, but subsequent to that things changed, and what we’d previously spotted no longer applies.
  • The data themselves were incomplete or unreliable.

Or it might be for not-such-good reasons:

  • We made a mistake in the modelling.
  • We made a mistake in the programming.

Or, just maybe, someone was careless when applying a simple mathematical identity in a situation for which it wasn’t really appropriate. Anyway, mistakes are inevitable, so here’s a handy guide about how to be wrong:

  1. Try very hard not to be wrong.
  2. Realise that, despite trying very hard, you might be wrong in any situation, so be constantly aware as new evidence becomes available that you may need to modify what you believed to be true.
  3. Once you realise you are wrong, let others know what was wrong and why you made the mistake you did. Humility and honesty is way more useful than evasiveness.
  4. Be aware that other people may be wrong too. Always use other people’s work with an element of caution, and if something seems wrong, politely discuss the possibility with them. (But remember also: you may be wrong about them being wrong).

Hmmm, hope that’s right.

I was encouraged to write a post along these lines by following a recent  chat where we were discussing the mistake I’d made as explained above. To help me not feel quite so bad about it, he mentioned a recent blog post where some of the research described in Daniel Kahneman’s book, ‘Thinking, Fast and Slow’, is also shown to be unreliable. You might remember I discussed this book briefly in a previous post. Anyway, the essence of that post is that the sample sizes used in much of the reported research are too small for the statistical conclusions reached to be valid. As such, some chapters from Kahneman’s book have to be considered unreliable. Actually, Kahneman himself seems to have been aware of the problem some years ago, writing an open letter to relevant researchers, setting out a possible protocol that would avoid the sorts of problems that occurred in the research on which his book chapters were based. However, while Kahneman himself can’t be blamed for the original failures in the research that he reported on, it’s argued in the blog post that his own earlier research might well have led him to foresee these types of problems. Hence, the rather aggressive tone of his letter seems to me like an attempt at ring-fencing himself from any particular blame for the errors in his book. In other words, this episode seems like a slightly different approach to ‘how to be wrong’ compared with my handy guide above.


… to Smartodds loves Statistics.

This blog is intended for employees and clients of Smartodds. Its aim is to raise awareness of statistical issues and to encourage statistical thinking within the company.

Posts will be of various types – references to statistical applications, explanations of statistical techniques, interactive exercises, discussion of statistical problems, references to statistics in the media,  and much else besides – but written in a way that recognises that not everyone connected to Smartodds is a statistics expert. Hopefully it will turn out to  be interesting and entertaining.

Though the aim is to raise the profile of statistics within Smartodds, and though Smartodds is primarily about sports, the items in this blog will not be exclusively about sports statistics. Some will be, but many won’t. The point is to help all of us understand better the way statistics is, or could be, used within Smartodds, and that can often be done best without specific reference to sporting issues. Probably you’ll find some posts more interesting than others, depending on your own tastes, but I hope throughout the blog there’ll be something for everybody. So please stick with it even there are some posts that don’t interest you much.

As well as following the blog regularly, there are different ways that you can contribute. You can:

  1. add comments to any of the posts;
  2. ask questions or request clarification, also through the comments section of each post;
  3. publish your own posts, perhaps giving links to a news article that has made an interesting (or terrible!) use of statistics; or perhaps simply discussing something statistically-related that’s of interest to you;
  4. mail me (at asking for a post that covers any particular topic;
  5. contact me (again at or using the contact page to provide feedback on the blog.

The more interactive the blog is, the more successful it’s likely to be. So, please do get involved. Even negative feedback is helpful: partly so I can try to improve things; partly so I can wrap the blog up if it’s not serving a useful purpose.

So, I hope you join the ride, read the blog and contribute yourself whenever you feel like it. Let’s see if we can find a better definition of statistics than this one: