Testing times

As referred to in an earliest post, it’s been understood from the start of the epidemic that testing for Coronavirus is crucial for limiting its initial spread and as a way of mitigating against a second wave once the first wave has been brought under control. It seems reasonable therefore, at least in part, to judge the efficacy of a government’s response to the crisis by the extent of their testing regime. So how does the UK compare against other countries in this respect? Though it’s not the whole picture we can look at some statistics across countries.

This is a graph of the total number of tests per 1,000 people carried out through time for a number of countries:

Some of the main points are:

  • Even allowing for the fact that South Korea had a number of cases before most other countries, it carried out a large number of tests early on in the crisis. This is almost certainly one reason that it has been relatively successful in containing the epidemic, and has not needed to massively extend its testing capacity in recent weeks.
  • Italy’s outbreak started later than that of South Korea, but even allowing for that, it was slower in developing its testing capabilities. More recently though it has developed an extensive testing framework.
  • The US was extremely slow in ramping up testing facilities, though in recent weeks it has started to do much better.
  • The UK and Greece continue to have relatively limited numbers of tests. However, Greece has been commended for tackling the epidemic through other fast and decisive measures; the UK less so.

Another way of looking at the numbers is as a single snapshot of the test numbers – this time per 1 million capita – as of a couple of days ago.

It’s maybe no surprise that the UK is unable to match a country with a far smaller population like Iceland, but less clear why it can’t match Ireland, Italy or Germany.

But what about the relationship between testing and infection rates? The following graph compares per capita test rates against per capita case numbers:

Interpretation is not completely straightforward. One might hope that more tests per capita lead to a better control of the epidemic and therefore fewer cases per capita. But equally, countries that have so far had relatively few cases are likely not to have built up an extensive testing network yet. Also, a country that carries out more tests is likely to identify more positive cases.

Nonetheless, there are a number of takeaway points:

  • Though Iceland does the most tests per capita, it also has a large number of cases. But, as explained above, one reason it has more confirmed cases is likely to be simply because it’s done more tests. Moreover, Iceland has been praised for tackling the epidemic through a rigid contact tracing and isolation regime, and has been successful in keeping the number of deaths very low (currently just 9).
  • Italy, similarly, has a large number of cases despite carrying out many tests. Again, this will be partly because more tests are likely to lead to more cases, but is likely to be more strongly influenced by the fact that Italy was the first European country to be badly affected by the epidemic. It therefore had less evidence with which to decide how best to respond to the outbreak and was therefore relatively slow in ramping up testing. Furthermore, Italy is further ahead in its trajectory than other countries, so in future versions of this graph, other countries are likely to move further to the right than Italy does.
  • Though South Korea has kept its number of cases fairly low – especially for a country that was affected early on in the pandemic – its actual testing capacity is no greater than that of the US. However, as we saw in the earlier graph, its initial testing response was very much quicker. So, it’s not just the number of tests that’s important, it’s the speed with which the testing framework is put in place.
  • For countries with a large number of cases, the UK is weakest in terms of the number of tests.  This won’t be the only reason it has a large number of cases per capita, but it’s likely to be one of them.

In summary, based purely on the statistical evidence, the UK comes out relatively poorly compared to many other countries in terms of its response to the Coronavirus epidemic through building a platform for population testing.

One caveat. The fine print in each of these graphs makes clear that comparison across countries is not completely fair. Data are collected and recorded in different ways in different countries. For example, in the UK numbers are on people tested; in Italy they are on the number of tests carried out. This is likely to make some difference, as people who test positive are likely to be subsequently tested again to ensure they are negative. This would apparently count as 2 tests in the Italy data, but only 1 in the UK data. However, most test results come back negative. In Italy, for example, the rate of positive tests is around 13%. So, although the difference in data protocols might explain some of the difference in the UK and Italy numbers, it doesn’t explain most of it.


Prior infection

In an earlier post I referred to an Austrian study in which a random sample of the inhabitants of a region had been tested for the Coronavirus. In that article I stressed the importance of random sampling in order to be able to extrapolate the conclusions from the sample to a wider population. But a limitation of that study was the fact that tests were ‘standard’ COVID-19 tests, providing information only on whether an individual currently carried the disease or not. So, although the infection rate was only 0.33% in the sample, it was impossible to say what proportion of individuals in that sample had ever had the disease, perhaps having only slight or no symptoms.

This week a similar study has been reported based on a study in the German municipality of Gangelt, close to the Netherlands border. But in this case, the 500 individuals in the sample were tested for Coronavirus antibodies, a test which provides information on whether an individual has a prior COVID-19 infection, regardless of whether they currently have the disease.

Now, there has been something of a mis-handling of such tests in the UK, with millions of bought tests of this variety which were not properly evaluated prior to purchase and shown subsequently to be worthless in terms of their results. But as far as I can tell, this isn’t an issue for the tests used in the German study.

The results of the study are interesting, and shed both a positive and negative light on the effects of the epidemic.

First, some context. On February 15th, a couple of weeks after Germany reported it’s first case of Coronavirus, Gangelt held its annual carnival. Subsequently, several attendants at the carnival tested positive for Coronavirus, and shortly after the town became a hotspot for the infection within Germany. It’s widely assumed that the carnival was focal in causing the local hotspot and for attendants then assisting the spread of the disease countrywide. So, it’s thought that the infection rate in Gangelt is likely to be relatively high, which is one of the reasons it was chosen for this study.

However, the random sampling antibody study found that ‘just’ 14% of the sample are likely to have a prior infection. Still, 14% is considerably higher than the proportion that have tested positive in the region (around 3.5%), implying that the true death rate due to COVID-19 there is around 0.37% as opposed to the 2% or so that’s stated for Germany as a whole based on standard data. That’s obviously a welcome piece of news.

On the negative side, as discussed in earlier posts – here for example – one solution to the pandemic will occur naturally when a sufficient proportion of the population have a prior infection and are hopefully immune – though this is not guaranteed – from subsequent infection. But this is thought to be as much as 80% for COVID-19. So, if in a community that’s thought to have been a bit of a hotspot the previously infected rate is only 14%, it’s likely to be a long way short of the required threshold in the country as a whole. In other words, on the basis of this study, Germany – and presumably other countries too – are likely to be very far off from being able to relax social restrictions and rely on herd immunity to get them out of the pandemic.

For the UK, the Harvard epidemiologist William Hanage shows – using some back-of-the-envelope calculations – that to acquire herd immunity in the UK, a minimum of 600,000 COVID-19 deaths would occur. Though the UK government’s initial strategy seemed to be aimed at achieving herd immunity without a vaccine, their realisation of the scale of this number of fatalities and the impact it would have on both communities and the health service is almost certainly what led to a change of heart. But until a vaccine is found, the alternative is maintaining social restrictions to a sufficient extent to keep the transmission rate of the disease sufficiently low. As Hanage says:

This crisis is not close to over, quite the reverse. The pandemic is only just getting started.

A couple of comments:

1. Hanage’s calculation goes like this… At the time of writing there were around 100,000 COVID-19 confirmed cases in the UK. But the British Medical Journal suggests that only around 20% of infected people are actually tested positive. (Based on the Gangelt study the estimate would be 3.5/14 =25%). This implies that the actual number of people in the UK who have been infected is around 500,000. But if the UK is currently at the peak of its epidemic, and a similar number of people will be infected as the epidemic declines, that implies one million infected people at the end of the epidemic. But that means around 65 million people will remain uninfected and without immunity. Similarly, there have been 10,000 confirmed deaths due to COVID-19 in the UK. Assuming fatalities have also now peaked, there will be a total of 20,000 deaths once the epidemic has faded. Now, a very conservative estimate of the proportion of people in the population who need to be infected for herd immunity to kick in is 50%, which implies around 30 million people. But so far, as we’ve seen, it’s likely that only one million have been infected. So, we need 30 times as many people to be infected, and this will imply 30 times as many deaths; i.e. around 600,000.

2. Hanage’s comments are not meant to be fatalistic. His point is simply that there are many positive signs that show social distancing is working in terms of controlling the spread of the epidemic in most countries.  But, these measures are doing only that: controlling the spread of the epidemic. Though numbers of infections are growing in the community, they are still a long way from the sorts of numbers that will inhibit further spread of the epidemic via herd immunity. So until vaccines and therapies are available, it’s inevitable that some form of ongoing social controls will be required to stop the epidemic growing exponentially once more.

A/B test

In an earlier post I showed how a comparison of two different, though similar, provinces of Italy which had adopted different approaches to containing the Coronavirus outbreak could serve as a kind of statistical trial on the relative effectiveness of the strategies. A similar argument has now been made in respect of Ireland and the UK by Elaine Doyle. This is the first tweet in her thread, showing that Ireland and the UK started off with similar resources to fight the pandemic:

But, at the time of writing, the UK had far more fatalities than Ireland. Elaine writes:

As of today, there have been 320 deaths from the coronavirus in Ireland, and 9,875 deaths in the UK.


As of Saturday 11 April, there have been 6.5 deaths per 100,000 people in Ireland. There have been 14.81 deaths per 100,000 people in the UK. Guys, people have been dying at more than *twice the rate* in the UK

Elaine argues that since the countries are comparable in most respects, it’s reasonable to assume that the difference in outcomes is attributable to the difference in approaches.

The complete thread of her argument is available here. She writes:

You have a real-time A/B test happening *right in front of you

In other words, you can think of Ireland and the UK as two equivalent units. To one unit (Ireland) you’ve applied ‘treatment A’, a fast and decisive lockdown. To the other (UK) you’ve applied a slower, more gradual, lockdown. And the difference in outcomes is due to the difference in treatments.

This argument isn’t universally agreed on. Other experts argue that there are other fundamental differences between Ireland and the UK – for example, Ireland has a much greater proportion of its population living in rural areas – and these are just as likely to impact on the numbers of COVID-19 cases as the differences in the strategies applied. This is no doubt true. And a true A/B test  would have comprised random allocation of treatments to more than two countries. The lack of randomness in the allocation is a serious hinderance to interpretation, though a comparison of just 2 countries is dangerous in any case.

In time, it might be possible to compare many different countries according to the strategies adopted, accounting for different geographical and demographic factors, and decide which strategies are most effective. This type of analysis was discussed in an earlier post for the 1918-19 influenza epidemic, though I’m guessing we won’t want to wait 100 years to carry out such an analysis for the current pandemic.

Update: since writing this post, Elaine Doyle has written an article for the Guardian setting out her arguments.

Relative risk

One issue that’s constantly pushed by the media about the effects of COVID-19 is that it’s much more deadly for older people than for younger people. The following graph, for example, shows the age distribution for COVID-19 deaths in Italy as of 11, April:

Bear in mind, also, that there are fewer people in the older cohorts, so the differences in the rate of deaths per cohort are even greater.

But does this suggest that COVID-19 is affecting younger and older people to a different extent? In one sense clearly yes – more older people are dying than younger people. But, in another sense, no. Or at least, possibly no.

In this article David Spiegelhalter – whose podcast I mentioned in an earlier post –  considers the hypothesis that everyone’s risk of dying in the short term is affected by the current epidemic in exactly the same way, regardless of age.

Look, for example, at this diagram taken from David’s article:

These are deaths by age, per 100,000 of the population in England for a week in March, shown separately for males (left) and females (right). The solid dots correspond to COVID-19 deaths, the hollow dots are all other deaths.There are several points to notice:

  • The scale of the graph is logarithmic, so since the death counts – both COVID-19 and not – are approximately linear, the raw numbers will be increasing exponentially with age;
  • This is true for both males and females;
  • The patterns for COVID-19 deaths and others are roughly parallel on this logarithmic scale. This means that on a linear scale, one will be a multiple of the other.

Consider, for example, males in the 45-64 cohort. The death rates for COVID-19 and other causes are around .05 and 10 per 100,000 respectively. This means that the total risk of death including COVID-19 is 10.5:10 or 105% real-time to what it would have been with COVID-19 excluded. Loosely speaking, COVID-19 has raised the death risk for someone in that cohort by around 5%. But if you look at any of the other cohorts, the change is around the same. For example, in the 65-74 cohort, the respective death rates are around 2 and 40 per 100,000, leading to a ratio of 42:40 which is again 105%.

So, looked at this way, COVID-19 is affecting people of all ages in exactly the same way: it’s increasing the risk of death by around 5%. It’s simply that the death risk is so low for young people that an increase of 5% doesn’t lead to many additional deaths in absolute terms, whereas for older people a 5% increase in death rate leads to substantial increases in the observed numbers of deaths. Nonetheless, this provides an interpretation by which COVID-19 is non-discriminatory in terms of age.

A few caveats:

  1. This analysis is based on relatively few data. Though David’s article includes other analyses to support the hypothesis, he also concludes that further data will be required to verify it.
  2. Most of the deaths observed in the data summarised above will have been for people infected prior to lockdown measures having been introduced. It’s likely to be the case that lockdown measures will offer more protection to certain age cohorts than others, in which case the effect on death rates will be disproportionate across age cohorts.
  3. David also mentions in his article that health workers appear to be disproportionately affected in terms of COVID-19 – i.e. their additional death risk due to COVID-19 is greater than 5%.

Random sampling


A recurrent theme in the COVID-19 posts to this blog is the difficulty in interpreting data and analyses due to the way data are collected. Different countries have completely different protocols for testing for the disease, and these protocols have also changed through time. Even the reported death counts are unreliable as a measure of the disease effects.

In one earlier post I mentioned two case studies where entire – though limited – populations had been tested: Vò, in northern Italy and the Diamond Princess cruise ship. Since these entire populations were studied, the data are 100% complete, but they are special cases, since they are closed populations where an outbreak of the epidemic is known to have occurred. But what about entire countries? It’s obviously impractical – at least in present circumstances – to test an entire population.

A statistically valid alternative in this case is random sampling – testing individuals randomly selected from the entire population. The proportion testing positive in the sample provides an estimate of the proportion in the entire population, and the bigger the sample, the better the estimate. Obviously there are logistical difficulties in testing genuinely randomly selected individuals, so various practical modifications are often implemented which have to be correct for in the analysis. But the principle is the same: to use information from a randomly selected sample of individuals to estimate the population level.

A study of this type has now been carried out for Austria. Full details of the analysis can be found here. In summary:

  • 1,544 individuals were included in the study which was carried out in the first week of April;
  • These individuals were identified by a stratification procedure: 249 Austrian districts were randomly selected; households in those districts were randomly selected; individuals within those households were randomly selected;
  • Such individuals were invited to participate in the study – the acceptance rate was 77%;
  • Hospitalised individuals were excluded from the study;
  • Final results were adjusted to correct for various factors including household size, gender and age.

The conclusion, after the correction for age, gender and other effects, is that the COVID-19 infection rate in the sample was 0.33%.

Now, bearing in mind that one solution to the epidemic is that a large number of the population acquire the disease, so building a ‘herd immunity‘, the figure of 0.33% is disappointingly small. Even allowing for sampling error, the true value in the population is predicted in the study to be at most 0.76%, whereas it’s thought that herd immunity will require around 60-80% of the population to have been infected.

However, this figure of 0.33% is just a snapshot in time of people who currently have the virus; it doesn’t say anything about the proportion of people who have had the virus – perhaps asymptomatically – and recovered. That figure, which is the figure of interest when discussing herd immunity, is bound to be bigger. But it’s impossible from this study to say by how much.

Moreover, extrapolating the 0.33% to the entire population of Austria would imply around 28,500 positive cases. By contrast, the number of active cases in Austria (as of today, 11 April) is recorded as 6,608:


So, even as low an estimate as 0.33% for the countrywide infection rate implies a roughly four-fold increase in the number of cases above and beyond the official numbers.


Why what we think we know is wrong

A recent Guardian article explained why many of the data that are reported about the spread of Coronavirus are bound to be wrong.

As discussed in previous posts – here, for example – a difficulty when studying data from the Coronavirus pandemic is the reliability and completeness of the data. This is especially true when looking at the number of confirmed cases, since this measure depends very much on the protocol used for testing. However, it’s also true for the number of fatalities. The following tweet leads to a thread by James Tozer of the Economist who has collated evidence from journalists across Europe that suggests the number of deaths due to COVID-19, at least indirectly, might be around double the officially reported number.

The thread is also available in complete form here.

The analysis is pretty much summarised in the following graphic:

The diagrams take different forms, but in each case there’s a black or grey level that corresponds to the number of deaths that would be expected in that period based on previous year’s data. The red level then shows the number of reported deaths due to COVID-19 in that period this year. And the pink region shows the total number of deaths due to any cause, again in the same period this year.

And it’s similar in the United States. The following graph – provided on Twitter by @Tangotiger – shows the excess number of deaths per month in New York, compared to the long-term average, over a period from 2000 onwards.

There is a large spike in September 2001 caused by the 9/11 tragedy. But there is a much larger spike for March/April 2020. But only around 60% of that excess is due to officially recorded COVID-19 related causes. So what explains the other 40%. It’s too large to be explained by random variation – compare its size to the variations that you see in other months over the same period – so it must be due to some specific effect in 2020, for which the only plausible explanation is COVID-19. That’s not to say that all of these deaths are directly attributable to the Coronavirus, though almost certainly many are from people who were positive but not tested. Others, though, are likely to be due to people dying from illnesses that in normal circumstances would have been treatable with medical support.

So, as with all data that are generated by this pandemic, what we think we know about fatality counts is almost certainly wrong. A reasonable run-of-thumb is to take the officially published numbers and double them.


Seasonal effects

There’s been plenty of speculation (here, here, here,…) that the novel Coronavirus might be seasonal, meaning that transmission rates will reduce significantly in the warmer summer months in temperate countries. This would help significantly in controlling the current epidemic wave, potentially buying considerable time in allowing vaccine development or other exit strategies from current lockdown conditions.  But so far there’s been little direct evidence that the Coronavirus is genuinely seasonal.

However, the following tweet links to a statistical analysis which, though circumstantial, provides reason to believe in a seasonal effect. The author of the study looked per-capita death rates due to COVID-19 in individual counties of the United States. They then fitted a regression model using demographic and climate-based statistics as potential explanatory variables for differences in county-to-county rates. What emerged is that temperature is the most significant factor. That’s to say, after allowance for other explanatory factors, the one that had the most impact was temperature: in counties with higher average temperature, everything else being equal, the per-capita death rated to COVID-19 was lower.

Of course, there are all sorts of caveats – see discussion here –  about extrapolating from the conclusions of the type here to assuming seasonality in the worldwide transmission behaviour of the virus. But it is, at least, another reason to be cheerful optimistic.

Risky talk

In a previous post I referenced a book by the eminent statistician David Spiegelhalter. Since earlier this year, Davis has also been producing a podcast ‘Risky Talk‘ on the relevance of Statistics for various issues of public interest. The latest of these is titled ‘Coronavirus: Understanding the Numbers’ and is full of useful information and discussion. It includes, among other things, a discussion of:

  1. Which data are most reliable for understanding the epidemic;
  2. How the different approaches to the epidemic adopted in Norway and Sweden provide a live experiment for assessing the impact of social controls;
  3. A comparison of the seriousness of COVID-19 relative to other flu-like illnesses in the UK;
  4. A discussion of the personal risk we all carry of dying from COVID-19 and other causes.

It’s a great listen and there’s probably nobody qualified to be explaining these issues


The graph above is the latest (as of 5th April) update from the FT showing a 7-day rolling average of the number of new COVID-19 confirmed cases through time for a number of countries. The point of using a 7-day rolling average – which means each value is the average of the previous 7-days’ values – is to reduce the effect of randomness in day-to-day variations, so as to get a smoother picture of trends. As discussed in previous posts, it’s possibly misleading to use confirmed cases as a strict measure of the epidemic scale, since the number of confirmed cases will depend in part on the protocol for testing, which varies from country to country, and even within each country through time. Nonetheless, it’s likely to be broadly interpretable as an indicator of epidemic strength.

Notwithstanding this issue, if the epidemic were growing exponentially in any country, the graph would show as a straight line on this logarithmic scale. To a greater or lesser extent, the curves for almost all countries show a tendency to flatten through time, especially from the time that social measures have been applied to limit potential transmissions through contact. The curve for the UK remains stubbornly close to linear, but its lockdown was introduced later – in relative terms – than for most other European countries. The curve for Italy seems to have flattened quicker than for other countries – again relative to when the country was placed on lockdown – but that’s probably because severe local restrictions were placed on the worst-affected regions some time before the entire country was placed on lockdown.

But anyway…. the point I wanted to make in this post is a little different. There are several reasons why it’s a good idea to use a logarithmic scale in graphs like the one above. Mostly this is because there are good epidemiological reasons to believe – as discussed here –  that an unchecked epidemic will grow exponentially. And exponential growth on a logarithmic scale will appear linear, which makes comparisons and contrasts much easier. But one disadvantage of the logarithmic scale in this context is that it can give a false impression as to the degree of similarity between countries. Looking at the above graph, it’s true that the trajectory for the United States looks currently worse than that for other countries, but not so much worse. But now look at the same graph, from a day or two earlier, on a linear, instead of logarithmic, scale:

On this scale the difference in trajectory for the United States relative to each of the other countries is much more apparent. The current level is very much greater, while the tendency for growth is also considerably more dramatic.

In summary, different scales for graphs are useful for different purposes. And though the logarithmic scale is better than a linear scale for most purposes in tracking an epidemic, it’s only once you put things back on a linear scale that you get a true sense of how different the epidemic currently is on the ground in different countries.


A changing world

In an earlier post, I discussed the ‘stringency index’, which has recently been developed as a way of measuring how severe – stringent – a country’s response has been to the Coronavirus epidemic.

The Financial Times, as part of its live coronavirus coverage, has now produced the following animated world map of the stringency index from the start of the year up to 24 March:

It’s striking how most of the world outside of China stays blue for most of February – arguably time thrown away – and how rapidly most of the world turns red and purple from the middle of March.

As an aside, the tweet below contains a great video where John Burn-Murdoch of the FT explains several of the decisions made by his team in the way they have chosen to present graphs showing the scale of the epidemic across countries: