The Base Rate Fallacy: why we should be cautious with anti-body testing results

Whether you think the UK is reopening too fast or too slowly, almost everyone agrees that antibody testing is critical to the next phase of our coronavirus existence. Serology tests could provide epidemiologists with vital data on how COVID-19 is spreading through a community, and also lead to the issuing of “immunity passports” for individuals who have beaten back the infection. “I think we’re going to see [antibody testing] explode,” commented Mitchell Grayson, chief of allergy and immunology at Nationwide Children’s Hospital and Ohio State University in Columbus. Although immunological assays appear to offer a promising path forward, does a positive test mean you should feel confident to work, shop, and socialise without getting sick or infecting others?

overlooking this fact is one of the most common decision-making errors, so much so that it has its own name – the base rate fallacy. 

The confidence that we should have in an antibody test depends on the base rate of the coronavirus, a key factor which is often ignored. The base rate (or disease prevalence) is the actual amount of COVID-19 infection in a known population. In the U.S., for example, this appears to be between five and 15%. This simple fact is essential to understanding the accuracy of serology-based testing. Even so, overlooking this fact is one of the most common decision-making errors, so much so that it has its own name – the base rate fallacy. 

In a classic and widely-referenced study, the following question was put to 60 students and staff at Harvard Medical School. “One in a thousand people have a prevalence for a particular heart disease. There is a test to detect this disease. The test is 100% accurate for people who have the disease and is 95% accurate for those who don’t (this means that 5% of people who do not have the disease will be wrongly diagnosed as having it). If a randomly selected person tests positive what is the probability that the person actually has the disease?” 

The lower prevalence there is of a trait in a studied population, the greater the chance that a test will return a false positive.

Almost half said 95%, with the average answer being 56%. On the surface, this makes sense – after all, a test accuracy above 90% is fairly high. In reality, however, the correct answer was just below 2%. The reason for this is a simple matter of statistics. The lower prevalence there is of a trait in a studied population, the greater the chance that a test will return a false positive. The inability of intelligent minds to apply simple mathematical reasoning and arrive at the correct value of 2% clearly demonstrates the aforementioned base rate fallacy.

In the context of coronavirus infection, the predictive value of a test with 90% accuracy could be as low as 32% if the true population prevalence is 5%. Put another way, there is an almost 70% probability in that instance that the immunological assay will falsely indicate a person has antibodies. Even deploying more accurate tests cannot change the statistical reality when the base rate of infection is very low. 

At this same disease prevalence, the CDC found that a test with 90% sensitivity and 95% specificity would yield a positive predictive value (PPV) of 49%. “In other words, less than half of those testing positive will truly have antibodies,” according to the agency. Additionally, a recent study published in the journal Public Health revealed that 16% of positive results would be false even when using a test with 99% sensitivity and specificity. In a notional population of 100,000 individuals, 950 people will therefore be incorrectly informed they have had the infection. 

these findings mean that we are all at risk of getting infected and spreading the virus, even if we’ve had a positive antibody test

Commenting on these results, the Infectious Disease Society of America stated that: “A positive test result is more likely a false-positive result than a true positive result.” This is particularly dangerous since it could lead to potentially susceptible hosts believing they have been infected with coronavirus, and acting as if they have immunity, when this is not the case. Put simply, these findings mean that we are all at risk of getting infected and spreading the virus, even if we’ve had a positive antibody test. 

Despite this, antibody tests remain an important tool in the fight against coronavirus and we should therefore encourage greater access to them; healthy people who have antibodies in their blood and have tested positive for the virus in the past (but are now symptom-free) can donate blood plasma, which may be used as a possible treatment for COVID-19. However, it is important to remember that a highly accurate test may not be as comforting as it first appears, and therefore the results of such assays should always be viewed with thoughtful reflection.