Number 10 placed an order for one million AbC-19 tests made by UK-based company Abingdon Health in October to be used for research purposes, in order to paint a better picture of how far the virus has spread across the country.
But its self-funded analysis of the pregnancy-stick type tests, published in the British Medical Journal, has suggested they are much less accurate than originally thought.
The study, which was conducted by Public Health England scientists and included almost 5,000 people, showed it was actually only 81.7 per cent accurate at spotting people who should test positive.
Controversially, officials did not publish data about how accurate the tests were before the Department of Health announced it had bought them.
Researchers estimated that if the test were used on one million people, of whom 100,000 had actually had coronavirus, 84,700 results would be correct positives, 15,300 people would get false negatives, and 18,900 people would get false positives.
False negatives mean people are told they have never had the disease when they actually have – of which there are not many negative consequences – while false positives tell people they have had the illness but they actually haven’t, which can lead people to wrongly believe they are immune and take more risky actions.
The results contrast with an earlier, but not yet peer-reviewed study, that claimed the test gives no false positive results. The study, published by the University of Ulster, has since been labelled as ‘critically flawed’.
The figures may be even worse if the antibody tests are self-administered at home because the general public are more likely to make mistakes, such as inaccurately placing a drop of blood into the test cartridge.
Researchers today claimed the Government had ‘jumped the gun’ on purchasing the tests in an undisclosed deal before thorough analysis was published.
The test gives results in 20 minutes and doesn’t need to be read by a lab technician. They have the potential for home use, but regulations say they should only be administered by health professionals.
The UK Government placed an order for one million for AbC-19 tests made by Abingdon Health in October for research purposes, helping to build a picture of how the virus has spread across the country
The AbC-19TM Rapid Test uses a drop of blood from a finger-prick to see if someone has antibodies against SARS-CoV-2 — the virus that causes Covid-19.
Antibodies are part of the immune system. If they are present in the blood, it suggests the person has already been infected with the coronavirus.
The device is known as a lateral flow immunoassay, which officials hoped would play a crucial role in understanding exactly how many people in Britain had been infected since the pandemic began.
WHAT IS AN ANTIBODY TEST AND WHAT IS IT USED FOR?
Antibody tests are ones which look for signs of past infection in someone’s blood.
Antibodies are substances produced by the immune system which store memories of how to fight off a specific virus. They can only be created if the body is exposed to the virus by getting infected for real, or through a vaccine or other type of specialist immune therapy.
Generally speaking, antibodies produce immunity to a virus because they are redeployed if it enters the body for a second time, defeating the bug faster than it can take hold and cause an illness.
An antibody test, which involves analysis of someone’s blood sample, has two purposes: to reveal whether an individual has been infected in the past and may therefore be protected against the virus, and to count those people.
Knowing you are immune to a virus – although whether people actually develop immunity to Covid-19 is still unknown – can affect how you act in the future. Someone may need to protect themselves less if they know they have been infected, for example, or medical staff may be able to return to work in the knowledge they are not at risk.
Counting the numbers of people who have antibodies is currently the most accurate way of calculating how many people in a population have had the virus already.
Officials tested the accuracy of the AbC-19 test on thousands of key workers, who have been given antibody testing since the summer.
It was commissioned by the Department of Health and Social Care and also involved scientists at the Universities of Bristol, Cambridge and Warwick.
First, the researchers looked at 1,995 blood samples that were taken in 1995 — well before the Covid-19 pandemic. These were used as ‘known negatives’ because they did not have antibodies against the coronavirus.
In the pre-pandemic samples, the antibody test’s specificity — the ability to correctly identify a true negative — was 97.9 per cent. This means 2.1 per cent of results were incorrectly reported as positive.
In the second arm of the study, scientists tested blood samples from 2,847 key workers in England in June. This included NHS workers, fire, and police staff.
Of these, 268 had already been told they were positive for the disease through a PCR swab — the gold-standard of testing. They were categorised as ‘known positives’.
Scientists compared the AbC-19 test with two more robust and accurate laboratory antibody tests.
They estimated the true sensitivity of the AbC-19 test — the ability to correctly identify a true positive sample — was 92.5 per cent.
It means 7.3 per cent of those people were wrongly told they did not have antibodies — known as a ‘false negative’. This is not as serious as a ‘false positive’.
The remaining 2,579 key workers were categorised as those who had not had a Covid-19 test before, who did not know if they had previously been infected.
This represents a real life situation, in which antibody testing would be conducted on people who have no idea if they have had Covid-19 or not.
Among this group, the test sensitivity was 84.7 per cent, based on comparison with other tests.
It means that 15.3 per cent of people with a previous SARS-CoV-2 infection would be missed by the test.
The specificity in the remaining key workers was around 98.9 per cent, meaning 1.1 per cent were given a false positive.
Sir John Bell, regius professor of medicine at Oxford University, who is the government’s main adviser on tests, has previously called the AbC-19 test ‘truly amazing’
TRUE PREDICTIVE VALUE: HOW EVEN A GOOD TEST CAN LEAD TO WRONG RESULTS 50% OF THE TIME
The US’s Centers for Disease Control & Prevention warned in May that coronavirus antibody tests may be wrong 50 per cent of the time even if they are high quality.
It warned that the antibody testing is not accurate enough for it to be used for any policy-making decisions, as even with 95 per cent test specificity, ‘less than half of those testing positive will truly have antibodies’.
This is why:
Antibody tests with what could be considered a high level of accuracy can still produce large margins of error if only a small proportion of a population has been infected.
A 95 per cent specific test, for example, will always produce five false positive results from a group of 100 people.
Even if it is sensitive enough to detect all the people who have genuinely had the disease, it will still return five false positives, and the effect this has on the results of a survey can be large if the number of true positives is low.
If the prevalence of antibodies is low – for example, only five per cent of people in the group have had the illness – the results could end up half wrong. The 95 per cent test, in that situation, would be expected to return 10 positives – five of them right, five of them wrong.
This means the functional accuracy of the test, known as its true predictive value, is only around 50 per cent.
The effect of these false positives is magnified if the prevalence of the virus in the population is low, and less noticeable if the prevalence is high.
For example, if 30 per cent of the population have been infected, those five false positive results would be counter-balanced by 30 true positives, making the test more like 85 per cent accurate.
A more specific test can reduce this effect; by comparison a 99.9 per cent specific test would return one wrong result per thousand – 100 per million.
Experts conceded that the test is probably more sensitive in those with a confirmed diagnosis because their antibody levels were higher. This is because their infection was more severe – assumed because they sought having a test.
The team said the lower sensitivity figure of 84.7 per cent is probably a more realistic estimate of test sensitivity in the real world, if people were to choose to take the test to find out their own previous infection status.
And those who already had a PCR diagnosis are not as likely to seek an antibody test because they already have confirmation they’ve had the coronavirus.
The team calculated how this would have an impact in the real world, working out what’s known as the ‘true positivity value’ – how accurate it would be when conducted on a population.
The team estimated how many false positives would occur in a situation where five per cent, 10 per cent, 15 per cent, 20 per cent or 25 per cent of the population had already had the disease.
If 10 per cent had been previously infected with Covid, it would mean around one in five positive results were wrong — a positive predictive value of 81.7 per cent.
If one million people were tested with the AbC-19 kit, there would be 18,900 false positive results, the team estimated.
If five per cent of the population had previously had the disease, then around one in three would get a false positive — a positive predictive value of 67.9 per cent.
As the proportion of people who have really had the disease gets smaller, the margin of error gets bigger and the results become more and more inaccurate – because the proportion of results that are wrong will never change, regardless of the proportions of people getting accurate positive results.
Previous research reported 97.7 per cent sensitivity and 100 per cent specificity of the AbC-19 Rapid Test, meaning no one would be given a false positive.
It was conducted by Ulster University at the laboratories of Abingdon Health – who said it ‘would be inaccurate to suggest our numbers are lower than we’ve stated’.
Experts pointed out the study excluded samples for which it was considered ‘unclear’ if a person had antibodies, which statisticians have warned will lead to over-estimates of accuracy.
No10 has already got one million AbC-19 tests under its belt for research purposes on behalf of the UK-Rapid Test Consortium (UK-RTC).
The CEO of Abingdon, Chris Yates claimed 10million tests could be manufactured within a six-month period.
Sir John Bell, a government testing adviser, has previously called it ‘truly amazing’.
But Dipender Gill, a clinical scientist at Imperial College London, and Dr Mark Ponsford, an immunologist at Cardiff University, warned that positive results from these tests does not necessarily mean someone is immune to catching Covid-19.
The team calculated how this would have an impact in the real world, working out what’s known as the ‘true positivity value’ – how accurate it would be when conducted on a population. This graphic shows how it can vary with scale
HOW MANY ‘FALSE POSITIVES’ ARE THERE IN THE TEST?
IF 10% OF THE POPULATION IS INFECTED
True predictive value 81.7%
One in five people get a false positive
IF 5% OF THE POPULATION IS INFECTED
True predictive value 69.8%
One in three people get a false positive
IF 15% OF THE POPULATION IS INFECTED
True predictive value 87.6%
Over one in ten people get a false positive
IF 20% OF THE POPULATION IS INFECTED
True predictive value 91%
One in ten people get a false positive
IF 25% OF THE POPULATION IS INFECTED
True predictive value 93.1%
Under one in ten get a false positive
In a linked editorial, the research experts said the study ‘identifies notable limitations of the UK government’s antibody test of choice and provides good evidence that its specificity in a ‘real life’ setting is highly unlikely to be 100 per cent’.
They said ‘a clear message must be communicated to the public that positive results from these assays do not provide evidence of immunity.’
‘Apart from limited surveillance to estimate the proportion of a population that has been infected, widespread use of this assay in any other role could risk considerable harm,’ they concluded.
Even when the news of the deal broke on October 7, scientists had concerns about the data.
Professor Jon Deeks, a biostatistician at the University of Birmingham, said: ‘The only available study of this test was undertaken by UK RTC members, and was shown to be critically flawed several weeks ago.
‘Whilst we are waiting to see valid evidence of how well it performs, the Government has jumped the gun in purchasing the test before the independent studies evaluating it are finished or available.’
The large study published today still has some limitations.
For example, the test was evaluated in a laboratory, rather than having participants perform the test themselves. Therefore, the accuracy may have overestimated.
It is possible that other pregnancy test style devices for antibodies may also be less accurate than claimed, the researchers said.
They said their work ‘highlights the scope for overestimation of SARS-CoV-2 antibody test sensitivity in other studies in which sensitivity has been estimated only from PCR confirmed cases’.
MailOnline has contacted Abingdon Health and Ulster University for comment.
WHAT IS ABINGDON HEALTH’S ANTIBODY TEST?
Abingdon’s antibody test uses a small drop of blood from an individual by a finger-prick, and shows results within 20 minutes without the need to go to a specialised laboratory.
The test detects IgG antibodies to ‘full trimeric spike proteins’ of the SARS-CoV-2 virus, essentially meaning it can detect antibodies which interfere with the coronavirus.
The manufacturer claims the AbC-19 test has a sensitivity of 98 per cent and specificity of 99.6 per cent for Covid-19 antibodies.
It can be administered by healthcare professionals, such as doctors, nurses, pharmacists, and healthcare workers, at the point-of-care. But its not clear yet if it will be used as a home kit.
The AbC-19 test has already received a CE mark, which means it conforms to health and safety standards for goods sold in the UK and EU.
The UK government announced a deal with Abingdon Health for the supply of tests for research purposes in October on behalf of the UK-Rapid Test Consortium (UK-RTC).
Abingdon Health CEO Chris Yates said it was a ‘breakthrough’ for UK science companies.
He added: ‘We have ramped up production since the start of August and will be ready to deliver the first tests to the Government by early October.’
The contract includes a first order for one million AbC-19 tests, with the tests set to be rolled out under the government’s Covid-19 surveillance studies.
The aim of these studies is to gain a clear picture of how the virus has spread across the country, and develop understanding of how antibodies work.
It came after the test underwent an independent evaluation, commissioned by the UK government, published today.