Roche, Siemens Healthineers Score Big on FDA SARS-CoV-2 Antibodies Test Reliability Evaluation

While real-time reverse transcription polymerase chain reaction (RT PCR) assays are the primary diagnostic for COVID-19, the U.S. Food and Drug Administration (FDA) has also granted Emergency Use Authorization (EUA) to dozens of antibody tests since the pandemic began. And that begs an important question: Which of these tests is most reliable? The FDA recently published comparison data purporting to answer that question.

The Role of Antibody Testing

Serology-based tests detect antibodies the body produces to fight SARS-CoV-2 rather than the virus itself. The problem is that it takes several days for people who are infected to develop antibodies to the virus. This limits their reliability in diagnosing COVID-19. But while they should not be used as the sole basis for diagnosis, antibody tests help healthcare professionals identify individuals who may have developed an immune response to SARS-CoV-2 and determine who can donate convalescent plasma to treat patients with infections.

The FDA Performance Evaluation Methodology

As with any other assay, some SARS-CoV-2 antibodies tests are more reliable than others. The FDA set out to evaluate performance of 64 EUA tests on the basis of data provided by the respective test manufacturers and using two key metrics:

• “Sensitivity” measures the ability of a test to identify persons with antibodies to SARS-CoV-2 (true positive rate); and
• “Specificity” measures the test’s ability to identify persons without SARS-CoV-2 antibodies (true negative rate).

To measure sensitivity, the FDA evaluated whether a test was able to detect antibodies in blood samples from patients confirmed as having COVID-19 via nucleic acid amplification test (NAAT). A test’s specificity can be estimated by testing large numbers of samples collected and frozen before SARS-CoV-2 is known to have circulated to demonstrate that the test does not produce positive results in response to the presence of other causes of a respiratory infection, such as other coronaviruses.

Because every test returns some false positives and negatives, tests are also measured by their Positive and Negative Predictive values (PPV and NPV), which are calculated using the test’s sensitivity and specificity based on “prevalence,” i.e., an assumption about the percentage of individuals in the population who have SARS-CoV-2 antibodies.

The Evaluation Results

Companies from central Europe garnered the bragging rights. One of the highest scoring tests was the Elecsys Anti-SARS-CoV-2 tests from Switzerland-based Roche Diagnostics. Applied to over 6,000 samples, including 233 positives, the semi-quantitative test correctly determined if SARS-CoV-2 antibodies were present in almost every case. When applied to a population in which 5 percent of samples had antibodies, the test had PPV of 99.7 percent and NPV of 99.8 percent. However, the qualitative version of the Elecsys test, while still among the top performers, was slightly less impressive.

Tests from Germany-based Siemens Healthineers also fared quite well, including a semi-quantitative test with a PPV of 98 percent and NPV of 100 percent at an antibody prevalence of 5 percent based on more than 1,000 samples.

Although several other tests had higher PPV and NPV, their scores are less impressive because they were based on smaller sample sizes. Thus, for example, an assay from Hangzhou Biotest Biotech posted 100 percent for both PPV and NPV but only tested 110 samples. Based on the 95 percent confidence interval, the real PPV for the test could be as low as 51 percent. By contrast, the lowest confidence interval the Roche and Siemens tests could dip would be PPV of, respectively, 93 percent and 88 percent.

Like Roche and Siemens, Abbott assessed most of its antibody tests on at least 1,000 samples. The PPVs of the three most comprehensively assessed Abbott tests were approximately 92 percent, with the lower and upper bounds of the confidence intervals of approximately 85 percent and 95 percent. While below the Roche, Siemens and other top tier tests, these performance numbers are comparable to kits from Beckman Coulter and other big diagnostics firms.

Siemens Healthineers SARS-CoV-2 Test Wins Bouquets on the Other Side of the Atlantic

The new FDA comparison is the latest in a series of wins for the Siemens Healthineers. In July, the UK Department of Health and Social Care published a study evaluating the sensitivity and specificity of four coronavirus antibodies assays available in the UK market: Abbott Laboratories’ SARS-CoV-2 Immunoassay, DiaSorin’s Liaison SARS-CoV-2 S1/S2 IgG assay, Roche’s Elecsys Anti-SARS-CoV-2 test, and Siemens’ SARS-CoV-2 Total (COV2T) test. The study found that the Siemens’ test was the only one to meet its 98 percent sensitivity and specificity standard. (See “Battle of the SARS-CoV-2 Antibody Immunoassays: Only One Test Meets Both UK Sensitivity and Specificity Targets,” see Diagnostic Tests and Emerging Technology (DTET), Aug. 12, 2020.

On the downside, Thermo Fisher Scientific’s OmniPATH COVID-19 Total Antibody came in with a disappointing PPV of 67 percent, far below those of its peers; but the smallness of the sample size—only 110 samples—creates a wide confidence interval that may belie the reliability of these results and make comparisons with tests evaluated on the basis of larger samples unreliable.

Takeaway

Available data suggests that almost all of the SARS-CoV-2 antibody tests that have received EUA from the FDA are at least 90 percent sensitive and specific. However, as the FDA evaluation illustrates, small differences in those figures can translate into big differences in PPV and NPV.