Developing a System for COVID-19 PCR Tests: "The Next Challenge is Improving Accuracy"

Can a negative test for COVID-19 be trusted? In a survey on the accuracy control of COVID-19 PCR tests, that was conducted by Professor Hayato Miyachi, Vice President of Nitobe Bunka College (Chairperson of the Technical Committee on Standardization for Gene-based Testing at the Japanese Committee for Clinical Laboratory Standards) (see picture) and others as a part of a project commissioned by Japan's Ministry of Health, Labour and Welfare (MHLW), a questionnaire survey and an on-site survey using samples were conducted at 1,191 facilities that use PCR or LAMP tests to determine whether a sample is positive or negative for COVID-19. The results found that some of the clinics issuing negative test certificates were not conducting performance evaluations or internal accuracy controls when the tests were introduced, and that they were making mistaken judgements even on control samples. According to Professor Miyachi, "The government's efforts to date have developed a testing system in terms of numbers, but the accuracy controls are not sufficient. Information disclosures are required so that users can make the right choices."

The survey investigated 1,191 facilities, including 928 medical institutions (883 hospitals and 45 clinics), sanitary laboratories (of which 25 were temporary), and 80 administrative institutions (24 regional health research institutes, 37 public health centers, and 19 quarantine stations). Among them, a high percentage of clinics (93.3%, 42 facilities) and temporary sanitary laboratories (80%, 20 facilities) were conducting testing at their own expense. 88.9% of the clinics issued negative test certificates.

In testing, the direct method, column method, dedicated systems, simple extraction, and magnetic beads were used for nucleic acid extraction. A variety of methods were used for amplification and detection, including the infectious laboratory method, direct PCR method, LAMP method, and fully automated measurements. There was also a large difference in detection sensitivity, from less than 10 copies to about 500 copies.

18.3% of facilities conducted basic performance evaluations at their own facilities when introducing the tests, and more than 80% of facilities used the manufacturer's nominal values. In addition, for the 52.3% of facilities that conducted validity confirmations/verifications, there was only a 20-30% implementation rate for accuracy (reproducibility), detection sensitivity, and specificity. Miyachi said, "Although internal quality controls are mandatory, the interviews revealed that many facilities do not implement internal quality controls because the control samples are not bundled with the reagents and are sold separately." There are a number of methods to ensure the accuracy of tests, but 39.2% of facilities referred to the MHLW's manual on accuracy control, 13.8% of facilities had specialized qualifications for genetic testing, and 68.6% of facilities were creating their own standard operating manuals. On the other hand, 16.4% of facilities did not conduct internal accuracy controls using control or patient specimens, and these were primarily clinics that did self-funded tests or that provided negative test certificates.

In a sample-based external accuracy control survey, 21 groups were analyzed, with 10 or more facilities that used the same instrument/sample combination being counted as one group. The correct response rate was good, ranging from 93% to 99.4%, but there were mistaken judgements (false positives/false negatives) at 98 facilities. Many of them were hospitals and clinics.

In particular, mistaken judgements (false negatives) in positive samples for nucleic acid extraction/amplification detection evaluation processes tended to depend on the measurement system, and were the most frequent for SmartGene (19 facilities), TRCReady (26 facilities), and μTASWako g1 (19 facilities). Looking at the 34 facilities that had mismatches, the ratio of clinics was high, at 17.6%, and a high percentage were facilities with inadequate accuracy controls, such as not having prepared standard operating manuals for accuracy. In one facility, a sample mix-up occurred.

Also observed in this survey was the use of guanidine hydrochloride solution as a medium when transporting samples. Guanidine hydrochloride solutions inactivate the virus and stabilize the RNA, but can lead to false negatives if not cleanly removed before testing. Miyachi pointed out that, "Safety is also an important factor in transport to reduce the risk of infection and to provide a wide range of testing services. However, guanidine hydrochloride cannot be removed via a simple extraction, instead requiring a suitable extraction system. Medical institutions and testing centers may be okay, but the test kits used by consumers may give quite a few false-negative results."

In addition, Professor Miyachi said, "In the MHLW-published 'List of Testing Institutions that Offer Private Testing,' there needs to be items such as whether guanidine hydrochloride solutions are used, what extraction method is used, and whether they have been evaluated by a third party. This kind of information is known only to experts, but, if there is public data, then knowledgeable individuals will be able to introduce reliable testing institutions."