Since March 2020, the Food and Drug Administration (FDA) has granted Emergency Use Authorizations (EUAs) for dozens of Real-Time Polymerase Chain Reaction (RT-PCR) diagnostic tests to identify patients infected with SARS-CoV-2, the novel SARS coronavirus strain of 2019 (COVID-19). However, not all RT-PCR tests are the same and can differ by the primer technology, polymerase enzyme, fluorescent probes and genetic targets. Because the FDA relies on self-reported data from the companies to make its EUA decision, many of these tests do not perform as reported. (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7350782/, https://www.nejm.org/doi/full/10.1056/NEJMp2015897) Clinical laboratories with COVID-19 testing programs are consequently in a bind, either faced with the decision to shut down the already overwhelmed testing pipeline to validate these tests, or to adopt the technology blindly, risking delivering false results to patients at a critical time.
TNG Dx, LLC has identified the Logix SmartTM COVID-19 RT-PCR test by Co-Diagnostics, Inc. as one of the most promising tests to address the national and global testing needs. Co-Diagnostics’ peer-reviewed CoPrimerTM technology has demonstrated applicability in the diagnosis of Zika, dengue, and chikungunya viruses, mycobacterium tuberculosis, and is now used to detect COVID-19. We ran tests on 93 replicates of COVID-19 and 93 replicates of Influenza B purified viral RNA and the Logix Smart COVID-19 RT-PCR test effectively achieved 100% sensitivity and selectivity metrics with only one genetic target and detection levels at the Poisson limit. The Logix Smart tests were independently validated to ensure they perform as expected. We also sought to determine the Level of Detection (LOD) of the tests and got a strong signal at as low as 0.3 viral copies per microliter (ul), well below the level reported by the company.
It is our opinion that the Logix Smart COVID-19 RT-PCR test performs as indicated, and not only should clinical testing programs feel confident in its immediate adoption, but also that the test could enable alterations in testing pipelines that fundamentally improve throughput and cost. Co-Diagnostics also practices compassionate pricing and sells their kits for a third to a quarter of the price of leading commercial brands in certain regions, thus making asymptomatic RT-PCR testing fiscally possible for public health organizations.
RNA extraction of inactivated COVID-19 at 1.7 billion copies per milliliter (mL) and Influenza B at 1.6 billion copies per mL (Bei Resources) was performed with the QIAmp Viral RNA mini kits (Qiagen) according to its protocol. RNA was then serially diluted to achieve a final concentration of 290 copies/ul for COVID-19 and 280 copies/ul for Influenza B.
46 5ul replicates of the RNA from COVID-19 were set up with 5ul of the Logix Smart Master Mix (Co-Diagnostics). One replicate of the provided positive control and one replicate with nuclease free water (negative control) were run in tandem on a CoDx Box Thermocycler (Bio Molecular Systems). Samples were held at 45°C for 15 minutes, 95°C for 2 minutes, and then cycled 50 times through a 3 second heat shock at 95°C (melting), and a 55°C for 32 seconds (annealing/extension) followed by recording in the green/orange channel (COVID target RdRp using a FAMTM probe, and the internal control RNaseP using a CAL Fluor Red® probe). Reported signal for RdRp was normalized by the internal RNaseP signal. Procedure was then repeated for 46 replicates of Influenza B in the same manner.
RT-PCR results were then recapitulated at Arches Research Laboratory.
Exhibit 1: Change in fluorescence during RT-PCR for 46 replicates of COVID-19 viral RNA
47 replicates of COVID-19 purified RNA and 47 replicates of Influenza B purified RNA were tested with the LogixSmart test on a QuantStudio Q3 qPCR machine (Applied Biosystems) along with positive and negative controls. Same qPCR parameters and normalization was followed. Samples were blinded and randomly distributed throughout plate.
LODs were calculated based on the viral concentration pre-extraction. Sample was diluted serially to obtain concentration of 290 copies/ul. 2-fold dilutions were then performed until the concentration reached 1.2 copies/ul. 4 replicates of each dilution, the positive control, and the nuclease-free water were then processed according to the methods described above.
LOD results were cross validated at Arches Research Laboratory. 8 replicates of 2-fold dilutions to a final concentration of 0.3 viral genome copies per ul were blinded and tested on a Quantum Studio 12K qPCR machine (Applied Biosystems). Samples were run under the same parameters as described above.
Determination of Assay Sensitivity
Ninety-three (93) replicates of COVID-19 purified viral RNA at 290 copies/ul were assayed in parallel with a positive and negative control. Every replicate passed the threshold between 27 and 31 cycles. The mean quantitation cycle (Cq, the point in which the target signal passes the threshold for statistical significance) was calculated to be 27.69 with a standard deviation of 0.59. The calculated efficiencies for each replicate fell between 72% and 81%, achieving a calculated mean of 77% with a standard deviation of 2%. This value is within 1 standard deviation of the mean efficiency for all positive controls run in this investigation (81% with a standard deviation of 9%). Exhibit 1 depicts the change in fluorescent signal through the PCR cycles. Because all samples yielded a positive signal, we can calculate the sensitivity to be 100%.
Exhibit 2: Change in fluorescence during RT-PCR for 46 replicates of Influenza B viral RNA
Determination of Assay Selectivity
Ninety-three (93) replicates of purified RNA from Influenza B at 280 copies/ul were run under the same conditions as the purified COVID-19 RNA, also with a positive control and nuclease-free water. The positive control performed as expected, with a Cq of 25.48 and efficiency of 100%. Ninety-two (92) of the Influenza B replicates showed no signal at all. One (1) replicate had a quantitation cycle reading of 47.14 and efficiency of 62% (Exhibit 2). Because the Cq of this sample fell outside of 40 cycles, the CDC recommended cutoff cycle, and its efficiency score fell outside of 1 standard deviation of the positive control efficiencies, we determined this sample to be a negative result despite crossing the threshold. Accordingly, because all replicates of Influenza B yielded negative results, we determined the selectivity of the test to be 100%.
Determination of the Level of Detection
Twelve (12) dilutions of purified COVID-19 RNA were prepared according to the concentrations presented in Exhibit 3. Concentrations were calculated based on the pre-extraction concentration. It can be assumed that the actual concentration is slightly lower than reported here, as RNA extraction is approximately 90% efficient. Twelve (12) replicates for each dilution, the positive control, and nuclease-free water were tested according to the established RT-PCR protocol. Means and standard deviations for the Cq and efficiencies were calculated from the replicates (Exhibit 3). Exhibit 4
depicts the change in fluorescence over the course of the PCR cycles. Replicates of all samples yielded positive test results from 290 copies/ul to 0.6 copies/ul. When testing 0.3 copies/ul, 5 out of 8, or 63% of the replicates tested positive. 5 ul of template is used in each RT-PCR reaction, translating to approximately 1.2 viral genome copies at this concentration. This places the Logix Smart LOD at the Poisson Limit for the sample size.
If the Cq values for these dilutions are accurate, they should have an inverse logarithmic relationship with concentration of each sample. Exhibit 5 graphs the Cq versus concentration, where concentration (x-axis) is shown on a logarithmic scale. The data fit a logistic regression with the equation y=-1.374ln(x)+35.587 and an R2 of 0.9969, indicating a strong trend. Mean efficiencies for all dilutions of COVID-19 RNA were then graphed with a “box and whiskers” plot along with the positive control data for the entire investigation (Exhibit 6).
Light colored lines represent the median value, boxes indicate the 25th and 75th percentile, and whiskers indicate the high and low values in the data set. Exhibit 5 demonstrates that all efficiencies for the COVID-19 dilutions fall within the 25th and 75th percentile of the positive controls. It can be concluded that the assay is performing robustly at both these extremely high and extremely low concentrations, with a LOD of 13 times lower than what Co-Diagnostics reports in their EUA and Logix Smart kit informational pamphlet.
Exhibit 3: Quantitation cycles and efficiencies for COVID-19 virus for a range of concentrations
Exhibit 4: Change in fluorescence during RT-PCR for a range of COVID-19 viral concentrations
Exhibit 5: Logarithmic relationship between quantitation cycles and viral concentration
Exhibit 6: Comparison of efficiencies for positive control replicates and samples varying in viral concentration
Exhibit 7: Results Summary Matrix (Sensitivity versus Selectivity)
Conclusion and Discussion
From this validation study we conclude that the Logix Smart COVID-19 RT-PCR test by Co-Diagnostics, Inc. not only performs as reported, but its performance exceeds the claims made by the company. We found the test to be 100% sensitive when performed on 93 COVID-19 RNA replicates in total, whereas the Co-Diagnostics reports a sensitivity of 99.52% on 631 replicates. Of the 93 replicates of Influenza B RNA, 92 showed no signal whatsoever. One sample crossed the established threshold but did not meet the Cq or efficiency criteria to be considered a positive result. This signal was likely a primer dimer, which are much rarer when using CoPrimers instead of traditional primers, but not entirely absent. Thus, our results confirm 100% selectivity for the Logix Smart COVID-19 RT-PCR test. Exhibit 7 summarizes the final numbers and how these calculations were performed.
From the LOD investigation, we concluded that the Logix Smart COVID-19 RT-PCR test is performing better than previously reported, achieving robust results at 0.3 copies/ul, compared to the 4.29 copies/ul reported
by Co-Diagnostics, Inc. Because concentrations were calculated pre-extraction, it can be assumed that the LOD of the test is actually lower than reported here. Efficiencies for all dilutions were well within one standard deviation of the median for the positive controls. We were also able to explore other parameters of the assay in our analysis, including the accuracy in which Cq values indicate the viral concentration.
The Logix Smart COVID-19 test is one the of highest performing commercially available RT-PCR diagnostic tests in the American market. Its performance makes it an ideal test to develop cost and time saving protocols for COVID-19 testing. Co-Diagnostics’ proprietary CoPrimer technology seems particularly amenable to rapid RNA extraction, sample pooling, self sample collection, and low cost to name a few.
Because the test performs so robustly with only one target, single patient testing does not require multiple tests, and the test is amenable to multiplexing (simultaneous diagnoses of multiple respiratory infections, such as Influenza). Some testing facilities, such as Clinical Reference Laboratories (CRL) and Access Genetics, have already capitalized on saliva sample collection and offer testing services under an EUA from the FDA. Development of patient-friendly, high-throughput/high-content testing like this is badly needed to maintain public health and safety while businesses, government services, religious institutions, and schools reopen. That being said, from our investigation, we have concluded that the Logix Smart COVID-19 test from Co-Diagnostics, Inc. performs as
well as the company claims, if not better. Clinical laboratories can feel confident incorporating it into existing testing pipelines and protocols efficiently and seamlessly to meet their needs.
Funding and Declaration of Competing Interests
This study was funded by the Utah Journalists Foundation (UJF). Materials were provided gratis by CoDiagnostics, Inc. Dr. Meghan Aileen Lockard is an independent advisor contracted for this study by TNG Dx, LLC, and has no competing financial interests to declare.