Harnessing Event Report Data to Identify Diagnostic Error During the COVID-19 Pandemic

Abstract

Introduction: COVID-19 exposed systemic gaps with increased potential for diagnostic error. This project implemented a new approach leveraging electronic safety reporting to identify and categorize diagnostic errors during the pandemic.

Methods: All safety event reports from March 1, 2020, to February 28, 2021, at an academic medical center were evaluated using two complementary pathways (Pathway 1: all reports with explicit mention of COVID-19; Pathway 2: all reports without explicit mention of COVID-19 where natural language processing [NLP] plus logic-based stratification was applied to identify potential cases). Cases were evaluated by manual review to identify diagnostic error/delay and categorize error type using a recently proposed classification framework of eight categories of pandemic-related diagnostic errors.

Results: A total of 14,230 reports were included, with 95 (0.7%) identified as cases of diagnostic error/delay. Pathway 1 (n = 1,780 eligible reports) yielded 45 reports with diagnostic error/delay (positive predictive value [PPV] = 2.5%), of which 35.6% (16/45) were attributed to pandemic-related strain. In Pathway 2, the NLP-based algorithm flagged 110 safety reports for manual review from 12,450 eligible reports. Of these, 50 reports had diagnostic error/delay (PPV = 45.5%); 94.0% (47/50) were related to strain. Errors from all eight categories of the taxonomy were found on analysis.

Conclusion: An event reporting-based strategy including use of simple-NLP-identified COVID-19-related diagnostic errors/delays uncovered several safety concerns related to COVID-19. An NLP-based approach can complement traditional reporting and be used as a just-in-time monitoring system to enable early detection of emerging risks from large volumes of safety reports.

Copyright © 2021 The Joint Commission. All rights reserved.

Figures

Figure 1

This flowchart shows that a total of 14,230 safety reports were filed between March 1, 2020, and February 28, 2021. These were processed through two pathways. Pathway 1 (1,780 reports) contained all reports with explicit mention of COVID-19, whereas Pathway 2 (12,450 reports) used automated natural language processing to highlight specific cases for manual review. Manual review was performed for 1,780 reports in Pathway 1 and 110 safety reports in Pathway 2. A total of 95 cases of diagnostic error or delay were identified.

Figure 2

Panel A: total hospital census (ambulatory and inpatient); Panel B: safety report volume; Panel C: volume for Pathway 1 (COVID-19–tagged safety reports), Pathway 2 (natural language processing–based reports), and diagnostic errors or delays found using each pathway. Safety reporting volume roughly mirrored total hospital volume. Pathway 1 volume peaked in April 2020 and declined by midsummer. Pathway 2–identified cases increased in the summer and remained steady thereafter. Important hospital policy events during this period include deferment of nonurgent care (March 13, 2021), resumption of nonurgent care (July 20, 2021), and second deferment of nonurgent care (December 26, 2020).

Figure 3

This chart illustrates that COVID-19–tagged safety reports (Pathway 1) had all types of errors represented, as compared to natural language processing–based reports (Pathway 2). Pathway 2 was most sensitive for detecting strain-type diagnostic errors.