Prognostic Models for COVID-19 Care

Generalizable Prognostic Models for Patient-Centered Decisions in COVID-19

Approximately 20% of patients hospitalized with COVID-19 require intensive care and possibly invasive mechanical ventilation (MV). Patient preferences with COVID-19 for MV may be different, because intubation for these patients is often prolonged (for several weeks), is administered in settings characterized by social isolation and is associated with very high average mortality rates. Supporting patients facing this decision requires providing an accurate forecast of their likely outcomes based on their individual characteristics.

The investigators therefore aim to:

1. Develop 3 CPMs in each of 2 hospital systems (i.e., 6 distinct models) to predict:

   i) the need for MV in patients hospitalized with COVID-19; ii) mortality in patients receiving MV; iii) length of stay in the ICU.

2. Evaluate the geographic and temporal transportability of these models and examine updating approaches.

   1. To evaluate geographic transportability, the investigators will apply the evaluation and updating framework developed (in the parent PCORI grant) to assess CPM validity and generalizability across the different datasets.
   2. To evaluate temporal transportability, the investigators will examine both the main effect of calendar time and also examine calendar time as an effect modifier.
3. Engage stakeholders to facilitate best use of these CPMs in the care of patients with COVID-19.

Study Overview

• Status: Completed
• Conditions: Covid19

Detailed Description

There has been a proliferation of COVID-19 clinical prediction models (CPMs) reported in the literature across health systems, but the validity and potential generalizability of these models to other settings is unknown. Generally, most hospitals (and systems) do not have a sufficient number of cases (and outcomes) to develop models fit to their local population, and predictor variables are not uniformly and reliably obtained across systems. Therefore, pooling and harmonizing data resources and assessing generalizability across different sites is urgently needed to create tools that may help support decision making across settings. In addition, since best practices are rapidly evolving over time (e.g., proning, minimizing paralytics, lung-protective volumes, remdesivir, dexamethasone or other treatments), updating and recalibrating these CPMs is crucially important.

In the current PCORI Methods project, the investigators developed a CPM evaluation and updating framework including both conventional and novel performance measures. The investigators will use this framework to evaluate COVID-19 prognostic models in the largest cohort of COVID-19 patients examined to date, spanning 2 datasets from very different settings. As the COVID-19 pandemic affects different regions, with subsequent waves expected, identifying the most accurate, robust and generalizable prognostic tools is needed to guide patient-centered decision making across diverse populations and settings.

• Study Type: Observational
• Enrollment (Actual): 21

Contacts and Locations

Study Locations

• United States
  • Massachusetts
    • Boston, Massachusetts, United States, 02111
      • Tufts Medical Center
  • New York
    • Manhasset, New York, United States, 11030
      • Northwell Health (The Feinstein Institutes for Medical Research)

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (ADULT, OLDER_ADULT)
• Accepts Healthy Volunteers: Yes
• Genders Eligible for Study: All

• Sampling Method: Non-Probability Sample

Study Population

The population will include COVID-19 survivors; family members of COVID-19 patients; caregivers for COVID-19 patients; critical care physicians; palliative care physicians; hospitalists; nurses; respiratory therapists; leaders of our clinical ethics committees and pastoral care representatives.

Description

Inclusion Criteria:

• COVID-19 patient survivor
• Family member/caregiver of patient hospitalized for COVID-19
• Physician with experience caring for COVID-19 patients
• Other provider (pastoral care, nursing, respiratory therapy) with experience caring for COVID-19 patients

Exclusion Criteria:

• Not proficient in reading or speaking English

Study Plan

How is the study designed?

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Changes in model discrimination (Model 1: need for MV in patients hospitalized with COVID-19)	Aim 1 Outcome: Changes in Area under receiver operating characteristic curve (AUC) [delta AUC] for models predicting the probability of: the need for MV in patients hospitalized with COVID-19.	30 days from hospitalization
Changes in model discrimination (Model 2: mortality in patients receiving MV)	Aim 1 Outcome: Changes in Area under receiver operating characteristic curve (AUC) [delta AUC] for models predicting the probability of: mortality in patients receiving MV.	30 days from hospitalization
Changes in model discrimination (Model 3: length of stay in the ICU)	Aim 1 Outcome: Changes in Area under receiver operating characteristic curve (AUC) [delta AUC] for models predicting the probability of: length of stay in the ICU.	30 days from hospitalization
Changes in model calibration (Model 1: need for MV in patients hospitalized with COVID-19)	Aim 1 Outcome-Changes in Harrell's E for models predicting the probability of: the need for MV in patients hospitalized with COVID-19.	30 days from hospitalization
Changes in model calibration (Model 2: mortality in patients receiving MV)	Aim 1 Outcome-Changes in Harrell's E for models predicting the probability of: mortality in patients receiving MV.	30 days from hospitalization
Changes in model calibration (Model 3: length of stay in the ICU)	Aim 1 Outcome-Changes in Harrell's E for models predicting the probability of: length of stay in the ICU.	30 days from hospitalization
Changes in net benefit (Model 1: need for MV in patients hospitalized with COVID-19)	Aim 1 Outcome-Changes in Net Benefit for models predicting the probability of: the need for MV in patients hospitalized with COVID-19.	30 days from hospitalization
Changes in net benefit (Model 2: mortality in patients receiving MV)	Aim 1 Outcome-Changes in Net Benefit for models predicting the probability of: mortality in patients receiving MV.	30 days from hospitalization
Changes in net benefit (Model 3: length of stay in the ICU)	Aim 1 Outcome-Changes in Net Benefit for models predicting the probability of: length of stay in the ICU.	30 days from hospitalization
Changes in model discrimination in external database after updating (Model 1: need for MV in patients hospitalized with COVID-19)	Aim 2 Outcome-Changes in Area under receiver operating characteristic curve (AUC) [delta AUC] for models predicting the probability of: the need for MV in patients hospitalized with COVID-19.	30 days from hospitalization
Changes in model discrimination in external database after updating (Model 2: mortality in patients receiving MV)	Aim 2 Outcome-Changes in Area under receiver operating characteristic curve (AUC) [delta AUC] for models predicting the probability of: mortality in patients receiving MV.	30 days from hospitalization
Changes in model discrimination in external database after updating (Model 3: length of stay in the ICU)	Aim 2 Outcome-Changes in Area under receiver operating characteristic curve (AUC) [delta AUC] for models predicting the probability of: length of stay in the ICU.	30 days from hospitalization
Changes in model calibration in external database after updating (Model 1: need for MV in patients hospitalized with COVID-19)	Aim 2 Outcome-Changes in Harrell's E for models predicting the probability of: the need for MV in patients hospitalized with COVID-19.	30 days from hospitalization
Changes in model calibration in external database after updating (Model 2: mortality in patients receiving MV)	Aim 2 Outcome-Changes in Harrell's E for models predicting the probability of: mortality in patients receiving MV.	30 days from hospitalization
Changes in model calibration in external database after updating (Model 3: length of stay in the ICU)	Aim 2 Outcome-Changes in Harrell's E for models predicting the probability of: length of stay in the ICU.	30 days from hospitalization
Changes in net benefit in external database after updating (Model 1: need for MV in patients hospitalized with COVID-19)	Aim 2 Outcome-Changes in Net Benefit for models predicting the probability of: the need for MV in patients hospitalized with COVID-19.	30 days from hospitalization
Changes in net benefit in external database after updating (Model 2: mortality in patients receiving MV)	Aim 2 Outcome-Changes in Net Benefit for models predicting the probability of: mortality in patients receiving MV.	30 days from hospitalization
Changes in net benefit in external database after updating (Model 3: length of stay in the ICU)	Aim 2 Outcome-Changes in Net Benefit for models predicting the probability of: length of stay in the ICU.	30 days from hospitalization

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Stakeholder perceptions, beliefs and opinions on COVID prediction models	Aim 3 Outcome-The outcome will be assessed with a codebook derived deductively from our structured interview guide to identify themes that emerge in the semi-structured sessions. Through focus groups held via synchronous video conferences, we will engage with patients and clinical providers to identify patient- and provider-reported themes that emerge in how clinical prediction models can support decision making in the care of patients with COVID-19. Themes will be identified through qualitative analysis of patient and provider feedback. We expect to elicit patient and provider beliefs, opinions and values around the scientific, ethical and pragmatic aspects of use of these models to support decision making.	6 months

Collaborators and Investigators

• Sponsor: Tufts Medical Center
• Collaborators: Northwell Health; Erasmus Medical Center
• Investigators: Principal Investigator: David M Kent, MD, MS, Tufts Medical Center

Publications and helpful links

General Publications

• Zhou F, Yu T, Du R, Fan G, Liu Y, Liu Z, Xiang J, Wang Y, Song B, Gu X, Guan L, Wei Y, Li H, Wu X, Xu J, Tu S, Zhang Y, Chen H, Cao B. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet. 2020 Mar 28;395(10229):1054-1062. doi: 10.1016/S0140-6736(20)30566-3. Epub 2020 Mar 11. Erratum In: Lancet. 2020 Mar 28;395(10229):1038. Lancet. 2020 Mar 28;395(10229):1038.
• Lim WS, van der Eerden MM, Laing R, Boersma WG, Karalus N, Town GI, Lewis SA, Macfarlane JT. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax. 2003 May;58(5):377-82. doi: 10.1136/thorax.58.5.377.
• Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, Bonten MMJ, Dahly DL, Damen JAA, Debray TPA, de Jong VMT, De Vos M, Dhiman P, Haller MC, Harhay MO, Henckaerts L, Heus P, Kammer M, Kreuzberger N, Lohmann A, Luijken K, Ma J, Martin GP, McLernon DJ, Andaur Navarro CL, Reitsma JB, Sergeant JC, Shi C, Skoetz N, Smits LJM, Snell KIE, Sperrin M, Spijker R, Steyerberg EW, Takada T, Tzoulaki I, van Kuijk SMJ, van Bussel B, van der Horst ICC, van Royen FS, Verbakel JY, Wallisch C, Wilkinson J, Wolff R, Hooft L, Moons KGM, van Smeden M. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020 Apr 7;369:m1328. doi: 10.1136/bmj.m1328. Erratum In: BMJ. 2020 Jun 3;369:m2204.
• Richardson S, Hirsch JS, Narasimhan M, Crawford JM, McGinn T, Davidson KW; the Northwell COVID-19 Research Consortium; Barnaby DP, Becker LB, Chelico JD, Cohen SL, Cookingham J, Coppa K, Diefenbach MA, Dominello AJ, Duer-Hefele J, Falzon L, Gitlin J, Hajizadeh N, Harvin TG, Hirschwerk DA, Kim EJ, Kozel ZM, Marrast LM, Mogavero JN, Osorio GA, Qiu M, Zanos TP. Presenting Characteristics, Comorbidities, and Outcomes Among 5700 Patients Hospitalized With COVID-19 in the New York City Area. JAMA. 2020 May 26;323(20):2052-2059. doi: 10.1001/jama.2020.6775. Erratum In: JAMA. 2020 May 26;323(20):2098.
• Clarification of Mortality Rate and Data in Abstract, Results, and Table 2. JAMA. 2020 May 26;323(20):2098. doi: 10.1001/jama.2020.7681. No abstract available.
• Levy TJ, Richardson S, Coppa K, et al. Development and Validation of a Survival Calculator for Hospitalized Patients with COVID-19. medRxiv. 2020:2020.2004.2022.20075416.
• Tibshirani R. Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society Series B (Methodological). 1996;58(1):267-288.
• Putter H, Fiocco M, Geskus RB. Tutorial in biostatistics: competing risks and multi-state models. Stat Med. 2007 May 20;26(11):2389-430. doi: 10.1002/sim.2712.
• de Wreede LC, Fiocco M, Putter H. The mstate package for estimation and prediction in non- and semi-parametric multi-state and competing risks models. Comput Methods Programs Biomed. 2010 Sep;99(3):261-74. doi: 10.1016/j.cmpb.2010.01.001. Epub 2010 Mar 15.
• van Klaveren D, Steyerberg EW, Gonen M, Vergouwe Y. The calibrated model-based concordance improved assessment of discriminative ability in patient clusters of limited sample size. Diagn Progn Res. 2019 Jun 6;3:11. doi: 10.1186/s41512-019-0055-8. eCollection 2019.
• Jones AE, Trzeciak S, Kline JA. The Sequential Organ Failure Assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation. Crit Care Med. 2009 May;37(5):1649-54. doi: 10.1097/CCM.0b013e31819def97.
• Griffith GJ, Morris TT, Tudball MJ, Herbert A, Mancano G, Pike L, Sharp GC, Sterne J, Palmer TM, Davey Smith G, Tilling K, Zuccolo L, Davies NM, Hemani G. Collider bias undermines our understanding of COVID-19 disease risk and severity. Nat Commun. 2020 Nov 12;11(1):5749. doi: 10.1038/s41467-020-19478-2.

Helpful Links

• Parent study funded through the Patient Centered Outcome Research Institute (PCORI) to develop evaluation and updating framework for clinical prediction models used in this study.
• Northwell COVID-19 Survival (NOCOS) Calculator

Study record dates

Study Major Dates

• Study Start (ACTUAL): December 7, 2020
• Primary Completion (ACTUAL): August 31, 2021
• Study Completion (ACTUAL): August 31, 2021

Study Registration Dates

• First Submitted: November 20, 2020
• First Submitted That Met QC Criteria: December 28, 2020
• First Posted (ACTUAL): December 30, 2020

Study Record Updates

• Last Update Posted (ACTUAL): January 20, 2022
• Last Update Submitted That Met QC Criteria: January 18, 2022
• Last Verified: January 1, 2022

More Information

Terms related to this study

• Additional Relevant MeSH Terms: Coronavirus Infections; Coronaviridae Infections; Nidovirales Infections; RNA Virus Infections; Virus Diseases; Infections; Respiratory Tract Infections; Respiratory Tract Diseases; Pneumonia, Viral; Pneumonia; Lung Diseases; COVID-19
• Other Study ID Numbers: PCORI-ME-1606-35555

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No