Developing predictive models using electronic medical records: challenges and pitfalls

Abstract

While Electronic Medical Records (EMR) contain detailed records of the patient-clinician encounter - vital signs, laboratory tests, symptoms, caregivers' notes, interventions prescribed and outcomes - developing predictive models from this data is not straightforward. These data contain systematic biases that violate assumptions made by off-the-shelf machine learning algorithms, commonly used in the literature to train predictive models. In this paper, we discuss key issues and subtle pitfalls specific to building predictive models from EMR. We highlight the importance of carefully considering both the special characteristics of EMR as well as the intended clinical use of the predictive model and show that failure to do so could lead to developing models that are less useful in practice. Finally, we describe approaches for training and evaluating models on EMR using early prediction of septic shock as our example application.

Figures

Figure 1:

Plot of average AUC across 5 folds for different numbers of clear hospital stays.

Figure 2:

Plot of average AUC across 5 folds for different numbers of clear hospital stays, ignoring stays with severe sepsis.