Prognosis Research Strategy (PROGRESS) 3: prognostic model research

Ewout W Steyerberg, Karel G M Moons, Danielle A van der Windt, Jill A Hayden, Pablo Perel, Sara Schroter, Richard D Riley, Harry Hemingway, Douglas G Altman, PROGRESS Group, Keith Abrams, Doug Altman, Andrew Briggs, Nils Brunner, Peter Croft, Jill Hayden, Aroon Hingorani, Harry Hemingway, Panayiotis Kyzas, Núria Malats, Karel Moons, George Peat, Pablo Perel, Richard Riley, Ian Roberts, Willi Sauerbrei, Sara Schroter, Ewout Steyerberg, Adam Timmis, Daniëlle van der Windt, Ewout W Steyerberg, Karel G M Moons, Danielle A van der Windt, Jill A Hayden, Pablo Perel, Sara Schroter, Richard D Riley, Harry Hemingway, Douglas G Altman, PROGRESS Group, Keith Abrams, Doug Altman, Andrew Briggs, Nils Brunner, Peter Croft, Jill Hayden, Aroon Hingorani, Harry Hemingway, Panayiotis Kyzas, Núria Malats, Karel Moons, George Peat, Pablo Perel, Richard Riley, Ian Roberts, Willi Sauerbrei, Sara Schroter, Ewout Steyerberg, Adam Timmis, Daniëlle van der Windt

Abstract

Prognostic models are abundant in the medical literature yet their use in practice seems limited. In this article, the third in the PROGRESS series, the authors review how such models are developed and validated, and then address how prognostic models are assessed for their impact on practice and patient outcomes, illustrating these ideas with examples.

Conflict of interest statement

SS is a full time employee of the BMJ Group but is not involved in the decision making on manuscripts. The authors declare no other competing interests.

Figures

Figure 1. Kaplan-Meier survival curves for four…
Figure 1. Kaplan-Meier survival curves for four risk groups derived from a prognostic model that provides a score to predict renal outcome in IgA nephropathy (reproduced from Goto et al [83]).
Figure 2. Web tool for prognosis of…
Figure 2. Web tool for prognosis of patients with head injury (CRASH trial) (reproduced from Perel et al with permission).
Figure 3. Distribution of published articles describing…
Figure 3. Distribution of published articles describing model development, validation, and impact assessment in four reviews (see Text S1).
Path element adapted from Chart 7.1 in the Cooksey report (2006) http://bit.ly/Ro27rL (made available for use and re-use through the Open Government License).

References

    1. Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, et al. (2013) Prognosis research strategy (PROGRESS) 1: A framework for researching clinical outcomes. BMJ 346: e5595.
    1. Riley RD, Hayden JA, Steyerberg EW, Moons KGM, Abrams K, et al. (2013) Prognosis research strategy (PROGRESS) 2: Prognostic factor research. PLoS Med 10: e1001380 doi:.
    1. Harrell FE Jr (2001) Regression modeling strategies with applications to linear models, logistic regression, and survival analysis. Springer.
    1. Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG (2009) Prognosis and prognostic research: what, why, and how? BMJ 338: b375.
    1. Steyerberg EW (2009) Clinical prediction models: a practical approach to development, validation, and updating. Springer.
    1. Haybittle JL, Blamey RW, Elston CW, Johnson J, Doyle PJ, et al. (1982) A prognostic index in primary breast cancer. Br J Cancer 45: 361–366.
    1. Perel P, Arango M, Clayton T, Edwards P, Komolafe E, et al. (2008) Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ 336: 425–429.
    1. Altman DG, Vergouwe Y, Royston P, Moons KG (2009) Prognosis and prognostic research: validating a prognostic model. BMJ 338: b605.
    1. Moons KG, Altman DG, Vergouwe Y, Royston P (2009) Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338: b606.
    1. Royston P, Moons KG, Altman DG, Vergouwe Y (2009) Prognosis and prognostic research: developing a prognostic model. BMJ 338: b604.
    1. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27: 157–172.
    1. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, et al. (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21: 128–138.
    1. Vickers AJ, Cronin AM (2010) Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework. Semin Oncol 37: 31–38.
    1. Counsell C, Dennis M (2001) Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis 12: 159–170.
    1. Perel P, Edwards P, Wentz R, Roberts I (2006) Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 6: 38.
    1. Leushuis E, van der Steeg JW, Steures P, Bossuyt PM, Eijkemans MJ, et al. (2009) Prediction models in reproductive medicine: a critical appraisal. Hum Reprod Update 15: 537–552.
    1. Minne L, Ludikhuize J, de Jonge E, de Rooij S, Abu-Hanna A (2011) Prognostic models for predicting mortality in elderly ICU patients: a systematic review. Intens Care Med 37: 1258–1268.
    1. Wyatt JC, Altman DG (1995) Commentary: Prognostic models: clinically useful or quickly forgotten? BMJ 311: 1539–1541.
    1. Kent DM, Hayward RA (2007) Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification. JAMA 298: 1209–1212.
    1. Linton SJ, Hallden K (1998) Can we screen for problematic back pain? A screening questionnaire for predicting outcome in acute and subacute back pain. Clin J Pain 14: 209–215.
    1. Mackway-Jones K (1997) Emergency triage. BMJ Publishing.
    1. Hernandez AV, Steyerberg EW, Habbema JD (2004) Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements. J Clin Epidemiol 57: 454–460.
    1. Roozenbeek B, Maas AI, Lingsma HF, Butcher I, Lu J, et al. (2009) Baseline characteristics and statistical power in randomized controlled trials: selection, prognostic targeting, or covariate adjustment? Crit Care Med 37: 2683–2690.
    1. Jarman B, Pieter D, van der Veen AA, Kool RB, Aylin P, et al. (2010) The hospital standardised mortality ratio: a powerful tool for Dutch hospitals to assess their quality of care? Qual Saf Health Care 19: 9–13.
    1. Lingsma HF, Roozenbeek B, Li B, Lu J, Weir J, et al. (2011) Large between-center differences in outcome after moderate and severe traumatic brain injury in the international mission on prognosis and clinical trial design in traumatic brain injury (IMPACT) study. Neurosurgery 68: 601–607.
    1. Sterne JAC, White IR, Carlin JB, Spratt M, Royston P, et al. (2009) Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 338: b2393.
    1. Royston P, Sauerbrei W (2008) Multivariable model-building: a pragmatic approach to regression analysis based on fractional polynomials for modelling continuous variables. John Wiley.
    1. Sauerbrei W, Royston P, Binder H (2007) Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med 26: 5512–5528.
    1. Sauerbrei W (1999) The use of resampling methods to simplify regression models in medical statistics. Appl Stat 48: 313–329.
    1. Schumacher M, Hollander N, Schwarzer G, Sauerbrei W (2006) Prognostic factor studies. In: Crowley J, Ankerst DP, editors. Handbook of statistics in clinical oncology. Chapman & Hall/CRC Press. pp. 289–333.
    1. Bleeker SE, Moll HA, Steyerberg EW, Donders AR, Derksen-Lubsen G, et al. (2003) External validation is necessary in prediction research: a clinical example. J Clin Epidemiol 56: 826–832.
    1. Justice AC, Covinsky KE, Berlin JA (1999) Assessing the generalizability of prognostic information. Ann Intern Med 130: 515–524.
    1. Reilly BM, Evans AT (2006) Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 144: 201–209.
    1. Toll DB, Janssen KJ, Vergouwe Y, Moons KG (2008) Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 61: 1085–1094.
    1. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD (2005) Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 58: 475–483.
    1. Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, et al. (2001) Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 54: 774–781.
    1. Altman DG (2009) Prognostic models: a methodological framework and review of models for breast cancer. Cancer Invest 27: 235–243.
    1. Hill JC, Whitehurst DG, Lewis M, Bryan S, Dunn KM, et al. (2011) Comparison of stratified primary care management for low back pain with current best practice (STarT Back): a randomised controlled trial. Lancet 378: 1560–1571.
    1. Feldman M, Stanford R, Catcheside A, Stotter A (2002) The use of a prognostic table to aid decision making on adjuvant therapy for women with early breast cancer. Eur J Surg Oncol 28: 615–619.
    1. Henriksson M, Palmer S, Chen R, Damant J, Fitzpatrick NK, et al. (2010) Assessing the cost effectiveness of using prognostic biomarkers with decision models: case study in prioritising patients waiting for coronary artery surgery. BMJ 340: b5606.
    1. Moons KG (2010) Criteria for scientific evaluation of novel markers: a perspective. Clin Chem 56: 537–541.
    1. Uijl SG, Leijten FS, Arends JB, Parra J, van Huffelen AC, et al. (2007) The added value of [18F]-fluoro-D-deoxyglucose positron emission tomography in screening for temporal lobe epilepsy surgery. Epilepsia 48: 2121–2129.
    1. Steyerberg EW, Mushkudiani N, Perel P, Butcher I, Lu J, et al. (2008) Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS Med 5: e165 doi:.
    1. De Silva MJ, Roberts I, Perel P, Edwards P, Kenward MG, et al. (2009) Patient outcome after traumatic brain injury in high-, middle- and low-income countries: analysis of data on 8927 patients in 46 countries. Int J Epidemiol 38: 452–458.
    1. Menon DK, Zahed C (2009) Prediction of outcome in severe traumatic brain injury. Curr Opin Crit Care 15: 437–441.
    1. Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD (2004) Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 23: 2567–2586.
    1. Van Houwelingen HC, Thorogood J (1995) Construction, validation and updating of a prognostic model for kidney graft survival. Stat Med 14: 1999–2008.
    1. Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y (2008) Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61: 76–86.
    1. Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, et al. (2009) Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circ Cardiovasc Qual Outcomes 119: 2408–2416.
    1. Cook NR (2010) Assessing the incremental role of novel and emerging risk factors. Curr Cardiovasc Risk Rep 4: 112–119.
    1. Pencina MJ, D'Agostino RB Sr, Steyerberg EW (2011) Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 30: 11–21.
    1. Vickers AJ, Cronin AM, Begg CB (2011) One statistical test is sufficient for assessing new predictive markers. BMC Med Res Methodol 11: 13.
    1. Melander O, Newton-Cheh C, Almgren P, Hedblad B, Berglund G, et al. (2009) Novel and conventional biomarkers for prediction of incident cardiovascular events in the community. JAMA 302: 49–57.
    1. Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P (2004) Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 159: 882–890.
    1. Rector TS, Anand IS (2010) Research needed to support clinical use of biomarkers as prognostic indicators for patients with heart failure. Cardiol Res Pract 2010: 453851.
    1. Marchevsky AM, Gupta R (2010) Interobserver diagnostic variability at “moderate” agreement levels could significantly change the prognostic estimates of clinicopathologic studies: evaluation of the problem using evidence from patients with diffuse lung disease. Ann Diagn Pathol 14: 88–93.
    1. Fernandes CM, Wuerz R, Clark S, Djurdjev O (1999) How reliable is emergency department triage? Ann Emerg Med 34: 141–147.
    1. Van Veen M, Steyerberg EW, Ruige M, van Meurs AH, Roukema J, et al. (2008) Manchester triage system in paediatric emergency care: prospective observational study. BMJ 337: a1501.
    1. Ross PL, Scardino PT, Kattan MW (2001) A catalog of prostate cancer nomograms. J Urol 165: 1562–1568.
    1. Maguire JL, Kulik DM, Laupacis A, Kuppermann N, Uleryk EM, et al. (2011) Clinical prediction rules for children: a systematic review. Pediatrics 128: e666–77.
    1. Mallett S, Royston P, Dutton S, Waters R, Altman DG (2010) Reporting methods in studies developing prognostic models in cancer: a review. BMC Med 8: 20.
    1. Mallett S, Royston P, Waters R, Dutton S, Altman DG (2010) Reporting performance of prognostic models in cancer: a review. BMC Med 8: 21.
    1. Collins GS, Mallett S, Omar O, Yu LM (2011) Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med 9: 103.
    1. Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, et al. (1998) Prediction of coronary heart disease using risk factor categories. Circulation 97: 1837–1847.
    1. Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, et al. (2001) Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA 285: 2864–2870.
    1. Sauerbrei W (2005) Prognostic factors. Confusion caused by bad quality design, analysis and reporting of many studies. Adv Otorhinolaryngol 62: 184–200.
    1. Harrell FE Jr, Lee KL, Mark DB (1996) Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15: 361–387.
    1. Riley RD, Lambert PC, Abo-Zaid G (2010) Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ 340: c221.
    1. Boulesteix AL, Sauerbrei W (2011) Added predictive value of high-throughput molecular data to clinical data and its validation. Brief Bioinform 12: 215–229.
    1. Mushkudiani NA, Hukkelhoven CW, Hernandez AV, Murray GD, Choi SC, et al. (2008) A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes. J Clin Epidemiol 61: 331–343.
    1. Wasson JH, Sox HC, Neff RK, Goldman L (1985) Clinical prediction rules. Applications and methodological standards. N Engl J Med 313: 793–9.
    1. Laupacis A, Sekar N, Stiell IG (1997) Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 277: 488–494.
    1. Ingui BJ, Rogers MA Searching for clinical prediction rules in Medline (2001). J Am Med Inform Assoc 8: 391–397.
    1. Galea MH, Blamey RW, Elston CE, Ellis IO (1992) The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Res Treat 22: 207–219.
    1. Balslev I, Axelsson CK, Zedeler K, Rasmussen BB, Carstensen B, et al. (1994) The Nottingham Prognostic Index applied to 9,149 patients from the studies of the Danish Breast Cancer Cooperative Group (DBCG). Breast Cancer Res Treat 32: 281–290.
    1. Williams C, Brunskill S, Altman D, Briggs A, Campbell H, et al. (2006) Cost-effectiveness of using prognostic information to select women with breast cancer for adjuvant systemic therapy. Health Technol Assess 10 34: 1–217.
    1. Maher CG, Grotle M (2009) Evaluation of the predictive validity of the Örebro Musculoskeletal Pain Screening Questionnaire. Clin J Pain 25: 666–670.
    1. Hockings RL, McAuley JH, Maher CG (2008) A systematic review of the predictive ability of the Örebro Musculoskeletal Pain Questionnaire. Spine (Phila Pa 1976) 33: E494–500.
    1. Melloh M, Elfering A, Egli Presland C, Roeder C, Barz T, et al. (2009) Identification of prognostic factors for chronicity in patients with low back pain: a review of screening instruments. Int Orthop 33: 301–313.
    1. Kendall NAS, Linton SJ, Main CJ (1997) Guide to assessing psychosocial yellow flags in acute low back pain: risk factors for long-term disability and work loss. Rehabilitation & Compensation Insurance Corporation of New Zealand, National Health Committee.
    1. Örebro musculoskeletal pain questionnaire (ÖMPQ). WorkCover Authority of New South Wales, 2011.
    1. Schmidt CO, Chenot JF, Pfingsten M, Fahland RA, Lindena G, et al. (2010) Assessing a risk tailored intervention to prevent disabling low back pain–protocol of a cluster randomized controlled trial. BMC Musculoskelet Disord 11: 5.
    1. Goto M, Wakai K, Kawamura T, Ando M, Endoh M, et al. (2009) A scoring system to predict renal outcome in IgA nephropathy: a nationwide 10-year prospective cohort study. Nephrol Dial Transplant 24: 3068–3074.

Source: PubMed

Sponsor e collaboratori

Condizioni mediche

Interventi farmacologici

3
Sottoscrivi