Validation of population-based disease simulation models: a review of concepts and methods

Jacek A Kopec, Philippe Finès, Douglas G Manuel, David L Buckeridge, William M Flanagan, Jillian Oderkirk, Michal Abrahamowicz, Samuel Harper, Behnam Sharif, Anya Okhmatovskaia, Eric C Sayre, M Mushfiqur Rahman, Michael C Wolfson, Jacek A Kopec, Philippe Finès, Douglas G Manuel, David L Buckeridge, William M Flanagan, Jillian Oderkirk, Michal Abrahamowicz, Samuel Harper, Behnam Sharif, Anya Okhmatovskaia, Eric C Sayre, M Mushfiqur Rahman, Michael C Wolfson

Abstract

Background: Computer simulation models are used increasingly to support public health research and policy, but questions about their quality persist. The purpose of this article is to review the principles and methods for validation of population-based disease simulation models.

Methods: We developed a comprehensive framework for validating population-based chronic disease simulation models and used this framework in a review of published model validation guidelines. Based on the review, we formulated a set of recommendations for gathering evidence of model credibility.

Results: Evidence of model credibility derives from examining: 1) the process of model development, 2) the performance of a model, and 3) the quality of decisions based on the model. Many important issues in model validation are insufficiently addressed by current guidelines. These issues include a detailed evaluation of different data sources, graphical representation of models, computer programming, model calibration, between-model comparisons, sensitivity analysis, and predictive validity. The role of external data in model validation depends on the purpose of the model (e.g., decision analysis versus prediction). More research is needed on the methods of comparing the quality of decisions based on different models.

Conclusion: As the role of simulation modeling in population health is increasing and models are becoming more complex, there is a need for further improvements in model validation methodology and common standards for evaluating model credibility.

References

    1. Elveback L, Varma A. Simulation of mathematical models for public health problems. Public Health Rep. 1965;80:1067–76.
    1. Handyside AJ, Morris D. Simulation of emergency bed occupancy. Health Serv Res. 1967;2:287–97.
    1. Fone D, Hollinghurst S, Temple M, Round A, Lester N, Weightman A, Roberts K, Coyle E, Bevan G, Palmer S. Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Public Health Med. 2003;25:325–35. doi: 10.1093/pubmed/fdg075.
    1. Weinstein MC, Toy EL, Sandberg EA, Neumann PJ, Evans JS, Kuntz KM, Graham JD, Hammitt JK. Modeling for health care and other policy decisions: uses, roles, and validity. Value Health. 2001;4:348–61. doi: 10.1046/j.1524-4733.2001.45061.x.
    1. Karnon JE, Goyder E, Tappenden P, McPhie S, Towers I, Brazier J, Madan J. A review and critique of modelling in prioritising and designing screening programmes. Health Technol Assess. 2007;11(52):1–145.
    1. Gupta A, Harding A. In: Modelling our future. Gupta A, Harding A, editor. Vol. 15. Amsterdam Elsevier, Inc; 2007. Introduction and overview; pp. 1–40.
    1. Hammond RA. Complex systems modeling for obesity research. Prev Chronic Dis. 2009;6:A97.
    1. Sargent RG. In: Proceedings of the 2005 Winter Simulation Conference. Kuhl ME, Steiger NM, Armstrong FB, Joines JA, editor. 2005. Verification and validation of simulation models; pp. 130–143. full_text.
    1. Citro CF, Hanushek EA. In: Improving information for social policy decisions The use of microsimulation modeling. Volume I: Review and recommendations. Citro CF, Hanushek EA, editor. Washington, DC: National Academy Press; 1994. Validation; pp. 231–264.
    1. Morrison R. DYNACAN Validation. DYNACAN Team, Statistics Canada, Ottawa, ON, Canada; 2007.
    1. Weinstein MC, O'Brien B, Hornberger J, Jackson J, Johannesson M, McCabe C, Luce BR. ISPOR; Task Force on Good Research Practices--Modeling Studies. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices -- Modeling Studies. Value Health. 2003;6:9–17. doi: 10.1046/j.1524-4733.2003.00234.x.
    1. Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, Woolacoot N, Glanville J. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004;8:iiiiv. ix xi, 1-158.
    1. Sculpher M, Fenwick E, Claxton K. Assessing quality in decision analytic cost-effectiveness models. A suggested framework and example of application. Pharmacoeconomics. 2000;17:461–77. doi: 10.2165/00019053-200017050-00005.
    1. Goeree R, O'Brien BJ, Blackhouse G. Principles of good modeling practice in healthcare cost-effectiveness studies. Expert Rev Pharmacoecon Outcomes Res. 2004;4:189–98. doi: 10.1586/14737167.4.2.189.
    1. Office of the Science Advisor, Council for Regulatory Environmental Modeling, U.S. Guidance on the development, evaluation, and application of environmental models. Environmental Protection Agency (EPA), Washington, DC 20460. EPA/100/K-09/003, 2009. Electronic version. Accessed: February 17, 2010.
    1. Feuer EJ, Etzioni R, Cronin KA, Mariotto A. The use of modeling to understand the impact of screening on U.S. mortality: examples from mammography and PSA testing. Stat Methods Med Res. 2004;13:421–42. doi: 10.1191/0962280204sm376ra.
    1. Wolfson MC. POHEM--a framework for understanding and modelling the health of human populations. World Health Stat Q. 1994;47:157–76.
    1. Will BP, Berthelot J-M, Nobrega KM, Flanagan W, Evans WK. Canada's Population Health Model (POHEM): A tool for performing economic evaluations of cancer control interventions. Eur J Cancer. 2001;37:1797–1804. doi: 10.1016/S0959-8049(01)00204-0.
    1. Will BP, Nobrega KM, Berthelot J-M, Flanagan W, Wolfson MC, Logan DM, Evans WK. First do no harm: Extending the debate on the provision of preventive tamoxifen. Br J Cancer. 2001;85:1280–8. doi: 10.1054/bjoc.2001.2125.
    1. Berthelot J-M, Will BP, Evans WK, Coyle D, Earle CC, Bordeleau L. Decision framework for chemotherapeutic interventions for metastatic non-small cell lung cancer. J Natl Cancer Inst. 2000;92:1321–29. doi: 10.1093/jnci/92.16.1321.
    1. Flanagan W, Le Petit C, Berthelot J-M, White KJ, Coombs BA, Jones-McLean E. Potential impact of population-based colorectal cancer screening in Canada. Chronic Dis Can. 2003;24:81–88.
    1. Kopec JA, Sayre EC, Berthelot JM, Flanagan W, Fines P, Le Petit C, Cibere J, Rahman M, Lima VD, Aghajanian J, Bansback NJ, Anis AH, Jordan JM, Sobolev B, Greidanus NV, Garbuz DS, Hawker GA, Badley EM. Development of a computer simulation model of osteoarthritis in Canada. Osteoarthritis Cartilage. 2010;18:303–11. doi: 10.1016/j.joca.2009.10.010.
    1. National Aeronautic and Space Administration (NASA) Glossary of verification and validation terms. NPARC Alliance CFD Verification and Validation Web Site. Internet Address. Accessed on Feb 8, 2010.
    1. Habbema JD, Schechter CB, Cronin KA, Clarke LD, Feuer EJ. Modeling cancer natural history, epidemiology, and control: reflections on the CISNET breast group experience. J Natl Cancer Inst Monogr. 2006;36:122–6.
    1. Eddy DM, Schlessinger L. Archimedes: a trial-validated model of diabetes. Diabetes Care. 2003;26:3093–101. doi: 10.2337/diacare.26.11.3093.
    1. Eddy DM, Schlessinger L. Validation of the Archimedes diabetes model. Diabetes Care. 2003;26:3102–10. doi: 10.2337/diacare.26.11.3102.
    1. Patten SB. An animated depiction of major depression epidemiology. BMC Psychiatry. 2007;7:23. doi: 10.1186/1471-244X-7-23.
    1. Unal B, Capewell S, Critchley JA. Coronary heart disease policy models: a systematic review. BMC Public Health. 2006;6:213. doi: 10.1186/1471-2458-6-213.
    1. Greenland S, Pearl J, Robins JM. Causal diagrams for epidemiologic research. Epidemiology. 1999;10:37–48. doi: 10.1097/00001648-199901000-00008.
    1. Li W, Agrawala M, Aalesin D. Interactive image-based exploded view diagrams. ACM International Conference Proceeding Series. 2004;62:203–12.
    1. Hunink MG, Goldman L, Tosteson AN, Mittleman MA, Goldman PA, Williams LW, Tsevat J, Weinstein MC. The recent decline in mortality from coronary heart disease, 1980-1990. The effect of secular trends in risk factors and treatment. JAMA. 1997;277:535–42. doi: 10.1001/jama.277.7.535.
    1. Altman DG, De Stavola BL, Love SB, Stepniewska KA. Review of survival analyses published in cancer journals. Br J Cancer. 1995;72:511–8.
    1. Quantin C, Abrahamowicz M, Moreau T, Bartlett G, MacKenzie T, Tazi MA, Lalonde L, Faivre J. Variation over time of the effects of prognostic factors in a population-based study of colon cancer: comparison of statistical models. Am J Epidemiol. 1999;150:1188–200.
    1. Kopec JA, Rahman MM, Berthelot J-M, Le Petit C, Aghajanian J, Sayre EC, Cibere J, Anis AH, Badley EM. Descriptive epidemiology of osteoarthritis in British Columbia, Canada. J Rheumatol. 2007;34:386–93.
    1. Kane RL, Saleh KJ, Wilt TJ, Bershadsky B, Cross WW, III, MacDonald RM, Rutks I. Total knee replacement. Evidence report/technology assessment. Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, Number 86, AHRQ Publication No. 04-E006-2. 2003.
    1. Stout NK, Knudsen AB, Kong CY, McMahon PM, Gazelle GS. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics. 2009;27:533–545. doi: 10.2165/11314830-000000000-00000.
    1. Weinstein MC. Recent developments in decision-analytic modelling for economic evaluation. Pharmacoeconomics. 2006;24:1043–53. doi: 10.2165/00019053-200624110-00002.
    1. Berry DA, Cronin KA, Plevritis SK, Fryback DG, Clarke L, Zelen M, Mandelblatt JS, Yakovlev AY, Habbema JD, Feuer EJ. Cancer Intervention and Surveillance Modeling Network (CISNET) Collaborators. Effect of screening and adjuvant therapy on mortality from breast cancer. New Engl J Med. 2005;353:1784–92. doi: 10.1056/NEJMoa050518.
    1. Groot Koerkamp B, Weinstein MC, Stijnen T, Heijenbrok-Kal MH, Hunink MG. Uncertainty and patient heterogeneity in medical decision models. Med Decis Making. 2010;30:194–205. doi: 10.1177/0272989X09342277.
    1. Attaway S. MATLAB: A practical approach. Oxford: Elsevier, Inc; 2009.
    1. Chambers J. Software for Data Analysis: Programming with R. New York: Springer Science+Business Media; 2008.
    1. Statistics Canada. Microsimulation: Modgen (Model generator) Accessed: Feb 8, 2010.
    1. Wikipedia. The Free Encyclopedia. Comparison of agent-based modeling software. Accessed: Feb 8, 2010.
    1. Wikipedia. The Free Encyclopedia. List of discrete event simulation software. Accessed: Feb 8, 2010.
    1. Railsback SF. Agent-based simulation platforms: review and development recommendations. Simulation. 2006;82:609–23. doi: 10.1177/0037549706073695.
    1. Diller AZ. Chichester: John Wiley & Sons Ltd; 1994. An introduction to formal methods.
    1. Saltelli A, Cha K, Scott M, (Eds.) Sensitivity analysis Wiley series in probability and statistics. New York: John Wiley and Sons; 2000.
    1. Klevmarken NA. Department of Economics Working Paper. Sweden: Uppsala University; 1998. Statistical inference in micro simulation models: Incorporating external information.
    1. Cronin KA, Legler JM, Etzioni RD. Assessing uncertainty in microsimulation modelling with application to cancer screening interventions. Stat Med. 1998;17:2509–23. doi: 10.1002/(SICI)1097-0258(19981115)17:21<2509::AID-SIM949>;2-V.
    1. Campolongo F, Kleijnen J, Andres T. In: Sensitivity Analysis. Saltelli A, Chan K, and Scott EM, editor. New York: John Wiley and Sons Publishers; 2000. Screening methods; pp. 65–80.
    1. Morris MD. Factorial sampling plans for preliminary computational experiments. Technometrics. 1991;33:161–174. doi: 10.2307/1269043.
    1. Cooke RM, Van Noordwejk JM. Generalized graphical methods for uncertainty and sensitivity analysis. Bashkiv Ecological Journal. 2000;1:54–57.
    1. Bojke L, Claxton K, Sculpher M, Palmer S. Characterizing structural uncertainty in decision analytic models: A review and application of methods. Value Health. 2009;12:739–49. doi: 10.1111/j.1524-4733.2008.00502.x.
    1. Mount Hood 4 Modeling Group. Computer modeling of diabetes and its complications: A report on the Fourth Mount Hood Challenge Meeting. Diabetes Care. 2007;30:1638–46. doi: 10.2337/dc07-9919.
    1. Gerds TA, Cai T, Schumacher M. The performance of risk prediction models. Biom J. 2008;50:457–79. doi: 10.1002/bimj.200810443.
    1. Schumacher M, Holländer N, Sauerbrei W. Resampling and cross-validation techniques: a tool to reduce bias caused by model building? Stat Med. 1997;16:2813–27. doi: 10.1002/(SICI)1097-0258(19971230)16:24<2813::AID-SIM701>;2-Z.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe