Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0

Shirley V Wang, Sebastian Schneeweiss, Marc L Berger, Jeffrey Brown, Frank de Vries, Ian Douglas, Joshua J Gagne, Rosa Gini, Olaf Klungel, C Daniel Mullins, Michael D Nguyen, Jeremy A Rassen, Liam Smeeth, Miriam Sturkenboom, joint ISPE-ISPOR Special Task Force on Real World Evidence in Health Care Decision Making, Marc L. Berger, Jeffrey Brown, Frank de Vries, Ian Douglas, Joshua J. Gagne, Rosa Gini, Olaf Klungel, C. Daniel Mullins, Michael D. Nguyen, Jeremy A. Rassen, Liam Smeeth, Miriam Sturkenboom, Andrew Bate, Alison Bourke, Suzanne Cadarette, Tobias Gerhard, Robert Glynn, Krista Huybrechts, Kiyoshi Kubota, Amr Makady, Fredrik Nyberg, Mary E. Ritchey, Ken Rothman, Sengwee Toh, Shirley V Wang, Sebastian Schneeweiss, Marc L Berger, Jeffrey Brown, Frank de Vries, Ian Douglas, Joshua J Gagne, Rosa Gini, Olaf Klungel, C Daniel Mullins, Michael D Nguyen, Jeremy A Rassen, Liam Smeeth, Miriam Sturkenboom, joint ISPE-ISPOR Special Task Force on Real World Evidence in Health Care Decision Making, Marc L. Berger, Jeffrey Brown, Frank de Vries, Ian Douglas, Joshua J. Gagne, Rosa Gini, Olaf Klungel, C. Daniel Mullins, Michael D. Nguyen, Jeremy A. Rassen, Liam Smeeth, Miriam Sturkenboom, Andrew Bate, Alison Bourke, Suzanne Cadarette, Tobias Gerhard, Robert Glynn, Krista Huybrechts, Kiyoshi Kubota, Amr Makady, Fredrik Nyberg, Mary E. Ritchey, Ken Rothman, Sengwee Toh

Abstract

Purpose: Defining a study population and creating an analytic dataset from longitudinal healthcare databases involves many decisions. Our objective was to catalogue scientific decisions underpinning study execution that should be reported to facilitate replication and enable assessment of validity of studies conducted in large healthcare databases.

Methods: We reviewed key investigator decisions required to operate a sample of macros and software tools designed to create and analyze analytic cohorts from longitudinal streams of healthcare data. A panel of academic, regulatory, and industry experts in healthcare database analytics discussed and added to this list.

Conclusion: Evidence generated from large healthcare encounter and reimbursement databases is increasingly being sought by decision-makers. Varied terminology is used around the world for the same concepts. Agreeing on terminology and which parameters from a large catalogue are the most essential to report for replicable research would improve transparency and facilitate assessment of validity. At a minimum, reporting for a database study should provide clarity regarding operational definitions for key temporal anchors and their relation to each other when creating the analytic dataset, accompanied by an attrition table and a design diagram. A substantial improvement in reproducibility, rigor and confidence in real world evidence generated from healthcare databases could be achieved with greater transparency about operational study parameters used to create analytic datasets from longitudinal healthcare databases.

Keywords: Transparency; healthcare databases; longitudinal data; methods; pharmacoepidemiology; replication; reproducibility.

© 2017 The Authors. Pharmacoepidemiology & Drug Safety Published by John Wiley & Sons Ltd.

Figures

Figure 1
Figure 1
Data provenance: transitions from healthcare delivery to analysis results. [Colour figure can be viewed at wileyonlinelibrary.com]
Figure 2
Figure 2
Example design diagram. [Colour figure can be viewed at wileyonlinelibrary.com]

References

    1. Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. Apr 2005;58(4):323‐337.
    1. Califf RM, Robb MA, Bindman AB, et al. Transforming Evidence Generation to Support Health and Health Care Decisions. N Engl J Med. 2016;375(24):2395‐2400.
    1. Psaty BM, Breckenridge AM. Mini‐Sentinel and regulatory science‐‐big data rendered fit and functional. N Engl J Med. Jun 5. 2014;370(23):2165‐2167.
    1. Fleurence RL, Curtis LH, Califf RM, Platt R, Selby JV, Brown JS. Launching PCORnet, a national patient‐centered clinical research network. J Am Med Inform Assoc. Jul‐Aug. 2014;21(4):578‐582.
    1. Oliveira JL, Lopes P, Nunes T, et al. The EU‐ADR Web Platform: delivering advanced pharmacovigilance tools. Pharmacoepidemiol Drug Saf. 2013 May. 2013;22(5):459‐467.
    1. As PENc, Andersen M, Bergman U, et al. The Asian Pharmacoepidemiology Network (AsPEN): promoting multi‐national collaboration for pharmacoepidemiologic research in Asia. Pharmacoepidemiol Drug Saf. 2013;22(7):700‐704.
    1. Behrman RE, Benner JS, Brown JS, McClellan M, Woodcock J, Platt R. Developing the Sentinel System‐‐a national resource for evidence development. N Engl J Med. Feb 10. 2011;364(6):498‐499.
    1. Suissa S, Henry D, Caetano P, et al. CNODES: the Canadian Network for Observational Drug Effect Studies. Open Medicine. 2012;6(4):e134‐e140.
    1. Trifiro G, Coloma PM, Rijnbeek PR, et al. Combining multiple healthcare databases for postmarketing drug and vaccine safety surveillance: why and how? J Intern Med. Jun. 2014;275(6):551‐561.
    1. Engel P, Almas MF, De Bruin ML, Starzyk K, Blackburn S, Dreyer NA. Lessons learned on the design and the conduct of Post‐Authorization Safety Studies: review of 3 years of PRAC oversight. Br J Clin Pharmacol. Apr. 2017;83(4):884‐893.
    1. Eichler H‐G, Hurts H, Broich K, Rasi G. Drug Regulation and Pricing — Can Regulators Influence Affordability? N Engl J Med. 2016;374(19):1807‐1809.
    1. Makady A, Ham RT, de Boer A, et al. Policies for Use of Real‐World Data in Health Technology Assessment (HTA): A Comparative Study of Six HTA Agencies. Value in health: J Int Soc Pharmacoecon Outcomes Res. Apr. 2017;20(4):520‐532.
    1. Benchimol EI, Manuel DG, To T, Griffiths AM, Rabeneck L, Guttmann A. Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. J Clin Epidemiol. 2011;64(8):821‐829.
    1. Nicholls SG, Quach P, von Elm E, et al. The REporting of Studies Conducted Using Observational Routinely‐Collected Health Data (RECORD) Statement: Methods for Arriving at Consensus and Developing Reporting Guidelines. PLoS One. 2015;10(5):e0125620
    1. Collins GS, Omar O, Shanyinde M, Yu LM. A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. J Clin Epidemiol. Mar. 2013;66(3):268‐277.
    1. Bouwmeester W, Zuithoff NP, Mallett S, et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1‐12.
    1. de Vries F, de Vries C, Cooper C, Leufkens B, van Staa T‐P. Reanalysis of two studies with contrasting results on the association between statin use and fracture risk: the General Practice Research Database. Int J Epidemiol. 2006;35(5):1301‐1308.
    1. Jick H, Kaye JA, Vasilakis‐Scaramozza C, Jick SS. Risk of venous thromboembolism among users of third generation oral contraceptives compared with users of oral contraceptives with levonorgestrel before and after 1995: cohort and case‐control analysis. BMJ: Br Med J. 2000;321(7270):1190‐1195.
    1. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature Mar 29. 2012;483(7391):531‐533.
    1. Kaiser J. The cancer test. Science. Jun 26 2015;348(6242):1411‐1413.
    1. Nosek BA, Alter G, Banks GC, et al. SCIENTIFIC STANDARDS. Promoting an open research culture. Science. Jun 26 2015;348(6242):1422‐1425.
    1. Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature Jan 30 2014;505(7485):612‐613.
    1. Goodman SK, Krumholz HM. Open Science: PCORI's Efforts to Make Study Results and Data More Widely Available. 2015.
    1. Science CfO . Transparency and Openness Promotion (TOP) Guidelines. . Accessed July 20, 2015.
    1. Meta‐Research Innovation Center at Stanford. /, 2016.
    1. . .
    1. Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JP. Reproducible Research Practices and Transparency across the Biomedical Literature. PLoS Biol. Jan 2016;14(1):e1002333
    1. Zika Open‐Research Portal.
    1. Hall GC, Sauer B, Bourke A, Brown JS, Reynolds MW, LoCasale R. Guidelines for good database selection and use in pharmacoepidemiology research. Pharmacoepidemiol Drug Saf Jan. 2012;21(1):1‐10.
    1. Lanes S, Brown JS, Haynes K, Pollack MF, Walker AM. Identifying health outcomes in healthcare databases. Pharmacoepidemiol Drug Saf Oct. 2015;24(10):1009‐1016.
    1. Brown JS, Kahn M, Toh S. Data quality assessment for comparative effectiveness research in distributed data networks. Med Care Aug. 2013;51(8 Suppl 3):S22‐S29.
    1. Kahn MG, Brown JS, Chun AT, et al. Transparent reporting of data quality in distributed data networks. Egems. 2015;3(1):1052
    1. EMA . ENCePP Guide on Methodological Standards in Pharmacoepidemiology. 2014.
    1. von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med Oct 16. 2007;4(10):e296
    1. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 03/15/online. 2016;3: 160018
    1. Gini R, Schuemie M, Brown J, et al. Data Extraction and Management in Networks of Observational Health Care Databases for Scientific Research: A Comparison of EU‐ADR, OMOP, Mini‐Sentinel and MATRICE Strategies. Egems. 2016;4(1):1189
    1. Berger ML, Sox H, Willke R, et al. Good Practices for Real‐World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR‐ISPE Special Task Force on Real‐World Evidence in Healthcare Decision‐Making. Pharmacoepidemiol Drug Saf. 2017;26(9):1033–1039.
    1. Nosek BA, Errington TM. Making sense of replications. Elife. 2017;6:e23383
    1. Benchimol EI, Smeeth L, Guttmann A, et al. The REporting of studies Conducted using Observational Routinely‐collected health Data (RECORD) statement. PLoS Med. Oct 2015;12(10):e1001885
    1. Committee PP‐CORIM . The PCORI Methodology Report. 2013.
    1. Gagne JW, Wang SV, Schneeweiss, S . FDA Mini‐Sentinel Prospective Routine Observational Monitoring Program Tool: Cohort Matching. Technical Users’ Guide version: 1.0. January 2014.
    1. Zhou X, Murugesan S, Bhullar H, et al. An evaluation of the THIN database in the OMOP Common Data Model for active drug safety surveillance. Drug Saf. Feb 2013;36(2):119‐134.
    1. Collins FS, Hudson KL, Briggs JP, Lauer MS. PCORnet: turning a dream into reality. J Am Med Inform Assoc Jul‐Aug. 2014;21(4):576‐577.
    1. Schuemie MJ, Gini R, Coloma PM, et al. Replication of the OMOP experiment in Europe: evaluating methods for risk identification in electronic health record databases. Drug Saf Oct. 2013;36(Suppl 1):S159‐S169.
    1. Schneeweiss S, Shrank WH, Ruhl M, Maclure M. Decision‐Making Aligned with Rapid‐Cycle Evaluation in Health Care. Int J Technol Assess Health Care. Jan 2015;31(4):214‐222.
    1. Public Policy Committee, International Society for Pharmacoepidemiology. Guidelines for good pharmacoepidemiology practice (GPP). Pharmacoepidemiol Drug Saf. 2016;25(1):2‐10.
    1. Wang SV, Verpillat P, Rassen JA, Patrick A, Garry EM, Bartels DB. Transparency and Reproducibility of Observational Cohort Studies Using Large Healthcare Databases. Clin Pharmacol Ther Mar. 2016;99(3):325‐332.
    1. Strom BL, Buyse M, Hughes J, Knoppers BM. Data sharing, year 1‐‐access to data from industry‐sponsored clinical trials. N Engl J Med Nov 27. 2014;371(22):2052‐2054.
    1. Science CfO . Open Science Framework now a recommended repository for the Nature Publishing Group's data journal, Scientific Data. , 2016.
    1. Observational Medical Outcomes Partnership. /, 2016.
    1. Innovation in Medical Evidence Development and Surveillance (IMEDS). , 2016.
    1. Mini‐Sentinel Prospective Routine Observational Monitoring Program Tools (PROMPT): Cohort Matching, SAS Code Package, Version 1.0. June 2014.
    1. Essential Evidence for Healthcare. 2016; .
    1. Center M‐SC . Routine Querying Tools (Modular Programs). 2014; . Accessed June 12, 2017.
    1. EMIF . EMIF Platform. .
    1. PROTECT. /.
    1. Aetion, Inc.'s Comparative Effectiveness and Safety platforms version 2.2 /, July 7, 2017.
    1. Observational Health Data Sciences and Informatics Atlas version 2.0.0 ) Accessed July 7, 2017.
    1. Observational Health Data Sciences and Informatics CohortMethod version 2.2.2 . Accessed July 7, 2017.
    1. Innovation in Medical Evidence in Development and Surveillance (IMEDs) Regularized Identification of Cohorts (RICO) version 1.2 . Accessed July 7, 2017.
    1. Mini‐Sentinel I . Mini‐Sentinel: Overview and Description of the Common Data Model v5.0.1. . Accessed 2016.
    1. OMOP Common Data Model. . Accessed March 21, 2016.
    1. Langan SM, Benchimol EI, Guttmann A, et al. Setting the RECORD straight: developing a guideline for the REporting of studies Conducted using Observational Routinely collected Data. Clin Epidemiol. 2013;5:29‐31.
    1. Ray WA. Evaluating Medication Effects Outside of Clinical Trials: New‐User Designs. Am J Epidemiol November 1, 2003. 2003;158(9):915‐920.
    1. Rothman KJ. Induction and latent periods. Am J Epidemiol. 1981;114(2):253‐259.
    1. Maclure M. The case‐crossover design: a method for studying transient effects on the risk of acute events. Am J Epidemiol. Jan 15. 1991;133(2):144‐153.
    1. Suissa S. Immortal time bias in pharmaco‐epidemiology. Am J Epidemiol. Feb 15. 2008;167(4):492‐499.
    1. Brookhart MA, Sturmer T, Glynn RJ, Rassen J, Schneeweiss S. Confounding control in healthcare database research: challenges and potential approaches. Med Care Jun. 2010;48(6 Suppl):S114‐S120.
    1. Glynn RJ, Schneeweiss S, Sturmer T. Indications for propensity scores and review of their use in pharmacoepidemiology. Basic Clin Pharmacol Toxicol Mar. 2006;98(3):253‐259.
    1. Agency EM . The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) 2014; . Accessed December 13, 2014.

Source: PubMed

Patrocinadores e Colaboradores

Condições médicas

Intervenções de drogas

3
Se inscrever