Using a web-based application to define the accuracy of diagnostic tests when the gold standard is imperfect

Cherry Lim, Prapass Wannapinij, Lisa White, Nicholas P J Day, Ben S Cooper, Sharon J Peacock, Direk Limmathurotsakul, Cherry Lim, Prapass Wannapinij, Lisa White, Nicholas P J Day, Ben S Cooper, Sharon J Peacock, Direk Limmathurotsakul

Abstract

Background: Estimates of the sensitivity and specificity for new diagnostic tests based on evaluation against a known gold standard are imprecise when the accuracy of the gold standard is imperfect. Bayesian latent class models (LCMs) can be helpful under these circumstances, but the necessary analysis requires expertise in computational programming. Here, we describe open-access web-based applications that allow non-experts to apply Bayesian LCMs to their own data sets via a user-friendly interface.

Methods/principal findings: Applications for Bayesian LCMs were constructed on a web server using R and WinBUGS programs. The models provided (http://mice.tropmedres.ac) include two Bayesian LCMs: the two-tests in two-population model (Hui and Walter model) and the three-tests in one-population model (Walter and Irwig model). Both models are available with simplified and advanced interfaces. In the former, all settings for Bayesian statistics are fixed as defaults. Users input their data set into a table provided on the webpage. Disease prevalence and accuracy of diagnostic tests are then estimated using the Bayesian LCM, and provided on the web page within a few minutes. With the advanced interfaces, experienced researchers can modify all settings in the models as needed. These settings include correlation among diagnostic test results and prior distributions for all unknown parameters. The web pages provide worked examples with both models using the original data sets presented by Hui and Walter in 1980, and by Walter and Irwig in 1988. We also illustrate the utility of the advanced interface using the Walter and Irwig model on a data set from a recent melioidosis study. The results obtained from the web-based applications were comparable to those published previously.

Conclusions: The newly developed web-based applications are open-access and provide an important new resource for researchers worldwide to evaluate new diagnostic tests.

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1. Schematic diagram of the web-based…
Figure 1. Schematic diagram of the web-based application (http://mice.tropmedres.ac).
(A) Users input the data set and settings into a table provided on the webpage, (B) The central web server invisibly transforms the data set and settings inputted into multiple text files suitable for the statistical software, and automatically runs the Bayesian latent class models (LCM) using the R and WinBUGS programs. (C) The results estimated by Bayesian LCM are provided on the webpage within few minutes.
Figure 2. Input and output screen for…
Figure 2. Input and output screen for the simplified interface of the two-tests in two-population model (Hui and Walter model) provided on the website (http://mice.tropmedres.ac).
See text for details.
Figure 3. Input and output screen for…
Figure 3. Input and output screen for the simplified interface of the three-tests in one-population model (Walter and Irwig model) provided on the website (http://mice.tropmedres.ac).
See text for details.

References

    1. Peeling RW, Smith PG, Bossuyt PM (2010) A guide for diagnostic evaluations. Nat Rev Microbiol 8: S2-S6. doi:10.1038/nrmicro2294. PubMed: .
    1. Rutjes AW, Reitsma JB, Coomarasamy A, Khan KS, Bossuyt PM (2007) Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess 11: iii, ix-51
    1. Enøe C, Georgiadis MP, Johnson WO (2000) Estimation of sensitivity and specificity of diagnostic tests and disease prevalence when the true disease state is unknown. Prev Vet Med 45: 61-81. doi:10.1016/S0167-5877(00)00117-3. PubMed: .
    1. Ochola LB, Vounatsou P, Smith T, Mabaso ML, Newton CR (2006) The reliability of diagnostic techniques in the diagnosis and management of malaria in the absence of a gold standard. Lancet Infect Dis 6: 582-588. doi:10.1016/S1473-3099(06)70579-5. PubMed: .
    1. Speybroeck N, Praet N, Claes F, Van Hong N, Torres K et al. (2011) True versus apparent malaria infection prevalence: the contribution of a Bayesian approach. PLOS ONE 6: e16705. doi:10.1371/journal.pone.0016705. PubMed: .
    1. Bernatsky S, Joseph L, Bélisle P, Boivin JF, Rajan R et al. (2005) Bayesian modelling of imperfect ascertainment methods in cancer studies. Statist Med 24: 2365-2379. doi:10.1002/sim.2116. PubMed: .
    1. Walter SD, Macaskill P, Lord SJ, Irwig L (2012) Effect of dependent errors in the assessment of diagnostic or screening test accuracy when the reference standard is imperfect. Statist Med 31: 1129-1138. doi:10.1002/sim.4444.
    1. Richeldi L (2006) An update on the diagnosis of tuberculosis infection. Am J Respir Crit Care Med 174: 736-742. doi:10.1164/rccm.200509-1516PP. PubMed: .
    1. Sadatsafavi M, Shahidi N, Marra F, FitzGerald MJ, Elwood KR et al. (2010) A statistical method was used for the meta-analysis of tests for latent TB in the absence of a gold standard, combining random-effect and latent-class methods to estimate test accuracy. J Clin Epidemiol 63: 257-269. doi:10.1016/j.jclinepi.2009.04.008. PubMed: .
    1. Buck AA, Gart JJ (1966) Comparison of a screening test and a reference test in epidemiologic studies. I. Indices of agreement and their relation to prevalence. Am J Epidemiol 83: 586-592. PubMed: .
    1. Gart JJ, Buck AA (1966) Comparison of a screening test and a reference test in epidemiologic studies. II. A probabilistic model for the comparison of diagnostic tests. Am J Epidemiol 83: 593-602. PubMed: .
    1. Hui SL, Walter SD (1980) Estimating the error rates of diagnostic tests. Biometrics 36: 167-171. doi:10.2307/2530508. PubMed: .
    1. Walter SD, Irwig LM (1988) Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review. J Clin Epidemiol 41: 923-937. doi:10.1016/0895-4356(88)90110-2. PubMed: .
    1. Joseph L, Gyorkos TW, Coupal L (1995) Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. Am J Epidemiol 141: 263-272. PubMed: .
    1. Johnson WO, Gastwirth JL, Pearson LM (2001) Screening without a "gold standard": the Hui-Walter paradigm revisited. Am J Epidemiol 153: 921-924. doi:10.1093/aje/153.9.921. PubMed: .
    1. Manning L, Laman M, Rosanas-Urgell A, Turlach B, Aipit S et al. (2012) Rapid antigen detection tests for malaria diagnosis in severely ill Papua New Guinean children: a comparative study using Bayesian latent class models. PLOS ONE 7: e48701. doi:10.1371/journal.pone.0048701. PubMed: .
    1. Gonçalves L, Subtil A, de Oliveira MR, do Rosário V, Lee PW et al. (2012) Bayesian Latent Class Models in malaria diagnosis. PLOS ONE 7: e40633. doi:10.1371/journal.pone.0040633. PubMed: .
    1. Dendukuri N, Schiller I, Joseph L, Pai M (2012) Bayesian meta-analysis of the accuracy of a test for tuberculous pleuritis in the absence of a gold standard reference. Biometrics 68: 1285-1293. doi:10.1111/j.1541-0420.2012.01773.x. PubMed: .
    1. Page AL, Alberti KP, Mondonge V, Rauzier J, Quilici ML et al. (2012) Evaluation of a rapid test for the diagnosis of cholera in the absence of a gold standard. PLOS ONE 7: e37360. doi:10.1371/journal.pone.0037360. PubMed: .
    1. Niederkohr RD, Levin LA (2007) A Bayesian analysis of the true sensitivity of a temporal artery biopsy. Invest Ophthalmol Vis Sci 48: 675-680. doi:10.1167/iovs.06-1106. PubMed: .
    1. Mossman D, Wygant DB, Gervais RO (2012) Estimating the accuracy of neurocognitive effort measures in the absence of a "gold standard". Psychol Assess 24: 815-822. doi:10.1037/a0028195. PubMed: .
    1. Limmathurotsakul D, Jamsen K, Arayawichanont A, Simpson JA, White LJ et al. (2010) Defining the true sensitivity of culture for the diagnosis of melioidosis using Bayesian latent class models. PLOS ONE 5: e12485. doi:10.1371/journal.pone.0012485. PubMed: .
    1. Limmathurotsakul D, Turner EL, Wuthiekanun V, Thaipadungpanit J, Suputtamongkol Y et al. (2012) Fool's gold: Why imperfect reference tests are undermining the evaluation of novel diagnostics: a reevaluation of 5 diagnostic tests for leptospirosis. Clin Infect Dis. 55: 322-331. doi:10.1093/cid/cis403. PubMed: .
    1. Pan-Ngum W, Blacksell SD, Lubell Y, Pukrittayakamee S, Bailey MS et al. (2013) Estimating the true accuracy of diagnostic tests for dengue infection using bayesian latent class models. PLOS ONE 8: e50765. doi:10.1371/journal.pone.0050765. PubMed: .
    1. R Development Core Team (2008) : A language and environment for statistical computing; Vienna, Austria. Available: . Accessed 1 November 2013.
    1. Gilks WR, Richardson S, Spiegelhalter D (1996) Markov chain Monte Carlo in practice. Chapman and Hall. 512 p.
    1. Joseph L (2012) BayesDiagnosticTests, Estimation of Disease Prevalence and the Parameters of One, Two or Three Diagnostic Tests. Available: . Accessed 1 November 2013.
    1. Dendukuri N, Joseph L (2001) Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests. Biometrics 57: 158-167. doi:10.1111/j.0006-341X.2001.00158.x. PubMed: .
    1. Toft N, Jørgensen E, Højsgaard S (2005) Diagnosing diagnostic tests: evaluating the assumptions underlying the estimation of sensitivity and specificity in the absence of a gold standard. Prev Vet Med 68: 19-33. doi:10.1016/j.prevetmed.2005.01.006. PubMed: .
    1. Zhou XH, Obuchowski NA, McClish DK (2008) Methods for Correcting Imperfect Standard Bias. In: Statistical Methods in Diagnostic Medicine. New York: John Wiley & Sons; . pp 307-348
    1. Lunn D, Spiegelhalter D, Thomas A, Best N (2009) The BUGS project: Evolution, critique and future directions. Stat Med 28: 3049-3067. doi:10.1002/sim.3680. PubMed: .
    1. Berkvens D, Speybroeck N, Praet N, Adel A, Lesaffre E (2006) Estimating disease prevalence in a Bayesian framework using probabilistic constraints. Epidemiology 17: 145-153. doi:10.1097/01.ede.0000198422.64801.8d. PubMed: .
    1. Branscum AJ, Gardner IA, Johnson WO (2005) Estimation of diagnostic-test sensitivity and specificity through Bayesian modeling. Prev Vet Med 68: 145-163. doi:10.1016/j.prevetmed.2004.12.005. PubMed: .
    1. Cheng AC, Currie BJ, Dance DA, Funnell SG, Limmathurotsakul D et al. (2013) Clinical Definitions of Melioidosis. Am J Trop Med Hyg 88: 411-413. doi:10.4269/ajtmh.12-0555. PubMed: .
    1. Cheng AC, Peacock SJ, Limmathurotsakul D, Wongsuvan G, Chierakul W et al. (2006) Prospective evaluation of a rapid immunochromogenic cassette test for the diagnosis of melioidosis in northeast Thailand. Trans R Soc Trop Med Hyg 100: 64-67. doi:10.1016/j.trstmh.2005.04.019. PubMed: .
    1. Chantratita N, Wuthiekanun V, Thanwisai A, Limmathurotsakul D, Cheng AC et al. (2007) Accuracy of enzyme-linked immunosorbent assay using crude and purified antigens for serodiagnosis of melioidosis. Clin Vaccine Immunol 14: 110-113. doi:10.1128/CVI.00289-06. PubMed: .
    1. Lunn D, Thomas A, Best N, Spigelhalter D (2000) WINBUGS - a Bayesian modelling framework: concepts, structure, and extensibility. Statist Comput 10: 325-337. doi:10.1023/A:1008929526011.
    1. Trikalinos TA, Balion CM (2012) Chapter 9: options for summarizing medical test performance in the absence of a "gold standard". Journal of general internal medicine 27 Suppl 1: S67-75. doi:10.1007/s11606-012-2031-7. PubMed: .

Source: PubMed

Sponsor e collaboratori

Condizioni mediche

Interventi farmacologici

3
Sottoscrivi