How to Determine the Accuracy of an Alternative Diagnostic Test when It Is Actually Better than the Reference Tests: A Re-Evaluation of Diagnostic Tests for Scrub Typhus Using Bayesian LCMs

Cherry Lim, Daniel H Paris, Stuart D Blacksell, Achara Laongnualpanich, Pacharee Kantipong, Wirongrong Chierakul, Vanaporn Wuthiekanun, Nicholas P J Day, Ben S Cooper, Direk Limmathurotsakul, Cherry Lim, Daniel H Paris, Stuart D Blacksell, Achara Laongnualpanich, Pacharee Kantipong, Wirongrong Chierakul, Vanaporn Wuthiekanun, Nicholas P J Day, Ben S Cooper, Direk Limmathurotsakul

Abstract

Background: The indirect immunofluorescence assay (IFA) is considered a reference test for scrub typhus. Recently, the Scrub Typhus Infection Criteria (STIC; a combination of culture, PCR assays and IFA IgM) were proposed as a reference standard for evaluating alternative diagnostic tests. Here, we use Bayesian latent class models (LCMs) to estimate the true accuracy of each diagnostic test, and of STIC, for diagnosing scrub typhus.

Methods/principal findings: Data from 161 patients with undifferentiated fever were re-evaluated using Bayesian LCMs. Every patient was evaluated for the presence of an eschar, and tested with blood culture for Orientia tsutsugamushi, three different PCR assays, IFA IgM, and the Panbio IgM immunochromatographic test (ICT). True sensitivity and specificity of culture (24.4% and 100%), 56kDa PCR assay (56.8% and 98.4%), 47kDa PCR assay (63.2% and 96.1%), groEL PCR assay (71.4% and 93.0%), IFA IgM (70.0% and 83.8%), PanBio IgM ICT (72.8% and 96.8%), presence of eschar (42.7% and 98.9%) and STIC (90.5% and 82.5%) estimated by Bayesian LCM were considerably different from those obtained when using STIC as a reference standard. The IgM ICT had comparable sensitivity and significantly higher specificity compared to IFA (p=0.34 and p<0.001, respectively).

Conclusions: The low specificity of STIC was caused by the low specificity of IFA IgM. Neither STIC nor IFA IgM can be used as reference standards against which to evaluate alternative diagnostic tests. Further evaluation of new diagnostic tests should be done with a carefully selected set of diagnostic tests and appropriate statistical models.

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

References

    1. Phongmany S, Rolain JM, Phetsouvanh R, Blacksell SD, Soukkhaseum V, et al. (2006) Rickettsial infections and fever, Vientiane, Laos. Emerg Infect Dis 12: 256–262.
    1. Choi YJ, Jang WJ, Ryu JS, Lee SH, Park KH, et al. (2005) Spotted fever group and typhus group rickettsioses in humans, South Korea. Emerg Infect Dis 11: 237–244.
    1. Yasunaga H, Horiguchi H, Kuwabara K, Hashimoto H, Matsuda S (2011) Delay in tetracycline treatment increases the risk of complications in Tsutsugamushi disease: data from the Japanese Diagnosis Procedure Combination database. Internal medicine 50: 37–42.
    1. Wang CC, Liu SF, Liu JW, Chung YH, Su MC, et al. (2007) Acute respiratory distress syndrome in scrub typhus. Am J Trop Med Hyg 76: 1148–1152.
    1. Thap LC, Supanaranond W, Treeprasertsuk S, Kitvatanachai S, Chinprasatsak S, et al. (2002) Septic shock secondary to scrub typhus: characteristics and complications. The Southeast Asian journal of tropical medicine and public health 33: 780–786.
    1. Kim DM, Won KJ, Park CY, Yu KD, Kim HS, et al. (2007) Distribution of eschars on the body of scrub typhus patients: a prospective study. Am J Trop Med Hyg 76: 806–809.
    1. Shiao CC, Lin SY (2011) Eschar: a clue to scrub typhus. CMAJ 183: E1152 10.1503/cmaj.101929
    1. Koh GC, Maude RJ, Paris DH, Newton PN, Blacksell SD (2010) Diagnosis of scrub typhus. Am J Trop Med Hyg 82: 368–370. 10.4269/ajtmh.2010.09-0233
    1. Prakash JA, Kavitha ML, Mathai E (2011) Nested polymerase chain reaction on blood clots for gene encoding 56 kDa antigen and serology for the diagnosis of scrub typhus. Indian journal of medical microbiology 29: 47–50. 10.4103/0255-0857.76524
    1. Kim DM, Park G, Kim HS, Lee JY, Neupane GP, et al. (2011) Comparison of conventional, nested, and real-time quantitative PCR for diagnosis of scrub typhus. J Clin Microbiol 49: 607–612. 10.1128/JCM.01216-09
    1. Paris DH, Aukkanit N, Jenjaroen K, Blacksell SD, Day NP (2009) A highly sensitive quantitative real-time PCR assay based on the groEL gene of contemporary Thai strains of Orientia tsutsugamushi . Clinical microbiology and infection 15: 488–495. 10.1111/j.1469-0691.2008.02671.x
    1. Paris DH, Blacksell SD, Stenos J, Graves SR, Unsworth NB, et al. (2008) Real-time multiplex PCR assay for detection and differentiation of rickettsiae and orientiae. Transactions of the Royal Society of Tropical Medicine and Hygiene 102: 186–193.
    1. Blacksell SD, Jenjaroen K, Phetsouvanh R, Tanganuchitcharnchai A, Phouminh P, et al. (2010) Accuracy of rapid IgM-based immunochromatographic and immunoblot assays for diagnosis of acute scrub typhus and murine typhus infections in Laos. Am J Trop Med Hyg 83: 365–369. 10.4269/ajtmh.2010.09-0534
    1. Tay ST, Kamalanathan M, Rohani MY (2003) Detection of rickettsial antibodies using Weil-Felix (OXK and OX19) antigens and the indirect immunoperoxidase assay. Southeast Asian J Trop Med Public Health 34: 171–174.
    1. Coleman RE, Sangkasuwan V, Suwanabun N, Eamsila C, Mungviriya S, et al. (2002) Comparative evaluation of selected diagnostic assays for the detection of IgG and IgM antibody to Orientia tsutsugamushi in Thailand. Am J Trop Med Hyg 67: 497–503.
    1. Blacksell SD, Bryant NJ, Paris DH, Doust JA, Sakoda Y, et al. (2007) Scrub typhus serologic testing with the indirect immunofluorescence method as a diagnostic gold standard: a lack of consensus leads to a lot of confusion. Clin Infect Dis 44: 391–401.
    1. Paris DH, Blacksell SD, Nawtaisong P, Jenjaroen K, Teeraratkul A, et al. (2011) Diagnostic accuracy of a loop-mediated isothermal PCR assay for detection of Orientia tsutsugamushi during acute Scrub Typhus infection. PLoS Negl Trop Dis 5: e1307 10.1371/journal.pntd.0001307
    1. Blacksell SD, Paris DH, Chierakul W, Wuthiekanun V, Teeratakul A, et al. (2012) Prospective evaluation of commercial antibody-based rapid tests in combination with a loop-mediated isothermal amplification PCR assay for detection of Orientia tsutsugamushi during the acute phase of scrub typhus infection. Clin Vaccine Immunol 19: 391–395. 10.1128/CVI.05478-11
    1. Dendukuri N, Joseph L (2001) Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests. Biometrics 57: 158–167.
    1. Limmathurotsakul D, Turner EL, Wuthiekanun V, Thaipadungpanit J, Suputtamongkol Y, et al. (2012) Fool's Gold: Why Imperfect Reference Tests Are Undermining the Evaluation of Novel Diagnostics: A Reevaluation of 5 Diagnostic Tests for Leptospirosis. Clin Infect Dis 55(3):322–31. 10.1093/cid/cis403
    1. Limmathurotsakul D, Jamsen K, Arayawichanont A, Simpson JA, White LJ, et al. (2010) Defining the true sensitivity of culture for the diagnosis of melioidosis using Bayesian latent class models. PLoS One 5(8): e12485 10.1371/journal.pone.0012485
    1. Limmathurotsakul D, Chantratita N, Teerawattanasook N, Piriyagitpaiboon K, Thanwisai A, et al. (2011) Enzyme-linked immunosorbent assay for the diagnosis of melioidosis: better than we thought. Clin Infect Dis 52: 1024–1028. 10.1093/cid/cir080
    1. Trikalinos TA, Balion CM (2012) Options for Summarizing Medical Test Performance in the Absence of a "Gold Standard". Rockville MD.
    1. Lim C, Wannapinij P, White L, Day NP, Cooper BS, et al. (2013) Using a web-based application to define the accuracy of diagnostic tests when the gold standard is imperfect. PLoS One 8: e79489 10.1371/journal.pone.0079489
    1. Luksameetanasan R, Blacksell SD, Kalambaheti T, Wuthiekanun V, Chierakul W, et al. (2007) Patient and sample-related factors that effect the success of in vitro isolation of Orientia tsutsugamushi . Southeast Asian J Trop Med Public Health 38: 91–96.
    1. Horinouchi H, Murai K, Okayama A, Nagatomo Y, Tachibana N, et al. (1996) Genotypic identification of Rickettsia tsutsugamushi by restriction fragment length polymorphism analysis of DNA amplified by the polymerase chain reaction. Am J Trop Med Hyg 54: 647–651.
    1. Jiang J, Chan TC, Temenak JJ, Dasch GA, Ching WM, et al. (2004) Development of a quantitative real-time polymerase chain reaction assay specific for Orientia tsutsugamushi . Am J Trop Med Hyg 70: 351–356.
    1. Ching WM, Rowland D, Zhang Z, Bourgeois AL, Kelly D, et al. (2001) Early diagnosis of scrub typhus with a rapid flow assay using recombinant major outer membrane protein antigen (r56) of Orientia tsutsugamushi. Clin Diagn Lab Immunol 8: 409–414.
    1. Joseph L, Gyorkos TW, Coupal L (1995) Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. Am J Epidemiol 141: 263–272.
    1. Zhuo X, Obuchowski N, McClish D (2008) Statistical Methods in Diagnostic Medicine: Wiley-Blackwell; pp. 359–395.
    1. Lunn DJ, Thomas A, Best N, Spiegelhalter D (2000) WinBUGS- A Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing 10: 325–337.
    1. Sonthayanon P, Chierakul W, Wuthiekanun V, Limmathurotsakul D, Amornchai P, et al. (2013) Molecular Confirmation of Co-Infection by Pathogenic Leptospira spp. and Orientia tsutsugamushi in Patients with Acute Febrile Illness in Thailand. Am J Trop Med Hyg 89: 797–799. 10.4269/ajtmh.13-0402
    1. Liu YX, Feng D, Suo JJ, Xing YB, Liu G, et al. (2009) Clinical characteristics of the autumn-winter type scrub typhus cases in south of Shandong province, northern China. BMC infectious diseases 9: 82 10.1186/1471-2334-9-82
    1. Reisner BS, DiBlasi J, Goel N (1999) Comparison of an enzyme immunoassay to an indirect fluorescent immunoassay for the detection of antinuclear antibodies. American journal of clinical pathology 111: 503–506.
    1. Phetsouvanh R, Thojaikong T, Phoumin P, Sibounheuang B, Phommasone K, et al. (2013) Inter- and intra-operator variability in the reading of indirect immunofluorescence assays for the serological diagnosis of scrub typhus and murine typhus. Am J Trop Med Hyg 88: 932–936. 10.4269/ajtmh.12-0325
    1. Phommasone K, Paris DH, Anantatat T, Castonguay-Vanier J, Keomany S, et al. (2013) Concurrent Infection with Murine Typhus and Scrub Typhus in Southern Laos-the Mixed and the Unmixed. PLoS Negl Trop Dis 7: e2163 10.1371/journal.pntd.0002163
    1. Ching WM, Wang H, Eamsila C, Kelly DJ, Dasch GA (1998) Expression and refolding of truncated recombinant major outer membrane protein antigen (r56) of Orientia tsutsugamushi and its use in enzyme-linked immunosorbent assays. Clin Diagn Lab Immunol 5: 519–526.
    1. Parola P, Miller RS, McDaniel P, et al. (2003) Emerging rickettsioses of the Thai-Myanmar border. Emerg Infect Dis 9(5): 592–5.

Source: PubMed

스폰서 및 공동 작업자

건강 상태

약물 개입

3
구독하다