ASPECTS Interobserver Agreement of 100 Investigators from the TENSION Study

Noel van Horn, Helge Kniep, Gabriel Broocks, Lukas Meyer, Fabian Flottmann, Matthias Bechstein, Julia Götz, Götz Thomalla, Martin Bendszus, Susanne Bonekamp, Johannes Alex Rolf Pfaff, Paulo Roberto Dellani, Jens Fiehler, Uta Hanning, Noel van Horn, Helge Kniep, Gabriel Broocks, Lukas Meyer, Fabian Flottmann, Matthias Bechstein, Julia Götz, Götz Thomalla, Martin Bendszus, Susanne Bonekamp, Johannes Alex Rolf Pfaff, Paulo Roberto Dellani, Jens Fiehler, Uta Hanning

Abstract

Purpose: Evaluating the extent of cerebral ischemic infarction is essential for treatment decisions and assessment of possible complications in patients with acute ischemic stroke. Patients are often triaged according to image-based early signs of infarction, defined by Alberta Stroke Program Early CT Score (ASPECTS). Our aim was to evaluate interrater reliability in a large group of readers.

Methods: We retrospectively analyzed 100 investigators who independently evaluated 20 non-contrast computed tomography (NCCT) scans as part of their qualification program for the TENSION study. Test cases were chosen by four neuroradiologists who had previously scored NCCT scans with ASPECTS between 0 and 8 and high interrater agreement. Percent and interrater agreements were calculated for total ASPECTS, as well as for each ASPECTS region.

Results: Percent agreements for ASPECTS ratings was 28%, with interrater agreement of 0.13 (95% confidence interval, CI 0.09-0.16), at zero tolerance allowance and 66%, with interrater agreement of 0.32 (95% CI: 0.21-0.44), at tolerance allowance set by TENSION inclusion criteria. ASPECTS region with highest level of agreement was the insular cortex (percent agreement = 96%, interrater agreement = 0.96 (95% CI: 0.94-0.97)) and with lowest level of agreement the M3 region (percent agreement = 68%, interrater agreement = 0.39 [95% CI: 0.17-0.61]).

Conclusion: Interrater agreement reliability for total ASPECTS and study enrollment was relatively low but seems sufficient for practical application. Individual region analysis suggests that some are particularly difficult to evaluate, with varying levels of reliability. Potential impairment of the supraganglionic region must be examined carefully, particularly with respect to the decision whether or not to perform mechanical thrombectomy.

Keywords: Acute stroke therapy; Brain; Endovascular treatment; Interrater reliability; Ischemic stroke; Krippendorff’s α.

Conflict of interest statement

G. Thomalla: consulting fees from Acandis, grant support and lecture fees from Bayer, lecture fees from Boehringer Ingelheim, Bristol-Myers Squibb/Pfizer, and Daiichi Sankyo, and consulting fees and lecture fees from Stryker. Grants from Bundesministerium für Wirtschaft und Energie (BMWi), Deutsche Forschungsgemeinschaft (DFG), European Union (EU), German Innovation Fund, Corona Foundation. J.A.R. Pfaff: personal fees from Stryker, outside the submitted work. J. Fiehler: consultant for Acandis, Boehringer Ingelheim, Codman, Microvention, Sequent, Stryker. Speaker for Bayer Healthcare, Bracco, Covidien/ev3, Penumbra, Philips, Siemens. Grants from Bundesministeriums für Wirtschaft und Energie (BMWi), Bundesministerium für Bildung und Forschung (BMBF), Deutsche Forschungsgemeinschaft (DFG), European Union (EU), Covidien, Stryker (THRILL study), Microvention (ERASER study), Philips. N. van Horn, H. Kniep, G. Broocks, L. Meyer, F. Flottmann, M. Bechstein, J. Götz, M. Bendszus, S. Bonekamp, P.R. Dellani, and U. Hanning declare that they have no competing interests.

© 2021. The Author(s).

Figures

Fig. 1
Fig. 1
Frequency distribution of Alberta Stroke Program Early CT Score (ASPECTS) ratings for each of the 20 cases
Fig. 2
Fig. 2
Illustrates ASPECTS per region agreement on the level of the basal ganglia (a) and immediately above (b) calculated using Gwet’s AC1 on NCCT with low (red), moderate (yellow) and high (green) agreement. I insula, IC internal capsule, C caput, L lentiform, M1–M6 media cerebral artery territory M1–M6, ASPECTS Alberta Stroke Program Early CT Score, NCCT non-contrast computed tomography
Fig. 3
Fig. 3
Demonstrates rating concordance among all raters regarding ASPECTS rating of all 20 cases for each ASPECTS region, mean concordance for all cases and concordance for enrolment decision for mechanical thrombectomy. Mean ASPECTS is illustrated according to ASPECTS with standard deviation (standard dev) sorted by lowest to highest standard dev (blue, high concordance; red, low concordance). I insula, IC internal capsule, C caput, L lentiform, M1–M6 media cerebral artery territory M1–M6, ASPECTS Alberta Stroke Program Early CT Score

References

    1. Barber PA, Demchuk AM, Zhang J, Buchan AM. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy. ASPECTS Study Group. Alberta Stroke Programme Early CT Score. Lancet. 2000;355:1670–1674. doi: 10.1016/S0140-6736(00)02237-6.
    1. Yoo AJ, Zaidat OO, Chaudhry ZA, Berkhemer OA, González RG, Goyal M, Demchuk AM, Menon BK, Mualem E, Ueda D, Buell H, Sit SP, Bose A, Penumbra Pivotal and Penumbra Imaging Collaborative Study (PICS) Investigators Impact of pretreatment noncontrast CT Alberta Stroke Program Early CT Score on clinical outcome after intra-arterial stroke therapy. Stroke. 2014;45:746–751. doi: 10.1161/STROKEAHA.113.004260.
    1. Goyal M, Menon BK, Coutts SB, Hill MD, Demchuk AM, Penumbra Pivotal Stroke Trial Investigators and the Seaman MR Research Center Effect of baseline CT scan appearance and time to recanalization on clinical outcomes in endovascular thrombectomy of acute ischemic strokes. Stroke. 2011;42:93–97. doi: 10.1161/STROKEAHA.110.594481.
    1. Powers WJ, Rabinstein AA, Ackerson T, Adeoye OM, Bambakidis NC, Becker K, Biller J, Brown M, Demaerschalk BM, Hoh B, Jauch EC, Kidwell CS, Leslie-Mazwi TM, Ovbiagele B, Scott PA, Sheth KN, Southerland AM, Summers DV, Tirschwell DL. Guidelines for the Early Management of Patients With Acute Ischemic Stroke: 2019 Update to the 2018 Guidelines for the Early Management of Acute Ischemic Stroke: A Guideline for Healthcare Professionals From the American Heart Association/American Stroke Association. Stroke. 2019;50:e344–418. doi: 10.1161/STROKEAHA.118.022606.
    1. Turc G, Bhogal P, Fischer U, Khatri P, Lobotesis K, Mazighi M, Schellinger PD, Toni D, de Vries J, White P. European Stroke Organisation (ESO)—European Society for Minimally Invasive Neurological Therapy (ESMINT) Guidelines on Mechanical Thrombectomy in Acute Ischemic Stroke. J Neurointerv Surg. 2019 doi: 10.1136/neurintsurg-2018-014569.
    1. Jovin TG, Chamorro A, Cobo E, de Miquel MA, Molina CA, Rovira A, San Román L, Serena J, Abilleira S, Ribó M, Millán M, Urra X, Cardona P, López-Cancio E, Tomasello A, Castaño C, Blasco J, Aja L, Dorado L, Quesada H, Rubiera M, Hernandez-Pérez M, Goyal M, Demchuk AM, von Kummer R, Gallofré M, Dávalos A, REVASCAT Trial Investigators Thrombectomy within 8 hours after symptom onset in ischemic stroke. N Engl J Med. 2015;372:2296–2306. doi: 10.1056/NEJMoa1503780.
    1. Saver JL, Goyal M, Bonafe A, Diener HC, Levy EI, Pereira VM, Albers GW, Cognard C, Cohen DJ, Hacke W, Jansen O, Jovin TG, Mattle HP, Nogueira RG, Siddiqui AH, Yavagal DR, Baxter BW, Devlin TG, Lopes DK, Reddy VK, du Mesnil de Rochemont R, Singer OC, Jahan R, SWIFT PRIME Investigators Stent-retriever thrombectomy after intravenous t-PA vs. t-PA alone in stroke. N Engl J Med. 2015;372:2285–2295. doi: 10.1056/NEJMoa1415061.
    1. Campbell BC, Mitchell PJ, Kleinig TJ, Dewey HM, Churilov L, Yassi N, Yan B, Dowling RJ, Parsons MW, Oxley TJ, Wu TY, Brooks M, Simpson MA, Miteff F, Levi CR, Krause M, Harrington TJ, Faulder KC, Steinfort BS, Priglinger M, Ang T, Scroop R, Barber PA, McGuinness B, Wijeratne T, Phan TG, Chong W, Chandra RV, Bladin CF, Badve M, Rice H, de Villiers L, Ma H, Desmond PM, Donnan GA, Davis SM, EXTEND-IA Investigators Endovascular therapy for ischemic stroke with perfusion-imaging selection. N Engl J Med. 2015;372:1009–1028. doi: 10.1056/NEJMoa1414792.
    1. Goyal M, Demchuk AM, Menon BK, Eesa M, Rempel JL, Thornton J, Roy D, Jovin TG, Willinsky RA, Sapkota BL, Dowlatshahi D, Frei DF, Kamal NR, Montanera WJ, Poppe AY, Ryckborst KJ, Silver FL, Shuaib A, Tampieri D, Williams D, Bang OY, Baxter BW, Burns PA, Choe H, Heo JH, Holmstedt CA, Jankowitz B, Kelly M, Linares G, Mandzia JL, Shankar J, Sohn SI, Swartz RH, Barber PA, Coutts SB, Smith EE, Morrish WF, Weill A, Subramaniam S, Mitha AP, Wong JH, Lowerison MW, Sajobi TT, Hill MD, ESCAPE Trial Investigators Randomized assessment of rapid endovascular treatment of ischemic stroke. N Engl J Med. 2015;372:1019–1030. doi: 10.1056/NEJMoa1414905.
    1. Bracard S, Ducrocq X, Mas JL, Soudant M, Oppenheim C, Moulin T, Guillemin F, THRACE investigators Mechanical thrombectomy after intravenous alteplase versus alteplase alone after stroke (THRACE): a randomised controlled trial. Lancet Neurol. 2016;15:1138–1147. doi: 10.1016/S1474-4422(16)30177-6.
    1. Berkhemer OA, Fransen PS, Beumer D, van den Berg LA, Lingsma HF, Yoo AJ, Schonewille WJ, Vos JA, Nederkoorn PJ, Wermer MJ, van Walderveen MA, Staals J, Hofmeijer J, van Oostayen JA, Lycklama à Nijeholt GJ, Boiten J, Brouwer PA, Emmer BJ, de Bruijn SF, van Dijk LC, Kappelle LJ, Lo RH, van Dijk EJ, de Vries J, de Kort PL, van Rooij WJ, van den Berg JS, van Hasselt BA, Aerden LA, Dallinga RJ, Visser MC, Bot JC, Vroomen PC, Eshghi O, Schreuder TH, Heijboer RJ, Keizer K, Tielbeek AV, den Hertog HM, Gerrits DG, van den Berg-Vos RM, Karas GB, Steyerberg EW, Flach HZ, Marquering HA, Sprengers ME, Jenniskens SF, Beenen LF, van den Berg R, Koudstaal PJ, van Zwam WH, Roos YB, van der Lugt A, van Oostenbrugge RJ, Majoie CB, Dippel DW, MR CLEAN Investigators A randomized trial of intraarterial treatment for acute ischemic stroke. N Engl J Med. 2015;372:11–20. doi: 10.1056/NEJMoa1411587.
    1. Nogueira RG, Jadhav AP, Haussen DC, Bonafe A, Budzik RF, Bhuva P, Yavagal DR, Ribo M, Cognard C, Hanel RA, Sila CA, Hassan AE, Millan M, Levy EI, Mitchell P, Chen M, English JD, Shah QA, Silver FL, Pereira VM, Mehta BP, Baxter BW, Abraham MG, Cardona P, Veznedaroglu E, Hellinger FR, Feng L, Kirmani JF, Lopes DK, Jankowitz BT, Frankel MR, Costalat V, Vora NA, Yoo AJ, Malik AM, Furlan AJ, Rubiera M, Aghaebrahim A, Olivot JM, Tekle WG, Shields R, Graves T, Lewis RJ, Smith WS, Liebeskind DS, Saver JL, Jovin TG, DAWN Trial Investigators Thrombectomy 6 to 24 Hours after Stroke with a Mismatch between Deficit and Infarct. N Engl J Med. 2018;378:11–21. doi: 10.1056/NEJMoa1706442.
    1. Albers GW, Marks MP, Kemp S, Christensen S, Tsai JP, Ortega-Gutierrez S, McTaggart RA, Torbey MT, Kim-Tenser M, Leslie-Mazwi T, Sarraj A, Kasner SE, Ansari SA, Yeatts SD, Hamilton S, Mlynash M, Heit JJ, Zaharchuk G, Kim S, Carrozzella J, Palesch YY, Demchuk AM, Bammer R, Lavori PW, Broderick JP, Lansberg MG, DEFUSE 3 Investigators Thrombectomy for Stroke at 6 to 16 Hours with Selection by Perfusion Imaging. N Engl J Med. 2018;378:708–718. doi: 10.1056/NEJMoa1713973.
    1. Meyers PM, Schumacher HC, Higashida RT, Barnwell SL, Creager MA, Gupta R, McDougall CG, Pandey DK, Sacks D, Wechsler LR, American Heart Association Indications for the performance of intracranial endovascular neurointerventional procedures: a scientific statement from the American Heart Association Council on Cardiovascular Radiology and Intervention, Stroke Council, Council on Cardiovascular Surgery and Anesthesia, Interdisciplinary Council on Peripheral Vascular Disease, and Interdisciplinary Council on Quality of Care and Outcomes Research. Circulation. 2009;119:2235–2249. doi: 10.1161/CIRCULATIONAHA.109.192217.
    1. Nagel S, Herweh C, Pfaff JAR, Schieber S, Schönenberger S, Möhlenbruch MA, Bendszus M, Ringleb PA. Simplified selection criteria for patients with longer or unknown time to treatment predict good outcome after mechanical thrombectomy. J Neurointerv Surg. 2019;11:559–562. doi: 10.1136/neurintsurg-2018-014347.
    1. Gupta AC, Schaefer PW, Chaudhry ZA, Leslie-Mazwi TM, Chandra RV, González RG, Hirsch JA, Yoo AJ. Interobserver reliability of baseline noncontrast CT Alberta Stroke Program Early CT Score for intra-arterial stroke treatment selection. AJNR Am J Neuroradiol. 2012;33:1046–1049. doi: 10.3174/ajnr.A2942.
    1. Coutts SB, Demchuk AM, Barber PA, Hu WY, Simon JE, Buchan AM, Hill MD, VISION Study Group Interobserver variation of ASPECTS in real time. Stroke. 2004;35:e103–e105.
    1. Fleiss J, Levin B, Paik MC. Statistical methods for rates and proportions. Hoboken. Sons, Inc: John Wiley; 2003.
    1. Bendszus M, Bonekamp S, Berge E, Boutitie F, Brouwer P, Gizewski E, Krajina A, Pierot L, Randall G, Simonsen CZ, Zeleňák K, Fiehler J, Thomalla G. A randomized controlled trial to test efficacy and safety of thrombectomy in stroke with extended lesion and extended time window. Int J Stroke. 2019;14:87–93. doi: 10.1177/1747493018798558.
    1. Bendszus M, Fiehler J. New Interventional Stroke Trials. Clin Neuroradiol. 2019;29:1. doi: 10.1007/s00062-019-00763-7.
    1. Feng GC. Mistakes and how to avoid mistakes in using intercoder reliability indices. Methodology. 2015;11:13–22. doi: 10.1027/1614-2241/a000086.
    1. Gwet K. Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters. 2010.
    1. R Core Team. A language and environment for statistical computing. 2018. . Accessed 13 Nov 2019.
    1. Nicholson P, Hilditch CA, Neuhaus A, Seyedsaadat SM, Benson JC, Mark I, Tsang COA, Schaafsma J, Kallmes DF, Krings T, Brinjikji W. Per-region interobserver agreement of Alberta Stroke Program Early CT Scores (ASPECTS) J Neurointerv Surg. 2020;12:1069–1071. doi: 10.1136/neurintsurg-2019-015473.
    1. Cagnazzo F, Derraz I, Dargazanli C, Lefevre PH, Gascou G, Riquelme C, Bonafe A, Costalat V. Mechanical thrombectomy in patients with acute ischemic stroke and ASPECTS ≤6: a meta-analysis. J Neurointerv Surg. 2020;12:350–355. doi: 10.1136/neurintsurg-2019-015237.
    1. Pexman JH, Barber PA, Hill MD, Sevick RJ, Demchuk AM, Hudon ME, Hu WY, Buchan AM. Use of the Alberta Stroke Program Early CT Score (ASPECTS) for assessing CT scans in patients with acute stroke. AJNR Am J Neuroradiol. 2001;22:1534–1542.
    1. Finlayson O, John V, Yeung R, Dowlatshahi D, Howard P, Zhang L, Swartz R, Aviv RI. Interobserver agreement of ASPECT score distribution for noncontrast CT, CT angiography, and CT perfusion in acute stroke. Stroke. 2013;44:234–236. doi: 10.1161/STROKEAHA.112.665208.
    1. Ghandehari K, Rezvani MR, Shakeri MT, Mohammadifard M, Ehsanbakhsh A, Mohammadifard M, Mirgholami A, Boostani R, Ghandehari K, Izadi-Mood Z. Inter-rater reliability of modified Alberta Stroke program early computerized tomography score in patients with brain infarction. J Res Med Sci. 2011;16:1326–1331.
    1. Naylor J, Churilov L, Rane N, Chen Z, Campbell BCV, Yan B. Reliability and Utility of the Alberta Stroke Program Early Computed Tomography Score in Hyperacute Stroke. J Stroke Cerebrovasc Dis. 2017;26:2547–2552. doi: 10.1016/j.jstrokecerebrovasdis.2017.05.042.
    1. Zapf A, Castell S, Morawietz L, Karch A. Measuring inter-rater reliability for nominal data—which coefficients and confidence intervals are appropriate? BMC Med Res Methodol. 2016;16:93. doi: 10.1186/s12874-016-0200-9.
    1. Payabvash S, Souza LC, Wang Y, Schaefer PW, Furie KL, Halpern EF, Gonzalez RG, Lev MH. Regional ischemic vulnerability of the brain to hypoperfusion: the need for location specific computed tomography perfusion thresholds in acute stroke patients. Stroke. 2011;42:1255–1260. doi: 10.1161/STROKEAHA.110.600940.
    1. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37:360–363.
    1. Bristow MS, Simon JE, Brown RA, Eliasziw M, Hill MD, Coutts SB, Frayne R, Demchuk AM, Mitchell JR. MR perfusion and diffusion in acute ischemic stroke: human gray and white matter have different thresholds for infarction. J Cereb Blood Flow Metab. 2005;25:1280–1287. doi: 10.1038/sj.jcbfm.9600135.
    1. Mitomi M, Kimura K, Aoki J, Iguchi Y. Comparison of CT and DWI findings in ischemic stroke patients within 3 hours of onset. J Stroke Cerebrovasc Dis. 2014;23:37–42. doi: 10.1016/j.jstrokecerebrovasdis.2012.08.014.
    1. Sundaram VK, Goldstein J, Wheelwright D, Aggarwal A, Pawha PS, Doshi A, Fifi JT, Leacy R, Mocco J, Puig J, Nael K. Automated ASPECTS in Acute Ischemic Stroke: A Comparative Analysis with CT Perfusion. AJNR Am J Neuroradiol. 2019;40:2033–2038.
    1. Tsang ACO, Lenck S, Hilditch C, Nicholson P, Brinjikji W, Krings T, Pereira VM, Silver FL, Schaafsma JD. Automated CT Perfusion Imaging Versus Non-contrast CT for Ischemic Core Assessment in Large Vessel Occlusion. Clin Neuroradiol. 2020;30:109–114. doi: 10.1007/s00062-018-0745-6.

Source: PubMed

Sponsorit ja yhteistyökumppanit

Sairaudet

Huumeiden interventiot

3
Tilaa