External validation of the Chicago Chiari Outcome Scale

Abstract

Object: Historically, assessment of clinical outcomes following surgical management of Chiari malformation Type I (CM-I) has been challenging due to the lack of a validated instrument for widespread use. The Chicago Chiari Outcome Scale (CCOS) is a novel system intended to provide a less subjective evaluation of outcomes for patients with CM-I. The goal of this study was to externally validate the performance of the CCOS.

Methods: Patients undergoing surgery for CM-I between 2001 and 2012 were reviewed (n = 292). Inclusion criteria for this study were as follows: 1) patients receiving primary posterior fossa decompression; 2) at least 5.5 months of postoperative clinical follow-up; and 3) patients ≤ 18 years of age at the time of surgery. Outcomes were evaluated using the CCOS, along with a "gestalt" impression of whether patients experienced significant improvement after surgery. A subgroup of 118 consecutive patients undergoing operations between 2008 and 2010 was selected for analysis of interrater reliability (n = 73 meeting inclusion/exclusion criteria). In this subgroup, gestalt and CCOS scores were independently determined by 2 reviewers, and interrater reliability was assessed using the intraclass correlation coefficient (ICC) and kappa (κ) statistic.

Results: The median CCOS score was 14, and 67% of patients had improved gestalt scores after surgery. Overall, the CCOS was effective at identifying patients with improved outcome after surgery (area under curve = 0.951). The interrater reliability of the CCOS (ICC = 0.71) was high, although the reliability of the component scores ranged from poor to good (ICC 0.23-0.89). The functionality subscore demonstrated a low ICC and did not add to the predictive ability of the logistic regression model (likelihood ratio = 1.8, p = 0.18). When analyzing gestalt outcome, there was moderate agreement between raters (κ = 0.56).

Conclusions: In this external validation study, the CCOS was effective at identifying patients with improved outcomes and proved more reliable than the authors' gestalt impression of outcome. However, certain component subscores (functionality and nonpain symptoms) were found to be less reliable, and may benefit from further definition in score assignment. In particular, the functionality subscore does not add to the predictive ability of the CCOS, and may be unnecessary. Overall, the authors found the CCOS to be an improvement over the previously used assessment of outcome at their institution.

Keywords: CCOS = Chicago Chiari Outcome Scale; CM-I = Chiari malformation Type I; Chiari malformation; ICC = intraclass correlation coefficient; ROC = receiver operating characteristic; SLCH = St. Louis Children's Hospital; outcome assessment; posterior fossa decompression.

Figures

Figure 1. Comparison of CCOS and gestalt outcomes

Clear alignment of higher CCOS with better gestalt outcome is shown, with all patients scoring 15 or 16 on CCOS having improved gestalt outcome. Conversely, all except one patient with CCOS 10 or lower were worse than preoperative status on gestalt assessment.

Figure 2. Receiver operator curve for CCOS

ROC had an area under the curve of 0.951, showing excellent fit of the CCOS to predict outcome on the gestalt assessment.