Blinded Validation of Breath Biomarkers of Lung Cancer, a Potential Ancillary to Chest CT Screening

Abstract

Background: Breath volatile organic compounds (VOCs) have been reported as biomarkers of lung cancer, but it is not known if biomarkers identified in one group can identify disease in a separate independent cohort. Also, it is not known if combining breath biomarkers with chest CT has the potential to improve the sensitivity and specificity of lung cancer screening.

Methods: Model-building phase (unblinded): Breath VOCs were analyzed with gas chromatography mass spectrometry in 82 asymptomatic smokers having screening chest CT, 84 symptomatic high-risk subjects with a tissue diagnosis, 100 without a tissue diagnosis, and 35 healthy subjects. Multiple Monte Carlo simulations identified breath VOC mass ions with greater than random diagnostic accuracy for lung cancer, and these were combined in a multivariate predictive algorithm. Model-testing phase (blinded validation): We analyzed breath VOCs in an independent cohort of similar subjects (n = 70, 51, 75 and 19 respectively). The algorithm predicted discriminant function (DF) values in blinded replicate breath VOC samples analyzed independently at two laboratories (A and B). Outcome modeling: We modeled the expected effects of combining breath biomarkers with chest CT on the sensitivity and specificity of lung cancer screening.

Results: Unblinded model-building phase. The algorithm identified lung cancer with sensitivity 74.0%, specificity 70.7% and C-statistic 0.78. Blinded model-testing phase: The algorithm identified lung cancer at Laboratory A with sensitivity 68.0%, specificity 68.4%, C-statistic 0.71; and at Laboratory B with sensitivity 70.1%, specificity 68.0%, C-statistic 0.70, with linear correlation between replicates (r = 0.88). In a projected outcome model, breath biomarkers increased the sensitivity, specificity, and positive and negative predictive values of chest CT for lung cancer when the tests were combined in series or parallel.

Conclusions: Breath VOC mass ion biomarkers identified lung cancer in a separate independent cohort, in a blinded replicated study. Combining breath biomarkers with chest CT could potentially improve the sensitivity and specificity of lung cancer screening.

Trial registration: ClinicalTrials.gov NCT00639067.

Conflict of interest statement

Competing Interests: Michael Phillips is President and CEO of Menssana Research, Inc. Renee N. Cataneo and Mayur Mundada are employed by Menssana Research, Inc.

Figures

Fig 1. Overview of study design.

Fig 2. Breath VOC sample analysis.

Total ion chromatogram of breath VOCs (upper panel) [3, 24]. VOCs are thermally desorbed from the sorbent trap, separated by gas chromatography, and injected into a mass sensitive detector where they are bombarded with energetic electrons in a vacuum and degraded into a set of ionic fragments, each with its own mass/charge (m/z) ratio. This figure displays the total ion current as a function of time, as a series of VOCs enter the detector sequentially. The total ion current from a peak containing toluene is marked, and the mass spectrum of the constituent mass ions is shown in the lower panel. A typical total ion chromatogram derived from a sample of human breath VOCs usually displays ~150 to 200 separate peaks. Mass spectrum of ions in a chromatograph peak (lower panel). The mass spectrum of ions derived from toluene (shown in the middle panel) comprises a characteristic pattern of fragments. Matching this pattern to a similar mass spectrum in a computer-based library enables identification of the chemical structure of the source VOC. In complex mixtures like breath, identification is usually tentative because biomarkers may be misidentified if co-eluting VOCs contaminate a mass spectrum, and if the spectral pattern matches inexactly with a library standard. However, individual mass ions from a VOC can be identified with confidence and provide robust biomarkers even when the identity of the parent VOC biomarker is uncertain.

Fig 3. Unblinded development of predictive algorithm.

Monte Carlo statistical analysis of mass ions (top panel) A list of more than 70,000 candidate mass ion biomarkers of lung cancer was obtained from a series of 5 sec segments in aligned chromatograms. The diagnostic accuracy of each mass ion was quantified by its C-statistic i.e. by the area under curve (AUC) of its associated receiver operating characteristic (ROC) curve (the “Correct assignment” curve). In order to exclude false biomarkers, the ‘‘Random assignment” curve employed multiple Monte Carlo simulations comprising 40 random assignments of diagnosis (“cancer” or “cancer-free”) to determine the random behavior of each candidate mass ion. The cutoff point in the “Correct assignment” curve was taken as the vertical intercept of the point where the number of mass ions in the ‘‘Random assignment” curve declined to zero (at C-statistic = 0.63). At this point, the vertical distance between the two curves indicated that 544 mass ions identified lung cancer with greater than random accuracy, and the separation between the curves exceeded 5 sigma. Linear clustering of mass ion biomarkers (middle panel). This figure displays vertical and horizontal linear clustering in a group of mass ion biomarkers of lung cancer with retention times between 1,500 and 2,500 sec. These mass ions were identified by Monte Carlo statistical analysis (upper panel) as having C-statistic values that were greater than random. M/z is the mass divided by the charge number of an ion, and the retention time indicates when a VOC eluted from the GC column and entered the MS detector where it was bombarded with electrons and converted to mass ion fragments. Vertical linear clusters indicate mass ions with similar retention times. These groupings are consistent with one or more breath VOCs entering the MS detector simultaneously, prior to breakdown to mass ions. This observation suggests that a comparatively small number of parent breath VOCs may account for several of the mass ion biomarkers of lung cancer. Horizontal linear clusters with m/z values of 43 and 57 are consistent with breakdown products of alkanes and methylated alkanes. Receiver operating characteristic (ROC) curve (bottom panel). The AUC of a ROC curve (or its C-statistic) indicates the overall accuracy of a test, and may vary from 0.5 (a straight line from bottom left to top right of the graph) to 1.0 (a right angle with its apex at the top left of the graph). A C-statistic of 0.5 indicates that the test performance was no better than random e.g. flipping a coin, while a C-statistic of 1.0 indicates a perfect test with 100% sensitivity and specificity. In clinical practice, a C-statistic of 0.78 is generally regarded as clinically useful.

Fig 4. Blinded prediction of lung cancer.

Inter-laboratory concordance of discriminant functions (DF) in replicate samples (top panel). DF values of chromatograms analyzed at laboratory A were plotted as a function of the DF value of the duplicate sample analyzed at laboratory B. There was a linear relationship between the two sets of DF values (r = 0.88, 95% confidence intervals shown). Predicted sensitivity and specificity in subjects with biopsy-proven lung cancer and chest CT negative for lung cancer (middle panel). The DF value derived from the predictive algorithm provides a variable cutoff point for the breath test. Test results greater than a DF value were scored as positive for lung cancer while those less than the DF were scored as negative. When DF = 0, the test has 100% sensitivity because all results are scored as positive for lung cancer, but zero specificity because no results are scored as negative. The sum of sensitivity plus specificity is maximal at the point where the two curves intersect, and was therefore selected as the optimal DF cutoff value for a binary test (i.e. cancer versus no cancer). In this graph (results from Laboratory A), the curves intersected at DF = 22, with sensitivity 68.0% and specificity 68.4%. ROC curves (lower panel). The ROC curves of the predicted outcomes of the breath test are shown for samples analyzed at laboratories A and B. The overall accuracy (C-statistic) of the lung cancer predictions was similar at both sites.

Fig 5. Projected outcome of chest CT combined with breath testing.

These predictions employ values reported in the National Lung Screening Trial for lung cancer prevalence (1.1%) and screening chest CT (sensitivity 93.8%, specificity 73.4%) [18]. Effect of combining two tests (top left panel). TP = true positives, FN = false negatives, TN = true negatives, FP = false positives. The equations demonstrate the effects on sensitivity and specificity when two tests A and B are combined. If the diagnostic criterion is a positive test result for both test A and test B, then sensitivity decreases and specificity increases, compared to either test employed alone. If the diagnostic criterion is a positive test result for either test A or test B, then sensitivity increases and specificity decreases, compared to either test employed alone. The figure demonstrates the expected outcome of lung cancer screening in one million high-risk people (smokers or former smokers aged 50 yr or older). The main limiting factor in population screening programs is the potentially overwhelming number of false-positive test results. Screening one million people with chest CT alone would result in 263,074 false positive test results, but if chest CT and breath testing are both positive, the increased specificity would reduce this number to 88,919 i.e. by 66.2%. If only one of the tests is positive, then the increased sensitivity would reduce the number of false-negatives from 682 to 198 i.e. by 71.0%. Effect of parallel and series testing on sensitivity and specificity (top right panel). This figure displays the expected improvement in sensitivity and specificity of chest CT for lung cancer if it is combined in parallel with a breath testing. If both tests are positive for lung cancer, then specificity increases from 73.4% to 91.49%. If either test is positive, then sensitivity increases from 93.8% to 98.15%. If the two tests are employed in series and the breath test is negative, there may be no need to proceed to chest CT because 98.15% sensitivity is greater than the sensitivity of either test employed alone. Positive predictive value (PPV) of chest CT combined with breath testing (bottom left panel). This figure displays the expected improvement in PPV of chest CT for lung cancer if combined in parallel with a breath test. Employed alone, the PPV of chest CT is 3.77%. If breath testing is employed in parallel with chest CT and both tests are positive, then the PPV increases to 7.91% i.e. it increases by a factor of 2.1. The improvement is due to the higher specificity of the combined test and the consequent reduction in false positive results. The PPV of a test depends upon the prevalence (prev) of a disease, and is computed as PPV = (sen X prev)/[(sen X prev + (1-spec) X (1-prev)]. The PPV of chest CT for lung cancer is 3.77% [i.e. 0.938 X 011/(0.938 X.011+(1–0.734 X (1–0.011)) = 0.0377]. Negative predictive value (NPV) of chest CT combined with breath testing (bottom right panel). If the two tests are employed in series, a negative breath test result rules out lung cancer with NPV 99.6%, which is greater than the NPV of either test employed alone. Despite the increased sensitivity of the combined test, only a modest increment in NPV is possible because the pre-test NPV based on prevalence of lung cancer is 98.9%.