Implications of Nine Risk Prediction Models for Selecting Ever-Smokers for Computed Tomography Lung Cancer Screening

Abstract

Background: Lung cancer screening guidelines recommend using individualized risk models to refer ever-smokers for screening. However, different models select different screening populations. The performance of each model in selecting ever-smokers for screening is unknown.

Objective: To compare the U.S. screening populations selected by 9 lung cancer risk models (the Bach model; the Spitz model; the Liverpool Lung Project [LLP] model; the LLP Incidence Risk Model [LLPi]; the Hoggart model; the Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial Model 2012 [PLCOM2012]; the Pittsburgh Predictor; the Lung Cancer Risk Assessment Tool [LCRAT]; and the Lung Cancer Death Risk Assessment Tool [LCDRAT]) and to examine their predictive performance in 2 cohorts.

Design: Population-based prospective studies.

Setting: United States.

Participants: Models selected U.S. screening populations by using data from the National Health Interview Survey from 2010 to 2012. Model performance was evaluated using data from 337 388 ever-smokers in the National Institutes of Health-AARP Diet and Health Study and 72 338 ever-smokers in the CPS-II (Cancer Prevention Study II) Nutrition Survey cohort.

Measurements: Model calibration (ratio of model-predicted to observed cases [expected-observed ratio]) and discrimination (area under the curve [AUC]).

Results: At a 5-year risk threshold of 2.0%, the models chose U.S. screening populations ranging from 7.6 million to 26 million ever-smokers. These disagreements occurred because, in both validation cohorts, 4 models (the Bach model, PLCOM2012, LCRAT, and LCDRAT) were well-calibrated (expected-observed ratio range, 0.92 to 1.12) and had higher AUCs (range, 0.75 to 0.79) than 5 models that generally overestimated risk (expected-observed ratio range, 0.83 to 3.69) and had lower AUCs (range, 0.62 to 0.75). The 4 best-performing models also had the highest sensitivity at a fixed specificity (and vice versa) and similar discrimination at a fixed risk threshold. These models showed better agreement on size of the screening population (7.6 million to 10.9 million) and achieved consensus on 73% of persons chosen.

Limitation: No consensus on risk thresholds for screening.

Conclusion: The 9 lung cancer risk models chose widely differing U.S. screening populations. However, 4 models (the Bach model, PLCOM2012, LCRAT, and LCDRAT) most accurately predicted risk and performed best in selecting ever-smokers for screening.

Primary funding source: Intramural Research Program of the National Institutes of Health/National Cancer Institute.

Conflict of interest statement

Conflicts of Interest: Dr. Christine Berg receives consulting fees from Medial ES, LLC, a company that is developing algorithms from routine blood tests that may indicate an increased risk of malignancy. Two of the models compared in this manuscript were previously proposed by co-authors of this manuscript: Lung Cancer Risk Assessment Tool (LCRAT) and Lung Cancer Death Risk Assessment Tool (LCDRAT).

Figures

Figure 1:. Calibration and discrimination of all 9 models in the NIH-AARP and CPS-II

Abbreviations: NIH-AARP: NIH-AARP Diet and Health Study; CPS-II: Cancer Prevention Study II Nutrition Survey Cohort; USPSTF: US Preventive Services Task Force. PLCO M2012: Prostate, Lung, Colorectal, and Ovarian Screening Trial Model 2012; LCDRAT: Lung Cancer Death Risk Assessment Tool; LCRAT: Lung Cancer Risk Assessment Tool; LLP: Liverpool Lung Project Risk Model; LLPi: Liverpool Lung Project Incidence Risk Model. Footnote: Expected/Observed ratios less than 1 indicate underestimation of risk and Expected/Observed>1 indicates overestimation. The Area Under Curve (AUC) statistic examines how well each model discriminates risks between individuals; AUC=1 indicates perfect model prediction and AUC=0.5 indicates poor model prediction equivalent to purely random selection. AUC for the USPSTF criteria represents identifying lung cancer diagnoses within 5 years of enrolment.

Figure 2:. Number of US ever-smokers selected for CT lung-cancer screening by different risk-models at a fixed risk threshold*

Abbreviations: NLST= National Lung Screening Trial; CT= computed tomography; US= United States; USPSTF= United States Preventive Services Task Force; PLCO M2012: Prostate, Lung, Colorectal, and Ovarian Screening Trial Model 2012; LCDRAT: Lung Cancer Death Risk Assessment Tool; LCRAT: Lung Cancer Risk Assessment Tool; LLP: Liverpool Lung Project Risk Model; LLPi: Liverpool Lung Project Incidence Risk Model. Footnote: Our chosen 5-year risk-thresholds for lung-cancer and lung-cancer death (2.0% and 1.2% respectively) yield totals close to the USPSTF-eligible total (8.9 million; dotted line). 2.0% lung-cancer risk over 5-years is equivalent to approximately 0.4% of the population acquiring lung-cancer each year. Approximately equivalent risk-thresholds for those models that do not provide 5-year risks are: 4.0% risk after 10 years for Bach; 0.4% risk after 1 year for Spitz; 3.5% risk after 8.7 years for LLPi; 0.4% risk after 1 year for Hoggart; 2.4% risk after 6 years for PLCOM2012; and 2.4% after 6 years for Pittsburgh. LCRAT, LCDRAT, and LLP models make projections over 5 years, so no adjustment is needed. Footnote 2: Risk thresholds needed to select 8.9 million ever-smokers (USPSTF-eligible total) are: 4.8% risk after 10 years for Bach; 1.2% risk after 1 year for Spitz; 3.5% risk after 5 years for LLP; 8.1% risk after 8.7 years for LLPi; 1.3% risk after 1 year for Hoggart; 2.0% after 6 years for PLCOM2012; 2.9% risk after 6 years for Pittsburgh; 2.0% risk after 5 years for LCRAT; and 1.2% death risk after 5 years for LCDRAT.