Effect sizes and test-retest reliability of the fMRI-based neurologic pain signature

Abstract

Identifying biomarkers that predict mental states with large effect sizes and high test-retest reliability is a growing priority for fMRI research. We examined a well-established multivariate brain measure that tracks pain induced by nociceptive input, the Neurologic Pain Signature (NPS). In N = 295 participants across eight studies, NPS responses showed a very large effect size in predicting within-person single-trial pain reports (d = 1.45) and medium effect size in predicting individual differences in pain reports (d = 0.49). The NPS showed excellent short-term (within-day) test-retest reliability (ICC = 0.84, with average 69.5 trials/person). Reliability scaled with the number of trials within-person, with ≥60 trials required for excellent test-retest reliability. Reliability was tested in two additional studies across 5-day (N = 29, ICC = 0.74, 30 trials/person) and 1-month (N = 40, ICC = 0.46, 5 trials/person) test-retest intervals. The combination of strong within-person correlations and only modest between-person correlations between the NPS and pain reports indicate that the two measures have different sources of between-person variance. The NPS is not a surrogate for individual differences in pain reports but can serve as a reliable measure of pain-related physiology and mechanistic target for interventions.

Keywords: Evoked pain; Individual differences; Measurement properties; Multivariate brain signature; Trial number.

Copyright © 2021. Published by Elsevier Inc.

Figures

Figure 1.

NPS pattern and measurement properties. (A) The multivariate brain pattern of NPS. The map shows thresholded voxel weights (at q<0.05 false discovery rate (FDR)) for display only; all weights were used in the subsequent analyses. Two examples of aMCC/SMA and midbrain unthresholded patterns is presented in the insets; small squares indicate individual voxel weight. Ins denotes Insula, V1 primary visual area, S2 secondary somatosensory cortex, MCC midcingulate cortex, Thal thalamus, STS superior temporal sulcus, PCC posterior cingulate cortex, LOC lateral occipital complex and IPL inferior parietal lobule. Direction is indicated with preceding lowercase letters as follows: r denotes right, l left, m middle, d dorsal, p posterior, pg perigenual. (B) A diagram to summarize analyses of measurement properties and the corresponding data used in different studies. Measurement properties include effect sizes in predicting external variables and test-retest reliability of the NPS. We analyzed four types of effect sizes, including: (1) the mean response of the NPS; (2) the within-person correlation between the NPS response and the temperature of the heat stimuli; (3) the within-person correlation between the NPS response and the pain ratings; and (4) the between-person correlation between the NPS response and the pain ratings. Besides assessing the test-retest reliability of the NPS, we also analyzed four factors that might influence the test-retest reliability, including the number of trials, stimuli intensities, contrast types, and the time interval between sessions. The influence of the time intervals between sessions was assessed with the data in Studies 9 and 10. All other analyses of the effect sizes and test-retest reliabilities were assessed with the data in Studies 1 to 8 (i.e., the single-trial dataset). (C) Four types of NPS effect size. Each big dot represents a type of averaged effect size of studies 1 to 8; the vertical bar represents the standard error; each small dot represents the effect size of one study. See Figure S1 for the tests of each study. See Figure S2 for the effect sizes of local regions of the NPS. (D) Short-term test-retest reliability of subjective pain reports, NPS, and local regions. Each big dot represents the mean reliability of studies 1 to 8; the vertical bar represents the standard error; each small dot represents the reliability of one study. The downward-pointing arrows indicate ICC < 0. See Figure S3 for the illustration of short-term test-retest reliability of the NPS and subjective pain reports. (E) Illustration of longer-term test-retest reliability of NPS with a 5-day interval. Correlations of the NPS responses between session 1, session 2 and session 3 in study 9 (ICC = 0.73). Each dot represents one participant; the line represents the linear relationship between the NPS response in sessions 1, 2 and 3, and the shadow represents the standard error. (F) Illustration of longer-term test-retest reliability of NPS with a 1-month interval. Correlation of the NPS responses between session 1 and session 2 in the treatment-as-usual control group of study 10 (ICC = 0.46). Each dot represents one participant; the line represents the linear relationship between the NPS response in sessions 1 and 2, and the shadow represents the standard error. *** p < 0.001; ** p<0.005.

Figure 2.

Factors that influence the reliability of the NPS response (left column) and subjective pain reports (right column). The small numbers from 1 to 10 correspond to studies 1 to 10. (A) The Influence of the trial number and time interval between sessions. The ICC values were calculated based on different trial numbers. Each line with color shows the nonlinear relationship between the trial number and the ICC values of the corresponding study (fitted using the loess function in R). The ICC values estimated with less than 10 participants were excluded due to poor estimation. The black line showed the average of studies 1 to 8, which was weighted by the square root of the number of participants in each study. The grey shadow presents the standard error, which was also weighted by the square root of the number of participants in each study. On average, to achieve excellent reliability, at least 60 trials were required to calculate the NPS response. Reliability was comparable in studies 9 and 10 with a longer time interval across 5-day and 1-month given the same number of trials (trial number = 30 and 5). The reliability of pain reports were excellent in general but were poor in study 10. (B) The influence of the temperature of the heat stimuli. Only participants with more than 4 trials in each temperature were included in the ICC calculation. The ICC values estimated with less than 13 participants were excluded due to poor estimation. Under these criteria, the study 4 and 7 were with no ICC value presented in the plot. NPS responses are more reliable in higher temperature stimuli. Whereas pain reports are reliable across all temperature stimuli. (C) The influence of the types of contrast. The larger dots represent the ICC values of the measurements calculated by comparing a temperature condition with the baseline, and the smaller dots represent the ICC values of the measurements calculated by comparing a temperature condition with the lowest temperature condition in each study. The length of the dashed line represents the difference between the ICC values of measurements calculated with different types of contrast. The downward-pointing arrow indicates ICC < 0. The measurements calculated by comparing with a control condition are less reliable than by comparing with the implicit baseline in virtually every case.

Figure 3.

Summary of variances and factors that influence the effect size and reliability. (A) Different sources of variance at the between-person level for the NPS and self-report pain. Rectangles represent the observed variables, i.e., pain reports and NPS. Ellipses represent the latent variables that we aim to measure, i.e., the core nociceptive feeling. The circle represents sources of variance that add to each observed measure. Both pain reports and NPS activity measure the core nociceptive circuits that generate pain experience. However, different sources of variance at the between-person level reduce the correlation between pain reports and the NPS response. It suggests that the NPS is not as useful as a surrogate measure for pain reports. In contrast, the NPS could be useful as an objective biological target to measure physiological pain in combination with subjective pain reports. (B) Factors that influence reliability. Rectangles represent the observed variables, such as the NPS response, across different sessions. Ellipses represent the latent variables that we are interested in modeling. Results suggest that stimuli with larger effect sizes have higher test-retest reliability, indicated by the up red arrow, and have the same effect on all sessions, indicated by η. Some active change across sessions could decrease the test-retest reliability, indicated by the down blue arrow. They might have different effects on different sessions, indicated by α_1, α_2, and α_n. The circle represents the measurement error that could decrease the test-retest reliability, indicated by the down blue arrow. There might be different errors on different sessions, indicated by σ1, σ2, and σn.