Test-retest reliability of graph metrics in functional brain networks: a resting-state fNIRS study

Haijing Niu, Zhen Li, Xuhong Liao, Jinhui Wang, Tengda Zhao, Ni Shu, Xiaohu Zhao, Yong He, Haijing Niu, Zhen Li, Xuhong Liao, Jinhui Wang, Tengda Zhao, Ni Shu, Xiaohu Zhao, Yong He

Abstract

Recent research has demonstrated the feasibility of combining functional near-infrared spectroscopy (fNIRS) and graph theory approaches to explore the topological attributes of human brain networks. However, the test-retest (TRT) reliability of the application of graph metrics to these networks remains to be elucidated. Here, we used resting-state fNIRS and a graph-theoretical approach to systematically address TRT reliability as it applies to various features of human brain networks, including functional connectivity, global network metrics and regional nodal centrality metrics. Eighteen subjects participated in two resting-state fNIRS scan sessions held ∼20 min apart. Functional brain networks were constructed for each subject by computing temporal correlations on three types of hemoglobin concentration information (HbO, HbR, and HbT). This was followed by a graph-theoretical analysis, and then an intraclass correlation coefficient (ICC) was further applied to quantify the TRT reliability of each network metric. We observed that a large proportion of resting-state functional connections (∼90%) exhibited good reliability (0.6< ICC <0.74). For global and nodal measures, reliability was generally threshold-sensitive and varied among both network metrics and hemoglobin concentration signals. Specifically, the majority of global metrics exhibited fair to excellent reliability, with notably higher ICC values for the clustering coefficient (HbO: 0.76; HbR: 0.78; HbT: 0.53) and global efficiency (HbO: 0.76; HbR: 0.70; HbT: 0.78). Similarly, both nodal degree and efficiency measures also showed fair to excellent reliability across nodes (degree: 0.52∼0.84; efficiency: 0.50∼0.84); reliability was concordant across HbO, HbR and HbT and was significantly higher than that of nodal betweenness (0.28∼0.68). Together, our results suggest that most graph-theoretical network metrics derived from fNIRS are TRT reliable and can be used effectively for brain network research. This study also provides important guidance on the choice of network metrics of interest for future applied research in developmental and clinical neuroscience.

Conflict of interest statement

Competing Interests: Dr. YH is an Editor of PLOS ONE journal. This does not alter the authors' adherence to all the PLOS ONE policies on sharing data and materials.

Figures

Figure 1. Schematic of fNIRS channel localization.
Figure 1. Schematic of fNIRS channel localization.
(A) and (C) display the left and right view of measurement channels on an anatomical brain image. (B) The arrangement of the whole-head 46 measurement channels on a brain template. Note that the red and purple solid circles represent sources and detectors, respectively.
Figure 2. Demonstration of R-fNIRS brain network…
Figure 2. Demonstration of R-fNIRS brain network reliability analysis.
Figure 3. Spatial similarity of RSFC maps.
Figure 3. Spatial similarity of RSFC maps.
Group-level RSFC maps for session 1 and session 2 and their Pearson correlation are displayed in the first to third columns. Figures (A) to (C) correspond to the RSFC data derived from HbO, HbR, and HbT, respectively. High similarity between sessions was observed in both the qualitative visual inspection and quantitative correlational analysis.
Figure 4. Reliability analysis of RSFC maps.
Figure 4. Reliability analysis of RSFC maps.
The first to third columns correspond to the data derived from HbO, HbR, and HbT, respectively. (A, B) The TRT reliability of RSFC maps and their corresponding reliability distributions. The reliability displays approximately normal configuration for all 1035 (i.e., 46×45/2) connections. The connections exhibit good reliability across HbO (mean ICC values 0.70), HbR (0.65) and HbT (0.71). (C) The relationship between RSFC strength and reliability as assessed by scatterplots. Each dot represents the group-level RSFC strength and the corresponding ICC value at the same connections. The trend lines were obtained by a linear least-squares fit method. Significant (p<0.05) positive correlations were found for HbO and HbR signals, suggesting stronger RSFC leads to higher reliability for both these signals.
Figure 5. TRT reliability of global network…
Figure 5. TRT reliability of global network metrics as a function of sparsity threshold.
(A–C) The global metric reliability was derived from HbO, HbR, and HbT, respectively. Five colors correspond to five different reliability grades. The red, yellow, green, cyan, and blue colors represent excellent (0.75ICC <1), good (0.6< ICC <0.75), fair (0.4< ICC <0.6), low (0.25< ICC <0.4), and poor (ICC<0.25) reliability of global network metrics, respectively. Cp, Lp, γ, λ, and σ denote the clustering coefficient, characteristic path length, normalized clustering coefficient, normalized characteristic path length, and small-world, respectively. Eloc and Eglob denote local efficiency and global efficiency, respectively. Q, β, and r denote modularity, hierarchy, and assortativity, respectively.
Figure 6. Threshold-independent reliability analysis of global…
Figure 6. Threshold-independent reliability analysis of global network metrics.
The areas under the curves (AUCs) of each global metric were used to provide threshold-independent reliability evaluation. (A–C) The global metric reliability was derived from HbO, HbR, and HbT, respectively. Cp, Lp, γ, λ, and σ denote the clustering coefficient, characteristic path length, normalized clustering coefficient, normalized characteristic path length, and small-world, respectively. Eloc and Eglob denote local efficiency and global efficiency, respectively. Q, β, and r denote modularity, hierarchy, and assortativity, respectively.
Figure 7. TRT reliability of nodal centrality…
Figure 7. TRT reliability of nodal centrality metrics as a function of sparsity threshold.
(A–C) The nodal metric reliability was derived from HbO, HbR, and HbT, respectively. The five colors correspond to five different reliability grades: red, yellow, green, cyan, and blue represent excellent (0.75ICC <1), good (0.6< ICC <0.75), fair (0.4< ICC <0.6), low (0.25< ICC <0.4), and poor (ICC<0.25) reliability of the nodal centrality metrics, respectively.
Figure 8. Threshold-independent reliability analysis of nodal…
Figure 8. Threshold-independent reliability analysis of nodal centrality metrics.
The areas under the curves (AUCs) of each nodal metric were used to provide threshold-independent reliability evaluation. (A–C) The nodal reliability was derived from HbO, HbR, and HbT, respectively. Different colors in the nodes correspond to different reliability grades: red, yellow, green, cyan, and blue colors represent excellent (0.75ICC <1), good (0.6< ICC <0.75), fair (0.4< ICC <0.6), low (0.25< ICC <0.4), and poor (ICC<0.25) reliability of the nodal centrality metrics, respectively.
Figure 9. Significant differences (paired t-test) in…
Figure 9. Significant differences (paired t-test) in nodal metric reliability.
(A) Reliability for three nodal centrality metrics (degree, efficiency, and betweenness). (B) Reliability for three concentration signals (HbO, HbR, and HbT). Note that the reliability of nodal degree and efficiency is concordant across HbO, HbR, and HbT and is also significantly (p<0.0001) higher than that of nodal betweenness. Error bars correspond to the standard deviation of the mean across the total nodes. The double asterisk indicates p<0.0001.
Figure 10. Threshold-independent reliability analysis of ICA-…
Figure 10. Threshold-independent reliability analysis of ICA- derived global network metrics.
The areas under the curves (AUCs) of each global metric were used to provide threshold-independent reliability evaluation. (A–C) The global metric reliability was derived from HbO, HbR, and HbT, respectively. Cp, Lp, γ, λ, and σ denote the clustering coefficient, characteristic path length, normalized clustering coefficient, normalized characteristic path length, and small-world, respectively. Eloc and Eglob denote local efficiency and global efficiency, respectively. Q, β, and r denote modularity, hierarchy, and assortativity, respectively.
Figure 11. Significant differences (paired t-test) in…
Figure 11. Significant differences (paired t-test) in ICA-derived nodal metric reliability.
(A) Reliability for three nodal centrality metrics (degree, efficiency, and betweenness). (B) Reliability for three concentration signals (HbO, HbR, and HbT). Each of these three concentration signals was denoised using ICA. Note that the reliability of nodal degree and efficiency is concordant across HbO, HbR, and HbT and is also significantly (p<0.0001) higher than that of nodal betweenness. Error bars correspond to the standard deviation of the mean across the total nodes. The double asterisk indicates p<0.0001.
Figure 12. Reliability analysis of global network…
Figure 12. Reliability analysis of global network metrics derived from the correlation matrix randomization method.
The areas under the curves (AUCs) of each global metric were used to provide threshold-independent reliability evaluation. (A–C) show the frequency-derived (i.e. band-pass filter) global metric reliability and (D–F) show the ICA-derived global metric reliability.γ, λ, and σ denote the normalized clustering coefficient, normalized characteristic path length, and small-world, respectively.

References

    1. Bullmore E, Sporns O (2009) Complex brain networks: graph theoretical analysis of structural and functional systems. Nat Rev Neurosci 10: 186–198.
    1. Bullmore E, Sporns O (2012) The economy of brain network organization. Nat Rev Neurosci 13: 336–349.
    1. He Y, Evans A (2010) Graph theoretical modeling of brain connectivity. Current Opinion in Neurology 23: 341–350.
    1. Rubinov M, Sporns O (2010) Complex network measures of brain connectivity: Uses and interpretations. NeuroImage 52: 1059–1069.
    1. He Y, Chen ZJ, Evans AC (2007) Small-World Anatomical Networks in the Human Brain Revealed by Cortical Thickness from MRI. Cerebral Cortex 17: 2407–2419.
    1. Gong G, He Y, Concha L, Lebel C, Gross DW, et al. (2009) Mapping Anatomical Connectivity Patterns of Human Cerebral Cortex Using In Vivo Diffusion Tensor Imaging Tractography. Cerebral Cortex 19: 524–536.
    1. Hagmann P, Cammoun L, Gigandet X, Meuli R, Honey C, et al. (2008) Mapping the structural core of human cerebral cortex. PLoS biology 6: e159.
    1. Hagmann P, Kurant M, Gigandet X, Thiran P, Wedeen VJ, et al. (2007) Mapping Human Whole-Brain Structural Networks with Diffusion MRI. PLoS ONE 2: e597.
    1. Achard S, Salvador R, Whitcher B, Suckling J, Bullmore E (2006) A Resilient, Low-Frequency, Small-World Human Brain Functional Network with Highly Connected Association Cortical Hubs. The Journal of Neuroscience 26: 63–72.
    1. He Y, Dagher A, Chen Z, Charil A, Zijdenbos A, et al. (2009) Impaired small-world efficiency in structural cortical networks in multiple sclerosis associated with white matter lesion load. Brain 132: 3366–3379.
    1. Salvador R, Suckling J, Coleman MR, Pickard JD, Menon D, et al. (2005) Neurophysiological Architecture of Functional Magnetic Resonance Images of Human Brain. Cerebral Cortex 15: 1332–1342.
    1. Niu H, Wang J, Zhao T, Shu N, He Y (2012) Revealing Topological Organization of Human Brain Functional Networks with Resting-State Functional near Infrared Spectroscopy. PLoS ONE 7: e45771.
    1. Bassett DS, Bullmore ET, Meyer-Lindenberg A, Apud JA, Weinberger DR, et al. (2009) Cognitive fitness of cost-efficient brain functional networks. Proceedings of the National Academy of Sciences 106: 11747–11752.
    1. Pievani M, de Haan W, Wu T, Seeley WW, Frisoni GB (2011) Functional network disruption in the degenerative dementias. The Lancet Neurology 10: 829–843.
    1. Wang J, Zuo X, He Y (2010) Graph-based network analysis of resting-state functional MRI. Frontiers in Systems Neuroscience 4.
    1. Xia M, He Y (2011) Magnetic Resonance Imaging and Graph Theoretical Analysis of Complex Brain Networks in Neuropsychiatric Disorders. Brain connectivity 1: 349–365.
    1. Jobsis F (1977) Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science 198: 1264–1267.
    1. Villringer A, Chance B (1997) Non-invasive optical spectroscopy and imaging of human brain function. Trends in Neurosciences 20: 435–442.
    1. Homae F, Watanabe H, Otobe T, Nakano T, Go T, et al. (2010) Development of Global Cortical Networks in Early Infancy. The Journal of Neuroscience 30: 4877–4882.
    1. Lu C-M, Zhang Y-J, Biswal BB, Zang Y-F, Peng D-L, et al. (2010) Use of fNIRS to assess resting state functional connectivity. Journal of Neuroscience Methods 186: 242–249.
    1. Mesquita RC, Franceschini MA, Boas DA (2010) Resting state functional connectivity of the whole head with near-infrared spectroscopy. Biomed Opt Express 1: 324–336.
    1. White BR, Liao SM, Ferradal SL, Inder TE, Culver JP (2011) Bedside optical imaging of occipital resting-state functional connectivity in neonates. NeuroImage.
    1. White BR, Snyder AZ, Cohen AL, Petersen SE, Raichle ME, et al. (2009) Resting-state functional connectivity in the human brain revealed with diffuse optical tomography. NeuroImage 47: 148–156.
    1. Zhang H, Duan L, Zhang Y-J, Lu C-M, Liu H, et al. (2011) Test–retest assessment of independent component analysis-derived resting-state functional connectivity based on functional near-infrared spectroscopy. NeuroImage 55: 607–615.
    1. Zhang H, Zhang Y-J, Lu C-M, Ma S-Y, Zang Y-F, et al. (2010) Functional connectivity as revealed by independent component analysis of resting-state fNIRS measurements. NeuroImage 51: 1150–1161.
    1. Cope M, Delpy D (1988) System for long-term measurement of cerebral blood and tissue oxygenation on newborn infants by near infra-red transillumination. Medical and Biological Engineering and Computing 26: 289–294.
    1. Biswal B, Zerrin Yetkin F, Haughton VM, Hyde JS (1995) Functional connectivity in the motor cortex of resting human brain using echo-planar mri. Magnetic Resonance in Medicine 34: 537–541.
    1. Sasai S, Homae F, Watanabe H, Sasaki AT, Tanabe HC, et al. (2012) A NIRS–fMRI study of resting state network. NeuroImage 63: 179–193.
    1. Fox MD, Snyder AZ, Vincent JL, Corbetta M, Van Essen DC, et al. (2005) The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proceedings of the National Academy of Sciences of the United States of America 102: 9673–9678.
    1. Latora V, Marchiori M (2001) Efficient Behavior of Small-World Networks. Physical Review Letters 87: 198701.
    1. Watts DJ, Strogatz SH (1998) Collective dynamics of /`small-world/' networks. Nature 393: 440–442.
    1. Latora V, Marchiori M (2003) Economic small-world behavior in weighted networks. The European Physical Journal B – Condensed Matter and Complex Systems 32: 249–263.
    1. Maslov S, Sneppen K (2002) Specificity and Stability in Topology of Protein Networks. Science 296: 910–913.
    1. Humphries MD, Gurney K, Prescott TJ (2006) The brainstem reticular formation is a small-world, not scale-free, network. Proceedings of the Royal Society B: Biological Sciences 273: 503–511.
    1. Newman MEJ (2006) Modularity and community structure in networks. Proceedings of the National Academy of Sciences 103: 8577–8582.
    1. Ravasz E, Barabási A-L (2003) Hierarchical organization in complex networks. Physical Review E 67: 026112.
    1. Newman MEJ (2002) Assortative Mixing in Networks. Physical Review Letters 89: 208701.
    1. Achard S, Bullmore E (2007) Efficiency and Cost of Economical Brain Functional Networks. PLoS Comput Biol 3: e17.
    1. Linton CF (1977) A Set of Measures of Centrality Based on Betweenness. Sociometry 40: 35–41.
    1. Shrout PE, Fleiss JL (1979) Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 86: 420–428.
    1. Sampat MP, Whitman GJ, Stephens TW, Broemeling LD, Heger NA, et al. (2006) The reliability of measuring physical characteristics of spiculated masses on mammography. British Journal of Radiology 79: S134–S140.
    1. Winer BJ (1962) Statistical principles in experimental design. New York, NYUS: McGraw-Hill Book Company. x, 672 p.
    1. Rousson V, Gasser T, Seifert B (2002) Assessing intrarater, interrater and test–retest reliability of continuous measurements. Statistics in medicine 21: 3431–3446.
    1. Müller R, Büttner P (1994) A critical discussion of intraclass correlation coefficients. Statistics in medicine 13: 2465–2476.
    1. Kong J, Gollub RL, Webb JM, Kong J-T, Vangel MG, et al. (2007) Test–retest study of fMRI signal change evoked by electroacupuncture stimulation. NeuroImage 34: 1171–1181.
    1. He Y, Wang J, Wang L, Chen ZJ, Yan C, et al. (2009) Uncovering Intrinsic Modular Organization of Spontaneous Brain Activity in Humans. PLoS ONE 4: e5226.
    1. Stone JV (2002) Independent component analysis: an introduction. Trends in Cognitive Sciences 6: 59–64.
    1. Kohno S, Miyai I, Seiyama A, Oda I, Ishikawa A, et al. (2007) Removal of the skin blood flow artifact in functional near-infrared spectroscopic imaging data through independent component analysis. Journal of Biomedical Optics 12: 062111–062111.
    1. Hyvarinen A (1999) Fast and robust fixed-point algorithms for independent component analysis. Neural Networks, IEEE Transactions on 10: 626–634.
    1. van de Ven VG, Formisano E, Prvulovic D, Roeder CH, Linden DEJ (2004) Functional connectivity as revealed by spatial independent component analysis of fMRI measurements during rest. Human Brain Mapping 22: 165–178.
    1. Shehzad Z, Kelly AMC, Reiss PT, Gee DG, Gotimer K, et al. (2009) The Resting Brain: Unconstrained yet Reliable. Cerebral Cortex 19: 2209–2229.
    1. Wang J-H, Zuo X-N, Gohel S, Milham MP, Biswal BB, et al. (2011) Graph Theoretical Analysis of Functional Brain Networks: Test-Retest Evaluation on Short- and Long-Term Resting-State Functional MRI Data. PLoS ONE 6: e21976.
    1. Braun U, Plichta MM, Esslinger C, Sauer C, Haddad L, et al. (2012) Test–retest reliability of resting-state connectivity network characteristics using fMRI and graph theoretical measures. NeuroImage 59: 1404–1412.
    1. Deuker L, Bullmore ET, Smith M, Christensen S, Nathan PJ, et al. (2009) Reproducibility of graph metrics of human brain functional networks. NeuroImage 47: 1460–1468.
    1. Liang X, Wang J, Yan C, Shu N, Xu K, et al. (2012) Effects of Different Correlation Metrics and Preprocessing Factors on Small-World Brain Functional Networks: A Resting-State Functional MRI Study. PLoS ONE 7: e32766.
    1. Telesford QK, Morgan AR, Hayasaka S, Simpson SL, Barret W, et al. (2010) Reproducibility of graph metrics in fMRI networks. Frontiers in NEUROINFORMATICS 4: 1–10.
    1. Chu CJ, Kramer MA, Pathmanathan J, Bianchi MT, Westover MB, et al. (2012) Emergence of Stable Functional Networks in Long-Term Human Electroencephalography. The Journal of Neuroscience 32: 2703–2713.
    1. Zalesky A, Fornito A, Bullmore E (2012) On the use of correlation as a measure of network connectivity. NeuroImage 60: 2096–2106.
    1. Gagnon L, Yücel MA, Dehaes M, Cooper RJ, Perdue KL, et al. (2012) Quantification of the cortical contribution to the NIRS signal over the motor cortex using concurrent NIRS-fMRI measurements. NeuroImage 59: 3933–3940.
    1. Kirilina E, Jelzow A, Heine A, Niessing M, Wabnitz H, et al. (2012) The physiological origin of task-evoked systemic artefacts in functional near infrared spectroscopy. NeuroImage 61: 70–81.
    1. Strangman G, Culver JP, Thompson JH, Boas DA (2002) A quantitative comparison of simultaneous BOLD fMRI and NIRS recordings during functional brain activation. NeuroImage 17: 719–731.
    1. Boas DA, Dale AM, Franceschini MA (2004) Diffuse optical imaging of brain activation: approaches to optimizing image sensitivity, resolution, and accuracy. NeuroImage 23 Suppl 1S275–288.
    1. Toronov V, Franceschini MA, Filiaci M, Fantini S, Wolf M, et al. (2000) Near-infrared study of fluctuations in cerebral hemodynamics during rest and motor stimulation: temporal analysis and spatial mapping. Medical physics 27: 801–815.
    1. Diamond SG, Perdue KL, Boas DA (2009) A cerebrovascular response model for functional neuroimaging including dynamic cerebral autoregulation. Mathematical Biosciences 220: 102–117.
    1. Obrig H, Neufang M, Wenzel R, Kohl M, Steinbrink J, et al. (2000) Spontaneous low frequency oscillations of cerebral hemodynamics and metabolism in human adults. NeuroImage 12: 623–639.
    1. Payne SJ, Selb J, Boas DA (2009) Effects of Autoregulation and CO2 Reactivity on Cerebral Oxygen Transport. Annals of Biomedical Engineering 37: 2288–2298.
    1. Zhang Y-J, Lu C-M, Biswal BB, Zang Y-F, Peng D-L, et al. (2010) Detecting resting-state functional connectivity in the language system using functional near-infrared spectroscopy. Journal of Biomedical Optics 15: 047003–047008.
    1. Greicius MD, Krasnow B, Reiss AL, Menon V (2003) Functional connectivity in the resting brain: A network analysis of the default mode hypothesis. Proceedings of the National Academy of Sciences 100: 253–258.
    1. Lowe MJ, Mock BJ, Sorenson JA (1998) Functional Connectivity in Single and Multislice Echoplanar Imaging Using Resting-State Fluctuations. NeuroImage 7: 119–132.
    1. Cooper RJ, Gagnon L, Goldenholz DM, Boas DA, Greve DN (2012) The utility of near-infrared spectroscopy in the regression of low-frequency physiological noise from functional magnetic resonance imaging data. NeuroImage 59: 3128–3138.
    1. Gagnon L, Perdue K, Greve DN, Goldenholz D, Kaskhedikar G, et al. (2011) Improved recovery of the hemodynamic response in diffuse optical imaging using short optode separations and state-space modeling. NeuroImage 56: 1362–1371.
    1. Saager RB, Telleri NL, Berger AJ (2011) Two-detector Corrected Near Infrared Spectroscopy (C-NIRS) detects hemodynamic activation responses more robustly than single-detector NIRS. NeuroImage 55: 1679–1685.
    1. Zhang Q, Brown EN, Strangman GE (2007) Adaptive filtering for global interference cancellation and real-time recovery of evoked brain activity: a Monte Carlo simulation study. Journal of Biomedical Optics 12: 044014–044014.
    1. Zhang Q, Strangman GE, Ganis G (2009) Adaptive filtering to reduce global interference in non-invasive NIRS measures of brain activation: How well and when does it work? NeuroImage 45: 788–794.
    1. Gagnon L, Cooper RJ, Yücel MA, Perdue KL, Greve DN, et al. (2012) Short separation channel location impacts the performance of short channel regression in NIRS. NeuroImage 59: 2518–2528.
    1. Leff DR, Orihuela-Espina F, Elwell CE, Athanasiou T, Delpy DT, et al. (2011) Assessment of the cerebral cortex during motor task behaviours in adults: A systematic review of functional near infrared spectroscopy (fNIRS) studies. NeuroImage 54: 2922–2936.
    1. Niu H, Khadka S, Tian F, Lin Z-J, Lu C, et al. (2011) Resting-state functional connectivity assessed with two diffuse optical tomographic systems. Journal of Biomedical Optics 16: 046006–046005.
    1. Hoshi Y (2007) Functional near-infrared spectroscopy: current status and future prospects. Journal of Biomedical Optics 12: 062106–062106.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe