Reliability and validity of the NeuroCognitive Performance Test, a web-based neuropsychological assessment

Glenn E Morrison, Christa M Simone, Nicole F Ng, Joseph L Hardy, Glenn E Morrison, Christa M Simone, Nicole F Ng, Joseph L Hardy

Abstract

The NeuroCognitive Performance Test (NCPT) is a brief, repeatable, web-based cognitive assessment platform that measures performance across several cognitive domains. The NCPT platform is modular and includes 18 subtests that can be arranged into customized batteries. Here we present normative data from a sample of 130,140 healthy volunteers for an NCPT battery consisting of 8 subtests. Participants took the NCPT remotely and without supervision. Factor structure and effects of age, education, and gender were evaluated with this normative dataset. Test-retest reliability was evaluated in a subset of participants who took the battery again an average of 78.8 days later. The eight NCPT subtests group into 4 putative cognitive domains, have adequate to good test-retest reliability, and are sensitive to expected age- and education-related cognitive effects. Concurrent validity to standard neuropsychological tests was demonstrated in 73 healthy volunteers. In an exploratory analysis the NCPT battery could differentiate those who self-reported Mild Cognitive Impairment or Alzheimer's disease from matched healthy controls. Overall these results demonstrate the reliability and validity of the NCPT battery as a measure of cognitive performance and support the feasibility of web-based, unsupervised testing, with potential utility in clinical and research settings.

Keywords: concurrent validity; fluid reasoning; memory; neuropsychological assessment; normative data; psychomotor speed; reliability; web-based.

Figures

Figure 1
Figure 1
NCPT normalization. Score distributions for Trail Making A before (left) and after (right) the normalization procedure. Each NCPT subtest is scaled following a percentile rank-based inverse normal transformation. The position of that percentile on a normal distribution is used to convert the raw score to a scaled score where the distribution has a mean of 100 and a standard deviation of 15.
Figure 2
Figure 2
Effect of age on individual NCPT subtest scores and overall sum score. (A) The effect of age on the Grand Index was significant with peak performance occurring around age 25 and then declining in a linear fashion. (B) The effect of age on the individual subtests followed the same general trend. The decline in performance with age after 25 years appears linear for most subtests with the exception of Arithmetic Reasoning, Grammatical Reasoning, and Trail Making B. The curves were smoothed with a General Additive Model (GAM), a type of general linear model in which the linear predictor depends on linear smoothed functions (Wood, 2011). GAM was selected over a simple linear smoother because we observed that the effect of age on NCPT scores was non-linear. TA, Trail Making A; TB, Trail Making B; DSC, Digit Symbol Coding; FMS, Forward Memory Span; RMS, Reverse Memory Span; PM, Progressive Matrices; AR, Arithmetic Reasoning; GR, Grammatical Reasoning.
Figure 3
Figure 3
Effect of education on NCPT Grand Index score and individual NCPT subtest scores. (A) The effect of education on the Grand Index was significant with scores increasing with number of years education. (B) The same trend was observed for each subtest, with increasing years of education correlated to higher subtest scores. The curves were smoothed with a GAM function. GAM was selected over a simple linear smoother because we observed that the effect of education on NCPT scores was non-linear. TA, Trail Making A; TB, Trail Making B; DSC, Digit Symbol Coding; FMS, Forward Memory Span; RMS, Reverse Memory Span; PM, Progressive Matrices; AR, Arithmetic Reasoning; GR, Grammatical Reasoning.
Figure 4
Figure 4
Effect of gender on NCPT Grand Index score. (A) The interaction between age and gender was significant and suggests that the negative main effect of gender diminishes with age. The y axis is the non-age-normed NCPT sum score. (B) The interaction between education and gender was significant and suggests that education has a greater positive impact on males compared to females. The y axis is the non-age-normed NCPT sum score. The curves were smoothed with a GAM function. GAM was selected over a simple linear model because we observed that the effect of gender on NCPT scores was non-linear.
Figure 5
Figure 5
Inter-assessment correlations. Heat map showing that each NCPT subtest was significantly correlated with performance on more than one other subtest. The heat map suggests three groupings that share higher correlations, as follows: (1) Arithmetic Reasoning (AR), Grammatical Reasoning (GR), and Digit Symbol Coding (DSC); (2) Forward (FMS) and Reverse Memory Span (RMS); and (3) Trail Making A (TA) and Trail Making B (TB). Progressive Matrices (PM) does not appear to correlate strongly with any other subtest.
Figure 6
Figure 6
Box plots of NCPT Grand Index scores for self-report MCI, AD, and healthy controls. Compared to all healthy controls in the Normative Sample, the Grand Index score was 0.78 SD (p < 0.05) lower for those who self-reported MCI and 1.17 SD (p < 0.05) lower for those who self-reported AD suggesting the NCPT is able to differentiate those who self-report MCI or AD from those who don't.

References

    1. American Educational Research Association (1999). Standards for Educational and Psychological Testing. Washington, DC: AERA.
    1. Anastasi A., Urbina S. (1997). Psychological Testing, 7th Edn. Upper Saddle River, NJ: Prentice Hall.
    1. Army Individual Test Battery (1944). Army Individual Test Battery: Manual of directions and scoring. Washington, DC: War Department, Adjutant General's Office.
    1. Baddeley A. D. (1968). A 3 min reasoning test based on grammatical transformation. Psychon. Sci. 10, 341–342. 10.3758/BF03331551
    1. Bauer R. M., Iverson G. L., Cernich A. N., Binder L. M., Ruff R. M., Naugle R. I. (2012). Computerized neuropsychological assessment devices: joint position paper of the american academy of clinical neuropsychology and the national academy of neuropsychology. Arch. Clin. Neuropsychol. 27, 362–373. 10.1093/arclin/acs027
    1. Bird C. M., Papadopoulou K., Ricciardelli P., Rossor M. N., Cipolotti L. (2003). Test-retest reliability, practice effects and reliable change indices for the recognition memory test. Br. J. Clin. Psychol. 42(Pt 4), 407–425. 10.1348/014466503322528946
    1. Burke H. R. (2010). Raven's progressive matrices: validity, reliability, and norms. J. Psychol. 82, 253–257. 10.1080/00223980.1972.9923815
    1. Capitani E., Barbarotto R., Laicana M. (1996). Does education influence age-related cognitive decline? A further inquiry. Dev. Neuropsychol. 12, 231–240. 10.1080/87565649609540648
    1. Cattell R. B. (1987). Intelligence: Its Structure, Growth and Action: Its Structure, Growth and Action. Amsterdam: Elsevier Science Publishers B.V.
    1. Collie A., Maruff P., Darby D. G., McStephen M. (2003). The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test-retest intervals. J. Int. Neuropsychol. Soc. 9, 419–428. 10.1017/S1355617703930074
    1. Deloche G., Seron X., Larroque C., Magnien C., Metz-Lutz M. N., Noel M. N., et al. . (1994). Calculation and number processing: assessment battery; role of demographic factors. J. Clin. Exp. Neuropsychol. 16, 195–208. 10.1080/01688639408402631
    1. Dikmen S. S., Heaton R. K., GRANT I., Temkin N. R. (1999). Test-retest reliability and practice effects of expanded Halstead-Reitan Neuropsychological Test Battery. J. Int. Neuropsychol. Soc. 5, 346–356. 10.1017/S1355617799544056
    1. Feldstein S. N., Keller F. R., Portman R. E., Durham R. L., Klebe K. J., Davis H. P. (1999). A comparison of computerized and standard versions of the Wisconsin Card Sorting Test. Clin. Neuropsychol. 13, 303–313. 10.1076/clin.13.3.303.1744
    1. Ford J. K., MacCallum R. C., Tail M. (1986). The application of exploratory factor analysis in applied psychology: a critical review and analysis. Pers. Psychol. 39, 291–314. 10.1111/j.1744-6570.1986.tb00583.x
    1. Gualtieri C., Johnson L. (2006). Reliability and validity of a computerized neurocognitive test battery, CNS Vital Signs. Arch. Clin. Neuropsychol. 21, 623–643. 10.1016/j.acn.2006.05.007
    1. Gulliksen H. (1950). Effect of Group Heterogeneity on Test Reliability. Theory of Mental Tests. Hoboken: John Wiley & Sons Inc; 108–127.
    1. Gur R. C., Richard J., Hughett P., Calkins M. E., Macy L., Bilker W. B., et al. . (2010). A cognitive neuroscience-based computerized battery for efficient measurement of individual differences: standardization and initial construct validation. J. Neurosci. Methods 187, 254–262. 10.1016/j.jneumeth.2009.11.017
    1. Hardy J. L., Nelson R. A., Thomason M. E., Sternberg D. A., Katovich K., Farzin F., et al. . (2015). Enhancing cognitive abilities with comprehensive training: a large, online, randomized, active-controlled trial. PLoS ONE 10:e0134467. 10.1371/journal.pone.0134467
    1. Hu L., Bentler P. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct. Equat. Model. Multidis. J. 6, 1–55. 10.1080/10705519909540118
    1. Kane R. L., Kay G. G. (1992). Computerized assessment in neuropsychology: a review of tests and test batteries. Neuropsychol. Rev. 3, 1–117. 10.1007/BF01108787
    1. Keefe R. S., Goldberg T. E., Harvey P. D., Gold J. M., Poe M. P., Coughenour L. (2004). The Brief Assessment of Cognition in Schizophrenia: reliability, sensitivity, and comparison with a standard neurocognitive battery. Schizophr. Res. 68, 283–297. 10.1016/j.schres.2003.09.011
    1. Kueider A. M., Parisi J. M., Gross A. L., Rebok G. W. (2012). Computerized cognitive training with older adults: a systematic review. PLoS ONE 7:e40588. 10.1371/journal.pone.0040588
    1. Lampit A., Hallock H., Valenzuela M. (2014). Computerized cognitive training in cognitively healthy older adults: a systematic review and meta-analysis of effect modifiers. PLoS Med. 11:e1001756. 10.1371/journal.pmed.1001756
    1. Lemay S., Bédard M. A., Rouleau I. (2004). Practice effect and test-retest reliability of attentional and executive tests in middle-aged to elderly subjects. Clin. Neuropsychol. 18, 1–19. 10.1080/13854040490501718
    1. Manly J. (2008). Race, culture, education, and cognitive test performance among older adults, in Handbook of Cognitive Aging: Interdisciplinary Perspectives, eds Hofer S., Alwin D. (Thousand Oaks, CA: SAGE Publications, Inc; ), 398–418.
    1. Maruff P., Collie A., Darby D., Weaver-Cargin J., Masters C., Currie J. (2004). Subtle memory decline over 12 months in mild cognitive impairment. Dement. Geriatr. Cogn. Disord. 18, 342–348. 10.1159/000080229
    1. McCrae R. R., Kurtz J. E., Yamagata S., Terracciano A. (2011). Internal consistency, retest reliability, and their implications for personality scale validity. Pers. Soc. Psychol. Rev. 15, 28–50. 10.1177/1088868310366253
    1. Milner B. (1971). Interhemispheric differences in the localization of psychological processes in man. Br. Med. Bull. 27, 272–277.
    1. Nakayama Y., Covassin T., Schatz P., Nogle S., Kovan J. (2014). Examination of the Test-Retest Reliability of a Computerized Neurocognitive Test Battery. Am. J. Sports Med. 42, 2000–2005. 10.1177/0363546514535901
    1. Nuechterlein K. H., Green M. F., Kern R. S., Baade L. E., Barch D. M., Cohen J. D., et al. . (2008). The MATRICS Consensus Cognitive Battery, part 1: test selection, reliability, and validity. Am. J. Psychiatry 165, 203–213. 10.1176/appi.ajp.2007.07010042
    1. Prabhakaran V., Rypma B., Gabrieli J. D. E. (2001). Neural substrates of mathematical reasoning: a functional magnetic resonance imaging study of neocortical activation during performance of the necessary arithmetic operations test. Neuropsychology 15, 115–127. 10.1037/0894-4105.15.1.115
    1. Prabhakaran V., Smith J. A., Desmond J. E., Glover G. H., Gabrieli J. D. (1997). Neural substrates of fluid reasoning: an fMRI study of neocortical activation during performance of the Raven's Progressive Matrices Test. Cogn. Psychol. 33, 43–63.
    1. Rabbitt P., Diggle P., Smith D., Holland F., Mc Innes L. (2001). Identifying and separating the effects of practice and of cognitive ageing during a large longitudinal study of elderly community residents. Neuropsychologia 39, 532–543. 10.1016/S0028-3932(00)00099-3
    1. Raven J. (2000). The Raven's progressive matrices: change and stability over culture and time. Cogn Psychol. 41, 1–48. 10.1006/cogp.1999.0735
    1. R Core Team (2015). R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing; Available online at:
    1. Reitan R. M., Wolfson D. (1995). The Halstead–Reitan Neuropsycholgical Test Battery: Therapy and Clinical Interpretation. Tucson, AZ: Neuropsychological Press.
    1. Revelle W. (2015). Psych: Procedures for Personality and Psychological Research. Evanston, IL: Northwestern University; Available online at: Version = 1.5.8.
    1. Richardson J. T. E. (2007). Measures of short-term memory: a historical review. Cortex 43, 635–650. 10.1016/S0010-9452(08)70493-3
    1. Robbins T. W., James L. F., Owen A. M., Sahakian B. J., Lawrence A. D., McInnes L., et al. . (1998). A study of performance on tests from the CANTAB battery sensitive to frontal lobe dysfunction in a large sample of normal volunteers: implications for theories of executive functioning and cognitive aging. Cambridge Neuropsychological Test Automated Battery. J. Int. Neuropsychol. Soc. 4, 474–490. 10.1017/s1355617798455073
    1. Rossetti H. C., Lacritz L. H., Cullum C. M., Weiner M. F. (2011). Normative data for the Montreal Cognitive Assessment (MoCA) in a population-based sample. Neurology 77, 1272–1275. 10.1212/WNL.0b013e318230208a
    1. Royer F. L. (1971). Spatial orientational and figural information in free recall of visual figures. J. Exp. Psychol. 91, 326–332. 10.1037/h0031846
    1. Salthouse T. A. (1996). The processing-speed theory of adult age differences in cognition. Psychol. Rev. 103, 403–428. 10.1037/0033-295X.103.3.403
    1. Schlegel R., Gilliland K. (2007). Development and quality assurance of computer-based assessment batteries. Arch. Clin. Neuropsychol. 22, 49–61. 10.1016/j.acn.2006.10.005
    1. Sekhon J. S. (2011). Multivariate and propensity score matching software with automated balance optimization: the matching package for R. J. Statist. Softw. 42, 1–52.
    1. Wechsler D. (1945). Wechsler Memory Scale. San Antonio, TX: Psychological Corporation.
    1. Wechsler D. (1955). Manual for the Wechsler Adult Intelligence Scale. Oxford, England: Psychological Corp; 110.
    1. Wechsler D. (1981). Wechsler Adult Intelligence Scale-Revised. San Antonio, TX: Psychological Corporation.
    1. Wild K., Howieson D., Webbe F., Seelye A., Kaye J. (2008). Status of computerized cognitive testing in aging: a systematic review. Alzheim. Demen. 4, 428–437. 10.1016/j.jalz.2008.07.003
    1. Williams J. E., McCord D. M. (2006). Equivalence of standard and computerized versions of the Raven Progressive Matrices Test. Comput. Hum. Behav. 22, 791–800. 10.1016/j.chb.2004.03.005
    1. Williams L. M., Simms E., Clark C. R., Paul R. H., Rowe D., Gordon E. (2005). The test-retest reliability of a standardized neurocognitive and neurophysiological test battery: "neuromarker". Int. J. Neurosci. 115, 1605–1630. 10.1080/00207450590958475
    1. Wood S. N. (2011). Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. J. R. Statist. Soc. B 73, 3–36. 10.1111/j.1467-9868.2010.00749.x
    1. Zygouris S., Tsolaki M. (2014). Computerized cognitive testing for older adults: a review. Am. J. Alzheimers. Dis. Other Demen. 30, 13–28. 10.1177/1533317514522852

Source: PubMed

Sponsors en medewerkers

Medische omstandigheden

Geneesmiddelinterventies

3
Abonneren