Development and Replication of Objective Measurements of Social Visual Engagement to Aid in Early Diagnosis and Assessment of Autism

Abstract

Importance: Autism spectrum disorder is a common and early-emerging neurodevelopmental condition. While 80% of parents report having had concerns for their child's development before age 2 years, many children are not diagnosed until ages 4 to 5 years or later.

Objective: To develop an objective performance-based tool to aid in early diagnosis and assessment of autism in children younger than 3 years.

Design, setting, and participants: In 2 prospective, consecutively enrolled, broad-spectrum, double-blind studies, we developed an objective eye-tracking-based index test for children aged 16 to 30 months, compared its performance with best-practice reference standard diagnosis of autism (discovery study), and then replicated findings in an independent sample (replication study). Discovery and replication studies were conducted in specialty centers for autism diagnosis and treatment. Reference standard diagnoses were made using best-practice standardized protocols by specialists blind to eye-tracking results. Eye-tracking tests were administered by staff blind to clinical results. Children were enrolled from April 27, 2013, until September 26, 2017. Data were analyzed from March 28, 2018, to January 3, 2019.

Main outcomes and measures: Prespecified primary end points were the sensitivity and specificity of the eye-tracking-based index test compared with the reference standard. Prespecified secondary end points measured convergent validity between eye-tracking-based indices and reference standard assessments of social disability, verbal ability, and nonverbal ability.

Results: Data were collected from 1089 children: 719 children (mean [SD] age, 22.4 [3.6] months) in the discovery study, and 370 children (mean [SD] age, 25.4 [6.0] months) in the replication study. In discovery, 224 (31.2%) were female and 495 (68.8%) male; in replication, 120 (32.4%) were female and 250 (67.6%) male. Based on reference standard expert clinical diagnosis, there were 386 participants (53.7%) with nonautism diagnoses and 333 (46.3%) with autism diagnoses in discovery, and 184 participants (49.7%) with nonautism diagnoses and 186 (50.3%) with autism diagnoses in replication. In the discovery study, the area under the receiver operating characteristic curve was 0.90 (95% CI, 0.88-0.92), sensitivity was 81.9% (95% CI, 77.3%-85.7%), and specificity was 89.9% (95% CI, 86.4%-92.5%). In the replication study, the area under the receiver operating characteristic curve was 0.89 (95% CI, 0.86-0.93), sensitivity was 80.6% (95% CI, 74.1%-85.7%), and specificity was 82.3% (95% CI, 76.1%-87.2%). Eye-tracking test results correlated with expert clinical assessments of children's individual levels of ability, explaining 68.6% (95% CI, 58.3%-78.6%), 63.4% (95% CI, 47.9%-79.2%), and 49.0% (95% CI, 33.8%-65.4%) of variance in reference standard assessments of social disability, verbal ability, and nonverbal cognitive ability, respectively.

Conclusions and relevance: In two diagnostic studies of children younger than 3 years, objective eye-tracking-based measurements of social visual engagement quantified diagnostic status as well as individual levels of social disability, verbal ability, and nonverbal ability in autism. These findings suggest that objective measurements of social visual engagement can be used to aid in autism diagnosis and assessment.

Conflict of interest statement

Conflict of Interest Disclosures: Dr Jones reported receiving grants from the Georgia Research Alliance, the Joseph B. Whitehead Foundation, the Marcus Foundation, and the National Institute of Mental Health (NIMH) during the conduct of the study (all via his institution); being a scientific cofounder of and owning equity in EarliTec Diagnostics; holding patents licensed to EarliTec Diagnostics; and receiving personal fees for consulting from EarliTec Diagnostics and a lecture honorarium from Washington University in St Louis School of Medicine outside the submitted work. Dr Klaiman reported receiving grants from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the Georgia Research Alliance, the Joseph B. Whitehead Foundation, the Marcus Foundation, and the NIMH (all via her institution) during the conduct of the study and personal fees from ABA Centers of America, Dekalb County School District, Cherokee County School District, Fulton County School District, Beaming Health, and EarliTec Diagnostics outside the submitted work. Dr Richardson reported receiving funding and/or equipment from the Georgia Research Alliance, the John B. Whitehead Foundation, and the Marcus Foundation (all via her institution) during the conduct of the study. Dr Lambha reported receiving funding and/or equipment from the Georgia Research Alliance, the Joseph B. Whitehead Foundation, the Marcus Foundation, the NICHD, and the NIMH (all via her institution) during the conduct of the study. Dr Beacham reported receiving funding and/or equipment from the John B. Whitehead Foundation and the Marcus Foundation during the conduct of the study. Mr Lewis reported receiving grants from the Georgia Research Alliance, the Joseph B. Whitehead Foundation, and the Marcus Foundation (all via his institution) during the conduct of the study; being a scientific cofounder of and owning equity in EarliTec Diagnostics; and holding patents licensed to EarliTec Diagnostics outside the submitted work. Mr Paredes reported receiving funding and/or equipment from the Joseph B. Whitehead Foundation and the Marcus Foundation (both via his institution) during the conduct of the study. Dr Marrus reported receiving grants from the NIMH and from Washington University in St Louis School of Medicine during the conduct of the study. Dr Constantino reported receiving grants from the NICHD during the conduct of the study and royalties from Western Psychological Services outside the submitted work. Dr Klin reported receiving grants from the Georgia Research Alliance, the Joseph B. Whitehead Foundation, the Marcus Foundation, the NICHD, and the NIMH (all via his institution) during the conduct of the study; being a scientific cofounder of and owning equity in EarliTec Diagnostics; holding patents licensed to EarliTec Diagnostics; and receiving personal fees from the Alliance for Early Success, the Brazilian Society of Speech Therapy, EarliTec Diagnostics, the McKnight Endowment Fund for Neuroscience, the National Autism Conference, and Washington University in St Louis School of Medicine outside the submitted work. No other disclosures were reported.

Figures

Figure 1.. Participant Enrollment and Outcomes for Comparing Objective Measurements of Social Visual Engagement With Expert Clinical Diagnosis of Autism in Discovery and Replication Studies

A, Participant flow for the discovery study. B, Participant flow for the replication study. In both studies, during a single visit at the testing site, enrolled participants received expert clinical diagnosis using standardized assessments (reference standard diagnosis) as well as eye-tracking–based measurement of social visual engagement (index test). Index test quality control indicator (QCIs) failures occurred when participants’ data failed to meet automated preset data QCIs (additional details are available in eTables 2 and 3 in Supplement 1).

Figure 2.. Diagnostic Performance Comparing Eye-Tracking–Based Measurement of Social Visual Engagement (Index Test) With Expert Clinical Diagnosis of Autism (Reference Standard)

Performance among 711 children in the discovery study and 361 children in the replication study. A, The diamond represents the optimal test positivity threshold for the discovery study (Youden index). B, The test positivity threshold determined in the discovery study was fixed and applied independently in the replication study. The diamond represents the achieved sensitivity and specificity in the replication study using the test positivity threshold from the discovery study. The solid blue circle represents the post hoc theoretical optimal threshold. C, Tabulation corresponds to the diamond in panel A. D, Tabulation corresponds to the diamond in panel B. AUC indicates area under the curve; ROC, receiver operating characteristic. Negative predictive value (NPV) and positive predictive value (PPV) estimates reported here are calculated based on study sample prevalence.

Figure 3.. Convergent Validity Between Eye-Tracking–Based Measurement of Social Visual Engagement (Index Test) and Expert Clinician–Administered Standardized Assessments of Social Disability, Verbal Ability, and Nonverbal Cognitive Ability

A, Discovery study correlation between eye-tracking–based indices of social disability versus children’s total scores on the Autism Diagnostic Observation Schedule, second edition (ADOS-2). B, Discovery study correlation between eye-tracking–based indices of verbal ability versus children’s verbal age equivalent scores as measured by the Mullen Scales of Early Learning (Mullen). C, Discovery study correlation between eye-tracking–based indices of nonverbal cognitive ability versus children’s nonverbal age equivalent scores as measured by the Mullen. D, E, F, Replication study correlations between eye-tracking–based indices and reference standard assessments. In all scatterplots, circles represent individual data and diamonds represent regression outliers (bivariate outliers identified using Cook distance and difference-in-fits regression diagnostic assessment). The adjusted R2 values were adjusted for measurement error variance of the reference standard (yielding percentage of reference standard nonerror variance explained by the index test). Additional information is provided in the Secondary End Point Analyses subsection of the eMethods in Supplement 1.

Figure 4.. Performance-Based Measures of Children’s Individual Strengths, Vulnerabilities, and Opportunities for Skill Development

Measurement of social visual engagement quantifies how a child engages with social and nonsocial cues occurring continuously within naturalistic environmental contexts (left column, shown as still frames from testing videos). In relation to those contexts, normative reference measures provide objective quantification of nonautism age-expected visual engagement (middle columns, shown as density distributions in both pseudocolor format and as color to grayscale fades overlaid on corresponding still frames). The age-expected reference measures can be used to measure and visualize patient comparisons, revealing individual strengths, vulnerabilities, and opportunities for skill building (right columns, sample patient data shown as overlaid circular apertures that encompass the portion of video foveated by each patient [each aperture spans the central 5.2 degrees of a patient’s visual field]). Children with autism present as engaging with toys of interest (1, 3, 5, and 7), color and contrast cues (2, 6, 8, and 9), and objects and background elements not directly relevant to social context (4 and 10-14). Elapsed times at the bottom right of still frames highlight the rapidly changing nature of social interaction in which many hundreds of verbal and nonverbal communicative cues are presented, each eliciting age-expected patterns of engagement and offering corresponding opportunities for objective quantitative comparisons of patient behavior.