Working memory representations in visual cortex mediate distraction effects

Abstract

Although the contents of working memory can be decoded from visual cortex activity, these representations may play a limited role if they are not robust to distraction. We used model-based fMRI to estimate the impact of distracting visual tasks on working memory representations in several visual field maps in visual and frontoparietal association cortex. Here, we show distraction causes the fidelity of working memory representations to briefly dip when both the memorandum and distractor are jointly encoded by the population activities. Distraction induces small biases in memory errors which can be predicted by biases in neural decoding in early visual cortex, but not other regions. Although distraction briefly disrupts working memory representations, the widespread redundancy with which working memory information is encoded may protect against catastrophic loss. In early visual cortex, the neural representation of information in working memory and behavioral performance are intertwined, solidifying its importance in visual memory.

Conflict of interest statement

The authors declare no competing interests.

© 2021. The Author(s).

Figures

Fig. 1. Attending a distractor stimulus impairs working memory performance.

A Participants (n = 7) performed a memory-guided saccade task while brain activity and gaze were recorded inside the scanner. Each trial began with a condition cue, reliably indicating whether a distractor would appear on that trial (70% of trials) or not (30%). On each trial, participants maintained the precise spatial position of a briefly presented visual target (12° eccentricity, random polar angle) over an extended 12 s memory delay. At the end of the delay, they executed a memory-guided saccade to the remembered position. The memory target was then re-presented, and participants fixated this location before returning to central fixation. During distractor-present trials, participants discriminated whether dots presented within a 2° diameter aperture were rotating clockwise or counterclockwise with a button press. Across runs, motion coherence was varied to achieve ~75% correct performance (actual mean = 73%). The distracting stimulus could appear within one of seven position bins (24° polar angle wide) around the screen relative to the WM target, evenly presented across trials, denoted by blue intervals relative to an example WM target position (inset). B Timing of task events and example gaze data. Top: trial events (start of delay, distractor, and response) were synchronized to the beginning of 750 ms imaging volumes. For subsequent fMRI analyses, we defined three trial epochs for further analyses (see below, Figs. 5–7) assuming ~4 s hemodynamic delay (PRE: volumes before distractor, DIST: volumes during distractor, POST: volumes after distractor). Bottom: eye-trace of all trials of each condition for an example participant (p02). Eye position eccentricity is plotted as a function of time; distractor-absent trials are plotted with positive values, and distractor-present trials are plotted with negative values. Note that gaze remains at fixation during distraction keeping the retinal position of the memory target constant. C Aligned final saccadic endpoints (all participants) for trials in which distractors were absent or present. All endpoints are aligned by rotating to a common spatial position (along the horizontal meridian at 12° eccentricity). D Memory error (standard deviation of the polar angle of saccade endpoints) varied with distractor presence (t-test, two-tailed, p = 0.039). Gray lines show individual participants (n = 7); colored circles show group mean (error bars reflect ±SEM). E Response time also varied based on distractor presence (t-test, two-tailed, p = 0.011). Colored circles show group mean (error bars reflect ±SEM); Gray lines show individual participants (n = 7). Analysis of behavioral performance across individual distractor location bins shown in Supplementary Fig. 1.

Fig. 2. BOLD responses sorted by voxel RF position during WM delay period.

A During distractor-absent trials, the average (±SEM) amplitude of BOLD responses was greater in voxels whose receptive fields—estimated using nonlinear population receptive field mapping, see Fig. 3A for an example hemisphere)—aligned with the WM target (RFin) compared to when the target was 180° away from voxels’ receptive fields (RFout). The inset to the right depicts an example of the RFin and RFout regions of the visual field with respect to the WM target location (see Methods section for more details). The amplitudes of persistent activity increased moving anterior in the dorsal stream ROIs from early visual cortex (V1–V3; V3AB) to parietal cortex (IPS0/1) to frontal cortex (IPS2/3), while the spatial selectivity (difference between RFin and RFout) decreased. Data from ventral (hV4) and lateral retinotopic regions (LO1) is also included for completeness. Time series were baseline-corrected by removing the mean activation from −2.25 to 0 s prior to delay period onset from each time series. B During distractor-present trials, we observed an additional phasic response time-locked to the distractor onset across all ROIs. C To further illustrate the distractor response, we averaged the BOLD responses in voxels whose RFs were aligned to the distractor position, regardless of the position of the WM target. The phasic responses were more robust in voxels with RFs that matched (RFin) compared to opposite to the distractor (RFout). The shaded areas denote the pre-distractor, distractor, and post-distractor epochs that are the target of later analyses. Results for individual ROIs shown in Supplementary Fig. 2, along with visualized voxel RF parameters, and all p-values are available in Supplementary Table 1.

Fig. 3. IEM-based reconstruction of WM and distractor representations.

A Each participant underwent retinotopic mapping to define ROIs in visual, parietal, and frontal cortex (V1–V3, V3AB, hV4, LO1, IPS0-3, and sPCS). Example hemisphere and participant shown (p02, LH). Each voxel’s time course is fit with a receptive field model. Color depicts preferred polar angle; thresholded at R2 ≥ 10%. B We estimated an inverted encoding model (IEM) for polar angle for each participant and ROI using a dataset reserved for this purpose (single-item memory-guided saccade task, 3–4.5 h/participant). Because we used a fixed encoding model,, reconstructions across different timepoints and task conditions can be directly compared to one another. Each timepoint of the spatial distractor dataset was reconstructed using this independently estimated model, and each row shows a reconstruction over polar angles computed as the sum of basis functions weighted by the corresponding reconstructed channel response (see refs. ,. Two example trials shown, and trials are not aligned to a common position. C In the example participant, reconstructions were aligned based on WM target positions (orange triangle), and separately averaged for each distractor bin position (cyan triangle at onset time). Note that both target and distractor representations can be seen in the reconstructions. D WM target reconstruction averaged over all distractor location bins. Because distractors are evenly presented around the screen with respect to WM locations (Fig. 1a), averaging across relative distractor positions reveals target-related spatial representations because distractor representations are washed out during the averaging procedure. E The same data as (C, D) now aligned to each trial’s distractor position (cyan triangles), and averaged separately for each relative distractor location bin (WM targets are at different locations relative to distractor; orange triangles). F Distractor location reconstruction averaged over all relative WM target location bins.

Fig. 4. Impact of distraction on the dynamics of WM representations.

Average reconstruction of WM target locations on distractor-absent (A) and distractor-present trials (B) across all participants (n = 7). c Reconstruction of distractor locations on distractor-present trials, where all trials were aligned to a fixed distractor location. Note that B and C include the same data, just aligned to different locations (see Fig. 3). Reconstruction strength (arbitrary units, a.u.) is greatest at the aligned location in each instance and represents the polar angle location of the WM target maintained over the entire delay period or the briefly presented distractor. D, E Fidelity of the neural representation of WM targets (D) and distractors (E). When activation peaks in the direction of the remembered target (after alignment), fidelity is positive; when there is no consistent activation peak, fidelity is near zero. Target fidelity on distractor-absent trials is robust and statistically significant throughout the delay period in all ROIs. When the distractor is present, fidelity drops, but remains significantly above zero for all ROIs except for one 750 ms TR in sPCS. Distractor fidelity is also statistically significant in all regions and is qualitatively most robust across extrastriate visual cortex (e.g., V3AB). Closed and open circles denote significance of p < 0.05, one-sided, FDR corrected and p < 0.05, one-sided, FDR uncorrected, respectively (one-sample t-test using null distribution derived from shuffled IEM; see Methods section). Error bars ±SEM.

Fig. 5. WM representations are transiently disrupted by an attended distractor.

A Independently trained model-based reconstructions of the WM target locations on distractor-absent (magenta) and distractor-present trials (blue) each averaged over three epochs of the memory delay. The epochs were composed of TRs before the distractor (3.75–5.25 s), during the distractor (8.25–9.75 s), and after the distractor (10.5–12 s), accounting for the hemodynamic delay. Error bars ±SEM. Changes in the baseline (vertical offset) of these reconstructions, which may be due to differences in mean univariate BOLD activation, do not impact fidelity computations (Fig. 5B). Note that during the distractor epoch, the reconstructions of the WM target locations appear weaker on distractor-present compared to distractor-absent trials. In some regions, this effect of the distractor lasts into the post-distractor epoch. B Group mean (error bars, ±SEM) fidelity of reconstructed WM targets on distractor-absent (magenta) and distractor-present (blue) trials separately for the pre-distraction, distraction, and post-distraction epochs. Thin gray lines connect mean distractor-absent (small magenta dots) and distractor-present (small blue dots) fidelity for individual participants (n = 7) for each delay epoch. The results from 2-way ANOVAs for each ROI (epoch and condition as factors; compared against a shuffled null) are marked by symbols to denote the significant main effects of condition (C), epoch (E), and the interaction between epoch and condition (X). The significant results of paired (two-tailed) t-tests between distractor-present and distractor-absent reconstructions per epoch, for each ROI, are marked with asterisks. In both cases, gray symbols denote p < 0.05, uncorrected, and black p < 0.05, FDR corrected across ROIs within-test. Results for individual ROIs shown in Supplementary Fig. 4, and all p-values available in Supplementary Table 4.

Fig. 6. Loss of WM fidelity during distraction cannot be explained by a different coding format.

A To evaluate the format of WM representations throughout distractor-present trials, we conducted a temporal generalization analysis using distractor-present trials to estimate an IEM (each timepoint in turn) which was used to reconstruct held-out distractor-present trials (each timepoint in turn; leave-one-run-out cross-validation). For each combination of training and testing timepoints, we compute the WM target representation fidelity. Four cartoon examples illustrate predicted results from this analysis under various (non-exhaustive) coding schemes. B Fidelity is strong across a large combination of training/testing timepoints during the delay period with no evidence of a transition to a new coding format during or after the distractor. In many ROIs (e.g., V3AB), results are consistent with a transient disruption in WM representation, but no change or morphing in representational format following distraction. White bars indicate epochs used for analyses in C, D. C Model-based reconstructions from a cross-temporal generalization analysis in which training and testing was performed on corresponding epochs of the delay (i.e., train IEM with PRE timepoints, reconstruct using PRE timepoints from trials in held-out run). Rows show reconstructions from each ROI from each epoch (error bars ±SEM). Qualitatively, a substantial dip in WM reconstruction strength is apparent during the DIST epoch, as in Fig. 5A. D Comparison of group mean (error bars ±SEM) fidelity during each trial epoch across model estimation procedures. Blue line shows data computed using an independent model (replotted from Fig. 5B); orange line shows data computed using the leave-one-run-out cross-validation procedure. Gray lines connect datapoints from individual participants (n = 7). We performed a 2-way repeated-measures ANOVA against a shuffled null for each ROI (factors model and trial epoch). Main effects of model are indicated by M, main effects of epoch are indicated by E, and interactions are indicated by X. Significant tests are shown in black (p < 0.05, FDR corrected across ROIs within test); trends are shown in gray (p < 0.05, no correction). Error bars ±SEM. No ROIs show a significant interaction between model and epoch (though a trend is seen in V3AB, which is largely driven by stronger WM target representations measured using the independent model). E Comparison of off-diagonal training/testing combinations, (group mean, error bars ±SEM, derived from n = 7 participants). To determine if models trained and tested on non-matched epochs were able to recover information, we measured fidelity from models trained on PRE and tested on POST, as well as trained on POST and tested on PRE delay epochs. All p’s < 0.05 when comparing each average fidelity against a null distribution of fidelity values computed as in Fig. 4. Data from all individual ROIs available in Supplementary Fig. 5; p-values for all tests available in Supplementary Table 6.

Fig. 7. Memory errors correlate with distractor-induced biases in WM representations in visual cortex.

A On distractor-present trials in which the distractor was presented within 12° polar angle from the WM target, we found an attractive bias such that behavioral WM responses were drawn toward the distractor (positive values indicate errors in the same direction as distractor; two-tailed permutation t-test, p = 0.006). B We quantified the trial-by-trial error of each WM reconstruction based on its circular mean (see Methods section) during the post-distractor epoch on distractor-present trials when the distractor was presented near the WM location. To determine whether behavioral WM responses were impacted by any offsets in these neural WM representations, for each ROI and participant we correlated each trial’s decoded WM representation error with the corresponding behavioral memory error. Example scatterplots shown for one participant and ROI; trend line shows least squares linear fit (all participants and ROIs are plotted in Supplementary Fig. 6). C Average (±SEM) neural/behavioral error correlation across participants based on decoded error from each ROI. Behavioral responses significantly correlated with errors in neural representations in V1–V3, but not other ROIs (p = 0.005, FDR corrected across ROIs; trial-level permutation test; see Methods section). There was no significant main effect of ROI (p = 0.08, permuted 1-way ANOVA). All p-values available in Supplementary Table 8. D Additionally, we sorted each trial into quartiles based on each neural bias and subsequently took the mean of neural bias and behavioral memory error within each group of trials per participant. Then, we computed the Pearson correlation coefficient using these binned data (one mean per participant per quartile, 28 data points total). Asterisks denote significant results, p < 0.05, FDR corrected.