Reach and grasp by people with tetraplegia using a neurally controlled robotic arm

Abstract

Paralysis following spinal cord injury, brainstem stroke, amyotrophic lateral sclerosis and other disorders can disconnect the brain from the body, eliminating the ability to perform volitional movements. A neural interface system could restore mobility and independence for people with paralysis by translating neuronal activity directly into control signals for assistive devices. We have previously shown that people with long-standing tetraplegia can use a neural interface system to move and click a computer cursor and to control physical devices. Able-bodied monkeys have used a neural interface system to control a robotic arm, but it is unknown whether people with profound upper extremity paralysis or limb loss could use cortical neuronal ensemble signals to direct useful arm actions. Here we demonstrate the ability of two people with long-standing tetraplegia to use neural interface system-based control of a robotic arm to perform three-dimensional reach and grasp movements. Participants controlled the arm and hand over a broad space without explicit training, using signals decoded from a small, local population of motor cortex (MI) neurons recorded from a 96-channel microelectrode array. One of the study participants, implanted with the sensor 5 years earlier, also used a robotic arm to drink coffee from a bottle. Although robotic reach and grasp actions were not as fast or accurate as those of an able-bodied person, our results demonstrate the feasibility for people with tetraplegia, years after injury to the central nervous system, to recreate useful multidimensional control of complex devices directly from a small sample of neural signals.

Figures

Figure 1

Experimental setup and performance metrics. (a) Diagram showing an overhead view of participant’s location at the table (grey rectangle) from which the targets (purple spheres) were elevated by a motor. The robotic arm was positioned to the right and slightly in front of the participant (the DLR and DEKA arms were mounted in slightly different locations to maximize the correspondence of their workspaces over the table; for details, see Supplementary Fig. 9). Both video cameras were used for all DLR and DEKA sessions; labels indicate which camera was used for the photographs in (b). (b) Photographs of the DLR (left panel) and DEKA (right panel) robots. (c) Reconstruction of an example trial in which the participant moved the DEKA arm in all three dimensions to successfully reach and grasp a target. The top panel illustrates the trajectory of the hand in 3D space. The middle panel shows the position of the wrist joint for the same trajectory decomposed into each of its three dimensions relative to the participant: the left-to-right axis (dashed blue line), the near-to-far axis (purple line) and the up-down axis (green line). The bottom panel shows the threshold crossing events from all units that contributed to decoding the movement. Each row of tick marks represents the activity of one unit and each tick mark represents a threshold crossing. The grey shaded area shows the first 1 sec of the grasp. (d) An example trajectory from a DLR session in which the participant needed to move the robot hand, which started to the left of the target, around and to the right of the target in order to approach it with the open part of the hand. The middle and bottom panels are analogous to (c). (e) Percentage of trials in which the participant successfully touched the target with the robotic hand (blue bars) and successfully grasped the target (red bars). (f) Average time required to touch (blue bars) or grasp (red bars) the targets. Each circle shows the acquisition time for one successful trial.

Figure 2

Participant S3 drinking from a bottle using the DLR robotic arm. (a) Four sequential images from the first successful trial showing participant S3 using the robotic arm to grasp the bottle, bring it towards her mouth, drink coffee from the bottle through a straw (her standard method of drinking), and place the bottle back on the table. The researcher in the background was positioned to monitor the participant and robotic arm. (See Supplementary Movie 1 from which these frames are extracted).

Figure 3

Examples of neural signals from three sessions and two participants: a 3D reach and grasp session from S3 (a–c) and T2 (d–f), and the 2D drinking session from S3 (g–i). (a,d,g) Average waveforms (thick black lines) ± 2 standard deviations (grey shadows) from two units from each session with a large directional modulation of activity. (b,e,h) Rasters and histograms of threshold crossings showing directional modulation. Each row of tick marks represents a trial, and each tick mark represents a threshold crossing event. The histogram summarizes the average activity across all trials in that direction. Rasters are displayed for arm movements to and from the pair of opposing targets that most closely aligned with the selected units’ preferred directions. (b) and (e) include both closed-loop filter calibration trials and assessment trials and (h) includes only filter calibration trials. Time 0 indicates the start of the trial. The dashed vertical line 1.8 seconds before the start of the trial identifies the time when the target for the upcoming trial began to rise. Activity occurring before this time corresponded to the end of the previous trial, which often included a grasp, followed by the lowering of the previous target and the computer moving the hand to the next starting position if it wasn’t already there. (c,f,i) Rasters and histograms from calibration and assessment trials for units that modulated with intended grasp state. During closed-loop filter calibration trials, the hand automatically closed starting at time 0, cueing the participant to grasp; during assessment trials, the grasp state was decoded at time 0. Expanded data appear in Supplementary Fig 5.