Blending of brain-machine interface and vision-guided autonomous robotics improves neuroprosthetic arm performance during grasping

Abstract

Background: Recent studies have shown that brain-machine interfaces (BMIs) offer great potential for restoring upper limb function. However, grasping objects is a complicated task and the signals extracted from the brain may not always be capable of driving these movements reliably. Vision-guided robotic assistance is one possible way to improve BMI performance. We describe a method of shared control where the user controls a prosthetic arm using a BMI and receives assistance with positioning the hand when it approaches an object.

Methods: Two human subjects with tetraplegia used a robotic arm to complete object transport tasks with and without shared control. The shared control system was designed to provide a balance between BMI-derived intention and computer assistance. An autonomous robotic grasping system identified and tracked objects and defined stable grasp positions for these objects. The system identified when the user intended to interact with an object based on the BMI-controlled movements of the robotic arm. Using shared control, BMI controlled movements and autonomous grasping commands were blended to ensure secure grasps.

Results: Both subjects were more successful on object transfer tasks when using shared control compared to BMI control alone. Movements made using shared control were more accurate, more efficient, and less difficult. One participant attempted a task with multiple objects and successfully lifted one of two closely spaced objects in 92 % of trials, demonstrating the potential for users to accurately execute their intention while using shared control.

Conclusions: Integration of BMI control with vision-guided robotic assistance led to improved performance on object transfer tasks. Providing assistance while maintaining generalizability will make BMI systems more attractive to potential users.

Trial registration: NCT01364480 and NCT01894802 .

Keywords: Assistive technology; Brain-computer interface; Brain-machine interface; Neuroprosthetic; Shared mode control.

Figures

Fig. 1

Array location. The approximate location of the microelectrode recording arrays for both subjects on a template brain. Subject 1 had 2 96-channel arrays implanted in M1 (green squares). Subject 2 had 2 88-channel arrays implanted in S1 (yellow squares) and 2 32-channel arrays implanted more posterior (yellow rectangles)

Fig. 2

Shared control system diagram and robot testing set up. a System diagram for the vision-guided shared control. The blue boxes show the BMI system decoding endpoint translational and grasp velocity. The green boxes show the components of the vision-guided robotic system for grasping. If shared control was not in use, only the output of the BMI system was used to send commands to the arm, but with shared control, the control signal of the vision-guided system was blended with that of the BMI system to create the final robot command. b The 7.5 cm cube (yellow) and the target box (clear box) were positioned on the table, as shown, to start the ARAT trials. The subject sat approximately 1 m to the left of the robot. c An example of the central cross-section of the grasp envelope for a stable grasp position on a 7.5 cm cube is outlined by the blue dotted line. The shading shows the gradient of shared control (α value), with white areas being completely controlled by the BMI user and darker areas having more robot control. d A trial progression schematic showing when translation and grasp control are under BMI control (blue) or robot control (green). Wrist orientation was always maintained in a neutral posture under computer control

Fig. 3

Target positions for the multiple object task. The 7.5 cm target cubes filled the squares in the diagram and were separated by 10 cm. For a single trial, the cubes were placed at 2 positions connected by dashed lines, and the subject was instructed to pick up 1 of the 2 cubes. The position numbers correspond to the target numbers in Table 2. The cube in Fig. 2b is at the same point on the table as the intersection of the dashed lines here. The “Cameras” box and hand position arrow indicate the location of those components of the robot at the start of the trial

Fig. 4

ARAT performance and difficulty. a The frequency of each trial result for Subject 1 (left) and Subject 2 (right). Completion times are shown for successful trials and the failure mode (time out or out of bounds) is noted for failed trials. Assisted (blue bars) and unassisted (red bars) trials are shown separately. b The frequency of each reported difficulty score for assisted and unassisted trial sets (1 = extremely easy, 10 = extremely difficult). Both subjects were more successful and reported that the task was easier during the trials with shared control

Fig. 5

Analysis of trajectory properties with and without shared control. a A box plot distribution of hand translation speeds across all time bins while the hand was less than 10 cm above the table during successful trials. The red line is the median speed, the blue box show the interquartile region, and the whiskers span the 5th-95th percentile. The speed distribution for assisted trials for both subjects skews low indicating that the hand was steadier when approaching the object. b Subject 1’s path lengths during successful trials, first for the full trials, then separated by the path length before the first grasp attempt and the path length after the object was grasped. Error bars span the interquartile region. The assisted trials benefit the most during the pre-grasp portion of the trial. c Subject 1’s hand trajectories with median path lengths for their assistance condition. The color shows the grasp aperture. The release point is marked where the hand opened to allow the object to drop onto the platform. We did not specify to the subject how the object had to be placed, or released, onto the platform. Additional file 2: Movie S2 shows both trials