A Characterization of Brain-Computer Interface Performance Trade-Offs Using Support Vector Machines and Deep Neural Networks to Decode Movement Intent

Nicholas D Skomrock, Michael A Schwemmer, Jordyn E Ting, Hemang R Trivedi, Gaurav Sharma, Marcia A Bockbrader, David A Friedenberg, Nicholas D Skomrock, Michael A Schwemmer, Jordyn E Ting, Hemang R Trivedi, Gaurav Sharma, Marcia A Bockbrader, David A Friedenberg

Abstract

Laboratory demonstrations of brain-computer interface (BCI) systems show promise for reducing disability associated with paralysis by directly linking neural activity to the control of assistive devices. Surveys of potential users have revealed several key BCI performance criteria for clinical translation of such a system. Of these criteria, high accuracy, short response latencies, and multi-functionality are three key characteristics directly impacted by the neural decoding component of the BCI system, the algorithm that translates neural activity into control signals. Building a decoder that simultaneously addresses these three criteria is complicated because optimizing for one criterion may lead to undesirable changes in the other criteria. Unfortunately, there has been little work to date to quantify how decoder design simultaneously affects these performance characteristics. Here, we systematically explore the trade-off between accuracy, response latency, and multi-functionality for discrete movement classification using two different decoding strategies-a support vector machine (SVM) classifier which represents the current state-of-the-art for discrete movement classification in laboratory demonstrations and a proposed deep neural network (DNN) framework. We utilized historical intracortical recordings from a human tetraplegic study participant, who imagined performing several different hand and finger movements. For both decoders, we found that response time increases (i.e., slower reaction) and accuracy decreases as the number of functions increases. However, we also found that both the increase of response times and the decline in accuracy with additional functions is less for the DNN than the SVM. We also show that data preprocessing steps can affect the performance characteristics of the two decoders in drastically different ways. Finally, we evaluated the performance of our tetraplegic participant using the DNN decoder in real-time to control functional electrical stimulation (FES) of his paralyzed forearm. We compared his performance to that of able-bodied participants performing the same task, establishing a quantitative target for ideal BCI-FES performance on this task. Cumulatively, these results help quantify BCI decoder performance characteristics relevant to potential users and the complex interactions between them.

Keywords: brain-computer interface; decoding; deep learning; machine learning; response time; support vector machines.

Figures

Figure 1
Figure 1
Experimental design. (A) The participant performing the six-movement task using the BCI-FES system. The cued movement is displayed and the participant is asked to imagine replicating the cued movement. (B) Design of the six-movement task where each movement is repeated for a total of four replicates in random order. Each cue remains for 2.5 s with variable rest time of 2.5–4.5 s between the cues. (C) Sample decoder output demonstrating the response time. Response time is the difference between the start of the cue to when the decoder output initiates the correct movement. To be successful, the movement must be sustained for a minimum of 1 s. (D) Impact of the boxcar filter. The yellow line shows the wavelet coefficient across time for a single channel and wavelet scale, whereas the gray line shows the same data with the boxcar filter applied. While the two lines track the same general trends, there is substantially more variation without the filter.
Figure 2
Figure 2
(A) Success rate for both the two-movement and four-movement offline tasks during the testing period. The redlines represent the performance of the DNN and the blue line represents the SVM. Solid lines and circles use the boxcar filter and dashed lines with triangles are without the filter. (B) Accuracy as a function of days since last training session. The redlines are the performance of the DNN and the blue line is the performance of the SVM. Solid lines and circles are decoders using the boxcar filter and dashed lines with triangles are without the boxcar filter.
Figure 3
Figure 3
Histograms of response times for the offline tasks. The distribution response times for each successful hand movement across the testing period, broken out by decoder-filter combination and task. The two-movement task is in blue and the four-movement task is in green. Mean response times for a decoder-filter combination and hand movement are represented by solid black lines.
Figure 4
Figure 4
(A) Accuracy as a function of the number of movements during the offline simulation. The accuracy across each of the testing days in the simulation, with the DNN is in red and the SVM is in blue. A jitter is applied to easy visualization and a linear regression model is fit to the original data. (B) Response time as a function of the number of movements during the offline simulation. The mean response time of successful movements for the DNN and SVM color coded by the different hand movements. As the number of movements increase, so does the mean response time for both model types.
Figure 5
Figure 5
Response time as a function of accuracy for the offline simulation. For each number of hand movements in the simulation, the mean response time is plotted against the accuracy for each separate test days. The SVM with the boxcar is in blue and the DNN is in red. Linear regression models were fit to each set of points to provide interpretation of the trends. For the SVM, all of the models appear to be flat, allowing for large variation in accuracy while maintaining similar response times. For the DNN, the trend shows a decrease in response time with an increase in accuracy.
Figure 6
Figure 6
(A) Demonstration of data processing for the real-time demonstration. The decoder is used to decipher the neural signals to determine the desired hand movement. This information is passed to a stimulator the uses electrical signal to stimulate the participants arm to evoke the desired hand movement. (B) Response time layout. The two hands on the screen are the cue hand (lower-left corner) and the decoded output (center) that play in real-time. In the upper-left corner a stop watch is synched with the cue to obtain time measurements for the hand movements. (C) Able-bodied compared to SCI participant response times measured from videos. This is a boxplot of the response time for the six-movement task broken out by each of the hand movements. The SCI is in yellow, and the aggregate of the three able-bodied participants are in gray.
Figure 7
Figure 7
Response times of the SCI participant as measured by the decoder output. The decoder output was used to determine response times in order to separate the time that can be attributed to the algorithm vs. the rest of the system. The response times based upon the decoder are about 200 ms less than the response times determined by the video, attributing 200 ms delay to the FES system.

References

    1. Abadi M., Agarwal A., Barham P., Brevdo E., Chen Z., Citro C., et al. (2016). TensorFlow: large-scale machine learning on heterogeneous distributed systems. ArXiv:1603.04467 [Cs], March. Available online at:
    1. Ajiboye A. B., Willett F. R., Young D. R., Memberg W. D., Murphy B. A., Miller J. P., et al. . (2017). Restoration of reaching and grasping movements through brain-controlled muscle stimulation in a person with tetraplegia: a proof-of-concept demonstration. Lancet 389, 1821–1830. 10.1016/S0140-6736(17)30601-3
    1. Bouton C. E., Shaikhouni A., Annetta N. V., Bockbrader M. A., Friedenberg D. A., Nielson D. M., et al. . (2016). Restoring cortical control of functional movement in a human with quadriplegia. Nature 533, 247–50. 10.1038/nature17435
    1. Chaudhary U., Birbaumer N., Ramos-Murguialday A. (2016). Brain-computer interfaces for communication and rehabilitation. Nat. Rev. Neurol. 12, 513–525. 10.1038/nrneurol.2016.113
    1. Chollet F. (2015). Keras. GitHub. Available online at:
    1. Colachis S., IV., Bockbrader M., Zhang M., Friedenberg D., Annetta N., Schwemmer M., et al. (2018). Dexterous control of seven functional hand movements using cortically-controlled non-invasive muscle stimulation in a tetraplegic person. Front. Neurosci. 12:208 10.3389/fnins.2018.00208
    1. Collinger J. L., Boninger M. L., Bruns T. M., Curley K., Wang W., Weber D. J. (2013a). Functional priorities, assistive technology, and brain-computer interfaces after spinal cord injury. J. Rehabil. Res. Dev. 50, 145–60. 10.1682/JRRD.2011.11.0213
    1. Collinger J. L., Wodlinger B., Downey J. E., Wang W., Tyler-Kabara E. C., Weber D. J., et al. . (2013b). High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381, 557–64. 10.1016/S0140-6736(12)61816-9
    1. Evans N., Gale S., Schurger A., Blanke O. (2015). Visual feedback dominates the sense of agency for brain-machine actions. PLOS ONE 10:e0130019. 10.1371/journal.pone.0130019
    1. Fernández-Delgado M., Cernadas E., Barro S., Amorim D. (2014). Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15, 3133–3181. Available online at:
    1. Friedenberg D. A., Bouton C. E., Annetta N. V., Skomrock N., Zhang M., Schwemmer M., et al. (2016). Big data challenges in decoding cortical activity in a human with quadriplegia to inform a brain computer interface, in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Orlando, FL: ), 3084–3087.
    1. Friedenberg D. A., Schwemmer M. A. (2016). Moving a paralyzed hand—a biomedical big data success story. Chance 29, 4–13. 10.1080/09332480.2016.1263093
    1. Friedenberg D. A., Schwemmer M. A., Landgraf A. J., Annetta N. V., Bockbrader M. A., Bouton C. E., et al. . (2017). Neuroprosthetic-enabled control of graded arm muscle contraction in a paralyzed human. Sci. Rep. 7:8386. 10.1038/s41598-017-08120-9
    1. Gatys L. A., Ecker A. S., Bethge M. (2016). Image style transfer using convolutional neural networks, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Las Vegas, NV: IEEE; ), 2414–2423.
    1. Gilja V., Pandarinath C., Blabe C. H., Nuyujukian P., Simeral J. D., Sarma A. A., et al. . (2015). Clinical translation of a high performance neural prosthesis. Nat. Med. 21, 1142–1145. 10.1038/nm.3953
    1. Glaser J. I., Chowdhury R. H., Perich M. G., Miller L. E., Kording K. P. (2017). Machine Learning for Neural Decoding. ArXiv:1708.00909 [Cs, q-Bio, Stat], August. Available online at:
    1. Goodfellow I., Bengio Y., Courville A. (2016). Deep Learning. Adaptative Computation and Machine Learning Series. Cambridge, MA: The MIT Press.
    1. Hochberg L. R., Bacher D., Jarosiewicz B., Masse N. Y., Simeral J. D., Vogel J., et al. . (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485, 372–375. 10.1038/nature11076
    1. Hochreiter S., Schmidhuber J. (1997). Long short-term memory. Neural Comput. 9, 1735–80. 10.1162/neco.1997.9.8.1735
    1. Huggins J. E., Moinuddin A. A., Chiodo A. E., Wren P. A. (2015). What would brain-computer interface users want: opinions and priorities of potential users with spinal cord injury. Archiv. Phys. Med. Rehabil. 96(3 Suppl.), S38–S45.e5. 10.1016/j.apmr.2014.05.028
    1. Huggins J. E., Wren P. A., Gruis K. L. (2011). What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotroph. Lateral Scler. 12, 318–24. 10.3109/17482968.2011.572978
    1. Jarosiewicz B., Sarma A. A., Bacher D., Masse N. Y., Simeral J. D., Sorice B., et al. . (2015). Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface. Sci. Transl. Med. 7:313ra179. 10.1126/scitranslmed.aac7328
    1. Kageyama Y., Hirata M., Yanagisawa T., Shimokawa T., Sawada J., Morris S., et al. . (2014). Severely affected ALS patients have broad and high expectations for brain-machine interfaces. Amyotroph. Lateral Scler. Frontotemp. Degener. 15, 513–519. 10.3109/21678421.2014.951943
    1. Kao J. C., Stavisky S. D., Sussillo D., Nuyujukian P., Shenoy K. V. (2014). Information systems opportunities in brain-machine interface decoders. Proc. IEEE 102, 666–682. 10.1109/JPROC.2014.2307357
    1. Kilgore K. L., Scherer M., Bobblitt R., Dettloff J., Dombrowski D. M., Godbold N., et al. . (2001). Neuroprosthesis consumers' forum: consumer priorities for research directions. J. Rehabil. Res. Dev. 38, 655–660. Available online at:
    1. Lebedev M. (2014). Brain-machine interfaces: an overview. Transl. Neurosci. 5, 99–110. 10.2478/s13380-014-0212-z
    1. LeCun Y., Bengio Y., Hinton G. (2015). Deep learning. Nature 521, 436–444. 10.1038/nature14539
    1. Lotte F., Congedo M., Lécuyer A., Lamarche F., Arnaldi B. (2007). A review of classification algorithms for EEG-based brain–computer interfaces. J. Neural Eng. 4:R1. 10.1088/1741-2560/4/2/R01
    1. Moore J. W. (2016). What Is the sense of agency and why does it matter? Front. Psychol. 7:1272. 10.3389/fpsyg.2016.01272
    1. Nuyujukian P., Kao J. C., Fan J. M., Stavisky S. D., Ryu S. I., Shenoy K. V. (2014). Performance sustaining intracortical neural prostheses. J. Neural Eng. 11:066003. 10.1088/1741-2560/11/6/066003
    1. Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Pedregosa F., et al. (2011). Scikit-learn: machine learning in python. J. Mach. Learn. Python 12, 2825–2830 Available online at:
    1. Ruder S. (2016). An Overview of Gradient Descent Optimization Algorithms. ArXiv:1609.04747 [Cs], September. Available online at:
    1. Santhanam G., Ryu S. I., Yu B. M., Afshar A., Shenoy K. V. (2006). A high-performance brain–computer interface. Nature 442, 195–98. 10.1038/nature04968
    1. Schwemmer M. A., Skomrock N. D., Sederberg B. P., Ting J. E., Sharma G., Bockbrader M. A., et al. . (2018). Meeting brain-computer interface user performance expectations using a deep neural network decoding framework. Nat. Med. (in press). 10.1038/s41591-018-0171-y
    1. Sharma G., Annetta N., Friedenberg D., Blanco T., Vasconcelos D., Shaikhouni A., et al. (2015). Time stability and coherence analysis of multiunit, single-unit and local field potential neuronal signals in chronically implanted brain electrodes. Bioelectron. Med. 2, 63–71. 10.15424/bioelectronmed.2015.00010
    1. Sharma G., Friedenberg D. A., Annetta N., Glenn B., Bockbrader M., Majstorovic C., et al. . (2016). Using an artificial neural bypass to restore cortical control of rhythmic movements in a human with quadriplegia. Sci. Rep. 6:33807. 10.1038/srep33807
    1. Simeral J. D., Kim S. P., Black M. J., Donoghue J. P., Hochberg L. R. (2011). Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. J. Neural Eng. 8:025027. 10.1088/1741-2560/8/2/025027
    1. Sitaram R., Ros T., Stoeckel L., Haller S., Scharnowski F., Lewis-Peacock J., et al. . (2017). Closed-loop brain training: the science of neurofeedback. Nat. Rev. Neurosci. 18, 86–100. 10.1038/nrn.2016.164
    1. Siuly S., Li Y. (2012). Improving the separability of motor imagery EEG signals using a cross correlation-based least square support vector machine for brain 2013; computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 20, 526–538. 10.1109/TNSRE.2012.2184838
    1. Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R. (2014). Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958. Available online at:
    1. Sussillo D., Churchland M. M., Kaufman M. T., Shenoy K. V. (2015). A neural network that finds a naturalistic solution for the production of muscle activity. Nat. Neurosci. 18, 1025–1033. 10.1038/nn.4042
    1. Sussillo D., Stavisky S. D., Kao J. C., Ryu S. I., Shenoy K. V. (2016). Making brain–machine interfaces robust to future neural variability. Nat. Commun. 7:13749. 10.1038/ncomms13749
    1. Thomas E., Dyson M., Clerc M. (2013). An analysis of performance evaluation for motor-imagery based BCI. J. Neural Eng. 10:031001. 10.1088/1741-2560/10/3/031001
    1. Thompson D. E., Quitadamo L. R., Mainardi L., Laghari K. U., Gao S., Kindermans P. J., et al. . (2014). Performance measurement for brain–computer or brain–machine interfaces: a tutorial. J. Neural Eng. 11:035001. 10.1088/1741-2560/11/3/035001
    1. Willett F. R., Pandarinath C., Jarosiewicz B., Murphy B. A., Memberg W. D., Blabe C. H., et al. . (2017). Feedback control policies employed by people using intracortical brain-computer interfaces. J. Neural Eng. 14:016001. 10.1088/1741-2560/14/1/016001

Source: PubMed

赞助者和合作者

医疗条件

药物干预

3
订阅