Hearables: Multimodal physiological in-ear sensing

Valentin Goverdovsky, Wilhelm von Rosenberg, Takashi Nakamura, David Looney, David J Sharp, Christos Papavassiliou, Mary J Morrell, Danilo P Mandic, Valentin Goverdovsky, Wilhelm von Rosenberg, Takashi Nakamura, David Looney, David J Sharp, Christos Papavassiliou, Mary J Morrell, Danilo P Mandic

Abstract

Future health systems require the means to assess and track the neural and physiological function of a user over long periods of time, and in the community. Human body responses are manifested through multiple, interacting modalities - the mechanical, electrical and chemical; yet, current physiological monitors (e.g. actigraphy, heart rate) largely lack in cross-modal ability, are inconvenient and/or stigmatizing. We address these challenges through an inconspicuous earpiece, which benefits from the relatively stable position of the ear canal with respect to vital organs. Equipped with miniature multimodal sensors, it robustly measures the brain, cardiac and respiratory functions. Comprehensive experiments validate each modality within the proposed earpiece, while its potential in wearable health monitoring is illustrated through case studies spanning these three functions. We further demonstrate how combining data from multiple sensors within such an integrated wearable device improves both the accuracy of measurements and the ability to deal with artifacts in real-world scenarios.

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Figure 1
Figure 1
Construction of the multimodal in-ear sensing device. (a) Detailed structure of the device, showing the placement of the microphone and the electrode on the substrate. (b) Construction of the multimodal sensor underneath one of the cloth electrodes. (c) Completed earpiece with electrodes and inward-facing microphone visible. (d) Placement of the earpiece in the user’s ear.
Figure 2
Figure 2
Mechanical tests comparing the viscoelastic foam substrate and silicone. (a) Stress-Strain curve for viscoelastic foam cylinders of varying density. (b) EEG signal captured with the foam-based earplug from the ear canal of a person with strong blood vessel pulsation, observe no pulsation in the recorded EEG. (c) Stress-Strain curve for silicone cylinders of varying hardness. (d) EEG signal captured using the silicone-based earplug from the ear canal of the same person with strong blood vessel pulsation, observe strong contamination of the EEG with pulsatile artifacts, indicated by red crosses.
Figure 3
Figure 3
Simulated electric potentials in the head at time 0 s, generated by dipoles which oscillated at 39 Hz, with positive potentials shown in red and negative potentials in blue. (a) Potentials on the whole scalp; (b) Potentials in the coronal plane; (c) Potentials in the right ear region seen from inside the head (note that (a–c) have different scales); (d) Waveforms of potential differences between electrode pairs on the scalp and inside the ear canal. The T8 and Cz positions are standard on-scalp EEG electrodes and further abbreviations are: RE D: Right ear canal, downward direction; RE U: Right ear canal, upward direction; RE H: Right ear, root of the helix. The positions of the ear electrodes are marked with black circles in panel (c).
Figure 4
Figure 4
Acquisition of EEG and respiration from inside the ear canal. (a) Stability of the electrode-skin interface over the course of 8 hours, for 5 subjects. (b) ASSR response measured from the ear, M1 and Cz scalp locations. (c) SSVEP response measured from the ear, M1 and Cz locations. (d) Visual evoked potential measured with the earpiece. (e) Alpha rhythm recorded from the ear electrodes, 30 s into the trial subject closed their eyes. (f,g) (Top and Middle) Spectrograms of acoustic signals measured by the two microphones (inward- and wall-facing) integrated within the earpiece when the subject was asked to breath at 16 and 28 breaths per minute, respectively. (f,g) (Bottom) Breathing signals extracted from the associated spectrograms. (h) EEG signals from scalp and ear canal during the N2 stage of sleep. (i) EEG signals from scalp and ear canal when the subject was awake, but with eyes closed.
Figure 5
Figure 5
Denoising of ear-EEG from mechanical jaw clench artifacts. (a) Best-case denoising scenario for which the artifact measurement with the embedded microphone was accurate. (b) Worst-case accuracy of artifact measurement with the embedded microphone, note that the artifact was nevertheless reduced in the Clean EEG trace. (Top panels) Raw EEG corrupted by a strong artifact. (Middle panels) Output of the mechanical sensor within the MMS. (Bottom panels) Denoised EEG using the mechanical MMS signal as reference.
Figure 6
Figure 6
Recording of speech from the multimodal earpiece, illustrated through a 10 s spectrogram. (Top) Speech signal recorded using a microphone placed on the tip of the earpiece looking towards the eardrum. (Bottom) Speech signal from an external microphone placed on a subject’s chest. The Y-axis for both spectrograms spans the 300–3400 Hz band, the so-called voice frequency band.
Figure 7
Figure 7
Cardiac activity captured from inside the ear canal using the multimodal sensor embedded within the earpiece. (Top) ECG signal from the arms. (Middle) PPG signal from the finger. (Bottom) Integral of the mechanical signal produced by the microphone within the multimodal sensor.

References

    1. Lee, S. M. et al. Self-adhesive epidermal carbon nanotube electronics for tether-free long-term continuous recording of biosignals. Sci. Rep. 4 (2014).
    1. Ko D, Lee C, Lee EJ, Lee SH, Jung KY. A dry and flexible electrode for continuous-EEG monitoring using silver balls based polydimethylsiloxane (PDMS) Biomed. Eng. Lett. 2012;2:18–23. doi: 10.1007/s13534-012-0049-8.
    1. Myers AC, Huang H, Zhu Y. Wearable silver nanowire dry electrodes for electrophysiological sensing. RSC Adv. 2015;5:11627–11632. doi: 10.1039/C4RA15101A.
    1. Salvo P, et al. A 3D printed dry electrode for ECG/EEG recording. Sensors Actuators, A Phys. 2012;174:96–102. doi: 10.1016/j.sna.2011.12.017.
    1. Chi YM, Jung TP, Cauwenberghs G. Dry-contact and noncontact biopotential electrodes: Methodological review. IEEE Rev. Biomed. Eng. 2010;3:106–119. doi: 10.1109/RBME.2010.2084078.
    1. Chiou J-C, Ko L-W, Lin C-T. Using novel MEMS EEG nensors in detecting drowsiness application. Proc. IEEE Biomed. Circuits Syst. Conf. 2006;95:33–36.
    1. Grozea C, Voinescu CD, Fazli S. Bristle-sensors – low-cost flexible passive dry EEG electrodes for neurofeedback and BCI applications. J. Neural Eng. 2011;8:025008. doi: 10.1088/1741-2560/8/2/025008.
    1. Lopez-Gordo MA, Sanchez-Morillo D, Pelayo Valle F. Dry EEG electrodes. Sensors (Basel) 2014;14:12847–12870. doi: 10.3390/s140712847.
    1. Norton JJS, et al. Soft, curved electrode systems capable of integration on the auricle as a persistent brain–computer interface. Proc. Natl. Acad. Sci. 2015;112:3920–3925. doi: 10.1073/pnas.1424875112.
    1. Looney D, et al. The in-the-ear recording concept: User-centered and wearable brain monitoring. IEEE Pulse. 2012;3:32–42. doi: 10.1109/MPUL.2012.2216717.
    1. Hoon Lee J, et al. CNT/PDMS-based canal-typed ear electrodes for inconspicuous EEG recording. J. Neural Eng. 2014;11:046014. doi: 10.1088/1741-2560/11/4/046014.
    1. Debener S, Emkes R, De Vos M, Bleichner M. Unobtrusive ambulatory EEG using a smartphone and flexible printed electrodes around the ear. Sci. Rep. 2015;5:16743. doi: 10.1038/srep16743.
    1. Goverdovsky V, Looney D, Kidmose P, Mandic DP. In-ear EEG from viscoelastic generic earpieces: Robust and unobtrusive 24/7 monitoring. IEEE Sensors Journal. 2016;16:271–277. doi: 10.1109/JSEN.2015.2471183.
    1. Looney D, Goverdovsky V, Rosenzweig I, Morrell MJ, Mandic DP. Wearable in-ear encephalography sensor for monitoring sleep. Preliminary Observations from Nap Studies. Ann. Am. Thorac. Soc. 2016;13:2229–2233. doi: 10.1513/AnnalsATS.201605-342BC.
    1. Goverdovsky V, Looney D, Kidmose P, Papavassiliou C, Mandic DP. Co-located multimodal sensing: A next generation solution for wearable health. IEEE Sensors Journal. 2015;15:138–145. doi: 10.1109/JSEN.2014.2338612.
    1. Kidmose P, Looney D, Ungstrup M, Rank ML, Mandic DP. A study of evoked potentials from ear-EEG. IEEE Trans. Biomed. Eng. 2013;60:2824–2830. doi: 10.1109/TBME.2013.2264956.
    1. Herdman AT, et al. Intracerebral sources of human auditory steady-state responses. Brain Topography. 2002;15:69–86. doi: 10.1023/A:1021470822922.
    1. Di Russo F, Martínez A, Sereno MI, Pitzalis S, Hillyard SA. Cortical sources of the early components of the visual evoked potential. Hum. Brain Mapp. 2002;15:95–111. doi: 10.1002/hbm.10010.
    1. Eoh HJ, Chung MK, Kim SH. Electroencephalographic study of drowsiness in simulated driving with sleep deprivation. Int. J. Ind. Ergon. 2005;35:307–320. doi: 10.1016/j.ergon.2004.09.006.
    1. Simon M, et al. EEG alpha spindle measures as indicators of driver fatigue under real traffic conditions. Clin. Neurophysiol. 2011;122:1168–1178. doi: 10.1016/j.clinph.2010.10.044.
    1. Looney, D., Goverdovsky, V., Kidmose, P. & Mandic, D. P. Subspace denoising of EEG artefacts via multivariate EMD. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 4688–4692 (2014).
    1. Looney, D. & Mandic, D. A machine learning enhanced empirical mode decomposition. In Proc. IEEE Int. Conf. Acoust. Speech Signal Process., 1897–1900 (2008).
    1. Kubichek, R. Mel-cepstral distance measure for objective speech quality assessment. In Proc. IEEE Pacific Rim Conf. on Communications Computers and Signal Process., vol. 1, 125–128 vol. 1 (1993).
    1. Corbishley P, Rodríguez-Villegas E. Breathing detection: Towards a miniaturized, wearable, battery-operated monitoring system. IEEE Trans. Biomed. Eng. 2008;55:196–204. doi: 10.1109/TBME.2007.910679.
    1. Pressler, G., Mansfield, J., Pasterkamp, H. & Wodicka, G. Detection of respiratory sounds within the ear canal. In Proc. Second Jt. Annu. Conf. Annu. Fall Meet. Biomed. Eng. Soc., vol. 2, 1529–1530 (2002).
    1. Venema B, et al. Advances in reflective oxygen saturation monitoring with a novel in-ear sensor system: Results of a human hypoxia study. IEEE Trans. Biomed. Eng. 2012;59:2003–10. doi: 10.1109/TBME.2012.2196276.
    1. Rosenzweig I, et al. Changes in neurocognitive architecture in patients with obstructive sleep apnea treated with continuous positive airway pressure. EBioMedicine. 2016;7:221–229. doi: 10.1016/j.ebiom.2016.03.020.
    1. Andreuccetti, D., Fossi, R. & Petrucci, C. An internet resource for the calculation of the dielectric properties of body tissues in the frequency range 10 Hz–100 GHz. IFAC-CNR, Florence (Italy) (1997).
    1. Gabriel C, Gabriel S, Corthout E. The dielectric properties of biological tissues: I. Literature survey. Physics in Medicine and Biology. 1996;41:2231–2249. doi: 10.1088/0031-9155/41/11/001.
    1. Gabriel S, Lau RW, Gabriel C. The dielectric properties of biological tissues: II. Measurements in the frequency range 10 Hz to 20 GHz. Physics in Medicine and Biology. 1996;41:2251–2269. doi: 10.1088/0031-9155/41/11/002.
    1. Gabriel S, Lau RW, Gabriel C. The dielectric properties of biological tissues: III. Parametric models for the dielectric spectrum of tissues. Physics in Medicine and Biology. 1996;41:2271–2293. doi: 10.1088/0031-9155/41/11/003.

Source: PubMed

Sponsorit ja yhteistyökumppanit

Sairaudet

Huumeiden interventiot

3
Tilaa