Heart Rate Detection Using Microsoft Kinect: Validation and Comparison to Wearable Devices

Ennio Gambi, Angela Agostinelli, Alberto Belli, Laura Burattini, Enea Cippitelli, Sandro Fioretti, Paola Pierleoni, Manola Ricciuti, Agnese Sbrollini, Susanna Spinsante, Ennio Gambi, Angela Agostinelli, Alberto Belli, Laura Burattini, Enea Cippitelli, Sandro Fioretti, Paola Pierleoni, Manola Ricciuti, Agnese Sbrollini, Susanna Spinsante

Abstract

Contactless detection is one of the new frontiers of technological innovation in the field of healthcare, enabling unobtrusive measurements of biomedical parameters. Compared to conventional methods for Heart Rate (HR) detection that employ expensive and/or uncomfortable devices, such as the Electrocardiograph (ECG) or pulse oximeter, contactless HR detection offers fast and continuous monitoring of heart activities and provides support for clinical analysis without the need for the user to wear a device. This paper presents a validation study for a contactless HR estimation method exploiting RGB (Red, Green, Blue) data from a Microsoft Kinect v2 device. This method, based on Eulerian Video Magnification (EVM), Photoplethysmography (PPG) and Videoplethysmography (VPG), can achieve performance comparable to classical approaches exploiting wearable systems, under specific test conditions. The output given by a Holter, which represents the gold-standard device used in the test for ECG extraction, is considered as the ground-truth, while a comparison with a commercial smartwatch is also included. The validation process is conducted with two modalities that differ for the availability of a priori knowledge about the subjects' normal HR. The two test modalities provide different results. In particular, the HR estimation differs from the ground-truth by 2% when the knowledge about the subject's lifestyle and his/her HR is considered and by 3.4% if no information about the person is taken into account.

Keywords: EVM; Kinect; RGB-D sensors; contactless sensing; heart rate; photoplethysmography; videoplethysmography.

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
The main scheme of the proposed system for the computation of HR from the RGB video.
Figure 2
Figure 2
Spatial decomposition through the Gaussian pyramid of five levels with specific details on the pixel resolution. Since the dimensions of the square detected face area are not integer exponents of two, the size is rounded to the nearest integer.
Figure 3
Figure 3
Selected regions of interest: (a) Forehead (F); (b1,b2) Cheeks (C); (c) Neck (N). Some details are blurred to preserve the subject’s privacy. The ROIs are selected considering the percentage of the detected face area shown in the figure.
Figure 4
Figure 4
VPG signal (Y-component) obtained by averaging the signals of all of the ROIs (F, C, N or, rather, ROI T (Total)) over one test execution. The signal is represented in the time domain (a) and in the frequency domain (b), after the bandpass filtering process.
Figure 4
Figure 4
VPG signal (Y-component) obtained by averaging the signals of all of the ROIs (F, C, N or, rather, ROI T (Total)) over one test execution. The signal is represented in the time domain (a) and in the frequency domain (b), after the bandpass filtering process.
Figure 5
Figure 5
Laboratory setup at Università Politecnica delle Marche, where the tests have been conducted. Two different environmental conditions are shown: high (a) and low (b) light.
Figure 6
Figure 6
Average error (%) in HR estimation for the system based on RGB data with different light conditions on different ROIs; (a) supervised approach; (b) unsupervised approach.
Figure 7
Figure 7
Bland–Altman plot, correlation between HR values extracted by the Holter and through the proposed method in: (a) ROI C; (b) ROI F; (c) ROI N.
Figure 8
Figure 8
Best correlation values with the Bland–Altman plot, between HR values extracted by the Holter and through the proposed method in (a) ROI F + N; (b) ROI T.
Figure 9
Figure 9
Average error (%) in HR estimation revealed by the Smartwatch (SW) with respect to Holter measurements.

References

    1. Frese E.M., Fick A., Sadowsky S.H. Blood pressure measurement guidelines for physical therapists. Cardiopulm. Phys. Ther. J. 2011;22:5–12.
    1. Ward S., Hisley S. Maternal-Child Nursing Care Optimizing Outcomes for Mothers, Children, & Families. FA Davis; Philadelpia, PA, USA: 2015.
    1. Ewing D.J., Neilson J.M., Travis P. New method for assessing cardiac parasympathetic activity using 24 h electrocardiograms. Br. Heart J. 1984;52:396–402. doi: 10.1136/hrt.52.4.396.
    1. Farrell T.G., Bashir Y., Cripps T., Malik M., Poloniecki J., Bennett E.D., Ward D.E., Camm A.J. Risk stratification for arrhythmic events in postinfarction patients based on heart rate variability, ambulatory electrocardiographic variables and the signal-averaged electrocardiogram. J. Am. Coll. Cardiol. 1991;18:687–697. doi: 10.1016/0735-1097(91)90791-7.
    1. Cho M.C., Kim J.Y., Cho S. A bio-impedance measurement system for portable monitoring of heart rate and pulse wave velocity using small body area; Proceedings of the 2009 IEEE International Symposium on Circuits and Systems; Taipei, Taiwan. 24–27 May 2009; pp. 3106–3109.
    1. Kwon S., Kim H., Park K.S. Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone; Proceedings of the 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society; San Diego, CA, USA. 28 August–1 September 2012; pp. 2174–2177.
    1. Monkaresi H., Calvo R.A., Yan H. A Machine Learning Approach to Improve Contactless Heart Rate Monitoring Using a Webcam. IEEE J. Biomed. Health Inform. 2014;18:1153–1160.
    1. Yang L., Ren Y., Hu H., Tian B. New Fast Fall Detection Method Based on Spatio-Temporal Context Tracking of Head by Using Depth Images. Sensors. 2015;15:23004–23019. doi: 10.3390/s150923004.
    1. Hong K., Hong S. Real-time stress assessment using thermal imaging. Vis. Comput. 2016;32:1369–1377.
    1. Procházka A., Vyšata O., Vališ M., Ťupa O., Schätz M., Mařík V. Bayesian classification and analysis of gait disorders using image and depth sensors of Microsoft Kinect. Digit. Signal Process. 2015;47:169–177. doi: 10.1016/j.dsp.2015.05.011.
    1. Procházka A., Schätz M., Vyšata O., Vališ M. Microsoft Kinect Visual and Depth Sensors for Breathing and Heart Rate Analysis. Sensors. 2016;16:996. doi: 10.3390/s16070996.
    1. Al-Naji A., Gibson K., Lee S.H., Chahl J. Real Time Apnoea Monitoring of Children Using the Microsoft Kinect Sensor: A Pilot Study. Sensors. 2017;17:286
    1. Verkruysse W., Svaasand L.O., Nelson J.S. Remote plethysmographic imaging using ambient light. Opt. Express. 2008;16:21434–21445.
    1. Poh M.Z., McDuff D.J., Picard R.W. Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam. IEEE Trans. Biomed. Eng. 2011;58:7–11. doi: 10.1109/TBME.2010.2086456.
    1. Li C., Peng Z., Huang T.Y., Fan T., Wang F.K., Horng T.S., Muñoz-Ferreras J.M., Gómez-García R., Ran L., Lin J. A Review on Recent Progress of Portable Short-Range Noncontact Microwave Radar Systems. IEEE Trans. Microw. Theory Tech. 2017;65:1692–1706. doi: 10.1109/TMTT.2017.2650911.
    1. Ren L., Kong L., Foroughian F., Wang H., Theilmann P., Fathy A.E. Comparison Study of Noncontact Vital Signs Detection Using a Doppler Stepped-Frequency Continuous-Wave Radar and Camera-Based Imaging Photoplethysmography. IEEE Trans. Microw. Theory Tech. 2017;99:1–11. doi: 10.1109/TMTT.2017.2658567.
    1. Wu H.Y., Rubinstein M., Shih E., Guttag J., Durand F., Freeman W. Eulerian Video Magnification for Revealing Subtle Changes in the World. ACM Trans. Graph. 2012;31:1–8. doi: 10.1145/2185520.2185561.
    1. Fletcher R.R., Chamberlain D., Paggi N., Deng X. Implementation of smart phone video plethysmography and dependence on lighting parameters; Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); Milan, Italy. 25–29 August 2015; pp. 3747–3750.
    1. Lachat E., Macher H., Landes T., Grussenmeyer P. Assessment and Calibration of a RGB-D Camera (Kinect v2 Sensor) Towards a Potential Use for Close-Range 3D Modeling. Remote Sens. 2015;7:13070–13097. doi: 10.3390/rs71013070.
    1. Dal Mutto C., Zanuttigh P., Cortelazzo G.M. Time-of-Flight Cameras and Microsoft Kinect™. Springer Publishing Company, Incorporated; Berlin, Germany: 2012.
    1. Zhang Z. Microsoft Kinect Sensor and Its Effect. IEEE; New York, NY, USA: Apr 27, 2012. 12694803.
    1. Cippitelli E., Gasparrini S., Spinsante S., Gambi E. Kinect as a Tool for Gait Analysis: Validation of a Real-Time Joint Extraction Algorithm Working in Side View. Sensors. 2015;15:1417–1434. doi: 10.3390/s150101417.
    1. Gao Z., Yu Y., Zhou Y., Du S. Leveraging Two Kinect Sensors for Accurate Full-Body Motion Capture. Sensors. 2015;15:24297–24317. doi: 10.3390/s150924297.
    1. Cippitelli E., Gasparrini S., Gambi E., Spinsante S. A Human Activity Recognition System Using Skeleton Data from RGBD Sensors. Comput. Intell. Neurosci. 2016;2016:4351435. doi: 10.1155/2016/4351435.
    1. Viola P., Jones M.J. Robust Real-Time Face Detection. Int. J. Comput. Vis. 2004;57:137–154. doi: 10.1023/B:VISI.0000013087.49260.fb.
    1. Mann K.A., Wernere F.W., Palmer A.K. Frequency spectrum analysis of wrist motion for activities of daily living. J. Orthop. Res. 1989;7:304–306. doi: 10.1002/jor.1100070219.
    1. Natarajan V., Vyas A. Power efficient compressive sensing for continuous monitoring of ECG and PPG in a wearable system; Proceedings of the 2016 IEEE 3rd World Forum on Internet of Things (WF-IoT); Reston, VA, USA. 12–14 December 2016; pp. 336–341.
    1. Pan H., Temel D., AlRegib G. HeartBEAT: Heart beat estimation through adaptive tracking; Proceedings of the 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI); Las Vegas, NV, USA. 25–27 February 2016; pp. 587–590.
    1. Mestha L.K., Kyal S., Xu B., Lewis L.E., Kumar V. Towards continuous monitoring of pulse rate in neonatal intensive care unit with a webcam; Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; Chicago, IL, USA. 26–30 August 2014; pp. 3817–3820.
    1. Secerbegovic A., Bergsland J., Halvorsen P.S., Suljanovic N., Mujcic A., Balasingham I. Blood pressure estimation using video plethysmography; Proceedings of the 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI); Prague, Czech Republic. 13–16 April 2016; pp. 461–464.
    1. More S.V., Chaudhari P.C. Development of non-invasive diagnostic tool for diseases using Photo Plethysmography; Proceedings of the 2016 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET); Chennai, India. 23–25 March 2016; pp. 1499–1503.
    1. Allen J. Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 2007;28:R1-39. doi: 10.1088/0967-3334/28/3/R01.
    1. Joseph G., Joseph A., Titus G., Thomas R.M., Jose D. Photoplethysmogram (PPG) signal analysis and wavelet de-noising; Proceedings of the 2014 Annual International Conference on Emerging Research Areas: Magnetics, Machines and Drives (AICERA/iCMMD); Kottayam, India. 24–26 July 2014; pp. 1–5.
    1. Tamura T., Maeda Y., Sekine M., Yoshida M. Wearable photoplethysmographic sensors—Past and present. Electronics. 2014;3:282–302. doi: 10.3390/electronics3020282.
    1. Kranjec J., Beguš S., Geršak G., Drnovšek J. Non-contact heart rate and heart rate variability measurements: A review. Biomed. Signal Process. Control. 2014;13:102–112. doi: 10.1016/j.bspc.2014.03.004.
    1. Lin W.H., Wu D., Li C., Zhang H., Zhang Y.T. Comparison of Heart Rate Variability from PPG with That from ECG. In: Zhang Y.T., editor. Proceedings of the International Conference on Health Informatics: ICHI 2013; Vilamoura, Portugal. 7–9 November 2013; Cham, Switzerland: Springer; 2014. pp. 213–215.
    1. Freund Y., Schapire R.E. A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting. J. Comput. Syst. Sci. 1997;55:119–139. doi: 10.1006/jcss.1997.1504.
    1. Comon P. Independent component analysis, A new concept? Signal Process. 1994;36:287–314. doi: 10.1016/0165-1684(94)90029-9.
    1. Rahman H., Uddin Ahmed M., Begum S., Funk P. Real Time Heart Rate Monitoring From Facial RGB Color Video Using Webcam; Proceedings of the 29th Annual Workshop of the Swedish Artificial Intelligence Society (SAIS); Malmö, Sweden. 2–3 June 2016; Linköping, Sweden: Linköping University Electronic Press; 2016. p. 8.
    1. Kramer M.A. Nonlinear principal component analysis using autoassociative neural networks. AIChE J. 1991;37:233–243. doi: 10.1002/aic.690370209.
    1. Bernacchia N., Scalise L., Casacanditella L., Ercoli I., Marchionni P., Tomasini E.P. Non contact measurement of heart and respiration rates based on Kinect; Proceedings of the 2014 IEEE International Symposium on Medical Measurements and Applications (MeMeA); Lisboa, Portugal. 11–12 June 2014; pp. 1–5.
    1. Iozzia L., Cerina L., Mainardi L.T. Assessment of beat-to-beat heart rate detection method using a camera as contactless sensor; Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); Orlando, FL, USA. 16–20 August 2016; pp. 521–524.
    1. Yu Y.P., Kwan B.H., Lim C.L., Wong S.L., Raveendran P. Video-based heart rate measurement using short-time Fourier transform; Proceedings of the 2013 International Symposium on Intelligent Signal Processing and Communication Systems; Naha-shi, Japan. 12–15 November 2013; pp. 704–707.
    1. Alghoul K., Alharthi S., Osman H.A., Saddik A.E. Heart Rate Variability Extraction From Videos Signals: ICA vs. EVM Comparison. IEEE Access. 2017;5:4711–4719. doi: 10.1109/ACCESS.2017.2678521.
    1. Smilkstein T., Buenrostro M., Kenyon A., Lienemann M., Larson G. Heart rate monitoring using Kinect and color amplification; Proceedings of the 2014 IEEE Healthcare Innovation Conference (HIC); Seattle, WA, USA. 8–10 October 2014; pp. 60–62.
    1. Tulyakov S., Alameda-Pineda X., Ricci E., Yin L., Cohn J.F., Sebe N. Self-Adaptive Matrix Completion for Heart Rate Estimation From Face Videos Under Realistic Conditions; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Seattle, WA, USA. 27–30 June 2016.
    1. Lucas B.D., Kanade T. An Iterative Image Registration Technique with an Application to Stereo Vision; Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI’81); Vancouver, BC, Canada. 24–28 August 1981; San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.; 1981. pp. 674–679.
    1. Tomasi C., Kanade T. Detection and Tracking of Point Features. Carnegie Mellon University; Pittsburgh, PA, USA: 1991. Technical Report CMU-CS-91-132.
    1. Shi J., Tomasi C. Good features to track; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Seattle, WA, USA. 21–23 June 1994; pp. 593–600.
    1. Recommendations for Standardization of Instruments in Electrocardiography and Vectorcardiography. IEEE Trans. Biomed. Eng. 1967;BME-14:60–68.
    1. Pan J., Tompkins W.J. A Real-Time QRS Detection Algorithm. IEEE Trans. Biomed. Eng. 1985;BME-32:230–236. doi: 10.1109/TBME.1985.325532.
    1. Bland J.M., Altman D. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;327:307–310. doi: 10.1016/S0140-6736(86)90837-8.

Source: PubMed

Sponsor e collaboratori

Condizioni mediche

Interventi farmacologici

3
Sottoscrivi