Contactless Vital Signs Measurement System Using RGB-Thermal Image Sensors and Its Clinical Screening Test on Patients with Seasonal Influenza

Toshiaki Negishi, Shigeto Abe, Takemi Matsui, He Liu, Masaki Kurosawa, Tetsuo Kirimoto, Guanghao Sun, Toshiaki Negishi, Shigeto Abe, Takemi Matsui, He Liu, Masaki Kurosawa, Tetsuo Kirimoto, Guanghao Sun

Abstract

Background: In the last two decades, infrared thermography (IRT) has been applied in quarantine stations for the screening of patients with suspected infectious disease. However, the fever-based screening procedure employing IRT suffers from low sensitivity, because monitoring body temperature alone is insufficient for detecting infected patients. To overcome the drawbacks of fever-based screening, this study aims to develop and evaluate a multiple vital sign (i.e., body temperature, heart rate and respiration rate) measurement system using RGB-thermal image sensors. Methods: The RGB camera measures blood volume pulse (BVP) through variations in the light absorption from human facial areas. IRT is used to estimate the respiration rate by measuring the change in temperature near the nostrils or mouth accompanying respiration. To enable a stable and reliable system, the following image and signal processing methods were proposed and implemented: (1) an RGB-thermal image fusion approach to achieve highly reliable facial region-of-interest tracking, (2) a heart rate estimation method including a tapered window for reducing noise caused by the face tracker, reconstruction of a BVP signal with three RGB channels to optimize a linear function, thereby improving the signal-to-noise ratio and multiple signal classification (MUSIC) algorithm for estimating the pseudo-spectrum from limited time-domain BVP signals within 15 s and (3) a respiration rate estimation method implementing nasal or oral breathing signal selection based on signal quality index for stable measurement and MUSIC algorithm for rapid measurement. We tested the system on 22 healthy subjects and 28 patients with seasonal influenza, using the support vector machine (SVM) classification method. Results: The body temperature, heart rate and respiration rate measured in a non-contact manner were highly similarity to those measured via contact-type reference devices (i.e., thermometer, ECG and respiration belt), with Pearson correlation coefficients of 0.71, 0.87 and 0.87, respectively. Moreover, the optimized SVM model with three vital signs yielded sensitivity and specificity values of 85.7% and 90.1%, respectively. Conclusion: For contactless vital sign measurement, the system achieved a performance similar to that of the reference devices. The multiple vital sign-based screening achieved higher sensitivity than fever-based screening. Thus, this system represents a promising alternative for further quarantine procedures to prevent the spread of infectious diseases.

Keywords: RGB-thermal image processing; contactless measurement; infection diseases; vital signs.

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Contact and contactless vital sign measurement systems for infection screening. The figures were with copyright permission [8,10].
Figure 2
Figure 2
Overview of measurement principle that remotely senses multiple vital signs and an example of screening result.
Figure 3
Figure 3
Feature matching for region-of-interest (ROI) detection in thermal image. The figure reproduced with copyright permission from Reference [14].
Figure 4
Figure 4
Block diagram of signal processing for HR estimation. (a) RGB video with ROI detected by OpenCV. (b) RGB ROI image applied to tapered window. (c) Raw RGB time-series data and reconstruction vector V=(vr,vg, vb) determined by kurtosis of spectra. (d) Reconstructed signal using V. (e) Power spectra obtained by MUSIC.
Figure 5
Figure 5
Block diagram of signal processing for respiration rate (RR) estimation. (a) Thermal video frame with facial landmark detected by the fusion sensor system described in Section 2. (b) Time-series data extracted from nasal and oral areas. (c) Respiration signal that chooses from four signals (b) based on SQI. (d) Power spectra obtained by MUSIC.
Figure 6
Figure 6
Recovery of heartbeat signal by applying tapered window and signal reconstruction. (a) RGB color traces obtained by RGB video. (b) Spectra estimated by Fast Fourier Transform (FFT). (c) Signal reconstruction determined through kurtosis of the spectra. (d), (e) Reconstructed signal and its spectra.
Figure 7
Figure 7
Bland–Altman plots and scatter plots of heart rate (HR) obtained by RGB sensor and electrocardiogram (ECG) or pulse oximeter. (a) Bland–Altman plot of raw green trace method applying FFT. (b) Bland–Altman plot of the proposed method applying tapered window, signal reconstruction and MUSIC. (c) Scatter plot of raw green trace. (d) Scatter plot of proposed method.
Figure 8
Figure 8
Determination of respiration signal applying nasal and oral breathing decision based on SQI. (a) Thermal facial image with ROI. (b) Mean and minimum temperature fluctuations in nasal area. (c) SQI parameter obtained by power spectral density (PSD), autocorrelation (ACR) and cross-power spectral density (CPSD) of nasal temperature changes. (d) Mean and minimum temperature fluctuations in oral area. (e) SQI parameter obtained by PSD, ACR and CPSD.
Figure 9
Figure 9
Bland–Altman plots and scatter plots of RR obtained by infrared thermography (IRT) sensor and respiratory effort belt. (a) Bland–Altman plot of nasal temperature change under the application of FFT. (b) Bland–Altman plot of the proposed method applying nasal or oral signal selection using SQI and MUSIC. (c) Scatter plot of nasal temperature change under FFT application. (d) Scatter plot of the proposed method.
Figure 10
Figure 10
Bland–Altman plots and scatter plots of body temperature obtained by IRT sensor and electric thermometer. (a) Bland–Altman plot. (b) Scatter plot.
Figure 11
Figure 11
Classification model based on Support Vector Machine (SVM). (a) SVM classification. (b) Confusion matrix.
Figure 12
Figure 12
Box plot of vital signs between influenza patients and healthy control subjects. (a) Facial skin temperature. (b) HR. (c) RR.

References

    1. Parashar U.D., Anderson L.J. Severe acute respiratory syndrome: Review and lessons of the 2003 outbreak. Int. J. Epidemiol. 2004;33:628–634. doi: 10.1093/ije/dyh198.
    1. Hui D.S., IAzhar E., Madani T.A., Ntoumi F., Kock R., Dar O., Ippolito G., Mchugh T.D., Memish Z.A., Drosten C., et al. The continuing 2019-nCoV epidemic threat of novel coronaviruses to global health—The latest 2019 novel coronavirus outbreak in Wuhan, China. Int. J. Infect. Dis. 2020;91:264–266. doi: 10.1016/j.ijid.2020.01.009.
    1. Ng E.Y., Kaw G.J., Chang W.M. Analysis of IR thermal imager for mass blind fever screening. Microvasc. Res. 2004;68:104–109. doi: 10.1016/j.mvr.2004.05.003.
    1. Chiang M.F., Lin P.W., Lin L.F., Chiou H.Y., Chien C.W., Chu S.F. Mass screening of suspected febrile patients with remote-sensing infrared thermography: Alarm temperature and optimal distance. J. Formos. Med. Assoc. 2008;107:937–944. doi: 10.1016/S0929-6646(09)60017-6.
    1. Sun G., Matsui T., Kirimoto T., Yao Y., Abe S. Applications of infrared thermography for noncontact and noninvasive mass screening of febrile international travelers at airport quarantine stations. In: Ng E.Y.K., Etehadtavakol M., editors. Application of Infrared to Biomedical Sciences. Springer; Singapore: 2017. pp. 347–358.
    1. Nishiura H., Kamiya K. Fever screening during the influenza (H1N1-2009) pandemic at Narita International Airport, Japan. BMC Infect Dis. 2011;11:111. doi: 10.1186/1471-2334-11-111.
    1. Bitar D., Goubar A., Desenclos J.C. International travels and fever screening during epidemics: A literature review on the effectiveness and potential use of non-contact infrared thermometers. Eurosurveillance. 2009;12:19115.
    1. Sun G., Matsui T., Hakozaki Y., Abe S. An infectious disease/fever screening radar system which stratifies higher-risk patients within ten seconds using a neural network and the fuzzy grouping method. J. Infect. 2015;70:230–236. doi: 10.1016/j.jinf.2014.12.007.
    1. Yao Y., Sun G., Matsui T., Hakozaki Y., van Waasen S., Schiek M. Multiple vital-sign-based infection screening outperforms thermography independent of the classification algorithm. IEEE Trans. Biomed. Eng. 2016;63:1025–1033. doi: 10.1109/TBME.2015.2479716.
    1. Sun G., Nakayama Y., Dagdanpurev S., Abe S., Nishimura H., Kirimoto T., Matsui T. Remote sensing of multiple vital signs using a CMOS camera-equipped infrared thermography system and its clinical application in rapidly screening patients with suspected infectious diseases. Int. J. Infect. Dis. 2017;55:113–117. doi: 10.1016/j.ijid.2017.01.007.
    1. Kaukonen K.M., Bailey M., Pilcher D., Cooper D.J., Bellomo R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N. Engl. J. Med. 2015;372:1629–1638. doi: 10.1056/NEJMoa1415236.
    1. Sun G., Trung N.V., Matsui T., Ishibashi K., Kirimoto T., Furukawa H., Hoi L.T., Huyen N.N., Nguyen Q., Abe S., et al. Field evaluation of an infectious disease/fever screening radar system during the 2017 dengue fever outbreak in Hanoi, Vietnam: A preliminary report. J. Infect. 2017;75:593–595. doi: 10.1016/j.jinf.2017.10.005.
    1. Negishi T., Sun G., Liu H., Sato S., Matsui T., Kirimoto T. Stable contactless sensing of vital signs using RGB-thermal image fusion system with facial tracking for infection screening. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2018:4371–4374. doi: 10.1109/EMBC.2018.8513300.
    1. Negishi T., Sun G., Sato S., Liu H., Matsui T., Abe S., Nishimura H., Kirimoto T. Infection screening system using thermography and CCD camera with good stability and swiftness for non-contact vital-signs measurement by feature matching and MUSIC algorithm. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2019:3183–3186. doi: 10.1109/EMBC.2019.8857027.
    1. Poh M.Z., McDuff D.J., Picard R.W. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 2011;58:7–11. doi: 10.1109/TBME.2010.2086456.
    1. Liu H., Ivanov K., Wang Y., Wang L. A novel method based on two cameras for accurate estimation of arterial oxygen saturation. BioMed. Eng. Online. 2015;14:52. doi: 10.1186/s12938-015-0045-1.
    1. Newell A., Yang K., Deng J. Stacked hourglass networks for human pose estimation; Proceedings of the European Conference on Computer Vision; Amsterdam, The Netherlands. 8–16 October 2016.
    1. Liu J., Ding H., Shahroudy A., Duan L., Jiang X., Wang G., Kot A. Feature boosting network for 3D pose estimation. IEEE Trans. Pattern Anal. Mach. Intell. 2016;42:494–501. doi: 10.1109/TPAMI.2019.2894422.
    1. Nibali A., He Z., Morgan S., Prendergast L. 3D human pose estimation with 2D marginal heatmaps; Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV); Waikoloa Village, HI, USA. 8–10 January 2019.
    1. Kazemi V., Sulivan J. One millisecond face alignment with an ensemble of regression trees; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Columbus, OH, USA. 23–28 June 2014.
    1. Rother C., Kolmogorov V., Blake A. ACM Transactions on Graphics Siggraph. Association for Computing Machinery; New York, NY, USA: 2004. “GrabCut”―Interactive foreground extraction using iterated graph cuts; pp. 309–314.
    1. Rublee E., Rabaud V., Konolige K., Bradski G. ORB: An efficient alternative to SIFT or SURF; Proceedings of the 2011 International Conference on Computer Vision; Barcelona, Spain. 6–13 November 2011; pp. 2564–2571.
    1. Raguram R., Chum O., Pollefeys M., Matas J., Frahm J.M. USAC: A universal framework for random sample consensus. IEEE Trans. Pattern Anal. Mach. Intel. 2012;35:2022–2038. doi: 10.1109/TPAMI.2012.257.
    1. Wang W., den Brinker A.C., de Haan G. Single element remote-PPG. IEEE Trans. Biomed. Eng. 2018 doi: 10.1109/TBME.2018.2882396.
    1. Izuhara Y., Matsumoto H., Nagasaki T., Kanemitsu Y., Murase K., Ito I., Oguma T., Muro S., Asai K., Tabara Y., et al. Mouth breathing another risk factor for asthma: The Nagahama study. Eur. J. Allergy Clin. Immunol. 2016;71:1031–1036. doi: 10.1111/all.12885.
    1. Bland J.M., Altman D.G. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1:307–310. doi: 10.1016/S0140-6736(86)90837-8.

Source: PubMed

Sponsoren und Mitarbeiter

Krankheiten

Drogeninterventionen

3
Abonnieren