Investigating sources of inaccuracy in wearable optical heart rate sensors

Brinnae Bent, Benjamin A Goldstein, Warren A Kibbe, Jessilyn P Dunn, Brinnae Bent, Benjamin A Goldstein, Warren A Kibbe, Jessilyn P Dunn

Abstract

As wearable technologies are being increasingly used for clinical research and healthcare, it is critical to understand their accuracy and determine how measurement errors may affect research conclusions and impact healthcare decision-making. Accuracy of wearable technologies has been a hotly debated topic in both the research and popular science literature. Currently, wearable technology companies are responsible for assessing and reporting the accuracy of their products, but little information about the evaluation method is made publicly available. Heart rate measurements from wearables are derived from photoplethysmography (PPG), an optical method for measuring changes in blood volume under the skin. Potential inaccuracies in PPG stem from three major areas, includes (1) diverse skin types, (2) motion artifacts, and (3) signal crossover. To date, no study has systematically explored the accuracy of wearables across the full range of skin tones. Here, we explored heart rate and PPG data from consumer- and research-grade wearables under multiple circumstances to test whether and to what extent these inaccuracies exist. We saw no statistically significant difference in accuracy across skin tones, but we saw significant differences between devices, and between activity types, notably, that absolute error during activity was, on average, 30% higher than during rest. Our conclusions indicate that different wearables are all reasonably accurate at resting and prolonged elevated heart rate, but that differences exist between devices in responding to changes in activity. This has implications for researchers, clinicians, and consumers in drawing study conclusions, combining study results, and making health-related decisions using these devices.

Keywords: Biomedical engineering; Imaging and sensing; Technology.

Conflict of interest statement

Competing interestsThe authors declare no competing interests.

© The Author(s) 2020.

Figures

Fig. 1. Graphical abstract of research.
Fig. 1. Graphical abstract of research.
Graphical abstract of research study presented. We present a full characterization of HR accuracy across skin tones, clinical metrics of HRV accuracy across skin tones, and HR during activity, rest, deep breathing, and typing for six wearable devices representing both consumer wearables and research-grade wearables. HR metrics are compared to the clinical-grade electrocardiogram (ECG) as the standard for heart rate measurement.
Fig. 2. Error in heart rate across…
Fig. 2. Error in heart rate across skin tones and devices at rest and during activity.
Mean error in heart rate (bpm) across skin tones and devices at a rest and b during physical activity. The green horizontal line represents no error (no difference from the true measurement of HR from ECG). Mean absolute error in heart rate (bpm) across skin tones and devices at c rest and d during physical activity. Error is calculated as the difference between the ECG and wearable reported heart rate at every simultaneous measurement. Fitzpatrick skin tones 1–6 are represented with an approximately equal number of participants in each skin tone. Error bars represent the 95% confidence interval. Mean absolute error across devices and across skin tones at rest (e) and during activity (f). Error bars represent the 95% confidence interval.
Fig. 3. Error in heart rate across…
Fig. 3. Error in heart rate across all devices and analysis of missing values across consumer devices.
a Mean absolute error in heart rate (bpm) across devices during rest (teal) and activity (orange). This shows the true difference in HR from the ECG but does not show the sign of the difference. The green horizontal line represents no error (no difference from the true measurement of HR from ECG). Error bars show the 95% confidence interval. ** indicates significant difference in error between baseline and activity with a Bonferroni multiple hypothesis corrected p value of 0.0042. b Mean relative error in heart rate (bpm) across devices during rest (teal) and during activity (orange) shows the relative differences from the ECG. The green horizontal line represents no error (no difference from the true measurement of HR from ECG). Error bars show the 95% confidence interval. ** indicates significant difference in error between baseline and activity with a Bonferroni multiple hypothesis corrected p value of 0.0042. c Analysis of missing values across skin tones for rest and activity for consumer wearables. Research-grade wearables (Empatica, Biovotion) down-sample and/or interpolate to have exactly 1 Hz sampling rate and thus we could not calculate missingness values for those devices. Missingness is calculated from the expected sampling rate (reported sampling rate for Apple Watch and Garmin and study average sampling rate for Garmin and Miband, which do not report sampling rate). Missingness that is positive indicates percentage of values with missingness. Missingness that is negative indicates a greater than expected sampling rate (more values than expected).

References

    1. Chronic Diseases in America | CDC. (2019).
    1. Health Care Cost and Utilization Report 2017 (2019).
    1. Older Americans Drive Growth of Wearables. eMarketer (2018).
    1. Munos B, et al. Mobile health: the power of wearables, sensors, and apps to transform clinical trials. Ann. N. Y. Acad. Sci. 2016;1375:3–18. doi: 10.1111/nyas.13117.
    1. Witt DR, Kellogg RA, Snyder MP, Dunn J. Windows into human health through wearables data analytics. Curr. Opin. Biomed. Eng. 2019;9:28–46. doi: 10.1016/j.cobme.2019.01.001.
    1. Goldsack J, et al. Verification, Analytical Validation, and Clinical Validation (V3): The Foundation of Determining Fit-for-Purpose for Biometric Monitoring Technologies (BioMeTs). Preprint at (2020).
    1. Ruth Hailu. Fitbits, other wearables may not accurately track heart rates in people of color. STAT News. (2019).
    1. Jo, E., Lewis, K., Directo, D., Kim, M. J. & Dolezal, B. A. Validation of biofeedback wearables for photoplethysmographic heart rate tracking. J. Sports Sci. Med.15, 540–547 (2016).
    1. Reddy RK, et al. Accuracy of wrist-worn activity monitors during common daily physical activities and types of structured exercise: evaluation study. JMIR mHealth uHealth. 2018;6:e10338. doi: 10.2196/10338.
    1. Sartor F, et al. Wrist-worn optical and chest strap heart rate comparison in a heterogeneous sample of healthy individuals and in coronary artery disease patients. BMC Sports Sci. Med. Rehabil. 2018;10:10. doi: 10.1186/s13102-018-0098-0.
    1. Shcherbina A, et al. Accuracy in wrist-worn, sensor-based measurements of heart rate and energy expenditure in a diverse cohort. J. Pers. Med. 2017;7:3. doi: 10.3390/jpm7020003.
    1. Wallen MP, Gomersall SR, Keating SE, Wisløff U, Coombes JS. Accuracy of heart rate watches: implications for weight management. PLoS ONE. 2016;11:e0154420. doi: 10.1371/journal.pone.0154420.
    1. Bickler PE, Feiner JR, Severinghaus JW. Effects of skin pigmentation on pulse oximeter accuracy at low saturation. Anesthesiology. 2005;102:715–9. doi: 10.1097/00000542-200504000-00004.
    1. Apple Support. Your heart rate. What it means, and where on Apple Watch you’ll find it—Apple Support. (2019).
    1. Apple Support. Get the most accurate measurements using your Apple Watch—Apple Support. (2019).
    1. Fitbit. Purepulse Technology. (2019).
    1. Garmin. What Can Cause the Heart Rate Sensor from My Watch to Be Inaccurate? (2019).
    1. Garmin. Garmin | Accuracy Disclaimer. (2019).
    1. Empatica Support. E4 data—Empatica Support. (2019).
    1. Biovotion. Everion data quality—Biovotion AG. (2019).
    1. Biovotion. The accuracy of Heart Rate—Biovotion AG. (2019).
    1. Biovotion. What impacts and affects my vital sign readings?—Biovotion AG. (2019).
    1. Zonios G, Bykowski J, Kollias N. Skin melanin, hemoglobin, and light scattering properties can be quantitatively assessed in vivo using diffuse reflectance spectroscopy. J. Invest. Dermatol. 2001;117:1452–1457. doi: 10.1046/j.0022-202x.2001.01577.x.
    1. Tseng S-H, Grant A, Durkin AJ. In vivo determination of skin near-infrared optical properties using diffuse optical spectroscopy. J. Biomed. Opt. 2008;13:014016. doi: 10.1117/1.2829772.
    1. Fallow BA, Tarumi T, Tanaka H. Influence of skin type and wavelength on light wave reflectance. J. Clin. Monit. Comput. 2013;27:313–317. doi: 10.1007/s10877-013-9436-7.
    1. Treesirichod A, Chansakulporn S, Wattanapan P. Correlation between skin color evaluation by skin color scale chart and narrowband reflectance spectrophotometer. Indian J. Dermatol. 2014;59:339–342. doi: 10.4103/0019-5154.135476.
    1. Ries AL, Prewitt LM, Johnson JJ. Skin color and ear oximetry. Chest. 1989;96:287–290. doi: 10.1378/chest.96.2.287.
    1. Lister T, Wright PA, Chappell PH. Optical properties of human skin. J. Biomed. Opt. 2012;17:0909011. doi: 10.1117/1.JBO.17.9.090901.
    1. Weiler DT, Villajuan SO, Edkins L, Cleary S, Saleem JJ. Wearable heart rate monitor technology accuracy in research: a comparative study between PPG and ECG technology. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2017;61:1292–1296. doi: 10.1177/1541931213601804.
    1. Nitzan M, Romem A, Koppel R. Pulse oximetry: fundamentals and technology update. Med. Devices Evid. Res. 2014;7:231. doi: 10.2147/MDER.S47319.
    1. Yan, L., Hu, S., Alzahrani, A., Alharbi, S. & Blanos, P. A Multi-wavelength opto-electronic patch sensor to effectively detect physiological changes against human skin types. Biosensors7, 22 (2017).
    1. Everion. Biovotion Instruction for Consumer (2018).
    1. Castaneda D, Esparza A, Ghamari M, Soltanpur C, Nazeran H. A review on wearable photoplethysmography sensors and their potential future applications in health care. Int. J. Biosens. Bioelectron. 2018;4:195–202.
    1. Lemay Mathieu, Bertschi Mattia, Sola Josep, Renevey Philippe, Parak Jakub, Korhonen Ilkka. Wearable Sensors. 2014. Application of Optical Heart Rate Monitoring; pp. 105–129.
    1. Zhang Y, et al. Motion artifact reduction for wrist-worn photoplethysmograph sensors based on different wavelengths. Sensors. 2019;19:673. doi: 10.3390/s19030673.
    1. Tautan, A.-M., Young, A., Wentink, E. & Wieringa, F. Characterization and reduction of motion artifacts in photoplethysmographic signals from a wrist-worn device. in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 6146–6149. 10.1109/EMBC.2015.7319795 (IEEE, 2015).
    1. Tamura T, et al. Wearable photoplethysmographic sensors—past and present. Electronics. 2014;3:282–302. doi: 10.3390/electronics3020282.
    1. Zong, C. & Jafari, R. Robust heart rate estimation using wrist-based PPG signals in the presence of intense physical activities. in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 8078–8082. 10.1109/EMBC.2015.7320268 (IEEE, 2015).
    1. Reis VM, et al. Are wearable heart rate measurements accurate to estimate aerobic energy cost during low-intensity resistance exercise? PLoS ONE. 2019;14:e0221284. doi: 10.1371/journal.pone.0221284.
    1. Valencell Team. Valencell | Optical Heart Rate Monitoring: What You Need to Know. (2015).
    1. Six Dijkstra M, et al. Exploring a 1-minute paced deep-breathing measurement of heart rate variability as part of a workers’ health assessment. Appl. Psychophysiol. Biofeedback. 2019;44:83–96. doi: 10.1007/s10484-018-9422-4.
    1. Mayo Clinic. Mayo Clinic Staff. Exercise intensity: how to measure it—Mayo Clinic. (2019).
    1. Shaffer F, Ginsberg JP. An overview of heart rate variability metrics and norms. Front. Public Health. 2017;5:258. doi: 10.3389/fpubh.2017.00258.
    1. Nelson BW, Allen NB. Accuracy of consumer wearable heart rate measurement during an ecologically valid 24-hour period: intraindividual validation study. JMIR mHealth uHealth. 2019;7:e10828. doi: 10.2196/10828.
    1. Wang R, et al. Accuracy of wrist-worn heart rate monitors. JAMA Cardiol. 2017;2:104. doi: 10.1001/jamacardio.2016.3340.
    1. Prawiro EAPJ, Yeh C-I, Chou N-K, Lee M-W, Lin Y-H. Integrated wearable system for monitoring heart rate and step during physical activity. Mob. Inf. Syst. 2016;2016:1–10.
    1. Weiss Daniel, Rydland Håvard T., Øversveen Emil, Jensen Magnus Rom, Solhaug Solvor, Krokstad Steinar. Innovative technologies and social inequalities in health: A scoping review of the literature. PLOS ONE. 2018;13(4):e0195447. doi: 10.1371/journal.pone.0195447.
    1. Silonie Sachdeva. Fitzpatrick Skin Typing: Applications in Dermatology. Indian J. Dermatol Venereol Leprol.75, 93–96 (2009).
    1. Fitzpatrick TB. The validity and practicality of sun-reactive skin types I through VI. Arch. Dermatol. 1988;124:869–871. doi: 10.1001/archderm.1988.01670060015008.
    1. CTA Standard. Physical Activity Monitoring for Heart Rate CTA-2065 (2018).
    1. YouTube. Erin Klassen. Triangle breathing, 1 min—YouTube. (2015).
    1. Adams, C. & Pinkas, D. Internet X.509 Public Key Infrastructure Time-Stamp Protocol (TSP). .
    1. ISO—ISO 8601 Date and time format. (2019).
    1. Kevin Dooley. The Why and How of Syncing Clocks on Network Devices. (2017).
    1. John Martellaro. What Time is it? Your iPad & iOS 5 Finally Knows—The Mac Observer. (2011).
    1. TechTarget. What is Network Time Protocol (NTP)?—Definition from . (2019).
    1. Todd Johnson. Devices and Timestamps: Seriously Though, WTF? | Airship. (2016).
    1. Corino VDA, Matteucci M, Mainardi LT. Analysis of heart rate variability to predict patient age in a healthy population. Methods Inf. Med. 2007;46:191–5. doi: 10.1055/s-0038-1625405.
    1. McCarthy, C., Pradhan, N., Redpath, C. & Adler, A. Validation of the Empatica E4 wristband. in 2016 IEEE EMBS International Student Conference (ISC) 1–4. 10.1109/EMBSISC.2016.7508621 (IEEE, 2016).
    1. Hugh et al. Consumer-Grade Wrist-worn Ppg Sensors Can Be Used to Detect Differences in Heart Rate Variability among A Heterogenous Prediabetic Population (IEEE International Conference on Biomedical and Health Informatics, 2018).

Source: PubMed

スポンサーと協力者

医学的状態

薬物療法

3
購読する