Kinect-Based In-Home Exercise System for Lymphatic Health and Lymphedema Intervention

An-Ti Chiang, Qi Chen, Yao Wang, Mei R Fu, An-Ti Chiang, Qi Chen, Yao Wang, Mei R Fu

Abstract

Using Kinect sensors to monitor and provide feedback to patients performing intervention or rehabilitation exercises is an upcoming trend in healthcare. However, the joint positions measured by the Kinect sensor are often unreliable, especially for joints that are occluded by other parts of the body. Also, users' motion sequences differ significantly even when doing the same exercise and are not temporally aligned, making the evaluation of the correctness of their movement challenging. This paper aims to develop a Kinect-based intervention system, which can guide the users to perform the exercises more effectively. To circumvent the unreliability of the Kinect measurements, we developed a denoising algorithm using a Gaussian Process regression model. We simultaneously capture the joint positions using both a Kinect sensor and a motion capture (MOCAP) system during a training stage and train a Gaussian process regression model to map the noisy Kinect measurements to the more accurate MOCAP measurements. For the sequences alignment issue, we develop a gradient-weighted dynamic time warping approach that can automatically recognize the endpoints of different subsequences from the original user's motion sequence, and furthermore temporally align the subsequences from multiple actors. During a live exercise session, the system applies the same alignment algorithm to a live-captured Kinect sequence to divide it into subsequences, and furthermore compare each subsequence with its corresponding reference subsequence, and generates feedback to the user based on the comparison results. Our results show that the denoised Kinect measurements by the proposed denoising algorithm are more accurate than several benchmark methods and the proposed temporal alignment approach can precisely detect the end of each subsequence in an exercise with very small amount of delay. These methods have been integrated into a prototype system for guiding patients with risks for breast-cancer related lymphedema to perform a set of lymphatic exercises. The system can provide relevant feedback to the patient performing an exercise in real time.

Keywords: Dynamic time warping; Gaussian process regression; Intervention system; denoising of Kinect measurements.

Figures

FIGURE 1.
FIGURE 1.
Proposed system flowchart.
FIGURE 2.
FIGURE 2.
Data capture system set up . (a) software interface of MOCAP system, (b) a typical recording scenario.
FIGURE 3.
FIGURE 3.
Upper body skeletons . (a) raw MOCAP sample and raw Kinect sample. (b) standardized MOCAP sample and Kinect sample.
FIGURE 4.
FIGURE 4.
Computation time and error for different cluster numbers . (a) average computation time per time sample, (b) error per joint.
FIGURE 5.
FIGURE 5.
Traces of the left wrist x-coordinate while users performing a subsequence in the “Horizontal Pumping” exercise. (a) Standardized MOCAP traces; (b) Aligned traces by stretching all traces to the same length; (c) Aligned traces by aligning at two key transition points.
FIGURE 6.
FIGURE 6.
The reference sequence and a user’s motion sequence of the x coordinate of the left wrist during Exercise 4. Each“sec” indicate a subsequence.
FIGURE 7.
FIGURE 7.
Motion data of different users.
FIGURE 8.
FIGURE 8.
The raw captured skeleton by Kinect (green) and the denoised skeleton by the proposed method (red) overlaid on the RGB image . (a) horizontal pumping exercise; (b) push down pumping exercise.
FIGURE 9.
FIGURE 9.
The raw captured joint trace by Kinect (green) and MOCAP (yellow), and the denoised trace by the proposed method (red) and the method of (green) . (a) The left wrist during the horizontal pumping exercise. Same markers (in shape) on different joint traces correspond to the same time, (b) The left elbow during the push down pumping exercise.
FIGURE 10.
FIGURE 10.
The subsequences and repetitions in different exercises. (a) Exercises 2: “‘over the head pumping”.(b)Exercise 4: “horizontal pumping”.
FIGURE 11.
FIGURE 11.
Graphical Interface of the proposed system.
FIGURE 12.
FIGURE 12.
Analysis results of breathing and hand status. In this figure, green curve describes the change of depth value in user’s chest region, where the largest depth difference is 3.5 cm in this case. Blue and red curve represent the states of the left and right hand, respectively, where the higher value corresponds to “close” and small value corresponds to “open”. (a) A case when a user breathes deeply and has perfect synchronization between the breathing state and the hand state. (b) A case when a user does not perform perfectly.
FIGURE 13.
FIGURE 13.
Internal Analysis for exercise “over the head pumping”.
FIGURE 14.
FIGURE 14.
The subsequences and repetitions in different exercises. (a) Exercise 1: “muscle-tightening deep breathing”. (b)Exercise 3: “push down pumping”.

References

    1. Chiang A.-T., Chen Q., Li S., Wang Y., and Fu M., “Denoising of joint tracking data by Kinect sensors using clustered Gaussian process regression,” in Proc. 2nd Int. Workshop Multimedia Pers. Health Health Care, 2017, pp. 19–25.
    1. Microsoft Kinect v2. Accessed: Jul. 21, 2017. [Online]. Available:
    1. Intel realsense. Accessed: Jul. 21, 2017[Online]. Available:
    1. Shum H. P. H., Ho E. S. L., Jiang Y., and Takagi S., “Real-time posture reconstruction for Microsoft Kinect,” IEEE Trans. Cybern., vol. 43, no. 5, pp. 1357–1369, Oct. 2013.
    1. Wei X., Zhang P., and Chai J., “Accurate realtime full-body motion capture using a single depth camera,” ACM Trans. Graph., vol. 31, no. 6, pp. 188:1–188:12, Nov. 2012.
    1. Liu Z., Zhou L., Leung H., and Shum H. P., “Kinect posture reconstruction based on a local mixture of Gaussian process models,” IEEE Trans. Vis. Comput. Graphics, vol. 22, no. 11, pp. 2437–2450, Nov. 2016.
    1. He H. and Siu W.-C., “Single image super-resolution using Gaussian process regression,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2011, pp. 449–456.
    1. Tripathy S. R., Chakravarty K., Sinha A., Chatterjee D., and Saha S. K., “Constrained Kalman filter for improving Kinect based measurements,” in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), May 2017, pp. 1–4.
    1. Alexiadis D. S., Kelly P., Daras P., O’Connor N. E., Boubekeur T., and Moussa M. B., “Evaluating a dancer’s performance using Kinect-based skeleton tracking,” in Proc. 19th ACM Int. Conf. Multimedia, 2011, pp. 659–662.
    1. Yurtman A. and Barshan B., “Detection and evaluation of physical therapy exercises by dynamic time warping using wearable motion sensor units,” in Information Sciences and Systems. Cham, Switzerland: Springer, 2013, pp. 305–314.
    1. Jeong Y.-S., Jeong M. K., and Omitaomu O. A., “Weighted dynamic time warping for time series classification,” Pattern Recognit., vol. 44, no. 9, pp. 2231–2240, Sep. 2011.
    1. Su C.-J., Chiang C.-Y., and Huang J.-Y., “Kinect-enabled home-based rehabilitation system using dynamic time warping and fuzzy logic,” Appl. Soft Comput., vol. 22, pp. 652–666, Sep. 2014.
    1. Wei W., Lu Y., Printz C. D., and Dey S., “Motion data alignment and real-time guidance in cloud-based virtual training system,” in Proc. Conf. Wireless Health, 2015, Art. no. 13.
    1. Optitrack. Accessed: Sep. 21, 2017. [Online]. Available:
    1. Armstrong H. G., “Anthropometry and mass distribution for human analogues, volume I: Military male aviators,” USAARL Rep. 88-5, 1988.
    1. Fu M. al., “Proactive approach to Lymphedema risk reduction: A prospective study,” Ann. Surgical Oncol., vol. 21, no. 11, pp. 3481–3489, 2014.
    1. Fu M. al., “Usability and feasibility of health IT interventions to enhance self-care for Lymphedema symptom management in breast cancer survivors,” Internet Interventions, vol. 5, pp. 56–64, Sep. 2016.
    1. Fu M. al., “mhealth self-care interventions: Managing symptoms following breast cancer treatment,” Mhealth, vol. 2, p. 28, Jul. 2016.
    1. Rasmussen C. E., “Gaussian processes in machine learning,” in Advanced Lectures on Machine Learning. Berlin, Germany: Springer, 2004, pp. 63–71.
    1. Cao Y., Brubaker M. A., Fleet D. J., and Hertzmann A., “Efficient optimization for sparse Gaussian process regression,” in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 1097–1105.
    1. Snelson E. and Ghahramani Z., “Sparse Gaussian processes using pseudo-inputs,” in Proc. Adv. Neural Inf. Process. Syst., 2006, pp. 1257–1264.
    1. Sakoe H. and Chiba S., “Dynamic programming algorithm optimization for spoken word recognition,” IEEE Trans. Acoust., Speech, Signal Process., vol. 26, no. 1, pp. 43–49, Feb. 1978.
    1. Ratanamahatana C. A. and Keogh E., “Making time-series classification more accurate using learned constraints,” in Proc. SIAM Int. Conf. Data Mining, 2004, pp. 11–22.
    1. Keogh E. and Ratanamahatana C. A., “Exact indexing of dynamic time warping,” Knowl. Inf. Syst., vol. 7, no. 3, pp. 358–386, 2005.
    1. Chen L., Özsu M. T., and Oria V., “Robust and fast similarity search for moving object trajectories,” in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2005, pp. 491–502.
    1. Soheily-Khah S. and Marteau P.-F. (2017). “Sparsification of the alignment path search space in dynamic time warping.” [Online]. Available:
    1. Kumar V. M. and Thipesh D. S. H., “Robot ARM performing writing through speech recognition using dynamic time warping algorithm,” Int. J. Eng. Trans. B, Appl., vol. 30, no. 8, pp. 1238–1245, 2017.
    1. American Cancer Society. Accessed: Apr. 10, 2018. [Online]. Available:

Source: PubMed

Sponsorzy i współpracownicy

Warunki medyczne

Interwencje lekowe

3
Subskrybuj