Web-based assessments of physical activity in youth: considerations for design and scale calibration

Pedro F Saint-Maurice, Gregory J Welk, Pedro F Saint-Maurice, Gregory J Welk

Abstract

This paper describes the design and methods involved in calibrating a Web-based self-report instrument to estimate physical activity behavior. The limitations of self-report measures are well known, but calibration methods enable the reported information to be equated to estimates obtained from objective data. This paper summarizes design considerations for effective development and calibration of physical activity self-report measures. Each of the design considerations is put into context and followed by a practical application based on our ongoing calibration research with a promising online self-report tool called the Youth Activity Profile (YAP). We first describe the overall concept of calibration and how this influences the selection of appropriate self-report tools for this population. We point out the advantages and disadvantages of different monitoring devices since the choice of the criterion measure and the strategies used to minimize error in the measure can dramatically improve the quality of the data. We summarize strategies to ensure quality control in data collection and discuss analytical considerations involved in group- vs individual-level inference. For cross-validation procedures, we describe the advantages of equivalence testing procedures that directly test and quantify agreement. Lastly, we introduce the unique challenges encountered when transitioning from paper to a Web-based tool. The Web offers considerable potential for broad adoption but an iterative calibration approach focused on continued refinement is needed to ensure that estimates are generalizable across individuals, regions, seasons and countries.

Keywords: Youth Activity Profile; measurement; questionnaire.

Conflict of interest statement

Conflicts of Interest: None declared.

Figures

Figure 1
Figure 1
Screen capture of the online (top) and game (bottom) version of the Youth Activity Profile.
Figure 2
Figure 2
Examples of energy expenditure values measured by accelerometer for discrete time segments captured by the YAP. The data presented are from a middle school female participant enrolled in the YPAMS.
Figure 3
Figure 3
Gant chart illustrating the timeline and design for data collection of the preliminary work. Each rectangle represents a classroom grade full protocol timeline. Thick bars on the horizontal axis represent 1 calendar week. Each collection took approximately 3 weeks.
Figure 4
Figure 4
Reported nonwear time across 7 days obtained from individual logs. Data presented are from a subset of participants enrolled in the YPAMS. Results are based on a sample of 87 elementary, 27 middle school, and 29 high school students that used a SenseWear Armband for 7 consecutive days. There were 2 participants without accelerometer data who provided records of nonwear time.
Figure 5
Figure 5
Relation between percent time in moderate-to-vigorous physical activity (MVPA) measured by the SenseWear Armband accelerometer and YAP raw scores during physical education (PE; bottom) and during after-school time (top) (n=221 participants from grades 4 to 12). The solid black line represents the line of best fit with respective 95% confidence intervals; the dashed red line fits a smooth curve across the distribution of scores. The lack of overlap between these suggests a nonlinear trend relation between percent time in MVPA and YAP scores.

References

    1. Corder K, Ekelund U, Steele RM, Wareham NJ, Brage S. Assessment of physical activity in youth. J Appl Physiol (1985) 2008 Sep;105(3):977–87. doi: 10.1152/japplphysiol.00094.2008.
    1. Rowlands Av. Accelerometer assessment of physical activity in children: an update. Pediatr Exerc Sci. 2007 Aug;19(3):252–66.
    1. Bauman A, Phongsavan P, Schoeppe S, Owen N. Physical activity measurement--a primer for health promotion. Promot Educ. 2006;13(2):92–103.
    1. Troiano RP. A timely meeting: objective measurement of physical activity. Med Sci Sports Exerc. 2005 Nov;37(11 Suppl):S487–9.
    1. Trost SG, Ward DS, Moorehead SM, Watson PD, Riner W, Burke JR. Validity of the computer science and applications (CSA) activity monitor in children. Med Sci Sports Exerc. 1998 Apr;30(4):629–33.
    1. Eston RG, Rowlands AV, Ingledew DK. Validity of heart rate, pedometry, and accelerometry for predicting the energy cost of children's activities. J Appl Physiol (1985) 1998 Jan;84(1):362–71.
    1. Puyau MR, Adolph AL, Vohra FA, Butte NF. Validation and calibration of physical activity monitors in children. Obes Res. 2002 Mar;10(3):150–7. doi: 10.1038/oby.2002.24.
    1. Ekelund U, Sjöström M, Yngve A, Poortvliet E, Nilsson A, Froberg K, Wedderkopp N, Westerterp K. Physical activity assessed by activity monitor and doubly labeled water in children. Med Sci Sports Exerc. 2001 Feb;33(2):275–81.
    1. Treuth MS, Schmitz K, Catellier DJ, McMurray RG, Murray DM, Almeida MJ, Going S, Norman JE, Pate R. Defining accelerometer thresholds for activity intensities in adolescent girls. Med Sci Sports Exerc. 2004 Jul;36(7):1259–66.
    1. Mattocks C, Leary S, Ness A, Deere K, Saunders J, Tilling K, Kirkby J, Blair SN, Riddoch C. Calibration of an accelerometer during free-living activities in children. Int J Pediatr Obes. 2007;2(4):218–26. doi: 10.1080/17477160701408809.
    1. Schmitz KH, Treuth M, Hannan P, McMurray R, Ring KB, Catellier D, Pate R. Predicting energy expenditure from accelerometry counts in adolescent girls. Med Sci Sports Exerc. 2005 Jan;37(1):155–61.
    1. Bowles HR. Measurement of active and sedentary behaviors: closing the gaps in self-report methods. J Phys Act Health. 2012 Jan;9 Suppl 1:S1–4.
    1. Troiano RP, Pettee Gabriel KK, Welk GJ, Owen N, Sternfeld B. Reported physical activity and sedentary behavior: why do you ask? J Phys Act Health. 2012 Jan;9 Suppl 1:S68–75.
    1. Spook JE, Paulussen T, Kok G, Van Empelen P. Monitoring dietary intake and physical activity electronically: feasibility, usability, and ecological validity of a mobile-based Ecological Momentary Assessment tool. J Med Internet Res. 2013;15(9):e214. doi: 10.2196/jmir.2617.
    1. Riebl SK, Paone AC, Hedrick VE, Zoellner JM, Estabrooks PA, Davy BM. The comparative validity of interactive multimedia questionnaires to paper-administered questionnaires for beverage intake and physical activity: pilot study. JMIR Res Protoc. 2013;2(2):e40. doi: 10.2196/resprot.2830.
    1. da Costa FF, Schmoelz CP, Davies VF, Di Pietro PF, Kupek E, de Assis MA. Assessment of diet and physical activity of brazilian schoolchildren: usability testing of a web-based questionnaire. JMIR Res Protoc. 2013;2(2):e31. doi: 10.2196/resprot.2646.
    1. Pearce PF, Williamson J, Harrell JS, Wildemuth BM, Solomon P. The children's computerized physical activity reporter: children as partners in the design and usability evaluation of an application for self-reporting physical activity. Comput Inform Nurs. 2007;25(2):93–105. doi: 10.1097/01.NCN.0000263979.54048.85.
    1. Calabro MA, Welk GJ, Carriquiry AL, Nusser SM, Beyler NK, Mathews CE. Validation of a computerized 24-hour physical activity recall (24PAR) instrument with pattern-recognition activity monitors. J Phys Act Health. 2009 Mar;6(2):211–20.
    1. Namba H, Yamaguchi Y, Yamada Y, Tokushima S, Hatamoto Y, Sagayama H, Kimura M, Higaki Y, Tanaka H. Validation of Web-based physical activity measurement systems using doubly labeled water. J Med Internet Res. 2012;14(5):e123. doi: 10.2196/jmir.2253.
    1. Pfaeffli L, Maddison R, Jiang Y, Dalleck L, Löf M. Measuring physical activity in a cardiac rehabilitation population using a smartphone-based questionnaire. J Med Internet Res. 2013;15(3):e61. doi: 10.2196/jmir.2419.
    1. Kipnis V, Carroll RJ, Freedman LS, Li L. Implications of a new dietary measurement error model for estimation of relative risk: application to four calibration studies. Am J Epidemiol. 1999 Sep 15;150(6):642–51.
    1. Nusser SM, Beyler NK, Welk GJ, Carriquiry AL, Fuller WA, King BMN. Modeling errors in physical activity recall data. J Phys Act Health. 2012 Jan;9 Suppl 1:S56–67.
    1. Nusser Sm, Carriquiry Al, Dodd Kw, Fuller Wa. A semiparametric transformation approach to estimating usual daily intake distributions. Journal of the American Statistical Association. 1996;91:1440–1449.
    1. Tooze JA, Troiano RP, Carroll RJ, Moshfegh AJ, Freedman LS. A measurement error model for physical activity level as measured by a questionnaire with application to the 1999-2006 NHANES questionnaire. Am J Epidemiol. 2013 Jun 1;177(11):1199–208. doi: 10.1093/aje/kws379.
    1. Neuhouser ML, Di C, Tinker LF, Thomson C, Sternfeld B, Mossavar-Rahmani Y, Stefanick ML, Sims S, Curb JD, Lamonte M, Seguin R, Johnson KC, Prentice RL. Physical activity assessment: biomarkers and self-report of activity-related energy expenditure in the WHI. Am J Epidemiol. 2013 Mar 15;177(6):576–85. doi: 10.1093/aje/kws269.
    1. Crocker PR, Bailey DA, Faulkner RA, Kowalski KC, McGrath R. Measuring general levels of physical activity: preliminary evidence for the Physical Activity Questionnaire for Older Children. Med Sci Sports Exerc. 1997 Oct;29(10):1344–9.
    1. Kowalski K, Crocker PRE, Faulkner RA. Validation of the Physical Activity Questionnaire for Older Children. Pediatr Exerc Sci. 1997;9:174–186.
    1. Kowalski C, Crocker PRE, Kowalski NP. Convergent validity of the physical activity questionnaire for adolescents. Pediatr Exerc Sci. 1997;9:342–352.
    1. Bailey DA. The Saskatchewan Pediatric Bone Mineral Accrual Study: bone mineral acquisition during the growing years. Int J Sports Med. 1997 Jul;18 Suppl 3:S191–4. doi: 10.1055/s-2007-972713.
    1. Crocker PR, Eklund RC, Kowalski KC. Children's physical activity and physical self-perceptions. J Sports Sci. 2000 Jun;18(6):383–94. doi: 10.1080/02640410050074313.
    1. Moore JB, Hanes JC, Barbeau P, Gutin B, Treviño RP, Yin Z. Validation of the Physical Activity Questionnaire for Older Children in children of different races. Pediatr Exerc Sci. 2007 Feb;19(1):6–19.
    1. Janz KF, Lutuchy EM, Wenthe P, Levy SM. Measuring activity in children and adolescents using self-report: PAQ-C and PAQ-A. Med Sci Sports Exerc. 2008 Apr;40(4):767–72. doi: 10.1249/MSS.0b013e3181620ed1.
    1. Martínez-Gómez D, Martínez-de-Haro V, Pozo T, Welk GJ, Villagra A, Calle ME, Marcos A, Veiga OL. Reliability and validity of the PAQ-A questionnaire to assess physical activity in Spanish adolescents. Revista Espanola de Salud Publica. 2009;83:427–439.
    1. Carroll R, Ruppert D, Stefanski L, Crainiceanu C. Measurement Error in Nonlinear Models: A Modern Perspective 2nd edition. Boca Raton, FL: Taylor and Francis Group; 2006. Regression calibration; pp. 65–95.
    1. Cradock AL, Barrett JL, Carter J, McHugh A, Sproul J, Russo ET, Dao-Tran P, Gortmaker SL. Impact of the Boston Active School Day policy to promote physical activity among children. Am J Health Promot. 2014;28(3 Suppl):S54–64. doi: 10.4278/ajhp.130430-QUAN-204.
    1. Mendoza JA, Watson K, Nguyen N, Cerin E, Baranowski T, Nicklas TA. Active commuting to school and association with physical activity and adiposity among US youth. J Phys Act Health. 2011 May;8(4):488–95.
    1. Ruch N, Joss F, Jimmy G, Melzer K, Hänggi J, Mäder U. Neural network versus activity-specific prediction equations for energy expenditure estimation in children. J Appl Physiol (1985) 2013 Nov 1;115(9):1229–36. doi: 10.1152/japplphysiol.01443.2012.
    1. Pober DM, Staudenmayer J, Raphael C, Freedson PS. Development of novel techniques to classify physical activity mode using accelerometers. Med Sci Sports Exerc. 2006 Sep;38(9):1626–34. doi: 10.1249/01.mss.0000227542.43669.45.
    1. Choi L, Liu Z, Matthews CE, Buchowski MS. Validation of accelerometer wear and nonwear time classification algorithm. Med Sci Sports Exerc. 2011 Feb;43(2):357–64. doi: 10.1249/MSS.0b013e3181ed61a3.
    1. Lee IM, Shiroma EJ. Using accelerometers to measure physical activity in large-scale epidemiological studies: issues and challenges. Br J Sports Med. 2014 Feb;48(3):197–201. doi: 10.1136/bjsports-2013-093154.
    1. Arvidsson D, Slinde F, Larsson S, Hulthén L. Energy cost of physical activities in children: validation of SenseWear Armband. Med Sci Sports Exerc. 2007 Nov;39(11):2076–84. doi: 10.1249/mss.0b013e31814fb439.
    1. Jakicic JM, Marcus M, Gallagher KI, Randall C, Thomas E, Goss FL, Robertson RJ. Evaluation of the SenseWear Pro Armband to assess energy expenditure during exercise. Med Sci Sports Exerc. 2004 May;36(5):897–904.
    1. Bassett DR, Rowlands A, Trost SG. Calibration and validation of wearable monitors. Med Sci Sports Exerc. 2012 Jan;44(1 Suppl 1):S32–8. doi: 10.1249/MSS.0b013e3182399cf7.
    1. Welk GJ. Principles of design and analyses for the calibration of accelerometry-based activity monitors. Med Sci Sports Exerc. 2005 Nov;37(11 Suppl):S501–11.
    1. Gracia-Marco L, Ortega FB, Ruiz JR, Williams CA, Hagströmer M, Manios Y, Kafatos A, Béghin L, Polito A, De Henauw S, Valtueña J, Widhalm K, Molnar D, Alexy U, Moreno LA, Sjöström M, Helena Study Group Seasonal variation in physical activity and sedentary time in different European regions. The HELENA study. J Sports Sci. 2013;31(16):1831–40. doi: 10.1080/02640414.2013.803595.
    1. Whitlock G, Clark T, Vander Hoorn S, Rodgers A, Jackson R, Norton R, MacMahon S. Random errors in the measurement of 10 cardiovascular risk factors. Eur J Epidemiol. 2001;17(10):907–9.
    1. Carroll RJ, Ruppert D, Stefanski LA, Crainiceanu CM. Measurement Error in Nonlinear Models: A Modern Perspective. Volume 2. Boca Raton, FL: Chapman & Hall/CRC; 2006. Linear regression and atenuation; pp. 41–64.
    1. Ferrari P, Friedenreich C, Matthews CE. The role of measurement error in estimating levels of physical activity. Am J Epidemiol. 2007 Oct 1;166(7):832–40. doi: 10.1093/aje/kwm148.
    1. Harrell JS, McMurray RG, Baggett CD, Pennell ML, Pearce PF, Bangdiwala SI. Energy costs of physical activities in children and adolescents. Med Sci Sports Exerc. 2005 Feb;37(2):329–36.
    1. Ridley K, Olds TS. Assigning energy costs to activities in children: a review and synthesis. Med Sci Sports Exerc. 2008 Aug;40(8):1439–46. doi: 10.1249/MSS.0b013e31817279ef.
    1. Esliger DW, Copeland JL, Barnes JD, Tremblay MS. Standardizing and optimizing the use of accelerometer data for free-living physical activity monitoring. J Phys Act Health. 2005;3:366–383.
    1. De Meester F, De Bourdeaudhuij I, Deforche B, Ottevaere C, Cardon G. Measuring physical activity using accelerometry in 13-15-year-old adolescents: the importance of including non-wear activities. Public Health Nutr. 2011 Dec;14(12):2124–33. doi: 10.1017/S1368980011001868.
    1. Ottevaere C, Huybrechts I, De Meester F, De Bourdeaudhuij I, Cuenca-Garcia M, De Henauw S. The use of accelerometry in adolescents and its implementation with non-wear time activity diaries in free-living conditions. J Sports Sci. 2011 Jan;29(1):103–13. doi: 10.1080/02640414.2010.521169.
    1. Mâsse LC, Fuemmeler BF, Anderson CB, Matthews CE, Trost SG, Catellier DJ, Treuth M. Accelerometer data reduction: a comparison of four reduction algorithms on select outcome variables. Med Sci Sports Exerc. 2005 Nov;37(11 Suppl):S544–54.
    1. Toftager M, Kristensen PL, Oliver M, Duncan S, Christiansen LB, Boyle E, Brønd JC, Troelsen J. Accelerometer data reduction in adolescents: effects on sample retention and bias. Int J Behav Nutr Phys Act. 2013;10:140. doi: 10.1186/1479-5868-10-140.
    1. Hallal PC, Reichert FF, Clark VL, Cordeira KL, Menezes AM, Eaton S, Ekelund U, Wells JC. Energy expenditure compared to physical activity measured by accelerometry and self-report in adolescents: a validation study. PLoS One. 2013;8(11):e77036. doi: 10.1371/journal.pone.0077036.
    1. Corder K, van Sluijs EM, Wright A, Whincup P, Wareham NJ, Ekelund U. Is it possible to assess free-living physical activity and energy expenditure in young people by self-report? Am J Clin Nutr. 2009 Mar;89(3):862–70. doi: 10.3945/ajcn.2008.26739.
    1. Saint-Maurice PF, Welk GJ, Beyler NK, Bartee RT, Heelan KA. Calibration of self-report tools for physical activity research: the Physical Activity Questionnaire (PAQ) BMC Public Health. 2014;14:461. doi: 10.1186/1471-2458-14-461.
    1. Scholes S, Coombs N, Pedisic Z, Mindell JS, Bauman A, Rowlands AV, Stamatakis E. Age- and sex-specific criterion validity of the health survey for England Physical Activity and Sedentary Behavior Assessment Questionnaire as compared with accelerometry. Am J Epidemiol. 2014 Jun 15;179(12):1493–502. doi: 10.1093/aje/kwu087.
    1. Wang C, Chen P, Zhuang J. Validity and reliability of International Physical Activity Questionnaire-Short Form in Chinese youth. Res Q Exerc Sport. 2013 Dec;84 Suppl 2:S80–6. doi: 10.1080/02701367.2013.850991.
    1. Medina C, Barquera S, Janssen I. Validity and reliability of the International Physical Activity Questionnaire among adults in Mexico. Rev Panam Salud Publica. 2013 Jul;34(1):21–8.
    1. Manios Y, Androutsos O, Moschonis G, Birbilis M, Maragkopoulou K, Giannopoulou A, Argyri E, Kourlaba G. Criterion validity of the Physical Activity Questionnaire for Schoolchildren (PAQ-S) in assessing physical activity levels: the Healthy Growth Study. J Sports Med Phys Fitness. 2013 Oct;53(5):502–8.
    1. Matthews CE, Keadle SK, Sampson J, Lyden K, Bowles HR, Moore SC, Libertine A, Freedson PS, Fowke JH. Validation of a previous-day recall measure of active and sedentary behaviors. Med Sci Sports Exerc. 2013 Aug;45(8):1629–38. doi: 10.1249/MSS.0b013e3182897690.
    1. Dyrstad SM, Hansen BH, Holme IM, Anderssen SA. Comparison of self-reported versus accelerometer-measured physical activity. Med Sci Sports Exerc. 2014 Jan;46(1):99–106. doi: 10.1249/MSS.0b013e3182a0595f.
    1. Walker E, Nowacki AS. Understanding equivalence and noninferiority testing. J Gen Intern Med. 2011 Feb;26(2):192–6. doi: 10.1007/s11606-010-1513-8.
    1. Robinson AP, Froese RE. Model validation using equivalence tests. Ecological Modelling. 2004 Sep;176(3-4):349–358. doi: 10.1016/j.ecolmodel.2004.01.013.
    1. Schuirmann DJ. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. J Pharmacokinet Biopharm. 1987 Dec;15(6):657–80.
    1. Midha KK, McKay G. Bioequivalence; its history, practice, and future. AAPS J. 2009 Dec;11(4):664–70. doi: 10.1208/s12248-009-9142-z.
    1. Hall R, Hanna P. The impact of web page text-background colour combinations on readability, retention, aesthetics and behavioural intention. Behaviour & Information Technology. 2004;23:183–195.
    1. Gnambs T, Appel M, Batinic B. Color red in web-based knowledge testing. Computers in Human Behavior. 2010;26:1625–1631.
    1. Baranowski T, Thompson WO, DuRant RH, Baranowski J, Puhl J. Observations on physical activity in physical locations: age, gender, ethnicity, and month effects. Res Q Exerc Sport. 1993 Jun;64(2):127–33. doi: 10.1080/02701367.1993.10608789.
    1. Tucker P, Gilliland J. The effect of season and weather on physical activity: a systematic review. Public Health. 2007 Dec;121(12):909–22. doi: 10.1016/j.puhe.2007.04.009.
    1. Welk GJ, Corbin CB, Dale D. Measurement issues in the assessment of physical activity in children. Res Q Exerc Sport. 2000 Jun;71(2 Suppl):S59–73.
    1. Welk GJ, McClain J, Ainsworth BE. Protocols for evaluating equivalency of accelerometry-based activity monitors. Med Sci Sports Exerc. 2012 Jan;44(1 Suppl 1):S39–49. doi: 10.1249/MSS.0b013e3182399d8f.
    1. Welk GJ, Kim Y, Stanfill B, Osthus DA, Calabro MA, Nusser SM, Carriquiry A. Validity of 24-h physical activity recall: physical activity measurement survey. Med Sci Sports Exerc. 2014 Oct;46(10):2014–24. doi: 10.1249/MSS.0000000000000314.
    1. Tucker JM, Welk G, Nusser SM, Beyler NK, Dzewaltowski D. Estimating minutes of physical activity from the previous day physical activity recall: validation of a prediction equation. J Phys Act Health. 2011 Jan;8(1):71–8.
    1. Osthus D, Beyler NK, Stanfill B, Nusser SM, Fuller WA, Carriquiry AL, Welk GJ. Estimating the distribution of usual daily energy expenditure for subpopulations from a probability sample. Statistics in Medicine. 2014:1–16.
    1. Hallal PC, Andersen LB, Bull FC, Guthold R, Haskell W, Ekelund U, Lancet Physical Activity Series Working Group Global physical activity levels: surveillance progress, pitfalls, and prospects. Lancet. 2012 Jul 21;380(9838):247–57. doi: 10.1016/S0140-6736(12)60646-1.
    1. Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: a clear and user-friendly guideline. J Eval Clin Pract. 2011 Apr;17(2):268–74. doi: 10.1111/j.1365-2753.2010.01434.x.
    1. Bull FC, Maslin TS, Armstrong T. Global physical activity questionnaire (GPAQ): nine country reliability and validity study. J Phys Act Health. 2009 Nov;6(6):790–804.
    1. Tomioka K, Iwamoto J, Saeki K, Okamoto N. Reliability and validity of the International Physical Activity Questionnaire (IPAQ) in elderly adults: the Fujiwara-kyo Study. J Epidemiol. 2011;21(6):459–65.
    1. Dahl-Petersen IK, Hansen AW, Bjerregaard P, Jørgensen ME, Brage S. Validity of the international physical activity questionnaire in the arctic. Med Sci Sports Exerc. 2013 Apr;45(4):728–36. doi: 10.1249/MSS.0b013e31827a6b40.
    1. Lachat CK, Verstraeten R, Khanh le NB, Hagströmer M, Khan NC, Van Ndo A, Dung NQ, Kolsteren PW. Validity of two physical activity questionnaires (IPAQ and PAQA) for Vietnamese adolescents in rural and urban areas. Int J Behav Nutr Phys Act. 2008;5:37. doi: 10.1186/1479-5868-5-37.
    1. De Vera MA, Ratzlaff C, Doerfling P, Kopec J. Reliability and validity of an internet-based questionnaire measuring lifetime physical activity. Am J Epidemiol. 2010 Nov 15;172(10):1190–8. doi: 10.1093/aje/kwq273.

Source: PubMed

Sponsoren und Mitarbeiter

Krankheiten

Drogeninterventionen

3
Abonnieren