Reliability and validity of a nutrition and physical activity environmental self-assessment for child care

Sara E Benjamin, Brian Neelon, Sarah C Ball, Shrikant I Bangdiwala, Alice S Ammerman, Dianne S Ward, Sara E Benjamin, Brian Neelon, Sarah C Ball, Shrikant I Bangdiwala, Alice S Ammerman, Dianne S Ward

Abstract

Background: Few assessment instruments have examined the nutrition and physical activity environments in child care, and none are self-administered. Given the emerging focus on child care settings as a target for intervention, a valid and reliable measure of the nutrition and physical activity environment is needed.

Methods: To measure inter-rater reliability, 59 child care center directors and 109 staff completed the self-assessment concurrently, but independently. Three weeks later, a repeat self-assessment was completed by a sub-sample of 38 directors to assess test-retest reliability. To assess criterion validity, a researcher-administered environmental assessment was conducted at 69 centers and was compared to a self-assessment completed by the director. A weighted kappa test statistic and percent agreement were calculated to assess agreement for each question on the self-assessment.

Results: For inter-rater reliability, kappa statistics ranged from 0.20 to 1.00 across all questions. Test-retest reliability of the self-assessment yielded kappa statistics that ranged from 0.07 to 1.00. The inter-quartile kappa statistic ranges for inter-rater and test-retest reliability were 0.45 to 0.63 and 0.27 to 0.45, respectively. When percent agreement was calculated, questions ranged from 52.6% to 100% for inter-rater reliability and 34.3% to 100% for test-retest reliability. Kappa statistics for validity ranged from -0.01 to 0.79, with an inter-quartile range of 0.08 to 0.34. Percent agreement for validity ranged from 12.9% to 93.7%.

Conclusion: This study provides estimates of criterion validity, inter-rater reliability and test-retest reliability for an environmental nutrition and physical activity self-assessment instrument for child care. Results indicate that the self-assessment is a stable and reasonably accurate instrument for use with child care interventions. We therefore recommend the Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC) instrument to researchers and practitioners interested in conducting healthy weight intervention in child care. However, a more robust, less subjective measure would be more appropriate for researchers seeking an outcome measure to assess intervention impact.

References

    1. Ogden CL, Carroll MD, Curtin LR, McDowell MA, Tabak CJ, Flegal KM. Prevalence of Overweight and Obesity in the United States, 1999–2004. JAMA. 2006;295:1549–1555. doi: 10.1001/jama.295.13.1549.
    1. Tremblay MS, Katzmarzyk PT, Willms JD. Temporal trends in overweight and obesity in Canada, 1981–1996. Int J Obes Relat Metab Disord. 2002;26:538–543. doi: 10.1038/sj.ijo.0801923.
    1. Lissau I, Overpeck MD, Ruan WJ, Due P, Holstein BE, Hediger ML. Body mass index and overweight in adolescents in 13 European countries, Israel, and the United States. Arch Pediatr Adolesc Med. 2004;158:27–33. doi: 10.1001/archpedi.158.1.27.
    1. Venn AJ, Thomson RJ, Schmidt MD, Cleland VJ, Curry BA, Gennat HC, Dwyer T. Overweight and obesity from childhood to adulthood: a follow-up of participants in the 1985 Australian Schools Health and Fitness Survey. Med J Aust. 2007;186:458–460.
    1. Kelishadi R. Childhood Overweight, Obesity, and the Metabolic Syndrome in Developing Countries. Epidemiol Rev. 2007
    1. Pinhas-Hamiel O, Dolan LM, Daniels SR, Standiford D, Khoury PR, Zeitler P, Freedman DS, Serdula MK, Srinivasan SR, Berenson GS. Increased incidence of non-insulin-dependent diabetes mellitus among adolescents. J Pediatr. 1996;128:608–615. doi: 10.1016/S0022-3476(96)80124-7.
    1. Freedman DS, Serdula MK, Srinivasan SR, Berenson GS. Relation of circumferences and skinfold thicknesses to lipid and insulin concentrations in children and adolescents: the Bogalusa Heart Study. Am J Clin Nutr. 1999;69:308–317.
    1. Morrison JA, Sprecher DL, Barton BA, Waclawiw MA, Daniels SR. Overweight, fat patterning, and cardiovascular disease risk factors in black and white girls: The National Heart, Lung, and Blood Institute Growth and Health Study. J Pediatr. 1999;135:458–464. doi: 10.1016/S0022-3476(99)70168-X.
    1. Leung AK, Robson WL. Childhood obesity. Postgrad Med. 1990;87:123–130. 133.
    1. Dietz WH. Health consequences of obesity in youth: childhood predictors of adult disease. Pediatrics. 1998;101:518–525.
    1. Hill JO, Peters JC. Environmental contributions to the obesity epidemic. Science. 1998;280:1371–1374. doi: 10.1126/science.280.5368.1371.
    1. Sturm R. Childhood obesity – what we can learn from existing data on societal trends, part 2. Prev Chronic Dis. 2005;2:A20.
    1. Malig C. The Civil Registration System in Denmark; IIVRS Technical Paper. Vol. 66. Bethesda; IIVRS Statistics; 1996. pp. 1–6.
    1. Oldfield K, Siebers R, Crane J. Endotoxin and indoor allergen levels in kindergartens and daycare centres in Wellington, New Zealand. N Z Med J. 2007;120:U2400.
    1. Monasch R, Boerma JT. Orphanhood and childcare patterns in sub-Saharan Africa: an analysis of national surveys from 40 countries. Aids. 2004;18:S55–65. doi: 10.1097/00002030-200406002-00007.
    1. Statistics Canada and Human Resourees Development Canada
    1. Federal Interagency Forum on Child and Family Statistics . America's Children: Key National Indicators of Well-Being, 2002. Washington, DC: U.S. Government Printing Office; 2002.
    1. Williams CL, Bollella MC, Strobino BA, Spark A, Nicklas TA, Tolosi LB, Pittman BP. "Healthy-start": outcome of an intervention to promote a heart healthy diet in preschool children. J Am Coll Nutr. 2002;21:62–71.
    1. Williams CL, Strobino BA, Bollella M, Brotanek J. Cardiovascular risk reduction in preschool children: the "Healthy Start" project. J Am Coll Nutr. 2004;23:117–123.
    1. Dennison BA, Russo TJ, Burdick PA, Jenkins PL. An intervention to reduce television viewing by preschool children. Arch Pediatr Adolesc Med. 2004;158:170–176. doi: 10.1001/archpedi.158.2.170.
    1. Fitzgibbon ML, Stolley MR, Schiffer L, Van Horn L, KauferChristoffel K, Dyer A. Two-year follow-up results for Hip-Hop to Health Jr.: a randomized controlled trial for overweight prevention in preschool minority children. J Pediatr. 2005;146:618–625. doi: 10.1016/j.jpeds.2004.12.019.
    1. Sallis JF, Nader PR, Broyles SL, Berry CC, Elder JP, McKenzie TL, Nelson JA. Correlates of physical activity at home in Mexican-American and Anglo-American preschool children. Health Psychol. 1993;12:390–398. doi: 10.1037/0278-6133.12.5.390.
    1. Trost SG, Sirard JR, Dowda M, Pfeiffer KA, Pate RR. Physical activity in overweight and nonoverweight preschool children. Int J Obes Relat Metab Disord. 2003;27:834–839. doi: 10.1038/sj.ijo.0802311.
    1. Briley ME, Buller AC, Roberts-Gray CR, Sparkman A. What is on the menu at the child care center? J Am Diet Assoc. 1989;89:771–774.
    1. Briley ME, Jastrow S, Vickers J, Roberts-Gray C. Dietary intake at child-care centers and away: are parents and care providers working as partners or at cross-purposes? J Am Diet Assoc. 1999;99:950–954. doi: 10.1016/S0002-8223(99)00226-6.
    1. Dowda M, Pate RR, Trost SG, Almeida MJ, Sirard JR. Influences of preschool policies and practices on children's physical activity. J Community Health. 2004;29:183–196. doi: 10.1023/B:.
    1. Bradley RH. Children's home environments, health, behavior, and intervention efforts: a review using the HOME inventory as a marker measure. Genet Soc Gen Psychol Monogr. 1993;119:437–490.
    1. Strauss RS, Knight J. Influence of the home environment on the development of obesity in children. Pediatrics. 1999;103:e85. doi: 10.1542/peds.103.6.e85.
    1. Parcel GS, Perry CL, Kelder SH, Elder JP, Mitchell PD, Lytle LA, Johnson CC, Stone EJ. School climate and the institutionalization of the CATCH program. Health Educ Behav. 2003;30:489–502. doi: 10.1177/1090198103253650.
    1. Carter MA, Swinburn B. Measuring the 'obesogenic' food environment in New Zealand primary schools. Health Promot Int. 2004;19:15–20. doi: 10.1093/heapro/dah103.
    1. Boarnet MG, Day K, Alfonzo M, Forsyth A, Oakes M. The Irvine-Minnesota inventory to measure built environments: reliability tests. Am J Prev Med. 2006;30:153–159. doi: 10.1016/j.amepre.2005.09.018.
    1. Harms T, Clifford RM, Cryer D. Early Childhood Environment Rating Scale, revised edition. New York: Teachers College Press; 1998.
    1. Harms T, Cryer D, Clifford RM. Infant/Toddler Environment Rating Scale. New York: Teachers College Press; 1990.
    1. Ammerman ASWD, Benjamin SE, Ball SC, Sommers JS, Molloy M, Dodds JM. An Intervention to Promote Healthy Weight: Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC) Theory and Design. Preventing Chronic Disease. 2007.
    1. Glanz DK, Rimer BK, Lewis FM. Health Behavior and Health Education: Theory, Research, and Practice. Third. San Francisco: Jossey-Bass; 2002.
    1. McLeroy KR, Bibeau D, Steckler A, Glanz K. An ecological perspective on health promotion programs. Health Educ Q. 1988;15:351–377.
    1. Benjamin SE, Ammerman A, Sommers J, Dodds J, Neelon B, Ward DS. Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC): Results from a Pilot Intervention. J Nutr Educ Behav. 2007;39:142–149. doi: 10.1016/j.jneb.2006.08.027.
    1. Crowley AA. Child care health consultation: an ecological model. J Soc Pediatr Nurs. 2001;6:170–181. doi: 10.1111/j.1744-6155.2001.tb00241.x.
    1. Prochaska JJ, Sallis JF. Reliability and validity of a fruit and vegetable screening measure for adolescents. J Adolesc Health. 2004;34:163–165.
    1. Golan M, Weizman A. Reliability and validity of the Family Eating and Activity Habits Questionnaire. Eur J Clin Nutr. 1998;52:771–777. doi: 10.1038/sj.ejcn.1600647.
    1. Devellis RF. Scale Development: Theory and Applications. 2. Thousand Oaks, CA: Sage Publications, Inc; 2003.
    1. Cohen J. A coefficient for agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37–46. doi: 10.1177/001316446002000104.
    1. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. doi: 10.2307/2529310.
    1. Munoz SR, Bangdiwala SI. Interpretation of Kappa and B statistics measures of agreement. Journal of Applied Statistics. 1997;24:105–111. doi: 10.1080/02664769723918.
    1. Allison T, Cicchetti DV. Sleep in mammals: ecological and constitutional correlates. Science. 1976;194:732–734. doi: 10.1126/science.982039.
    1. Chinn S, Burney PG. On measuring repeatability of data from self-administered questionnaires. Int J Epidemiol. 1987;16:121–127. doi: 10.1093/ije/16.1.121.
    1. Fleischhacker S, Cason KL, Achterberg C. "You Had Peas Today?": A Pilot Study Comparing a Head Start Child-Care Center's Menu with the Actual Food Served. J Am Diet Assoc. 2006;106:277–280. doi: 10.1016/j.jada.2005.10.038.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe