A systematic review of publications assessing reliability and validity of the Behavioral Risk Factor Surveillance System (BRFSS), 2004-2011

Carol Pierannunzi, Shaohua Sean Hu, Lina Balluz, Carol Pierannunzi, Shaohua Sean Hu, Lina Balluz

Abstract

Background: In recent years response rates on telephone surveys have been declining. Rates for the behavioral risk factor surveillance system (BRFSS) have also declined, prompting the use of new methods of weighting and the inclusion of cell phone sampling frames. A number of scholars and researchers have conducted studies of the reliability and validity of the BRFSS estimates in the context of these changes. As the BRFSS makes changes in its methods of sampling and weighting, a review of reliability and validity studies of the BRFSS is needed.

Methods: In order to assess the reliability and validity of prevalence estimates taken from the BRFSS, scholarship published from 2004-2011 dealing with tests of reliability and validity of BRFSS measures was compiled and presented by topics of health risk behavior. Assessments of the quality of each publication were undertaken using a categorical rubric. Higher rankings were achieved by authors who conducted reliability tests using repeated test/retest measures, or who conducted tests using multiple samples. A similar rubric was used to rank validity assessments. Validity tests which compared the BRFSS to physical measures were ranked higher than those comparing the BRFSS to other self-reported data. Literature which undertook more sophisticated statistical comparisons was also ranked higher.

Results: Overall findings indicated that BRFSS prevalence rates were comparable to other national surveys which rely on self-reports, although specific differences are noted for some categories of response. BRFSS prevalence rates were less similar to surveys which utilize physical measures in addition to self-reported data. There is very little research on reliability and validity for some health topics, but a great deal of information supporting the validity of the BRFSS data for others.

Conclusions: Limitations of the examination of the BRFSS were due to question differences among surveys used as comparisons, as well as mode of data collection differences. As the BRFSS moves to incorporating cell phone data and changing weighting methods, a review of reliability and validity research indicated that past BRFSS landline only data were reliable and valid as measured against other surveys. New analyses and comparisons of BRFSS data which include the new methodologies and cell phone data will be needed to ascertain the impact of these changes on estimates in the future.

Figures

Figure 1
Figure 1
Method of assessing literature on reliability and validity of the BRFSS.

References

    1. US Department of Health and Human Services. Healthy People 2010. 2. Washington DC: U.S. Department of Health and Human Services; 2000.
    1. Centers for Disease Control and Prevention. Behavioral Risk Factor Surveillance System. .
    1. Galesic MG, Tourangeau R, Couper MP. Complementing random-digit-dialing telephone surveys with other approaches to collecting sensitive data. Am J Prev Med. 2006;31:437–443. doi: 10.1016/j.amepre.2006.07.023.
    1. Nelson DE, Holtzman D, Bolen J, Stanwyck CA, Mack KA. Reliability and validity of measures from the Behavioral Risk Factor Surveillance System (BRFSS) Int J Public Health. 2001;46:1–42.
    1. Mokdad A. The behavioral risk factors surveillance system: past, present and future. Annual Rev Public Health. 2009;30:43–54. doi: 10.1146/annurev.publhealth.031308.100226.
    1. Blumberg SJ, Luke JV, Cynamon ML. Telephone coverage and health survey estimates: evaluating the need for concern about wireless substitution. Am J Public Health. 2006;96:926–31. doi: 10.2105/AJPH.2004.057885.
    1. Centers for Disease Control and Prevention. National Center for Health Statistics. .
    1. US Census Bureau. Current Population Survey.
    1. Centers for Disease Control and Prevention. National Survey of Family Growth.
    1. Everson KR, McGinn AP. Test-retest reliability of adult surveillance measures for physical activity and inactivity. Am J Preventive Med. 2005;28:470–478. doi: 10.1016/j.amepre.2005.02.005.
    1. Link MW, Battaglia MP, Frankel MR, Osborn L, Mokdad AH. Address-based versus random-digit-dialing surveys: comparisons of key health and risk indicators. Am J Epidemiol. 2006;164:1019–1025. doi: 10.1093/aje/kwj310.
    1. Fahimi M, Link M, Schwartz DA, Levy P, Mokdad A. Tracking chronic disease and risk behavior prevalence as survey participation declines: statistics from the behavioral risk factor surveillance system and other national surveys. Prev Chronic Dis. 2008;5:1–16.
    1. Battaglia MP, Frankel MR, Link M. Improving post stratification techniques for random-digit-dialing telephone surveys. Survey Res Methods. 2008;2:11–19.
    1. Altman DG, Bland JM. Diagnostic tests: sensitivity and specificity. Med J. 1994;308:1552. doi: 10.1136/bmj.308.6943.1552.
    1. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74. doi: 10.2307/2529310.
    1. Deyo RA, Diehr P, Patrick DL. Reproducibility and responsiveness of health status measures. Control Clin Trials. 1991;12:142–158. doi: 10.1016/S0197-2456(05)80019-4.
    1. Pettee KK, Ham SA, Macera CA, Ainsworth BA. The reliability of a survey question on television viewing and associations with health risk factors in US adults. Obesity. 2008;17:487–493.
    1. Yore MM, Ham SA, Ainsworth BE, Kruger J, Reis JP, Kohl HW, Macera CA. Reliability and validity of the instrument used in BRFSS to assess physical activity. Med Sci Sports Exerc. 2007;39:1267–74. doi: 10.1249/mss.0b013e3180618bbe.
    1. Mucci LA, Wood PA, Cohen B, Clements KM, Brawarsky P, Brooks DR. Validity of self-reported health plan information in a population-based health survey. J Publ Health Manage Pract. 2006;12:570–7.
    1. Clements KM, Cohen BB, Brawarsky P, Brooks DR, Mucci LA, Wood PA. Identifying health maintenance organization membership through self-report of health plan name: ascertainment and reliability. J Publ Health Manage Pract. 2006;12:278–287.
    1. Salomon JA, Nordhagen S, Oza S, Murray CJL. Are Americans feeling less healthy? The puzzle in self-rated health. Am J Epidemiol. 2009;170:343–351. doi: 10.1093/aje/kwp144.
    1. Shenson D, DiMartino D, Bolen J, Campbell M, Ly P, Singleton JA. Validation of self-reported pneumococcal vaccination in behavioral risk factor surveillance surveys: experience from the sickness prevention through regional collaboration (SPARC) program. Vaccine. 2005;25:1015–1020.
    1. Bradbury BD, Brooks DR, Brawarsky P, Mucci LA. Test-retest reliability of colorectal testing question on the Massachusetts Behavioral Risk Factor Surveillance System (BRFSS) Prev Med. 2004;41:303–311.
    1. Cronin KA, Miglioretti DL, Yu B, Krapcho M, Geller BM, Carney PA, Onega T, Feuer EJ, Breen N, Ballard-Barbash R. Bias associated with self-report of prior screening mammography. Cancer Epidemiol Biomarkers Prev. 2009;18:1699–1705. doi: 10.1158/1055-9965.EPI-09-0020.
    1. Yore MM, Ham SA, Ainsworth BE, Macera CA, Jones DA, Kohl HW. J Phys Activ Health. 2005. pp. 358–365.
    1. Brown WJ, Trost SG, Bauman A, Mummery K, Owen N. Test-retest reliability of four physical activity measures used in population surveys. J Sci Med Sport. 2004;7:205–215. doi: 10.1016/S1440-2440(04)80010-0.
    1. Hutto B, Sharpe BA, Granner ML, Addy CL, Hooker S. The effect of question order on reporting physical activity and walking behavior. J PhysActiv. 2008;5:16–29.
    1. Carlson SA, Densmore D, Fulton JE, Yore MM, Kohl HW. Differences in physical activity prevalence and trends from 3 US surveillance systems: NHIS, NHANES and BRFSS. J Phys Activ Health. 2009;6:18–27.
    1. Reis JP, DuBose KD, Ainsworth BE, Macera CA, Yore MM. Reliability and validity of the occupational physical activity questionnaire. Med Sci Sports Exerc. 2005;37:2075–2083. doi: 10.1249/01.mss.0000179103.20821.00.
    1. Bombard JM, Powell KE, Martin LM, Helmick CG, Wilson WH. Validity and reliability of self-reported arthritis: Georgia senior centers 2000–2001. Am J Prev Med. 2005;28:251–258. doi: 10.1016/j.amepre.2004.12.004.
    1. Sacks JJ, Harrold LR, Helmick CG, Gurwitz JH, Emani S, Yood RA. Validation of a surveillance case definition for arthritis. J Rheumatol. 2005;32:340–347.
    1. Cossman RE, Cossman JS, James WL, Blanchard T, Thomas RK, Pol LG, Cosby AG, Mirvis DM. J Health Human Serv Admin. 2008. pp. 503–529.
    1. Andresen EM, Catlin TK, Wyrwich KW, Jackson-Thompson J. Retest reliability of surveillance questions on health related quality of life. J Epidemiol Commun Health. 2003;57:339–343. doi: 10.1136/jech.57.5.339.
    1. Kapp JM, Jacksonc T, Petroski GF, Schootman M. Reliability of health related quality of life indicators in cancer survivors from a population based sample, 2005, BRFSS. Public Health. 2009;123:321–325. doi: 10.1016/j.puhe.2008.10.005.
    1. Ezzati M, Martin H, Skjold SA, Noorn SV, Murray CJL. Trends in national and state-level obesity in the USA after correction for self-report bias: analysis of health surveys. JR Soc Med. 2006;99:250–257.
    1. Yun S, Zhu B-P, Black W, Brownson RC. A comparison of national estimates of obesity prevalence from the behavioral risk factor surveillance systems and the national health and nutrition examination survey. Inter J Obes. 2005;30:164–170.
    1. Klein JD, Thomas RK, Sutter EJ. Self-reported smoking in online surveys: prevalence estimate validity and item format effects. Med Care. 2007;45:691–695. doi: 10.1097/MLR.0b013e3180326145.
    1. Miller JW, Gfroerer JC, Brewer RD, Naimi TS, Mokdad A, Giles WH. Prevalence of adult binge drinking: a comparison of two national surveys. Am J Prev Med. 2004;27:197–204.
    1. Santelli J, Lindberg LD, Finer LB, Rickert VI, Bensyl D, Posner S, Makleff S, Kost K, Singh S. Comparability of contraceptive prevalence estimates for women from the 2002 Behavioral Risk Factor Surveillance System. Public Health Rep. 2008;123:147–154.
    1. Bonomi AE, Thompson RS, Anderson M, Rivera FP, Holt VL, Carrell D, Martin DP. Ascertainment of intimate partner violence using two abuse measurement frameworks. Inj Prev. 2006;12:121–124. doi: 10.1136/ip.2005.009563.
    1. Tourangeau R, Smith TM. Asking sensitive questions: the impact of data collection mode, question format, and question content. Public Opin Q. 1996;60:275–304. doi: 10.1086/297751.

Source: PubMed

Sponsor e collaboratori

Condizioni mediche

Interventi farmacologici

3
Sottoscrivi