Reliability of internet- versus telephone-administered questionnaires in a diverse sample of smokers

Amanda L Graham, George D Papandonatos, Amanda L Graham, George D Papandonatos

Abstract

Background: Smoking is more prevalent among lower-income individuals and certain racial/ethnic minorities. Addressing tobacco cessation among diverse populations is an urgent public health priority. As Internet use continues to rise among all segments of the US population, Web-based interventions have enormous potential to reach priority populations. Conducting Web-based smoking cessation research in priority populations requires psychometrically sound measurement instruments. To date, only one published study has examined the psychometric properties of Internet-administered measures commonly used in Web-based cessation trials. However, the sample was homogeneous with regard to race/ethnicity and income. We sought to replicate and extend these findings in a more diverse sample of smokers.

Objective: The aim was to examine the internal consistency and test-retest reliability of measures commonly used in smoking cessation clinical trials among racial/ethnic minorities and smokers with lower income.

Methods: Participants were enrolled in a randomized trial of the efficacy of an Internet smoking cessation program between June 2005 and September 2006. Following a baseline telephone assessment and randomization into the parent trial, participants were recruited to the reliability substudy. In phase I of recruitment, all participants in the parent trial were recruited to the substudy; in phase II, all consecutive racial/ethnic minority participants in the parent trial were recruited. Race and ethnicity were assessed via self-report using two standard items from the US Office of Management and Budget. An email was sent 2 days after the telephone assessment with a link to the Internet survey. Measures examined were quit methods, perceived stress, depression, social support, smoking temptations, alcohol use, perceived health status, and income. Internal consistency and test-retest reliability of Internet- versus telephone-administered measures were examined within four strata defined by race/ethnicity (non-Hispanic White, racial/ethnic minority) and annual household income (US $40,000 or less, more than $40,000).

Results: Of the 442 individuals invited, 319 participated (72% response rate): 52.4% were non-Hispanic White, 22.9% Black, 11.6% Hispanic, 7.8% Asian, 4.4% American Indian / Alaska Native, and 1% Native Hawaiian / Other Pacific Islander. About half (49.4%) reported an annual household income of US $40,000 or less, and 25.7% had a high school degree or less. Test-retest reliability was satisfactory to excellent across all strata for the majority of measures examined: 9 of 12 continuous variables had intraclass correlation coefficients > or = 0.70, and 10 of 18 binary variables and both ordinal variables had kappa coefficients > or = 0.70. Test-retest reliability of several quit methods varied across strata.

Conclusions: Race/ethnicity and income do not affect the psychometric properties of most Internet-administered measures examined. This knowledge adds to the confidence of conducting Web-based smoking cessation research and strengthens the scientific rigor of collecting information via the Internet on racial/ethnic minority and low-income subgroups.

Trial registration: clinicaltrials.gov NCT00282009 (parent trial).

Conflict of interest statement

None declared.

Figures

Figure 1
Figure 1
Flowchart of enrollment in phase I and phase II

References

    1. Centers for Disease Control and Prevention (CDC) Tobacco use among adults--United States, 2005. MMWR Morb Mortal Wkly Rep. 2006 Oct 27;55(42):1145–8. mm5542a1
    1. Burns DM, Anderson CM, Johnson M, Major JM, Biener L, Vaughn J, Shanks TG. Cessation and Cessation Measures among Adult Daily Smokers: National and State-Specific Data. In: Population Based Smoking Cessation: Proceedings of a Conference on What Works to Influence Cessation in the General Population. Smoking and Tobacco Control Monograph No. 12. Washington, DC: US Department of Health and Human Services; 2000. Nov:25-98.
    1. Levy David T, Romano Eduardo, Mumford Elizabeth. The relationship of smoking cessation to sociodemographic characteristics, smoking intensity, and tobacco control policies. Nicotine Tob Res. 2005 Jun;7(3):387–96. doi: 10.1080/14622200500125443.R6T70637405N8U31
    1. Centers for Disease Control and Prevention (CDC) Annual smoking-attributable mortality, years of potential life lost, and economic costs--United States, 1995-1999. MMWR Morb Mortal Wkly Rep. 2002 Apr 12;51(14):300–3.
    1. Centers for Disease Control and Prevention (CDC) Annual smoking-attributable mortality, years of potential life lost, and productivity losses--United States, 1997-2001. MMWR Morb Mortal Wkly Rep. 2005 Jul 1;54(25):625–8. mm5425a1
    1. Centers for Disease Control and Prevention (CDC), authors Tobacco Use Among US Racial/Ethnic Minority Groups–African Americans, American Indians and Alaska Natives, Asian Americans and Pacific Islanders, and Hispanics: A Report of the Surgeon General. Atlanta, GA: US Department of Health and Human Services; 1998. [2008 Feb 28]. .
    1. US Department of Health and Human Services, authors. Expanding the Reach and Impact of Consumer E Health Tools. Washington, DC: US Department of Health and Human Services, Office of Disease Prevention and Health Promotion; 2006. [2008 Feb 28]. .
    1. Gibbons Michael C. A historical overview of health disparities and the potential of eHealth solutions. J Med Internet Res. 2005;7(5):e50. doi: 10.2196/jmir.7.5.e50. v7i5e50
    1. Eng Thomas R. Population health technologies: emerging innovations for the health of the public. Am J Prev Med. 2004 Apr;26(3):237–42. doi: 10.1016/j.amepre.2003.12.004.S0749379703003696
    1. Ahern David K. Challenges and opportunities of eHealth research. Am J Prev Med. 2007 May;32(5 Suppl):S75–82. doi: 10.1016/j.amepre.2007.01.016.S0749-3797(07)00045-1
    1. US Department of Health and Human Services, authors. Healthy People 2010: Understanding and Improving Health and Objectives for Improving Health. 2nd edition. Washington, DC: US Government Printing Office; 2000.
    1. Graham Amanda L, Abrams David B. Reducing the cancer burden of lifestyle factors: opportunities and challenges of the Internet. J Med Internet Res. 2005 Jul 1;7(3):e26. doi: 10.2196/jmir.7.3.e26. v7i3e26
    1. Madden M. Internet penetration and impact. Washington, DC: Pew Internet & American Life Project; 2006. [2008 Feb 28]. .
    1. Fox S. Digital divisions. Washington, DC: Pew Internet & American Life Project; 2005. [2008 Feb 28]. .
    1. US Department of Commerce, authors. A Nation Online: How Americans Are Expanding Their Use of the Internet. Washington, DC: US Department of Commerce, Economics and Statistics Administration, National Telecommunications and Information Administration; 2002. [2008 Mar 3]. .
    1. Fox S. Health information online. Washington, DC: Pew Internet & American Life Project; 2005. [2008 Mar 3]. .
    1. Fox S, Fallows D. Internet health resources: health searches and email have become more commonplace, but there is room for improvement in searches and overall Internet access. Washington, DC: Pew Internet & American Life Project; 2003. [2008 Mar 3]. .
    1. Bock B. Web-Assisted Tobacco Interventions (WATI) Initiative: Where we are, where we’re going, and some challenges along the way. Paper presented at: 13th Annual Meeting of the Society for Research on Nicotine and Tobacco; February 21-24, 2007; Austin, TX. 2007
    1. Lenert Leslie, Muñoz Ricardo F, Perez John E, Bansod Aditya. Automated e-mail messaging as a tool for improving quit rates in an internet smoking cessation intervention. J Am Med Inform Assoc. 2004;11(4):235–40. doi: 10.1197/jamia.M1464. M1464
    1. Lenert Leslie, Muñoz Ricardo F, Stoddard Jackie, Delucchi Kevin, Bansod Aditya, Skoczen Steven, Pérez-Stable Eliseo J. Design and pilot evaluation of an internet smoking cessation program. J Am Med Inform Assoc. 2003;10(1):16–20. doi: 10.1197/jamia.M1128.
    1. Feil Edward G, Noell John, Lichtenstein Ed, Boles Shawn M, McKay H Garth. Evaluation of an Internet-based smoking cessation program: lessons learned from a pilot study. Nicotine Tob Res. 2003 Apr;5(2):189–94. doi: 10.1080/1462220031000073694.QWFL1HYWMNT69NPM
    1. Cobb Nathan K, Graham Amanda L, Bock Beth C, Papandonatos George, Abrams David B. Initial evaluation of a real-world Internet smoking cessation system. Nicotine Tob Res. 2005 Apr;7(2):207–16. doi: 10.1080/14622200500055319.M751036L54TH3674
    1. Cheh Julie A, Ribisl Kurt M, Wildemuth Barbara M. An assessment of the quality and usability of smoking cessation information on the Internet. Health Promot Pract. 2003 Jul;4(3):278–87. doi: 10.1177/1524839903004003012.
    1. Bock Beth, Graham Amanda, Sciamanna Christopher, Krishnamoorthy Jenelle, Whiteley Jessica, Carmona-Barros Rosa, Niaura Raymond, Abrams David. Smoking cessation treatment on the Internet: content, quality, and usability. Nicotine Tob Res. 2004 Apr;6(2):207–19. doi: 10.1080/14622200410001676332.TGW76WEC4655JFJQ
    1. Japuntich Sandra J, Zehner Mark E, Smith Stevens S, Jorenby Douglas E, Valdez José A, Fiore Michael C, Baker Timothy B, Gustafson David H. Smoking cessation via the internet: a randomized clinical trial of an internet intervention as adjuvant treatment in a smoking cessation intervention. Nicotine Tob Res. 2006 Dec;8 Suppl 1(1):S59–67. doi: 10.1080/14622200601047900.
    1. Strecher Victor J, Shiffman Saul, West Robert. Randomized controlled trial of a web-based computer-tailored smoking cessation program as a supplement to nicotine patch therapy. Addiction. 2005 May;100(5):682–8. doi: 10.1111/j.1360-0443.2005.01093.x.ADD1093
    1. Strecher Victor J, Shiffman Saul, West Robert. Moderators and mediators of a web-based computer-tailored smoking cessation program among nicotine patch users. Nicotine Tob Res. 2006 Dec;8 Suppl 1(1):S95–101. doi: 10.1080/14622200601039444.
    1. Graham Amanda L, Bock Beth C, Cobb Nathan K, Niaura Raymond, Abrams David B. Characteristics of smokers reached and recruited to an internet smoking cessation trial: a case of denominators. Nicotine Tob Res. 2006 Dec;8 Suppl 1(1):S43–8. doi: 10.1080/14622200601042521.
    1. Swartz L H G, Noell J W, Schroeder S W, Ary D V. A randomised control study of a fully automated internet based smoking cessation programme. Tob Control. 2006 Feb;15(1):7–12. doi: 10.1136/tc.2003.006189. 15/1/7
    1. Etter Jean-François. Comparing the efficacy of two Internet-based, computer-tailored smoking cessation programs: a randomized trial. J Med Internet Res. 2005;7(1):e2. doi: 10.2196/jmir.7.1.e2. v7e2
    1. Pike K Joanne, Rabius Vance, McAlister Alfred, Geiger Angela. American Cancer Society's QuitLink: randomized trial of Internet assistance. Nicotine Tob Res. 2007 Mar;9(3):415–20. doi: 10.1080/14622200701188877.772841762
    1. Etter Jean-François, le Houezec Jacques, Landfeldt Björn. Impact of messages on concomitant use of nicotine replacement therapy and cigarettes: a randomized trial on the Internet. Addiction. 2003 Jul;98(7):941–50. doi: 10.1046/j.1360-0443.2003.00406.x.406
    1. NIH State-of-the-Science Panel. National Institutes of Health State-of-the-Science conference statement: tobacco use: prevention, cessation, and control. Ann Intern Med. 2006 Dec 5;145(11):839–44. 0000605-200612050-00141
    1. Institute of Medicine, Committee on Quality of Health Care in America, authors. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press; 2001.
    1. Institute of Medicine, Committee on the National Quality Report on Health Care Delivery, authors. Envisioning the National Health Care Quality Report. Washington, DC: National Academies Press; 2001.
    1. Institute of Medicine, Committee on Communication for Behavior Change in the 21st Century, authors. Speaking of Health: Assessing Health Communication Strategies for Diverse Populations. Washington, DC: National Academies Press; 2002.
    1. Institute of Medicine, authors. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. Washington, DC: National Academies Press; 2002.
    1. Muñoz Ricardo F, Lenert Leslie L, Delucchi Kevin, Stoddard Jacqueline, Perez John E, Penilla Carlos, Pérez-Stable Eliseo J. Toward evidence-based Internet interventions: A Spanish/English Web site for international smoking cessation trials. Nicotine Tob Res. 2006 Feb;8(1):77–87. doi: 10.1080/14622200500431940.X446716T2418L625
    1. Byrne BM, Watkins D. The issue of measurement invariance revisited. J Cross Cult Psychol. 2003;34(2):155–175. doi: 10.1177/0022022102250225.
    1. Myers MB, Calantone RJ, Page TJ Jr, Taylor CR. An application of multiple-group causal models in assessing cross-cultural measurement equivalence. J Int Mark. 2000;8(4):108–121. doi: 10.1509/jimk.8.4.108.19790.
    1. Ramírez Mildred, Ford Marvella E, Stewart Anita L, Teresi Jeanne A. Measurement issues in health disparities research. Health Serv Res. 2005 Oct;40(5 Pt 2):1640–57. doi: 10.1111/j.1475-6773.2005.00450.x. HESR450
    1. Boeckner Linda S, Pullen Carol H, Walker Susan Noble, Abbott Gerald W Abbott, Block Torin. Use and reliability of the World Wide Web version of the Block Health Habits and History Questionnaire with older rural women. J Nutr Educ Behav. 2002;34 Suppl 1(Suppl 1):S20–4. doi: 10.1016/S1499-4046(06)60307-2.
    1. Bressani RV, Downs AC. Youth independent living assessment: testing the equivalence of Web and paper/pencil versions of the Ansell-Casey Life Skills Assessment. Comput Human Behav. 2002;18(4):453. doi: 10.1016/S0747-5632(01)00053-X.
    1. Ritter Philip, Lorig Kate, Laurent Diana, Matthews Katy. Internet versus mailed questionnaires: a randomized comparison. J Med Internet Res. 2004 Sep 15;6(3):e29. doi: 10.2196/jmir.6.3.e29. v6e29
    1. Vallejo Miguel A, Jordán Carlos M, Díaz Marta I, Comeche María I, Ortega José. Psychological assessment via the internet: a reliability and validity study of online (vs paper-and-pencil) versions of the General Health Questionnaire-28 (GHQ-28) and the Symptoms Check-List-90-Revised (SCL-90-R) J Med Internet Res. 2007;9(1):e2. doi: 10.2196/jmir.9.1.e2. v9i1e2
    1. Graham Amanda L, Papandonatos George D, Bock Beth C, Cobb Nathan K, Baskin-Sommers Arielle, Niaura Raymond, Abrams David B. Internet- vs. telephone-administered questionnaires in a randomized trial of smoking cessation. Nicotine Tob Res. 2006 Dec;8 Suppl 1(1):S49–57. doi: 10.1080/14622200601045367.
    1. American Psychological Association (APA), authors Ethical principles of psychologists and code of conduct. Am Psychol. 2002;57(12):1060–1073. doi: 10.1037/0003-066X.57.12.1060.
    1. Okazaki S, Sue S. In: Kazdin AE, editor. Methodological Issues and Strategies in Clinical Research. 2nd edition. Washington, DC: American Psychological Association; 1998. Methodological issues in assessment research with ethnic minorities; pp. 263–281.
    1. Office of Management and Budget, authors. Federal Register Notice. Revisions to the Standards for the Classification of Federal Data on Race and Ethnicity. 1997 Oct 30. OMB. [2008 Mar 3]. .
    1. Velicer W F, Diclemente C C, Rossi J S, Prochaska J O. Relapse situations and self-efficacy: an integrative model. Addict Behav. 1990;15(3):271–83. doi: 10.1016/0306-4603(90)90070-E.0306-4603(90)90070-E
    1. Coppotelli H C, Orleans C T. Partner support and other determinants of smoking cessation maintenance among women. J Consult Clin Psychol. 1985 Aug;53(4):455–60. doi: 10.1037/0022-006X.53.4.455.
    1. Mermelstein R, Lichtenstein E, McIntyre K. Partner support and relapse in smoking-cessation programs. J Consult Clin Psychol. 1983 Jun;51(3):465–6. doi: 10.1037/0022-006X.51.3.465.
    1. Cohen S, Lichtenstein E. Partner behaviors that support quitting smoking. J Consult Clin Psychol. 1990 Jun;58(3):304–9. doi: 10.1037/0022-006X.58.3.304.
    1. Roski J, Schmid L A, Lando H A. Long-term associations of helpful and harmful spousal behaviors with smoking cessation. Addict Behav. 1996;21(2):173–85. doi: 10.1016/0306-4603(95)00047-X.0306-4603(95)00047-X
    1. McMahon S D, Jason L A. Social support in a worksite smoking intervention. A test of theoretical models. Behav Modif. 2000 Apr;24(2):184–201. doi: 10.1177/0145445500242002.
    1. Gruder C L, Mermelstein R J, Kirkendol S, Hedeker D, Wong S C, Schreckengost J, Warnecke R B, Burzette R, Miller T Q. Effects of social support and relapse prevention training as adjuncts to a televised smoking-cessation intervention. J Consult Clin Psychol. 1993 Feb;61(1):113–20. doi: 10.1037/0022-006X.61.1.113.
    1. Cohen S, Lichtenstein E, Prochaska J O, Rossi J S, Gritz E R, Carr C R, Orleans C T, Schoenbach V J, Biener L, Abrams D. Debunking myths about self-quitting. Evidence from 10 prospective studies of persons who attempt to quit smoking by themselves. Am Psychol. 1989 Nov;44(11):1355–65. doi: 10.1037/0003-066X.44.11.1355.
    1. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. 1983 Dec;24(4):385–96. doi: 10.2307/2136404.
    1. Cohen S, Williamson GM. In: Spacapan S, Oskamp S, editors. The Social Psychology of Health. Newbury Park, CA: Sage; 1988. Perceived stress in a probability sample of the United States; pp. 31–67.
    1. Andresen E M, Malmgren J A, Carter W B, Patrick D L. Screening for depression in well older adults: evaluation of a short form of the CES-D (Center for Epidemiologic Studies Depression Scale) Am J Prev Med. 1994;10(2):77–84.
    1. Anda R F, Williamson D F, Escobedo L G, Mast E E, Giovino G A, Remington P L. Depression and the dynamics of smoking. A national perspective. JAMA. 1990 Sep 26;264(12):1541–5. doi: 10.1001/jama.264.12.1541.
    1. Niaura R, Britt D M, Borrelli B, Shadel W G, Abrams D B, Goldstein M G. History and symptoms of depression among smokers during a self-initiated quit attempt. Nicotine Tob Res. 1999 Sep;1(3):251–7. doi: 10.1080/14622299050011371.
    1. Pomerleau Cynthia S, Zucker Alyssa N, Stewart Abigail J. Patterns of depressive symptomatology in women smokers, ex-smokers, and never-smokers. Addict Behav. 2003 Apr;28(3):575–82. doi: 10.1016/S0306-4603(01)00257-X.S030646030100257X
    1. Lerman C, Audrain J, Orleans C T, Boyd R, Gold K, Main D, Caporaso N. Investigation of mechanisms linking depressed mood to nicotine dependence. Addict Behav. 1996;21(1):9–19. doi: 10.1016/0306-4603(95)00032-1.0306460395000321
    1. Zhu Shu-Hong, Valbø Annelill. Depression and smoking during pregnancy. Addict Behav. 2002;27(4):649–58. doi: 10.1016/S0306-4603(01)00199-X.
    1. Munafò Marcus R, Hitsman Brian, Rende Richard, Metcalfe Chris, Niaura Raymond. Effects of progression to cigarette smoking on depressed mood in adolescents: evidence from the National Longitudinal Study of Adolescent Health. Addiction. 2008 Jan;103(1):162–71. doi: 10.1111/j.1360-0443.2007.02052.x.ADD2052
    1. Benjet C, Wagner F A, Borges G G, Medina-Mora M E. The relationship of tobacco smoking with depressive symptomatology in the Third Mexican National Addictions Survey. Psychol Med. 2004 Jul;34(5):881–8. doi: 10.1017/S0033291703001600.
    1. McClure Jennifer B, Wetter David W, de Moor Carl, Cinciripini Paul M, Gritz Ellen R. The relation between alcohol consumption and smoking abstinence: results from the Working Well Trial. Addict Behav. 2002;27(3):367–79. doi: 10.1016/S0306-4603(01)00177-0.
    1. Shiffman S, Hickcox M, Paty J A, Gnys M, Kassel J D, Richards T J. Progression from a smoking lapse to relapse: prediction from abstinence violation effects, nicotine dependence, and lapse characteristics. J Consult Clin Psychol. 1996 Oct;64(5):993–1002. doi: 10.1037/0022-006X.64.5.993.
    1. Centers for Disease Control and Prevention, authors. 2002 Behavioral Risk Factor Surveillance System. 2001 Dec 27. [2008 Mar 3]. .
    1. Brown R L, Leonard T, Saunders L A, Papasouliotis O. A two-item conjoint screen for alcohol and other drug problems. J Am Board Fam Pract. 2001;14(2):95–106.
    1. Ware J E, Sherbourne C D. The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med Care. 1992 Jun;30(6):473–83. doi: 10.1097/00005650-199206000-00002.
    1. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–428. doi: 10.1037/0033-2909.86.2.420.
    1. SAS/STAT Software, authors. Changes and Enhancements. Cary, NC: SAS Institute, Inc; 2001.
    1. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd edition. Hillsdale, NJ: Lawrence Erlbaum; 1988.
    1. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37–46. doi: 10.1177/001316446002000104.
    1. Cook RJ. Kappa and its dependence on marginal rates. In: Armitage P, Colton T, editors. The Encyclopedia of Biostatistics. New York: Wiley; 1998. pp. 2166–2168.
    1. Landis J R, Koch G G. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159–74. doi: 10.2307/2529310.
    1. Lee JJ, Tu ZN. A better confidence for kappa on measuring agreement between two raters with binary outcomes. J Comput Graph Stat. 1994;3(3):301–321. doi: 10.2307/1390914.
    1. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychol Bull. 1969;72(5):323–327. doi: 10.1037/h0028106.
    1. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213–220. doi: 10.1037/h0026256.
    1. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297–333. doi: 10.1007/BF02310555.
    1. van Zyl J M, Neudecker H, Nel D G. On the distribution of the maximum likelihood estimator of Cronbach's alpha. Psychometrika. 2000;65(3):271–280. doi: 10.1007/BF02296146.
    1. DeNavas-Walt C, Proctor BD, Lee CH. US Census Bureau. Income, Poverty, and Health Insurance Coverage in the United States: 2006 Aug. . [2008 Mar 3]. .

Source: PubMed

赞助者和合作者

医疗条件

药物干预

3
订阅