Reliability and validity issues related to interactive tailored patient assessments: a case study

Cornelia M Ruland, Suzanne Bakken, Jo Røislien, Cornelia M Ruland, Suzanne Bakken, Jo Røislien

Abstract

Recently there has been a proliferation of interactive tailored patient assessment (ITPA) tools. However, evidence of the reliability and validity of these instruments is often missing, which makes their value in research studies questionable. Because several of the common methods to evaluate instrument reliability and validity are not applicable to interactive tailored patient assessments, informatics researchers may benefit from some guidance on which methods of reliability and validity assessment they can appropriately use. This paper describes the main differences between interactive tailored patient assessments and assessment instruments based on psychometric, or classical test, theory; it summarizes the measurement techniques normally used to ascertain the validity and reliability of assessment instruments based on psychometric theory; it discusses which methods are appropriate for interactive tailored patient assessments and which are not; and finally, it illustrates the application of some of the feasible techniques with a case study that describes how the reliability and validity of the tailored symptom assessment instrument called Choice were evaluated.

Conflict of interest statement

None declared.

References

    1. White T M, Hauan M J. The capture and use of detailed process information in the Dialogix system for structured web-based interactions. Proc AMIA Symp. 2001:761–5.D010001516
    1. Wittchen H U. Reliability and validity studies of the WHO--Composite International Diagnostic Interview (CIDI): a critical review. J Psychiatr Res. 1994;28(1):57–84. doi: 10.1016/0022-3956(94)90036-1.0022-3956(94)90036-1
    1. Norbeck J S. What constitutes a publishable report of instrument development? Nurs Res. 1985;34(6):380–2. doi: 10.1097/00006199-198511000-00022.
    1. Ruland C M. Decision support for patient preference-based care planning: effects on nursing care and patient outcomes. J Am Med Inform Assoc. 1999;6(4):304–12.
    1. Ruland Cornelia M. Handheld technology to improve patient care: evaluating a support system for preference-based care planning at the bedside. J Am Med Inform Assoc. 2002;9(2):192–201. doi: 10.1197/jamia.M0891.
    1. Ruland Cornelia M, White Thomas, Stevens Marguerite, Fanciullo Gilbert, Khilani Samir M. Effects of a computerized system to support shared decision making in symptom management of cancer patients: preliminary results. J Am Med Inform Assoc. 2003;10(6):573–9. doi: 10.1197/jamia.M1365. M1365
    1. DeVellis RF. Scale Development: Theory and Applications. 2nd edition. Thousand Oaks, CA: Sage Publications; 2003.
    1. Streiner David L. Being inconsistent about consistency: when coefficient alpha does and doesn't matter. J Pers Assess. 2003 Jun;80(3):217–22. doi: 10.1207/S15327752JPA8003_01.
    1. Bollen K, Lennox R. Conventional wisdom on measurement: a structural equation perspective. Psychol Bull. 1991;110(2):305–314. doi: 10.1037/0033-2909.110.2.305.
    1. Kreuter M W, Bull F C, Clark E M, Oswald D L. Understanding how people process health information: a comparison of tailored and nontailored weight-loss materials. Health Psychol. 1999 Sep;18(5):487–94. doi: 10.1037/0278-6133.18.5.487.
    1. Radloff LS. The CES-D Scale: a self-report depression scale for research in the general population. Appl Psychol Meas. 1977;1(3):385–401. doi: 10.1177/014662167700100306.
    1. Andersen Roar, Ruland Cornelia, Slaughter Laura, Andersen Trine, Jacobsen Wenche. Clustering techniques for organizing cancer-related concepts into meaningful groups for patients. AMIA Annu Symp Proc. 2005:882.58078
    1. Carmines EG, Zeller RA. Reliability and Validity Assessment. Newbury Park, CA: Sage Publications; 1979.
    1. Pedhazur EJ, Schmelkin LP. Measurement, Design, and Analysis: An Integrated Approach. Hillsdale, NJ: Lawrence Erlbaum Associates; 1991.
    1. Nunnally JC, Bernstein IH. Psychometric Theory. 3rd edition. New York: McGraw-Hill; 1994.
    1. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297–334. doi: 10.1007/BF02310555.
    1. Ruland Cornelia, Røslien Jo, Bakken Suzanne, Kristiansen Jørn. Comparing tailored computerized symptom assessments to interviews and questionnaires. AMIA Annu Symp Proc. 2006:1081. 86197
    1. Tinsley HEA, authors. Tinsley DJ. Uses of factor analysis in counselling psychological research. J Couns Psychol. 1987;34(4):414–424. doi: 10.1037/0022-0167.34.4.414.
    1. Comrey A L. Factor-analytic methods of scale development in personality and clinical psychology. J Consult Clin Psychol. 1988 Oct;56(5):754–61. doi: 10.1037/0022-006X.56.5.754.
    1. Campbell D T, Fiske D W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull. 1959 Mar;56(2):81–105. doi: 10.1037/h0046016.
    1. Miaskowski Christine. Gender differences in pain, fatigue, and depression in patients with cancer. J Natl Cancer Inst Monogr. 2004;(32):139–43. doi: 10.1093/jncimonographs/lgh024.2004/32/139
    1. Ware J E, Gandek B. Overview of the SF-36 Health Survey and the International Quality of Life Assessment (IQOLA) Project. J Clin Epidemiol. 1998 Nov;51(11):903–12. doi: 10.1016/S0895-4356(98)00081-X.S0895-4356(98)00081-X

Source: PubMed

Sponsoren und Mitarbeiter

Krankheiten

Drogeninterventionen

3
Abonnieren