A critical analysis of test-retest reliability in instrument validation studies of cancer patients under palliative care: a systematic review

Carlos Eduardo Paiva, Eliane Marçon Barroso, Estela Cristina Carneseca, Cristiano de Pádua Souza, Felipe Thomé Dos Santos, Rossana Verónica Mendoza López, Sakamoto Bianca Ribeiro Paiva, Carlos Eduardo Paiva, Eliane Marçon Barroso, Estela Cristina Carneseca, Cristiano de Pádua Souza, Felipe Thomé Dos Santos, Rossana Verónica Mendoza López, Sakamoto Bianca Ribeiro Paiva

Abstract

Background: Patient-reported outcome validation needs to achieve validity and reliability standards. Among reliability analysis parameters, test-retest reliability is an important psychometric property. Retested patients must be in a clinically stable condition. This is particularly problematic in palliative care (PC) settings because advanced cancer patients are prone to a faster rate of clinical deterioration. The aim of this study was to evaluate the methods by which multi-symptom and health-related qualities of life (HRQoL) based on patient-reported outcomes (PROs) have been validated in oncological PC settings with regards to test-retest reliability.

Methods: A systematic search of PubMed (1966 to June 2013), EMBASE (1980 to June 2013), PsychInfo (1806 to June 2013), CINAHL (1980 to June 2013), and SCIELO (1998 to June 2013), and specific PRO databases was performed. Studies were included if they described a set of validation studies. Studies were included if they described a set of validation studies for an instrument developed to measure multi-symptom or multidimensional HRQoL in advanced cancer patients under PC. The COSMIN checklist was used to rate the methodological quality of the study designs.

Results: We identified 89 validation studies from 746 potentially relevant articles. From those 89 articles, 31 measured test-retest reliability and were included in this review. Upon critical analysis of the overall quality of the criteria used to determine the test-retest reliability, 6 (19.4%), 17 (54.8%), and 8 (25.8%) of these articles were rated as good, fair, or poor, respectively, and no article was classified as excellent. Multi-symptom instruments were retested over a shortened interval when compared to the HRQoL instruments (median values 24 hours and 168 hours, respectively; p = 0.001). Validation studies that included objective confirmation of clinical stability in their design yielded better results for the test-retest analysis with regard to both pain and global HRQoL scores (p < 0.05). The quality of the statistical analysis and its description were of great concern.

Conclusion: Test-retest reliability has been infrequently and poorly evaluated. The confirmation of clinical stability was an important factor in our analysis, and we suggest that special attention be focused on clinical stability when designing a PRO validation study that includes advanced cancer patients under PC.

Figures

Figure 1
Figure 1
PRISMA flow diagram for search strategy.
Figure 2
Figure 2
Quality criteria of the included studies according to the COSMIN checklist.

References

    1. Ferrell B, Paice J, Koczywas M. New standards and implications for improving the quality of supportive oncology practice. J Clin Oncol. 2008;26(23):3824–3831. doi: 10.1200/JCO.2007.15.7552.
    1. Frost MH, Reeve BB, Liepa AM, Stauffer JW, Hays RD, Mayo FDAP-ROCMG. What is sufficient evidence for the reliability and validity of patient-reported outcome measures? Value Health. 2007;10(Suppl 2):S94–S105.
    1. Aaronson N, Alonso J, Burnam A, Lohr KN, Patrick DL, Perrin E, Stein RE. Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res. 2002;11(3):193–205. doi: 10.1023/A:1015291021312.
    1. Richardson LA, Jones GW. A review of the reliability and validity of the Edmonton Symptom Assessment System. Curr Oncol. 2009;16(1):55.
    1. The Patient-Reported Outcome and Quality of Life Instrument Database (PROQOLID)
    1. Australian Centre on Quality of Life (ACQOL) .
    1. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HC. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–549. doi: 10.1007/s11136-010-9606-8.
    1. Terwee CB, Mokkink LB, Knol DL, Ostelo RW, Bouter LM, de Vet HC. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Qual Life Res. 2012;21(4):651–657. doi: 10.1007/s11136-011-9960-1.
    1. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, Bouter LM, de Vet HC. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42. doi: 10.1016/j.jclinepi.2006.03.012.
    1. Axelsson B, Sjoden PO. Assessment of quality of life in palliative care–psychometric properties of a short questionnaire. Acta Oncol. 1999;38(2):229–237. doi: 10.1080/028418699431663.
    1. Cohen SR, Mount BM. Living with cancer: “good” days and “bad” days–what produces them? Can the McGill quality of life questionnaire distinguish between them? Cancer. 2000;89(8):1854–1865. doi: 10.1002/1097-0142(20001015)89:8<1854::AID-CNCR28>;2-C.
    1. Llobera J, Esteva M, Benito E, Terrasa J, Rifa J, Pons O, Maya A. Quality of life for oncology patients during the terminal period. Validation of the HRCA-QL index. Support Care Cancer. 2003;11(5):294–303.
    1. Mystakidou K, Tsilika E, Kouloulias V, Parpa E, Katsouda E, Kouvaris J, Vlahos L. The “Palliative Care Quality of Life Instrument (PQLI)” in terminal cancer patients. Health Qual Life Outcomes. 2004;2:8. doi: 10.1186/1477-7525-2-8.
    1. Steinhauser KE, Clipp EC, Bosworth HB, McNeilly M, Christakis NA, Voils CI, Tulsky JA. Measuring quality of life at the end of life: validation of the QUAL-E. Palliat Support Care. 2004;2(1):3–14.
    1. Lopes Ferreira P, Pinto Barros A. Measuring quality of life in palliative care. Acta Med Port. 2008;21(2):111–124.
    1. Suarez-del-Real Y, Allende-Perez S, Alferez-Mancera A, Rodriguez RB, Jimenez-Toxtle S, Mohar A, Onate-Ocana LF. Validation of the Mexican-Spanish version of the EORTC QLQ-C15-PAL questionnaire for the evaluation of health-related quality of life in patients on palliative care. Psychooncology. 2011;20(8):889–896. doi: 10.1002/pon.1801.
    1. Leppert W, Majkowicz M. Validation of the Polish version of the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire - Core 15 - Palliative Care in patients with advanced cancer. Palliat Med. 2013;27(5):470–477. doi: 10.1177/0269216312458823.
    1. Kim KU, Yoon SJ, Lee JL, Ahn HS, Park HJ, Lee SI, Jo MW. Validation of the Korean version of the McMaster Quality of Life Scale in terminal cancer patients. J Palliat Care. 2006;22(1):40–45.
    1. Kim SH, Choi YS, Lee J, Oh SC, Yeom CH, Lee MA, Kim DG, Moon Do H, Kim DY, Koh SJ. Reliability and validity of the Hospice Quality of Life Scale for Korean cancer patients. J Pain Symptom Manage. 2009;37(2):156–167. doi: 10.1016/j.jpainsymman.2008.02.008.
    1. Serra-Prat M, Nabal M, Santacruz V, Picaza JM, Trelis J. Grupo Catalan de Estudio de la Efectividad de los Cuidados P. Validation of the Spanish version of the palliative care outcome scale. Med Clin. 2004;123(11):406–412.
    1. Eisenchlas JH, Harding R, Daud ML, Perez M, De Simone GG, Higginson IJ. Use of the palliative outcome scale in Argentina: a cross-cultural adaptation and validation study. J Pain Symptom Manage. 2008;35(2):188–202. doi: 10.1016/j.jpainsymman.2007.02.045.
    1. Guo H, Fine PG, Mendoza TR, Cleeland CS. A preliminary study of the utility of the brief hospice inventory. J Pain Symptom Manage. 2001;22(2):637–648. doi: 10.1016/S0885-3924(01)00296-2.
    1. Lo RS, Woo J, Zhoc KC, Li CY, Yeo W, Johnson P, Mak Y, Lee J. Cross-cultural validation of the McGill Quality of Life questionnaire in Hong Kong Chinese. Palliat Med. 2001;15(5):387–397. doi: 10.1191/026921601680419438.
    1. Shahidi J, Khodabakhshi R, Gohari MR, Yahyazadeh H, Shahidi N. McGill Quality of Life Questionnaire: reliability and validity of the Persian version in Iranian patients with advanced cancer. J Palliat Med. 2008;11(4):621–626. doi: 10.1089/jpm.2007.0173.
    1. Harding R, Selman L, Agupio G, Dinat N, Downing J, Gwyther L, Mashao T, Mmoledi K, Moll T, Sebuyira LM. et al.Validation of a core outcome measure for palliative care in Africa: the APCA African Palliative Outcome Scale. Health Qual Life Outcomes. 2010;8:10. doi: 10.1186/1477-7525-8-10.
    1. Hearn J, Higginson IJ. Development and validation of a core outcome measure for palliative care: the palliative care outcome scale. Palliative Care Core Audit Project Advisory Group. Qual Health Care. 1999;8(4):219–227. doi: 10.1136/qshc.8.4.219.
    1. Wilson KG, Graham ID, Viola RA, Chater S, de Faye BJ, Weaver LA, Lachance JA. Structured interview assessment of symptoms and concerns in palliative care. Can J Psychiatry. 2004;49(6):350–358.
    1. Agra Y, Badia X. Spanish version of the Rotterdam Symptom Check List: cross-cultural adaptation and preliminary validity in a sample of terminal cancer patients. Psychooncology. 1998;7(3):229–239. doi: 10.1002/(SICI)1099-1611(199805/06)7:3<229::AID-PON302>;2-R.
    1. Chang VT, Hwang SS, Feuerman M. Validation of the Edmonton Symptom Assessment Scale. Cancer. 2000;88(9):2164–2171. doi: 10.1002/(SICI)1097-0142(20000501)88:9<2164::AID-CNCR24>;2-5.
    1. Stiel S, Matthes ME, Bertram L, Ostgathe C, Elsner F, Radbruch L. Validation of the new version of the minimal documentation system (MIDOS) for patients in palliative care : the German version of the edmonton symptom assessment scale (ESAS) Schmerz. 2010;24(6):596–604. doi: 10.1007/s00482-010-0972-5.
    1. Moro C, Brunelli C, Miccinesi G, Fallai M, Morino P, Piazza M, Labianca R, Ripamonti C. Edmonton symptom assessment scale: Italian validation in two palliative care settings. Support Care Cancer. 2006;14(1):30–37. doi: 10.1007/s00520-005-0834-3.
    1. Mystakidou K, Cleeland C, Tsilika E, Katsouda E, Primikiri A, Parpa E, Vlahos L, Mendoza T. Greek M.D. Anderson symptom inventory: validation and utility in cancer patients. Oncology. 2004;67(3–4):203–210.
    1. Pautex S, Berger A, Chatelain C, Herrmann F, Zulian GB. Symptom assessment in elderly cancer patients receiving palliative care. Crit Rev Oncol Hematol. 2003;47(3):281–286. doi: 10.1016/S1040-8428(03)00043-X.
    1. Stiel S, Pollok A, Elsner F, Lindena G, Ostgathe C, Nauck F, Radbruch L. Validation of the symptom and problem checklist of the German Hospice and Palliative Care Evaluation (HOPE) J Pain Symptom Manage. 2012;43(3):593–605. doi: 10.1016/j.jpainsymman.2011.04.021.
    1. Carvajal A, Centeno C, Watson R, Bruera E. A comprehensive study of psychometric properties of the Edmonton Symptom Assessment System (ESAS) in Spanish advanced cancer patients. Eur J Cancer. 2011;47(12):1863–1872. doi: 10.1016/j.ejca.2011.03.027.
    1. Radbruch L, Sabatowski R, Loick G, Jonen-Thielemann I, Elsner F, Hormann E. MIDOS–validation of a minimal documentation system for palliative medicine. Schmerz. 2000;14(4):231–239. doi: 10.1007/s004820070028.
    1. Kwon JH, Nam SH, Koh S, Hong YS, Lee KH, Shin SW, Hui D, Park KW, Yoon SY, Won JY. et al.Validation of edmonton symptom assessment system in korean patients with cancer. J Pain Symptom Manage. 2013;46(6):947–956. doi: 10.1016/j.jpainsymman.2013.01.012.
    1. Aoun SM, Monterosso L, Kristjanson LJ, McConigley R. Measuring symptom distress in palliative care: psychometric properties of the Symptom Assessment Scale (SAS) J Palliat Med. 2011;14(3):315–321. doi: 10.1089/jpm.2010.0412.
    1. Pereira F, Santos C. Estudo de adaptação cultural e validação da Functional Assessment of Cancer Therapy-General em cuidados paliativos. Rev Enferm Referênci. 2011;5:10.
    1. Sterkenburg CA, King B, Woodward CA. A reliability and validity study of the McMaster Quality of Life Scale (MQLS) for a palliative population. J Palliat Care. 1996;12(1):18–25.
    1. Fanelli D. Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PLoS One. 2010;5(4):e10271. doi: 10.1371/journal.pone.0010271.
    1. Guo Y, Logan HL, Glueck DH, Muller KE. Selecting a sample size for studies with repeated measures. BMC Med Res Methodol. 2013;13:100. doi: 10.1186/1471-2288-13-100.
    1. Hopkins WG. Measures of reliability in sports medicine and science. Sports Med. 2000;30(1):1–15. doi: 10.2165/00007256-200030010-00001.
    1. Atkinson G, Nevill A. Typical error versus limits of agreement. Sports Med. 2000;30(5):375–381.
    1. Fayers P, Machin D. Quality of Life. Assessment, Analysis and Interpretation. 2. Chichester: John Wiley; 2007.
    1. The COSMIN checklist manual. .
    1. Streiner DL, Norman GR. Health Measurement Scales. A Practical Guide to Their Development and use. 4. New York: Oxford University Press; 2008.
    1. Jim HS, Small B, Faul LA, Franzen J, Apte S, Jacobsen PB. Fatigue, depression, sleep, and activity during chemotherapy: daily and intraday variation and relationships among symptom changes. Ann Behav Med. 2011;42(3):321–333. doi: 10.1007/s12160-011-9294-9.
    1. Dimsdale JE, Ancoli-Israel S, Ayalon L, Elsmore TF, Gruen W. Taking fatigue seriously, II: variability in fatigue levels in cancer patients. Psychosomatics. 2007;48(3):247–252. doi: 10.1176/appi.psy.48.3.247.
    1. Vrijman C, Homan ML, Limpens J, van der Veen W, Wolkerstorfer A, Terwee CB, Spuls PI. Measurement properties of outcome measures for Vitiligo: a systematic review. Arch Dermatol. 2012;148(11):1302–1309. doi: 10.1001/archdermatol.2012.3065.
    1. Abma FI, van der Klink JJ, Terwee CB, Amick BC 3rd, Bultmann U. Evaluation of the measurement properties of self-reported health-related work-functioning instruments among workers with common mental disorders. Scand J Work Environ Health. 2012;38(1):5–18. doi: 10.5271/sjweh.3190.
    1. Reimers AK, Mess F, Bucksch J, Jekauc D, Woll A. Systematic review on measurement properties of questionnaires assessing the neighbourhood environment in the context of youth physical activity behaviour. BMC Public Health. 2013;13:461. doi: 10.1186/1471-2458-13-461.
    1. Werner MU, Petersen MA, Bischoff JM. Test-retest studies in quantitative sensory testing: a critical review. Acta Anaestheriol Scand. 2013;57(8):957–963. doi: 10.1111/aas.12150.
    1. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–428.

Source: PubMed

Sponsorer och medarbetare

Medicinska tillstånd

Läkemedelsinterventioner

3
Prenumerera