The Visual Analogue Scale for Rating, Ranking and Paired-Comparison (VAS-RRP): A new technique for psychological measurement
Yao-Ting Sung, Jeng-Shin Wu, Yao-Ting Sung, Jeng-Shin Wu
Abstract
Traditionally, the visual analogue scale (VAS) has been proposed to overcome the limitations of ordinal measures from Likert-type scales. However, the function of VASs to overcome the limitations of response styles to Likert-type scales has not yet been addressed. Previous research using ranking and paired comparisons to compensate for the response styles of Likert-type scales has suffered from limitations, such as that the total score of ipsative measures is a constant that cannot be analyzed by means of many common statistical techniques. In this study we propose a new scale, called the Visual Analogue Scale for Rating, Ranking, and Paired-Comparison (VAS-RRP), which can be used to collect rating, ranking, and paired-comparison data simultaneously, while avoiding the limitations of each of these data collection methods. The characteristics, use, and analytic method of VAS-RRPs, as well as how they overcome the disadvantages of Likert-type scales, ranking, and VASs, are discussed. On the basis of analyses of simulated and empirical data, this study showed that VAS-RRPs improved reliability, response style bias, and parameter recovery. Finally, we have also designed a VAS-RRP Generator for researchers' construction and administration of their own VAS-RRPs.
Keywords: CTCU model; Likert-type scale; Multi-item VAS; Paired comparison; Ranking; VAS-RRP.
Figures
References
- Albaum G. The Likert scale revisited: An alternate version. Journal of the Market Research Society. 1997;39:331–348. doi: 10.1177/147078539703900202.
- Allen IE, Seaman CA. Likert scales and data analyses. Quality Progress. 2007;40:64–65.
- Alwin DF. Information transmission in the survey interview: Number of response categories and the reliability of attitude measurement. In: Marsden PV, editor. Sociological methodology. Cambridge, MA: Blackwell; 1992. pp. 83–118.
- Babakus E, Ferguson CE, Jöreskog KG. The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research. 1987;37:72–141.
- Bagozzi RP, Yi Y. On the evaluation of structural equation models. Journal of the Academy of Marketing Science. 1988;16:74–94. doi: 10.1007/BF02723327.
- Baron H. Strengths and limitations of ipsative measurement. Journal of Occupational and Organizational Psychology. 1996;69:49–56. doi: 10.1111/j.2044-8325.1996.tb00599.x.
- Bollen KA. Structural equation models. New York, NY: Wiley; 1989.
- Bollen KA, Barb KH. Pearson’s r and coarsely categorized measures. American Sociological Review. 1981;46:232–239. doi: 10.2307/2094981.
- Brady HE. Factor and ideal point analysis for interpersonally incomparable data. Psychometrika. 1989;54:181–202. doi: 10.1007/BF02294514.
- Brown A. Item response models for forced-choice questionnaires: A common framework. Psychometrika. 2014;81:1–26.
- Brown A, Maydeu-Olivares A. Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement. 2011;71:460–502. doi: 10.1177/0013164410375112.
- Brown A, Maydeu-Olivares A. Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods. 2012;44:1135–1147. doi: 10.3758/s13428-012-0217-x.
- Brown A, Maydeu-Olivares A. How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods. 2013;18:36–52. doi: 10.1037/a0030641.
- Carmines EG, McIver JP. Analyzing models with unobserved variables: Analysis of covariance structure. In: Bohrnstedt GW, Borgatta EF, editors. Social measurement: Current issues. Beverly Hills, CA: Sage; 1981. pp. 65–115.
- Chan W, Bentler PM. Covariance structure analysis of ordinal ipsative data. Psychometrika. 1998;63:369–399. doi: 10.1007/BF02294861.
- Cheung MWL, Chan W. Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis. Structural Equation Modeling. 2002;9:55–77. doi: 10.1207/S15328007SEM0901_4.
- Chimi, C. J., & Russell, D. L. (2009, November). The Likert-type scale: A proposal for improvement using quasi-continuous variables. Paper presented at the ISECON 2009, Washington, DC.
- Chiu CK, Alliger GM. A proposed method to combine ranking and graphic rating in performance appraisal: The quantitative ranking scale. Educational and Psychological Measurement. 1990;50:493–503. doi: 10.1177/0013164490503003.
- Clemans, W. V. (1966). An analytical and empirical examination of some properties of ipsative measures (Psychometric Monograph No. 14). Richmond, VA: Psychometric Society. Retrieved from
- Cook C, Heath F, Thompson R, Thompson B. Score reliability in web- or internet-based surveys: Unnumbered graphic rating scales versus Likert-type scales. Educational and Psychological Measurement. 2001;61:697–706. doi: 10.1177/00131640121971356.
- Costa PT, McCrae RR. Professional manual: Revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI) Odessa, FL: Psychological Assessment Resources; 1992.
- Couper MP, Tourangeau R, Conrad FG, Singer E. Evaluating the effectiveness of visual analog scales: A Web experiment. Social Science Computer Review. 2006;24:227–245. doi: 10.1177/0894439305281503.
- Cox EP. The optimal number of response alternatives for a scale: A review. Journal of Marketing Research. 1980;17:407–422. doi: 10.2307/3150495.
- Cunningham WH, Cunningham ICM, Green RT. The ipsative process to reduce response set bias. Public Opinion Quarterly. 1977;41:379–384. doi: 10.1086/268394.
- Diedenhofen B, Musch J. Cocron: A web interface and R package for the statistical comparison of Cronbach’s alpha coefficients. International Journal of Internet Science. 2016;11:51–60.
- Dunlap WP, Cornwell JM. Factor analysis of ipsative measures. Multivariate Behavioral Research. 1994;29:115–126. doi: 10.1207/s15327906mbr2901_4.
- Ferrando PJ. A kernel density analysis of continuous typical-response scales. Educational and Psychological Measurement. 2003;63:809–824. doi: 10.1177/0013164403251323.
- Flynn D, van Schaik P, van Wersch A. A comparison of multi-item Likert and visual analogue scales for the assessment of transactionally defined coping function. European Journal of Psychological Assessment. 2004;20:49–58. doi: 10.1027/1015-5759.20.1.49.
- Funke F, Reips U-D. Why semantic differentials in Web-based research should be made from visual analogue scales and not from 5-point scales. Field Methods. 2012;24:310–327. doi: 10.1177/1525822X12444061.
- Gay, E. G., Weiss, D. J., Hendel, D. D., Dawis, R. V., & Lofquist, L. H. (1971). Manual for the Minnesota importance questionnaire (No. 54). Work Adjustment Project, University of Minnesota.
- Goffin RD, Olson JM. Is it all relative? Comparative judgments and the possible improvement of self-ratings and ratings of others. Perspectives on Psychological Science. 2011;6:48–60. doi: 10.1177/1745691610393521.
- Gordon LV. Gordon personal profile inventory (GPP-1): Manual. San Antonio, TX: Psychological Corporation; 1993.
- Granberg-Rademacker JS. An algorithm for converting ordinal scale measurement data to interval/ratio scale. Educational and Psychological Measurement. 2010;70:74–90. doi: 10.1177/0013164409344532.
- Greenleaf EA. Measuring extreme response style. Public Opinion Quarterly. 1992;56:328–351. doi: 10.1086/269326.
- Guyatt GH, Townsend M, Berman LB, Keller JL. A comparison of Likert and visual analogues scales for measuring change in function. Journal of Chronic Disability. 1987;40:1129–1133. doi: 10.1016/0021-9681(87)90080-4.
- Harwell MR, Gatti GG. Rescaling ordinal data to interval data in educational research. Review of Educational Research. 2001;71:105–131. doi: 10.3102/00346543071001105.
- Hicks LE. Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin. 1970;74:167–184. doi: 10.1037/h0029780.
- Holland JL. Making vocational choices: A theory of vocational personalities and work environments. Odessa, FL: Psychological Assessment Resources; 1997.
- Holyk GG. Context effect. In: Lavrakas PJ, editor. Encyclopedia of survey research methods. Thousand Oaks CA: Sage; 2008. p. 142.
- Hooper D, Coughlan J, Mullen MR. Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods. 2008;6:53–60.
- Jackson DJ, Alwin DF. The factor analysis of ipsative measures. Sociological Methods and Research. 1980;9:218–238. doi: 10.1177/004912418000900206.
- Jaeschke R, Singer J, Guyatt GH. A comparison of seven-point and visual analogue scales. Controlled Clinical Trials. 1990;11:43–51. doi: 10.1016/0197-2456(90)90031-V.
- Jamieson S. Likert scales: How to (ab)use them. Medical Education. 2004;38:1212–1218. doi: 10.1111/j.1365-2929.2004.02012.x.
- Kolb AY. The Kolb learning style inventory—Version 3.1:2005 technical specifications. Boston, MA: Hay Resources Direct; 2005.
- Krieg EF. Biases induced by coarse measurements scales. Educational and Psychological Measurement. 1999;59:749–766. doi: 10.1177/00131649921970125.
- Kuhlmann T, Dantlgraber M, Reips U-D. Investigating measurement equivalence of visual analogue scales and Likert-type scales in Internet-based personality questionnaires. Behavior Research Methods. 2017;49:2173–2181. doi: 10.3758/s13428-016-0850-x.
- Laming D. Human judgment: The eye of the beholder. London, UK: Thomson; 2004.
- Likert R. A technique for the measurement of attitudes. Archives of Psychology. 1932;140:5–55.
- Loo R. Confirmatory factor analyses of Kolb’s learning style inventory (LSI-1985) British Journal of Educational Psychology. 1999;69:213–219. doi: 10.1348/000709999157680.
- Marsh HW. Confirmatory factor analyses of multitrait–multimethod data: Many problems and a few solutions. Applied Psychological Measurement. 1989;13:335–361. doi: 10.1177/014662168901300402.
- Marsh HW, Bailey M. Confirmatory factor analyses of multitrait–multimethod data: A comparison of alternative models. Applied Psychological Measurement. 1991;15:47–70. doi: 10.1177/014662169101500106.
- Marsh HW, Grayson D. Latent variable models of multitrait–multimethod data. In: Hoyle RH, editor. Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage; 1995. pp. 177–198.
- McCloy R, Waugh G, Medsker G, Wall J, Rivkin D, Lewis P. Development of the O* NET computerized work importance profiler. Raleigh, NC: National Center for O* NET Development; 1999.
- McCloy R, Waugh G, Medsker G, Wall J, Rivkin D, Lewis P. Development of the O* NET paper-and pencil work importance locator. Raleigh, NC: National Center for O* NET Development; 1999.
- McKelvie SJ. Graphic rating scales: How many categories? British Journal of Psychology. 1978;69:185–202. doi: 10.1111/j.2044-8295.1978.tb01647.x.
- Meade AW. Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology. 2004;77:531–551. doi: 10.1348/0963179042596504.
- Munshi, J. (2014). A method for constructing Likert scales. Research report, Sonoma State University. Retrieved from
- Myles PS, Troedel S, Boquest M, Reeves M. The pain visual analog scale: Is it linear or nonlinear? Anesthesia and Analgesia. 1999;89:1517–1520.
- Nunnally JC. Psychometric theory. New York, NY: McGraw-Hill; 1967.
- Nyren O, Adami O, Bates S, Bergstrom R, Gustavsson S, Loof L, Sjoden PO. Self-rating of pain in non-ulcer dyspepsia. Journal of Clinical Gastroenterology. 1987;9:408–414. doi: 10.1097/00004836-198708000-00010.
- Tomás JM, Oliver A, Hontangas PM. Linear confirmatory models for MTMM matrices: The case of several indicators per trait–method combinations. In: Shohov SP, editor. Advances in psychology research. Huntington, NY: Nova Science; 2002. pp. 99–122.
- Paulhus DL. Control of social desirability in personality inventories: Principal-factor deletion. Journal of Research in Personality. 1981;15:383–388. doi: 10.1016/0092-6566(81)90035-0.
- Paulhus DL. Measures of personality and social psychological attitudes. In: Robinson JP, Shaver RP, editors. Measures of social psychological attitudes series. San Diego, CA: Academic; 1991. pp. 17–59.
- Preston CC, Colman AM. Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent. Acta Psychologica. 2000;104:1–15. doi: 10.1016/S0001-6918(99)00050-5.
- Price DD, McGrath PA, Rafii A, Buckingham B. The validation of visual analogue scales as ratio scale measures for chronic and experimental pain. Pain. 1983;17:45–56. doi: 10.1016/0304-3959(83)90126-4.
- Randall DM, Fernandes MF. The social desirability response bias in ethics research. Journal of Business Ethics. 1991;10:805–817. doi: 10.1007/BF00383696.
- Raykov T. Estimation of composite reliability for congeneric measures. Applied Psychological Measurement. 1997;21:173–184. doi: 10.1177/01466216970212006.
- Reips U-D, Funke F. Interval-level measurement with visual analogue scales in Internet-based research: VAS Generator. Behavior Research Methods. 2008;40:699–704. doi: 10.3758/BRM.40.3.699.
- Rounds JB, Miller TW, Dawis RV. Comparability of multiple rank order and paired comparison methods. Applied Psychological Measurement. 1978;2:415–422. doi: 10.1177/014662167800200316.
- Scherpenzeel AC, Saris WE. The validity and reliability of survey questions. Sociological Methods & Research. 1997;25:341–383. doi: 10.1177/0049124197025003004.
- Sheppard LD, Goffin RD, Lewis RJ, Olson J. The effect of target attractiveness and rating method on the accuracy of trait ratings. Journal of Personnel Psychology. 2011;10:24–33. doi: 10.1027/1866-5888/a000030.
- Spooren P, Mortelmans D, Thijssen P. “Content” versus “style”: Acquiescence in student evaluation of teaching? British Educational Research Journal. 2012;38:3–21. doi: 10.1080/01411926.2010.523453.
- Stevens SS. On the theory of scales of measurement. Science. 1946;103:677–680. doi: 10.1126/science.103.2684.677.
- Stevens SS. Mathematics, measurement, and psychophysics. In: Stevens SS, editor. Handbook of experimental psychology. New York, NY: Wiley; 1951. pp. 1–49.
- Sung, Y.-T., Chang, Y.-T. Y., Cheng, T.-Y., & Tien, H.-L. S. (2017). Development and validation of a work values scale for assessing high school students: A mixed methods approach. European Journal of Psychological Assessment. Advance online publication. 10.1027/1015-5759/a000408
- Sung Y-T, Cheng YW, Hsueh JH. Identifying the career-interest profiles of junior-high-school students through latent profile analysis. Journal of Psychology. 2017;151:229–246. doi: 10.1080/00223980.2016.1261076.
- Sung Y-T, Cheng YW, Wu JS. Constructing a situation-based career interest assessment for junior-high-school students and examining their interest structure. Journal of Career Assessment. 2016;24:347–365. doi: 10.1177/1069072715580419.
- Tabachnick BG, Fidell LS. Using multivariate statistics. 4. Boston, MA: Allyn & Bacon; 2001.
- Viswanathan M, Bergen M, Dutta S, Childers T. Does a single response category in a scale completely capture a response? Psychology and Marketing. 1996;13:457–479. doi: 10.1002/(SICI)1520-6793(199608)13:5<457::AID-MAR2>;2-8.
- Wewers ME, Lowe NK. A critical review of visual analogue scales in the measurement of clinical phenomena. Research in Nursing and Health. 1990;13:227–236. doi: 10.1002/nur.4770130405.
- Widaman KF. Hierarchically nested covariance structure models for multitrait–multimethod data. Applied Psychological Measurement. 1985;9:1–26. doi: 10.1177/014662168500900101.
- Wu CH. An empirical study on the transformation of Likert-type scale data to numerical scores. Applied Mathematical Sciences. 2007;1:2851–2862.
- Yusoff R, Janor RM. Generation of an interval metric scale to measure attitude. Sage Open. 2014;4:1–16. doi: 10.1177/2158244013516768.
- Zimmerman DW, Zumbo BD, Lalonde C. Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement. 1993;53:33–49. doi: 10.1177/0013164493053001003.
Source: PubMed