The Visual Analogue Scale for Rating, Ranking and Paired-Comparison (VAS-RRP): A new technique for psychological measurement

Yao-Ting Sung, Jeng-Shin Wu, Yao-Ting Sung, Jeng-Shin Wu

Abstract

Traditionally, the visual analogue scale (VAS) has been proposed to overcome the limitations of ordinal measures from Likert-type scales. However, the function of VASs to overcome the limitations of response styles to Likert-type scales has not yet been addressed. Previous research using ranking and paired comparisons to compensate for the response styles of Likert-type scales has suffered from limitations, such as that the total score of ipsative measures is a constant that cannot be analyzed by means of many common statistical techniques. In this study we propose a new scale, called the Visual Analogue Scale for Rating, Ranking, and Paired-Comparison (VAS-RRP), which can be used to collect rating, ranking, and paired-comparison data simultaneously, while avoiding the limitations of each of these data collection methods. The characteristics, use, and analytic method of VAS-RRPs, as well as how they overcome the disadvantages of Likert-type scales, ranking, and VASs, are discussed. On the basis of analyses of simulated and empirical data, this study showed that VAS-RRPs improved reliability, response style bias, and parameter recovery. Finally, we have also designed a VAS-RRP Generator for researchers' construction and administration of their own VAS-RRPs.

Keywords: CTCU model; Likert-type scale; Multi-item VAS; Paired comparison; Ranking; VAS-RRP.

Figures

Fig. 1
Fig. 1
Two examples of the Visual Analogue Scale for Rating, Ranking, and Paired-Comparison (VAS-RRP) after a user has placed each item on the continuum
Fig. 2
Fig. 2
Example of a correlated-traits–correlated-uniqueness model using the Visual Analogue Scale for Rating, Ranking, and Paired-Comparison (VAS-RRP). R = realistic type, I = investigative type, A = artistic type (Holland, 1997).
Fig. 3
Fig. 3
Example of a Situation-Based Career-Interest Assessment testlet.
Fig. 4
Fig. 4
First page of VAS-RRP Generator.
Fig. 5
Fig. 5
Snapshot of the procedure for the Design_your_VAS-RRP_scale functionality.
Fig. 6
Fig. 6
Snapshot of the VAS-RRP template file for three testlets with six items.
Fig. 7
Fig. 7
Snapshot of a Take_a_VAS-RRP_survey testlet.

References

    1. Albaum G. The Likert scale revisited: An alternate version. Journal of the Market Research Society. 1997;39:331–348. doi: 10.1177/147078539703900202.
    1. Allen IE, Seaman CA. Likert scales and data analyses. Quality Progress. 2007;40:64–65.
    1. Alwin DF. Information transmission in the survey interview: Number of response categories and the reliability of attitude measurement. In: Marsden PV, editor. Sociological methodology. Cambridge, MA: Blackwell; 1992. pp. 83–118.
    1. Babakus E, Ferguson CE, Jöreskog KG. The sensitivity of confirmatory maximum likelihood factor analysis to violations of measurement scale and distributional assumptions. Journal of Marketing Research. 1987;37:72–141.
    1. Bagozzi RP, Yi Y. On the evaluation of structural equation models. Journal of the Academy of Marketing Science. 1988;16:74–94. doi: 10.1007/BF02723327.
    1. Baron H. Strengths and limitations of ipsative measurement. Journal of Occupational and Organizational Psychology. 1996;69:49–56. doi: 10.1111/j.2044-8325.1996.tb00599.x.
    1. Bollen KA. Structural equation models. New York, NY: Wiley; 1989.
    1. Bollen KA, Barb KH. Pearson’s r and coarsely categorized measures. American Sociological Review. 1981;46:232–239. doi: 10.2307/2094981.
    1. Brady HE. Factor and ideal point analysis for interpersonally incomparable data. Psychometrika. 1989;54:181–202. doi: 10.1007/BF02294514.
    1. Brown A. Item response models for forced-choice questionnaires: A common framework. Psychometrika. 2014;81:1–26.
    1. Brown A, Maydeu-Olivares A. Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement. 2011;71:460–502. doi: 10.1177/0013164410375112.
    1. Brown A, Maydeu-Olivares A. Fitting a Thurstonian IRT model to forced-choice data using Mplus. Behavior Research Methods. 2012;44:1135–1147. doi: 10.3758/s13428-012-0217-x.
    1. Brown A, Maydeu-Olivares A. How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods. 2013;18:36–52. doi: 10.1037/a0030641.
    1. Carmines EG, McIver JP. Analyzing models with unobserved variables: Analysis of covariance structure. In: Bohrnstedt GW, Borgatta EF, editors. Social measurement: Current issues. Beverly Hills, CA: Sage; 1981. pp. 65–115.
    1. Chan W, Bentler PM. Covariance structure analysis of ordinal ipsative data. Psychometrika. 1998;63:369–399. doi: 10.1007/BF02294861.
    1. Cheung MWL, Chan W. Reducing uniform response bias with ipsative measurement in multiple-group confirmatory factor analysis. Structural Equation Modeling. 2002;9:55–77. doi: 10.1207/S15328007SEM0901_4.
    1. Chimi, C. J., & Russell, D. L. (2009, November). The Likert-type scale: A proposal for improvement using quasi-continuous variables. Paper presented at the ISECON 2009, Washington, DC.
    1. Chiu CK, Alliger GM. A proposed method to combine ranking and graphic rating in performance appraisal: The quantitative ranking scale. Educational and Psychological Measurement. 1990;50:493–503. doi: 10.1177/0013164490503003.
    1. Clemans, W. V. (1966). An analytical and empirical examination of some properties of ipsative measures (Psychometric Monograph No. 14). Richmond, VA: Psychometric Society. Retrieved from
    1. Cook C, Heath F, Thompson R, Thompson B. Score reliability in web- or internet-based surveys: Unnumbered graphic rating scales versus Likert-type scales. Educational and Psychological Measurement. 2001;61:697–706. doi: 10.1177/00131640121971356.
    1. Costa PT, McCrae RR. Professional manual: Revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI) Odessa, FL: Psychological Assessment Resources; 1992.
    1. Couper MP, Tourangeau R, Conrad FG, Singer E. Evaluating the effectiveness of visual analog scales: A Web experiment. Social Science Computer Review. 2006;24:227–245. doi: 10.1177/0894439305281503.
    1. Cox EP. The optimal number of response alternatives for a scale: A review. Journal of Marketing Research. 1980;17:407–422. doi: 10.2307/3150495.
    1. Cunningham WH, Cunningham ICM, Green RT. The ipsative process to reduce response set bias. Public Opinion Quarterly. 1977;41:379–384. doi: 10.1086/268394.
    1. Diedenhofen B, Musch J. Cocron: A web interface and R package for the statistical comparison of Cronbach’s alpha coefficients. International Journal of Internet Science. 2016;11:51–60.
    1. Dunlap WP, Cornwell JM. Factor analysis of ipsative measures. Multivariate Behavioral Research. 1994;29:115–126. doi: 10.1207/s15327906mbr2901_4.
    1. Ferrando PJ. A kernel density analysis of continuous typical-response scales. Educational and Psychological Measurement. 2003;63:809–824. doi: 10.1177/0013164403251323.
    1. Flynn D, van Schaik P, van Wersch A. A comparison of multi-item Likert and visual analogue scales for the assessment of transactionally defined coping function. European Journal of Psychological Assessment. 2004;20:49–58. doi: 10.1027/1015-5759.20.1.49.
    1. Funke F, Reips U-D. Why semantic differentials in Web-based research should be made from visual analogue scales and not from 5-point scales. Field Methods. 2012;24:310–327. doi: 10.1177/1525822X12444061.
    1. Gay, E. G., Weiss, D. J., Hendel, D. D., Dawis, R. V., & Lofquist, L. H. (1971). Manual for the Minnesota importance questionnaire (No. 54). Work Adjustment Project, University of Minnesota.
    1. Goffin RD, Olson JM. Is it all relative? Comparative judgments and the possible improvement of self-ratings and ratings of others. Perspectives on Psychological Science. 2011;6:48–60. doi: 10.1177/1745691610393521.
    1. Gordon LV. Gordon personal profile inventory (GPP-1): Manual. San Antonio, TX: Psychological Corporation; 1993.
    1. Granberg-Rademacker JS. An algorithm for converting ordinal scale measurement data to interval/ratio scale. Educational and Psychological Measurement. 2010;70:74–90. doi: 10.1177/0013164409344532.
    1. Greenleaf EA. Measuring extreme response style. Public Opinion Quarterly. 1992;56:328–351. doi: 10.1086/269326.
    1. Guyatt GH, Townsend M, Berman LB, Keller JL. A comparison of Likert and visual analogues scales for measuring change in function. Journal of Chronic Disability. 1987;40:1129–1133. doi: 10.1016/0021-9681(87)90080-4.
    1. Harwell MR, Gatti GG. Rescaling ordinal data to interval data in educational research. Review of Educational Research. 2001;71:105–131. doi: 10.3102/00346543071001105.
    1. Hicks LE. Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin. 1970;74:167–184. doi: 10.1037/h0029780.
    1. Holland JL. Making vocational choices: A theory of vocational personalities and work environments. Odessa, FL: Psychological Assessment Resources; 1997.
    1. Holyk GG. Context effect. In: Lavrakas PJ, editor. Encyclopedia of survey research methods. Thousand Oaks CA: Sage; 2008. p. 142.
    1. Hooper D, Coughlan J, Mullen MR. Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods. 2008;6:53–60.
    1. Jackson DJ, Alwin DF. The factor analysis of ipsative measures. Sociological Methods and Research. 1980;9:218–238. doi: 10.1177/004912418000900206.
    1. Jaeschke R, Singer J, Guyatt GH. A comparison of seven-point and visual analogue scales. Controlled Clinical Trials. 1990;11:43–51. doi: 10.1016/0197-2456(90)90031-V.
    1. Jamieson S. Likert scales: How to (ab)use them. Medical Education. 2004;38:1212–1218. doi: 10.1111/j.1365-2929.2004.02012.x.
    1. Kolb AY. The Kolb learning style inventory—Version 3.1:2005 technical specifications. Boston, MA: Hay Resources Direct; 2005.
    1. Krieg EF. Biases induced by coarse measurements scales. Educational and Psychological Measurement. 1999;59:749–766. doi: 10.1177/00131649921970125.
    1. Kuhlmann T, Dantlgraber M, Reips U-D. Investigating measurement equivalence of visual analogue scales and Likert-type scales in Internet-based personality questionnaires. Behavior Research Methods. 2017;49:2173–2181. doi: 10.3758/s13428-016-0850-x.
    1. Laming D. Human judgment: The eye of the beholder. London, UK: Thomson; 2004.
    1. Likert R. A technique for the measurement of attitudes. Archives of Psychology. 1932;140:5–55.
    1. Loo R. Confirmatory factor analyses of Kolb’s learning style inventory (LSI-1985) British Journal of Educational Psychology. 1999;69:213–219. doi: 10.1348/000709999157680.
    1. Marsh HW. Confirmatory factor analyses of multitrait–multimethod data: Many problems and a few solutions. Applied Psychological Measurement. 1989;13:335–361. doi: 10.1177/014662168901300402.
    1. Marsh HW, Bailey M. Confirmatory factor analyses of multitrait–multimethod data: A comparison of alternative models. Applied Psychological Measurement. 1991;15:47–70. doi: 10.1177/014662169101500106.
    1. Marsh HW, Grayson D. Latent variable models of multitrait–multimethod data. In: Hoyle RH, editor. Structural equation modeling: Concepts, issues, and applications. Thousand Oaks, CA: Sage; 1995. pp. 177–198.
    1. McCloy R, Waugh G, Medsker G, Wall J, Rivkin D, Lewis P. Development of the O* NET computerized work importance profiler. Raleigh, NC: National Center for O* NET Development; 1999.
    1. McCloy R, Waugh G, Medsker G, Wall J, Rivkin D, Lewis P. Development of the O* NET paper-and pencil work importance locator. Raleigh, NC: National Center for O* NET Development; 1999.
    1. McKelvie SJ. Graphic rating scales: How many categories? British Journal of Psychology. 1978;69:185–202. doi: 10.1111/j.2044-8295.1978.tb01647.x.
    1. Meade AW. Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology. 2004;77:531–551. doi: 10.1348/0963179042596504.
    1. Munshi, J. (2014). A method for constructing Likert scales. Research report, Sonoma State University. Retrieved from
    1. Myles PS, Troedel S, Boquest M, Reeves M. The pain visual analog scale: Is it linear or nonlinear? Anesthesia and Analgesia. 1999;89:1517–1520.
    1. Nunnally JC. Psychometric theory. New York, NY: McGraw-Hill; 1967.
    1. Nyren O, Adami O, Bates S, Bergstrom R, Gustavsson S, Loof L, Sjoden PO. Self-rating of pain in non-ulcer dyspepsia. Journal of Clinical Gastroenterology. 1987;9:408–414. doi: 10.1097/00004836-198708000-00010.
    1. Tomás JM, Oliver A, Hontangas PM. Linear confirmatory models for MTMM matrices: The case of several indicators per trait–method combinations. In: Shohov SP, editor. Advances in psychology research. Huntington, NY: Nova Science; 2002. pp. 99–122.
    1. Paulhus DL. Control of social desirability in personality inventories: Principal-factor deletion. Journal of Research in Personality. 1981;15:383–388. doi: 10.1016/0092-6566(81)90035-0.
    1. Paulhus DL. Measures of personality and social psychological attitudes. In: Robinson JP, Shaver RP, editors. Measures of social psychological attitudes series. San Diego, CA: Academic; 1991. pp. 17–59.
    1. Preston CC, Colman AM. Optimal number of response categories in rating scales: Reliability, validity, discriminating power, and respondent. Acta Psychologica. 2000;104:1–15. doi: 10.1016/S0001-6918(99)00050-5.
    1. Price DD, McGrath PA, Rafii A, Buckingham B. The validation of visual analogue scales as ratio scale measures for chronic and experimental pain. Pain. 1983;17:45–56. doi: 10.1016/0304-3959(83)90126-4.
    1. Randall DM, Fernandes MF. The social desirability response bias in ethics research. Journal of Business Ethics. 1991;10:805–817. doi: 10.1007/BF00383696.
    1. Raykov T. Estimation of composite reliability for congeneric measures. Applied Psychological Measurement. 1997;21:173–184. doi: 10.1177/01466216970212006.
    1. Reips U-D, Funke F. Interval-level measurement with visual analogue scales in Internet-based research: VAS Generator. Behavior Research Methods. 2008;40:699–704. doi: 10.3758/BRM.40.3.699.
    1. Rounds JB, Miller TW, Dawis RV. Comparability of multiple rank order and paired comparison methods. Applied Psychological Measurement. 1978;2:415–422. doi: 10.1177/014662167800200316.
    1. Scherpenzeel AC, Saris WE. The validity and reliability of survey questions. Sociological Methods & Research. 1997;25:341–383. doi: 10.1177/0049124197025003004.
    1. Sheppard LD, Goffin RD, Lewis RJ, Olson J. The effect of target attractiveness and rating method on the accuracy of trait ratings. Journal of Personnel Psychology. 2011;10:24–33. doi: 10.1027/1866-5888/a000030.
    1. Spooren P, Mortelmans D, Thijssen P. “Content” versus “style”: Acquiescence in student evaluation of teaching? British Educational Research Journal. 2012;38:3–21. doi: 10.1080/01411926.2010.523453.
    1. Stevens SS. On the theory of scales of measurement. Science. 1946;103:677–680. doi: 10.1126/science.103.2684.677.
    1. Stevens SS. Mathematics, measurement, and psychophysics. In: Stevens SS, editor. Handbook of experimental psychology. New York, NY: Wiley; 1951. pp. 1–49.
    1. Sung, Y.-T., Chang, Y.-T. Y., Cheng, T.-Y., & Tien, H.-L. S. (2017). Development and validation of a work values scale for assessing high school students: A mixed methods approach. European Journal of Psychological Assessment. Advance online publication. 10.1027/1015-5759/a000408
    1. Sung Y-T, Cheng YW, Hsueh JH. Identifying the career-interest profiles of junior-high-school students through latent profile analysis. Journal of Psychology. 2017;151:229–246. doi: 10.1080/00223980.2016.1261076.
    1. Sung Y-T, Cheng YW, Wu JS. Constructing a situation-based career interest assessment for junior-high-school students and examining their interest structure. Journal of Career Assessment. 2016;24:347–365. doi: 10.1177/1069072715580419.
    1. Tabachnick BG, Fidell LS. Using multivariate statistics. 4. Boston, MA: Allyn & Bacon; 2001.
    1. Viswanathan M, Bergen M, Dutta S, Childers T. Does a single response category in a scale completely capture a response? Psychology and Marketing. 1996;13:457–479. doi: 10.1002/(SICI)1520-6793(199608)13:5<457::AID-MAR2>;2-8.
    1. Wewers ME, Lowe NK. A critical review of visual analogue scales in the measurement of clinical phenomena. Research in Nursing and Health. 1990;13:227–236. doi: 10.1002/nur.4770130405.
    1. Widaman KF. Hierarchically nested covariance structure models for multitrait–multimethod data. Applied Psychological Measurement. 1985;9:1–26. doi: 10.1177/014662168500900101.
    1. Wu CH. An empirical study on the transformation of Likert-type scale data to numerical scores. Applied Mathematical Sciences. 2007;1:2851–2862.
    1. Yusoff R, Janor RM. Generation of an interval metric scale to measure attitude. Sage Open. 2014;4:1–16. doi: 10.1177/2158244013516768.
    1. Zimmerman DW, Zumbo BD, Lalonde C. Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement. 1993;53:33–49. doi: 10.1177/0013164493053001003.

Source: PubMed

Szponzorok és közreműködők

Egészségi állapot

Kábítószer-beavatkozások

3
Iratkozz fel