How to measure metacognition

Stephen M Fleming, Hakwan C Lau, Stephen M Fleming, Hakwan C Lau

Abstract

The ability to recognize one's own successful cognitive processing, in e.g., perceptual or memory tasks, is often referred to as metacognition. How should we quantitatively measure such ability? Here we focus on a class of measures that assess the correspondence between trial-by-trial accuracy and one's own confidence. In general, for healthy subjects endowed with metacognitive sensitivity, when one is confident, one is more likely to be correct. Thus, the degree of association between accuracy and confidence can be taken as a quantitative measure of metacognition. However, many studies use a statistical correlation coefficient (e.g., Pearson's r) or its variant to assess this degree of association, and such measures are susceptible to undesirable influences from factors such as response biases. Here we review other measures based on signal detection theory and receiver operating characteristics (ROC) analysis that are "bias free," and relate these quantities to the calibration and discrimination measures developed in the probability estimation literature. We go on to distinguish between the related concepts of metacognitive bias (a difference in subjective confidence despite basic task performance remaining constant), metacognitive sensitivity (how good one is at distinguishing between one's own correct and incorrect judgments) and metacognitive efficiency (a subject's level of metacognitive sensitivity given a certain level of task performance). Finally, we discuss how these three concepts pose interesting questions for the study of metacognition and conscious awareness.

Keywords: confidence; consciousness; metacognition; probability judgment; signal detection theory.

Figures

Figure 1
Figure 1
Schematic showing the theoretical dissociation between metacognitive sensitivity and bias. Each graph shows a hypothetical probability density of confidence ratings for correct and incorrect trials, with confidence increasing from left to right along each x-axis. Metacognitive sensitivity is the separation between the distributions—the extent to which confidence discriminates between correct and incorrect trials. Metacognitive bias is the overall level of confidence expressed, independent of whether the trial is correct or incorrect. Note that this is a cartoon schematic and we do not mean to imply any parametric form for these “Type 2” signal detection theoretic distributions. Indeed, as shown by Galvin et al. (2003), these distributions are unlikely to be Gaussian.
Figure 2
Figure 2
(A) Example type 2 ROC function for a single subject. Each point plots the type 2 false alarm rate on the x-axis against the type 2 hit rate on the y-axis for a given confidence criterion. The shaded area under the curve indexes metacognitive sensitivity. (B) Example underconfident and overconfident probability calibration curves, modified after Harvey (1997).

References

    1. Baird B., Smallwood J., Gorgolewski K. J., Margulies D. S. (2013). Medial and lateral networks in anterior prefrontal cortex support metacognitive ability for memory and perception. J. Neurosci. 33, 16657–16665 10.1523/JNEUROSCI.0786-13.2013
    1. Barrett A., Dienes Z., Seth A. K. (2013). Measures of metacognition on signal-detection theoretic models. Psychol. Methods 18, 535–552 10.1037/a0033268
    1. Benjamin A. S., Diaz M. (2008). Measurement of relative metamnemonic accuracy, in Handbook of Metamemory and Memory, eds Dunlosky J., Bjork R. A. (New York, NY: Psychology Press; ), 73–94
    1. Brier G. W. (1950). Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 78, 1–3 10.1175/1520-0493(1950)078%3C0001:VOFEIT%;2
    1. Charles L., Van Opstal F., Marti S., Dehaene S. (2013). Distinct brain mechanisms for conscious versus subliminal error detection. Neuroimage 73, 80–94 10.1016/j.neuroimage.2013.01.054
    1. Clarke F., Birdsall T., Tanner W. (1959). Two types of ROC curves and definition of parameters. J. Acoust. Soc. Am. 31, 629–630 10.1121/1.1907764
    1. David A. S., Bedford N., Wiffen B., Gilleen J. (2012). Failures of metacognition and lack of insight in neuropsychiatric disorders. Philos. Trans. R. Soc. Lond. B Biol. Sci. 367, 1379–1390 10.1098/rstb.2012.0002
    1. de Gardelle V., Mamassian P. (2014). Does confidence use a common currency across two visual tasks? Psychol. Sci. 25, 1286–1288 10.1177/0956797614528956
    1. De Martino B., Fleming S. M., Garrett N., Dolan R. J. (2013). Confidence in value-based choice. Nat. Neurosci. 16, 105–110 10.1038/nn.3279
    1. Dienes Z. (2008). Subjective measures of unconscious knowledge. Prog. Brain Res. 168, 49–64 10.1016/S0079-6123(07)68005-4
    1. Evans S., Azzopardi P. (2007). Evaluation of a “bias-free” measure of awareness. Spat. Vis. 20, 61–77 10.1163/156856807779369742
    1. Ferrell W. R., McGoey P. J. (1980). A model of calibration for subjective probabilities. Organ. Behav. Hum. Perform. 26, 32–53 10.1016/0030-5073(80)90045-8
    1. Fleming S. M., Dolan R. J. (2012). The neural basis of metacognitive ability. Philos. Trans. R. Soc. Lond. B Biol. Sci. 367, 1338–1349 10.1098/rstb.2011.0417
    1. Fleming S. M., Weil R. S., Nagy Z., Dolan R. J., Rees G. (2010). Relating introspective accuracy to individual differences in brain structure. Science 329, 1541–1543 10.1126/science.1191883
    1. Galvin S. J., Podd J. V., Drga V., Whitmore J. (2003). Type 2 tasks in the theory of signal detectability: discrimination between correct and incorrect decisions. Psychon. Bull. Rev. 10, 843–876 10.3758/BF03196546
    1. Gonzalez R., Nelson T. O. (1996). Measuring ordinal association in situations that contain tied scores. Psychol. Bull. 119, 159 10.1037//0033-2909.119.1.159
    1. Goodman L. A., Kruskal W. H. (1954). Measures of association for cross classifications. J. Am. Stat. Assoc. 49, 732–764
    1. Green D., Swets J. (1966). Signal Detection Theory and Psychophysics. New York, NY: Wiley
    1. Hampton R. R. (2001). Rhesus monkeys know when they remember. Proc. Natl. Acad. Sci. U.S.A. 98, 5359–5362 10.1073/pnas.071600998
    1. Harvey N. (1997). Confidence in judgment. Trends Cogn. Sci. 1, 78–82 10.1016/S1364-6613(97)01014-0
    1. Henmon V. (1911). The relation of the time of a judgment to its accuracy. Psychol. Rev. 18, 186 10.1037/h0074579
    1. Higham P. A. (2007). No special K! A signal detection framework for the strategic regulation of memory accuracy. J. Exp. Psychol. Gen. 136, 1 10.1037/0096-3445.136.1.1
    1. Higham P. A., Perfect T. J., Bruno D. (2009). Investigating strength and frequency effects in recognition memory using type-2 signal detection theory. J. Exp. Psychol. Learn. Mem. Cogn. 35, 57 10.1037/a0013865
    1. Hollard G., Massoni S., Vergnaud J. C. (2010). Subjective Belief Formation and Elicitation Rules: Experimental Evidence. Working paper.
    1. Howell D. C. (2009). Statistical Methods for Psychology. Pacific Grove, CA: Wadsworth Pub Co
    1. Jang Y., Wallsten T. S., Huber D. E. (2012). A stochastic detection and retrieval model for the study of metacognition. Psychol. Rev. 119, 186 10.1037/a0025960
    1. Kepecs A., Mainen Z. F. (2012). A computational framework for the study of confidence in humans and animals. Philos. Trans. R. Soc. Lond. B Biol. Sci. 367, 1322–1337 10.1098/rstb.2012.0037
    1. Keren G. (1991). Calibration and probability judgements: conceptual and methodological issues. Acta Psychol. 77, 217–273 10.1016/0001-6918(91)90036-Y
    1. Kolb F. C., Braun J. (1995). Blindsight in normal observers. Nature 377, 336–338 10.1038/377336a0
    1. Koriat A. (2007). Metacognition and consciousness, in The Cambridge Handbook of Consciousness, eds Zelazo P. D., Moscovitch M., Davies E. (New York, NY: Cambridge University Press; ), 289–326
    1. Kornell N., Son L. K., Terrace H. S. (2007). Transfer of metacognitive skills and hint seeking in monkeys. Psychol. Sci. 18, 64–71 10.1111/j.1467-9280.2007.01850.x
    1. Kruger J., Dunning D. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J. Pers. Soc. Psychol. 77, 1121–1134 10.1037/0022-3514.77.6.1121
    1. Kunimoto C., Miller J., Pashler H. (2001). Confidence and accuracy of near-threshold discrimination responses. Conscious. Cogn. 10, 294–340 10.1006/ccog.2000.0494
    1. Lachman J. L., Lachman R., Thronesbery C. (1979). Metamemory through the adult life span. Dev. Psychol. 15, 543 10.1037/0012-1649.15.5.543
    1. Lau H. (2008). Are we studying consciousness yet? in Frontiers of Consciousness: Chichele Lectures, eds Weiskrantz L., Davies M. (Oxford: Oxford University Press; ), 245–258
    1. Lau H. C., Passingham R. E. (2006). Relative blindsight in normal observers and the neural correlate of visual consciousness. Proc. Natl. Acad. Sci. U.S.A. 103, 18763–18768 10.1073/pnas.0607716103
    1. Lee T. G., Blumenfeld R. S., D'Esposito M. (2013). Disruption of dorsolateral but not ventrolateral prefrontal cortex improves unconscious perceptual memories. J. Neurosci. 33, 13233–13237 10.1523/JNEUROSCI.5652-12.2013
    1. Lichtenstein S., Fischhoff B., Phillips L. D. (1982). Calibration of probabilities: the state of the art to 1980, in Judgment Under Uncertainty: Heuristics and Biases, eds Kahneman D., Slovic P., Tversky A. (Cambridge, UK: Cambridge University Press; ), 306–334
    1. Maniscalco B., Lau H. (2012). A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Conscious. Cogn. 21, 422–430 10.1016/j.concog.2011.09.021
    1. Maniscalco B., Lau H. (2014). Signal detection theory analysis of type 1 and type 2 data: meta-d', response-specific meta-d', and the unequal variance SDT Model, in The Cognitive Neuroscience of Metacognition, eds Fleming S. M., Frith C. D. (Berlin: Springer; ), 25–66
    1. Mason I. B. (2003). Binary events, in Forecast Verification: A Practitioner's Guide in Atmospheric Science, eds Jolliffe I. T., Stephenson D. B. (Chichester: Wiley; ), 37–76
    1. Masson M. E. J., Rotello C. M. (2009). Sources of bias in the Goodman–Kruskal gamma coefficient measure of association: implications for studies of metacognitive processes. J. Exp. Psychol. Learn. Mem. Cogn. 35, 509–527 10.1037/a0014876
    1. McCurdy L. Y., Maniscalco B., Metcalfe J., Liu K. Y., de Lange F. P., Lau H. (2013). Anatomical coupling between distinct metacognitive systems for memory and visual perception. J. Neurosci. 33, 1897–1906 10.1523/JNEUROSCI.1890-12.2013
    1. Metcalfe J., Shimamura A. P. (1996). Metacognition: Knowing About Knowing. Cambridge, MA: MIT Press
    1. Middlebrooks P. G., Sommer M. A. (2011). Metacognition in monkeys during an oculomotor task. J. Exp. Psychol. Learn. Mem. Cogn. 37, 325–337 10.1037/a0021611
    1. Moore D. A., Healy P. J. (2008). The trouble with overconfidence. Psychol. Rev. 115, 502–517 10.1037/0033-295X.115.2.502
    1. Morgan M., Mason A. (1997). Blindsight in normal subjects? Nature 385, 401–402 10.1038/385401b0
    1. Murphy A. H. (1973). A new vector partition of the probability score. J. Appl. Meteor. 12, 595–600 10.1175/1520-0450(1973)012<0595:ANVPOT>;2
    1. Nelson T. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychol. Bull. 95, 109–133 10.1037/0033-2909.95.1.109
    1. Nelson T. O., Dunlosky J. (1991). When people's Judgments of Learning (JOLs) are extremely accurate at predicting subsequent recall: the ‘Delayed-JOL Effect.’ Psychol. Sci. 2, 267–270 10.1111/j.1467-9280.1991.tb00147.x
    1. Nelson T. O., Narens L. (1990). Metamemory: a theoretical framework and new findings. Psychol. Learn. Motiv. 26, 125–141 10.1016/S0079-7421(08)60053-5
    1. Peirce C. S., Jastrow J. (1885). On small differences in sensation. Mem. Natl. Acad. Sci. 3, 73–83
    1. Persaud N., Davidson M., Maniscalco B., Mobbs D., Passingham R. E., Cowey A., et al. (2011). Awareness-related activity in prefrontal and parietal cortices in blindsight reflects more than superior visual performance. Neuroimage 58, 605–611 10.1016/j.neuroimage.2011.06.081
    1. Rhodes M. G., Tauber S. K. (2011). The influence of delaying judgments of learning on metacognitive accuracy: a meta-analytic review. Psychol. Bull. 137, 131 10.1037/a0021705
    1. Rounis E., Maniscalco B., Rothwell J., Passingham R., Lau H. (2010). Theta-burst transcranial magnetic stimulation to the prefrontal cortex impairs metacognitive visual awareness. Cogn. Neurosci. 1, 165–175 10.1080/17588921003632529
    1. Sandberg K., Timmermans B., Overgaard M., Cleeremans A. (2010). Measuring consciousness: is one measure better than the other? Conscious. Cogn. 19, 1069–1078 10.1016/j.concog.2009.12.013
    1. Schmitz T. W., Rowley H. A., Kawahara T. N., Johnson S. C. (2006). Neural correlates of self-evaluative accuracy after traumatic brain injury. Neuropsychologia 44, 762–773 10.1016/j.neuropsychologia.2005.07.012
    1. Schwiedrzik C. M., Singer W., Melloni L. (2011). Subjective and objective learning effects dissociate in space and in time. Proc. Natl. Acad. Sci. U.S.A. 108, 4506–4511 10.1073/pnas.1009147108
    1. Souchay C., Isingrini M., Espagnet L. (2000). Aging, episodic memory feeling-of-knowing, and frontal functioning. Neuropsychology 14, 299 10.1037/0894-4105.14.2.299
    1. Weil L. G., Fleming S. M., Dumontheil I., Kilford E. J., Weil R. S., Rees G., et al. (2013). The development of metacognitive ability in adolescence. Conscious. Cogn. 22, 264–271 10.1016/j.concog.2013.01.004
    1. Weiskrantz L., Warrington E. K., Sanders M. D., Marshall J. (1974). Visual capacity in the hemianopic field following a restricted occipital ablation. Brain 97, 709–728 10.1093/brain/97.1.709
    1. Yaniv I., Yates J. F., Smith J. K. (1991). Measures of discrimination skill in probabilistic judgment. Can. J. Exp. Psychol. 110, 611

Source: PubMed

Sponzoři a spolupracovníci

Zdravotní podmínky

Drogové intervence

3
Předplatit