CoChat
Sign Up
Cited by 201Memory Processes and InfluencesVisual perception and processing mechanismsMemory and Neural Mechanisms

Regressive research: the pitfalls of post hoc data selection in the study of unconscious mental processes

Open access
Shanks, David R.·2017·Psychonomic Bulletin & Review
DOI 10.3758/s13423-016-1170-yPMC PMC5486877
Open this paper in CoChat — free accountOpen PDF

Abstract

Many studies of unconscious processing involve comparing a performance measure (e.g., some assessment of perception or memory) with an awareness measure (such as a verbal report or a forced-choice response) taken either concurrently or separately. Unconscious processing is inferred when above-chance performance is combined with null awareness. Often, however, aggregate awareness is better than chance, and data analysis therefore employs a form of extreme group analysis focusing post hoc on participants, trials, or items where awareness is absent or at chance. The pitfalls of this analytic approach are described with particular reference to recent research on implicit learning and subliminal perception. Because of regression to the mean, the approach can mislead researchers into erroneous conclusions concerning unconscious influences on behavior. Recommendations are made about future use of post hoc selection in research on unconscious cognition.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3758/s13423-016-1170-y

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Selecting subjects based on the same data used to demonstrate the absence of awareness is comparable to double-dipping and likely to confound the results with regression to the mean.

    “This practice is problematic because selecting subjects based on the same data used to demonstrate the absence of awareness is comparable to “double-dipping” (Kriegeskorte et al. 2009) and likely to confound the results with regression to the mean (Shanks 2017, Stein et al. 2024).”

    From Evidence for unconscious priming using a Bayesian single-subject approach (2026) · cites this paper as Shanks 2017

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 71

  1. Batterink, L. J., Reber, P. J., Neville, H. J., & Paller, K. A. (2015). Implicit and explicit contributions to statistical learning. Journal of Memory and Language, 83, 62–78. doi:10.1016/j.jml.2015.04.004 PMC4448134 26034344 · DOI
  2. Berry CJ Shanks DR Li S Rains LS Henson RNA Can “pure” implicit memory be isolated? A test of a single-system model of recognition and repetition priming Canadian Journal of Experimental Psychology 2010 64 241 255 10.1037/a0021525 21186908 · DOI
  3. Berry CJ Shanks DR Speekenbrink M Henson RNA Models of recognition, repetition priming, and fluency: Exploring a new framework Psychological Review 2012 119 40 79 10.1037/a0025464 22022831 · DOI
  4. Buchner, A., & Wippich, W. (2000). On the reliability of implicit and explicit memory measures. Cognitive Psychology, 40, 227–259. doi:10.1006/cogp.1999.0731 10790324 · DOI
  5. Butler, B. C., & Klein, R. (2009). Inattentional blindness for ignored words: Comparison of explicit and implicit memory tasks. Consciousness and Cognition, 18, 811–819. doi:10.1016/j.concog.2009.02.009 19328012 · DOI
  6. Campbell DT Kenny DA A primer on regression artifacts 1999 New York, NY Guilford Press
  7. Chun, M. M., & Jiang, Y. (1998). Contextual cueing: Implicit learning and memory of visual context guides spatial attention. Cognitive Psychology, 36, 28–71. doi:10.1006/cogp.1998.0681 9679076 · DOI · Paper page
  8. Chun, M. M., & Jiang, Y. (2003). Implicit, long-term spatial contextual memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29, 224–234. doi:10.1037/0278-7393.29.2.224 12696811 · DOI
  9. Conci M von Mühlenen A Limitations of perceptual segmentation on contextual cueing in visual search Visual Cognition 2011 19 203 233 10.1080/13506285.2010.518574 · DOI
  10. Daselaar, S. M., Fleck, M. S., Prince, S. E., & Cabeza, R. (2006). The medial temporal lobe distinguishes old from new independently of consciousness. Journal of Neuroscience, 26, 5835–5839. doi:10.1523/JNEUROSCI.0258-06.2006 PMC6675252 16723542 · DOI
  11. Dienes Z Overgaard M How Bayesian statistics are needed to determine whether mental states are unconscious Behavioural methods in consciousness research 2015 Oxford, UK Oxford University Press 199 220
  12. Dixon NF Subliminal perception: The nature of a controversy 1971 London, UK McGraw-Hill
  13. Dosher BA The response-window regression method—Some problematic assumptions: Comment on Draine and Greenwald (1998) Journal of Experimental Psychology: General 1998 127 311 317 10.1037/0096-3445.127.3.311 · DOI
  14. Duke, D., Fiacconi, C. M., & Köhler, S. (2014). Parallel effects of processing fluency and positive affect on familiarity-based recognition decisions for faces. Frontiers in Psychology, 5. doi:10.3389/fpsyg.2014.00328 PMC4001004 24795678 · DOI
  15. Dulany, D. E. (1961). Hypotheses and habits in verbal “operant conditioning”. Journal of Abnormal and Social Psychology, 63, 251–263. doi:10.1037/h0047703 13888343 · DOI
  16. Fiedler K Unkelbach C Regressive judgment: Implications of a universal property of the empirical world Current Directions in Psychological Science 2014 23 361 367 10.1177/0963721414546330 · DOI
  17. Geyer T Shi ZH Müller HJ Contextual cueing in multiconjunction visual search is dependent on color- and configuration-based intertrial contingencies Journal of Experimental Psychology: Human Perception and Performance 2010 36 515 532 20515186
  18. Greenspoon, J. (1955). The reinforcing effect of two spoken sounds on the frequency of two responses. American Journal of Psychology, 68, 409–416. doi:10.2307/1418524 13248974 · DOI
  19. Greenwald AG Klinger MR Schuh ES Activation by marginally perceptible (“subliminal”) stimuli: Dissociations of unconscious from conscious cognition Journal of Experimental Psychology: General 1995 124 22 42 10.1037/0096-3445.124.1.22 7897340 · DOI
  20. Greenwald AG McGhee DE Schwartz JLK Measuring individual differences in implicit cognition: The implicit association test Journal of Personality and Social Psychology 1998 74 1464 1480 10.1037/0022-3514.74.6.1464 9654756 · DOI · Paper page
  21. Hannula, D. E., & Ranganath, C. (2009). The eyes have it: Hippocampal activity predicts expression of memory in eye movements. Neuron, 63, 592–599. doi:10.1016/j.neuron.2009.08.025 PMC2747814 19755103 · DOI · Paper page
  22. Harris, J. J., Schwarzkopf, D. S., Song, C., Bahrami, B., & Rees, G. (2011). Contextual illusions reveal the limit of unconscious visual processing. Psychological Science, 22, 399–405. doi:10.1177/0956797611399293 PMC3278746 21317371 · DOI
  23. Heerey, E. A., & Velani, H. (2010). Implicit learning of social predictions. Journal of Experimental Social Psychology, 46, 577–581. doi:10.1016/j.jesp.2010.01.003 · DOI
  24. Holender, D. (1986). Semantic activation without conscious identification in dichotic listening, parafoveal vision, and visual masking: A survey and appraisal. Behavioral and Brain Sciences, 9, 1–66. doi:10.1017/S0140525X00021269 · DOI
  25. Hsu, L. M. (1995). Regression toward the mean associated with measurement error and the identification of improvement and deterioration in psychotherapy. Journal of Consulting and Clinical Psychology, 63, 141–144. doi:10.1037/0022-006X.63.1.141 7896979 · DOI
  26. Jones CR Fazio RH Olson MA Implicit misattribution as a mechanism underlying evaluative conditioning Journal of Personality and Social Psychology 2009 96 933 948 10.1037/a0014747 19379028 PMC2701550 · DOI
  27. Kaufman SB DeYoung CG Gray JR Jiménez L Brown J Mackintosh N Implicit learning as an ability Cognition 2010 116 321 340 10.1016/j.cognition.2010.05.011 20573341 · DOI
  28. Kenny DA Everitt BS Howell DC Regression artifacts Encyclopedia of statistics in behavioral science 2005 Chichester, UK Wiley 1723 1725
  29. Koivisto M Mäntylä T Silvanto J The role of early visual cortex (V1/V2) in conscious and unconscious visual perception NeuroImage 2010 51 828 834 10.1016/j.neuroimage.2010.02.042 20188199 · DOI
  30. Krause MS Reversion toward the mean independently of regression toward the mean Methodology: European Journal of Research Methods for the Behavioral and Social Sciences 2009 5 3 6 10.1027/1614-2241.5.1.3 · DOI
  31. Lazarus, R. S., & McCleary, R. A. (1951). Autonomic discrimination without awareness: A study of subception. Psychological Review, 58, 113–122. doi:10.1037/h0054104 14834294 · DOI
  32. LeBel, E. P., & Paunonen, S. V. (2011). Sexy but often unreliable: The impact of unreliability on the replicability of experimental findings with implicit measures. Personality and Social Psychology Bulletin, 37, 570–583. doi:10.1177/0146167211400619 21441219 · DOI
  33. Lovibond PF Shanks DR The role of awareness in Pavlovian conditioning: Empirical evidence and theoretical implications Journal of Experimental Psychology: Animal Behavior Processes 2002 28 3 26 11868231
  34. Merikle, P. M., & Reingold, E. M. (1991). Comparing direct (explicit) and indirect (implicit) measures to study unconscious memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 17, 224–233. doi:10.1037/0278-7393.17.2.224 · DOI
  35. Miller, J. (2000). Measurement error in subliminal perception experiments: Simulation analyses of two regression methods. Journal of Experimental Psychology: Human Perception and Performance, 26, 1461–1477. doi:10.1037/0096-1523.26.4.1461 10946725 · DOI
  36. Muscarella, C., Brintazzoli, G., Gordts, S., Soetens, E., & Van den Bussche, E. (2013). Short- and long-term effects of conscious, minimally conscious and unconscious brand logos. PLoS ONE, 8, e57738. doi:10.1371/journal.pone.0057738 PMC3642191 23658681 · DOI
  37. Newell BR Shanks DR Unconscious influences on decision making: A critical review Behavioral and Brain Sciences 2014 37 1 19 10.1017/S0140525X12003214 24461214 · DOI · Paper page
  38. Paciorek, A., & Williams, J. N. (2015). Semantic generalization in implicit language learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 41, 989–1002. doi:10.1037/xlm0000100 25581225 · DOI
  39. Pessiglione, M., Schmidt, L., Draganski, B., Kalisch, R., Lau, H., Dolan, R. J., & Frith, C. D. (2007). How the brain translates money into force: A neuroimaging study of subliminal motivation. Science, 316, 904–906. doi:10.1126/science.1140459 PMC2631941 17431137 · DOI · Paper page
  40. Poldrack, R. A. (1996). On testing for stochastic dissociations. Psychonomic Bulletin & Review, 3, 434–448. doi:10.3758/BF03214547 24213977 · DOI
  41. Pratte, M. S., & Rouder, J. N. (2009). A task-difficulty artifact in subliminal priming. Attention, Perception, & Psychophysics, 71, 1276–1283. doi:10.3758/APP.71.6.1276 19633343 · DOI
  42. Preacher, K. J., Rucker, D. D., MacCallum, R. C., & Nicewander, W. A. (2005). Use of the extreme groups approach: A critical reexamination and new recommendations. Psychological Methods, 10, 178–192. doi:10.1037/1082-989X.10.2.178 15998176 · DOI · Paper page
  43. Reingold, E. M., & Merikle, P. M. (1988). Using direct and indirect measures to study perception without awareness. Perception & Psychophysics, 44, 563–575. doi:10.3758/BF03207490 3200674 · DOI · Paper page
  44. Rugg, M. D., Mark, R. E., Walla, P., Schloerscheidt, A. M., Birch, C. S., & Allan, K. (1998). Dissociation of the neural correlates of implicit and explicit memory. Nature, 392, 595–598. doi:10.1038/33396 9560154 · DOI
  45. Rünger, D., Nagy, G., & Frensch, P. A. (2009). Do recognition and priming index a unitary knowledge base? Comment on Shanks et al. (2003). Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 572–585. doi:10.1037/a0014543 19271870 · DOI
  46. Ryan, J. D., Althoff, R. R., Whitlow, S., & Cohen, N. J. (2000). Amnesia is a deficit in relational memory. Psychological Science, 11, 454–461. doi:10.1111/1467-9280.00288 11202489 · DOI
  47. Salthouse, T. A., McGuthry, K. E., & Hambrick, D. Z. (1999). A framework for analyzing and interpreting differential aging patterns: Application to three measures of implicit learning. Aging, Neuropsychology, and Cognition, 6, 1–18. doi:10.1076/anec.6.1.1.789 · DOI
  48. Sanchez DJ Gobel EW Reber PJ Performing the unexplainable: Implicit task performance reveals individually reliable sequence learning without explicit knowledge Psychonomic Bulletin & Review 2010 17 790 796 10.3758/PBR.17.6.790 21169570 PMC4341998 · DOI
  49. Sand, A., & Nilsson, M. E. (2016). Subliminal or not? Comparing null-hypothesis and Bayesian methods for testing subliminal priming. Consciousness and Cognition, 44, 29–40. doi:10.1016/j.concog.2016.06.012 27351780 · DOI
  50. Schmidt T Invisible stimuli, implicit thresholds: Why invisibility judgments cannot be interpreted in isolation Advances in Cognitive Psychology 2015 11 31 41 10.5709/acp-0169-3 26203311 PMC4510198 · DOI
  51. Schmidt T Vorberg D Criteria for unconscious cognition: Three types of dissociation Perception & Psychophysics 2006 68 489 504 10.3758/BF03193692 16900839 · DOI · Paper page
  52. Schultz DH Helmstetter FJ Classical conditioning of autonomic fear responses is independent of contingency awareness Journal of Experimental Psychology: Animal Behavior Processes 2010 36 495 500 20973611 PMC6510249
  53. Shanks DR Berry CJ Are there multiple memory systems? Tests of models of implicit and explicit memory Quarterly Journal of Experimental Psychology 2012 65 1449 1474 10.1080/17470218.2012.691887 22804727 · DOI
  54. Shanks DR Perruchet P Dissociation between priming and recognition in the expression of sequential knowledge Psychonomic Bulletin & Review 2002 9 362 367 10.3758/BF03196294 12120801 · DOI
  55. Shanks DR St. John MF Characteristics of dissociable human learning systems Behavioral and Brain Sciences 1994 17 367 395 10.1017/S0140525X00035032 · DOI · Paper page
  56. Shanks DR Wilkinson L Channon S Relationship between priming and recognition in deterministic and probabilistic sequence learning Journal of Experimental Psychology: Learning, Memory, and Cognition 2003 29 248 261 12696813
  57. Sklar, A. Y., Levy, N., Goldstein, A., Mandel, R., Maril, A., & Hassin, R. R. (2012). Reading and doing arithmetic nonconsciously. Proceedings of the National Academy of Sciences, 109, 19614–19619. doi:10.1073/pnas.1211645109 PMC3511729 23150541 · DOI
  58. Slotnick SD Schacter DL A sensory signature that distinguishes true from false memories Nature Neuroscience 2004 7 664 672 10.1038/nn1252 15156146 · DOI · Paper page
  59. Smyth AC Shanks DR Awareness in contextual cuing with extended and concurrent explicit tests Memory & Cognition 2008 36 403 415 10.3758/MC.36.2.403 18426069 · DOI · Paper page
  60. Stark, C. E. L., & McClelland, J. L. (2000). Repetition priming of words, pseudowords, and nonwords. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26, 945–972. doi:10.1037/0278-7393.26.4.945 10946373 · DOI
  61. Sweeny TD Grabowecky M Suzuki S Paller KA Long-lasting effects of subliminal affective priming from facial expressions Consciousness and Cognition 2009 18 929 938 10.1016/j.concog.2009.07.011 19695907 PMC2784103 · DOI
  62. ten Brinke L Stimson D Carney DR Some evidence for unconscious lie detection Psychological Science 2014 25 1098 1105 10.1177/0956797614524421 24659190 · DOI
  63. Vadillo MA Konstantinidis E Shanks DR Underpowered samples, false negatives, and unconscious learning Psychonomic Bulletin & Review 2016 23 1 87 102 10.3758/s13423-015-0892-6 26122896 PMC4742512 · DOI · Paper page
  64. Vuilleumier, P., Schwartz, S., Duhoux, S., Dolan, R. J., & Driver, J. (2005). Selective attention modulates neural substrates of repetition printing and “implicit” visual memory: Suppressions and enhancements revealed by fMRI. Journal of Cognitive Neuroscience, 17, 1245–1260. doi:10.1162/0898929055002409 16197681 · DOI
  65. Ward EV Berry CJ Shanks DR An effect of age on implicit memory that is not due to explicit contamination: Implications for single and multiple-systems theories Psychology and Aging 2013 28 2 429 442 10.1037/a0031888 23586355 · DOI
  66. Weiermann B Meier B Incidental sequence learning across the lifespan Cognition 2012 123 380 391 10.1016/j.cognition.2012.02.010 22425486 · DOI
  67. Weiskrantz L Blindsight: A case study and implications 1986 Oxford, UK Oxford University Press
  68. Williams, A. C. (1938). Perception of subliminal visual stimuli. Journal of Psychology, 6, 187–199. doi:10.1080/00223980.1938.9917596 · DOI
  69. Willingham, D. B., Nissen, M. J., & Bullemer, P. (1989). On the development of procedural knowledge. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 1047–1060. doi:10.1037/0278-7393.15.6.1047 2530305 · DOI
  70. Woltz, D. J., & Shute, V. J. (1993). Individual difference in repetition priming and its relationship to declarative knowledge acquisition. Intelligence, 17, 333–359. doi:10.1016/0160-2896(93)90004-O · DOI
  71. Zeelenberg R Plomp G Raaijmakers JGW Can false memories be created through nonconscious processes? Consciousness and Cognition 2003 12 403 412 10.1016/S1053-8100(03)00021-7 12941285 · DOI

Cited by

Sorted by citations

Authors
Shanks, D. R.
Affiliations
University College London (GB)
Corresponding author
Shanks, David R.
ORCID
Shanks, David R.
Venue
Psychonomic Bulletin & Review
Publisher
Springer Science and Business Media LLC
Published
2016-10-17
Volume
24
Issue
3
Pages
752-775
ISSN
  • 1069-9384 (ISSN-L)
  • 1531-5320
DOI
10.3758/s13423-016-1170-y
PMC
PMC5486877
OpenAlex
W2538938041
Cited by
201
References
5
Citations by year
2026: 14 · 2025: 28 · 2024: 27 · 2023: 21 · 2022: 27 · 2021: 22 · 2020: 15 · 2019: 15 · 2018: 22 · 2017: 9 · 2016: 1
FWCI
1.66
Type
journal article
Language
en
Open access
Licence
Locations
  • Psychonomic Bulletin & Review (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
Keywords
unconscious processes, regression to the mean, implicit learning, subliminal perception, unconscious cognition
MeSH terms
Awareness, Choice Behavior, Cognition, Humans, Learning, Memory, Mental Processes, Perception, Regression Analysis, Research Design, Subliminal Stimulation, Unconscious, Psychology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review