Suppressing the Search Engine Manipulation Effect (SEME)
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
A recent series of experiments demonstrated that introducing ranking bias to election-related search engine results can have a strong and undetectable influence on the preferences of undecided voters. This phenomenon, called the Search Engine Manipulation Effect (SEME), exerts influence largely through order effects that are enhanced in a digital context. We present data from three new experiments involving 3,600 subjects in 39 countries in which we replicate SEME and test design interventions for suppressing the effect. In the replication, voting preferences shifted by 39.0%, a number almost identical to the shift found in a previously published experiment (37.1%). Alerting users to the ranking bias reduced the shift to 22.1%, and more detailed alerts reduced it to 13.8%. Users' browsing behaviors were also significantly altered by the alerts, with more clicks and time going to lower-ranked search results. Although bias alerts were effective in suppressing SEME, we found that SEME could be completely eliminated only by alternating search results -- in effect, with an equal-time rule. We propose a browser extension capable of deploying bias alerts in real-time and speculate that SEME might be impacting a wide range of decision-making, not just voting, in which case search engines might need to be strictly regulated.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012) · Political Analysis · doi:10.1093/pan/mpr057 · Cited by 4,008
Experimental evidence of massive-scale emotional contagion through social networks
Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1320040111 · Cited by 3,687
Exposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245
Memory: A Contribution to Experimental Psychology
Ebbinghaus, H. (2013) · Annals of Neurosciences · doi:10.5214/ans.0972.7531.200408 · Cited by 2,827
A 61-million-person experiment in social influence and political mobilization
Bond, R. M., Fariss, C. J., Jones, J. J., Kramer, A. D. I., Marlow, C., Settle, J. E., & Fowler, J. H. (2012) · Nature · doi:10.1038/nature11421 · Cited by 2,575
Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market
Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006) · Science · doi:10.1126/science.1121066 · Cited by 2,231 · cited 2×
“Experiments conducted on online ranking algorithms have demonstrated their influence on users’ music preferences [111, 112], use of emotional language [69], beliefs about scientific controversy [92], and undecided voters’ preferences [30].”
BackgroundPrivacy in Context: Technology, Policy, and the Integrity of Social Life
Nissenbaum, H. (2009) · doi:10.1515/9780804772891 · Cited by 2,129 · cited 1×
“More broadly, our work adds to the growing literature that provides an empirical basis to calls for algorithm accountability and transparency [24, 25, 90, 91] and contributes a quantitative approach that complements the qualitative literature on designing interventions for ranking algorithms [33, 35, 36, 93].”
BackgroundFILTER BUBBLE: Wie wir im Internet entmündigt werden
Pariser, E. (2012) · doi:10.3139/9783446431164 · Cited by 1,849
The serial position effect of free recall.
Murdock, B. B. (1962) · Journal of Experimental Psychology · doi:10.1037/h0045106 · Cited by 1,717 · cited 1×
“1 Order Effects and Ranking Algorithms Order effects are among the strongest and most reliable effects ever discovered in the psychological sciences [29, 88].”
BackgroundMemory: A contribution to experimental psychology.
Ebbinghaus, H. (1913) · doi:10.1037/10011-000 · Cited by 1,645
Reputation as a sufficient condition for data quality on Amazon Mechanical Turk
Peer, E., Vosgerau, J., & Acquisti, A. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0434-y · Cited by 1,618
The Fox News Effect: Media Bias and Voting
DellaVigna, S., & Kaplan, E. (2007) · The Quarterly Journal of Economics · doi:10.1162/qjec.122.3.1187 · Cited by 1,602 · cited 1×
“2 Attitude Change and Online Systems Compared to newspaper readers and television viewers, search engine and social media users are more susceptible to influence [11, 21, 23, 28, 30, 44].”
BackgroundIto, T. A., Larsen, J. T., Smith, N. K., & Cacioppo, J. T. (1998) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.75.4.887 · Cited by 1,340
The Black Box Society: The Secret Algorithms That Control Money and Information
(2016) · Contemporary Sociology: A Journal of Reviews · doi:10.1177/0094306116641409c · Cited by 1,334
Inoculating the Public against Misinformation about Climate Change
van der Linden, S., Leiserowitz, A., Rosenthal, S., & Maibach, E. (2017) · Global Challenges · doi:10.1002/gch2.201600008 · Cited by 1,058 · cited 3×
“Consistent with the debiasing literature [39, 78, 117] we found that more detailed alerts (Figure 1c) suppressed SEME more than less detailed alerts (Figure 1b).”
BackgroundIdentifying influential and susceptible members of social networks
Aral, S., & Walker, D. (2012) · Science · doi:10.1126/science.1215842 · Cited by 1,047 · cited 1×
“As with previous research on SEME [30], and with research on attitude change and influence more generally [3, 72, 120], we found that subjects vary in their susceptibility to SEME, as well as in their responsiveness to the alerts, based on their personal characteristics (Figure 6 and Figure 7 in the Appendix).”
MethodGillespie, T. (2014) · Media Technologies · doi:10.7551/mitpress/9042.003.0013 · Cited by 1,016
Accountability in algorithmic decision making
Diakopoulos, N. (2016) · Communications of the ACM · doi:10.1145/2844110 · Cited by 763
In Google We Trust: Users’ Decisions on Rank, Position, and Relevance
Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2007.00351.x · Cited by 715 · cited 2×
“Highly ranked search results attract longer gaze durations [48, 50, 77] and receive the majority of clicks [108], even when superior results are present in lower ranked positions [60, 61, 95].”
BackgroundThe nature of psychological reactance revisited: A meta-analytic review
Rains, S. A. (2013) · Human Communication Research · doi:10.1111/j.1468-2958.2012.01443.x · Cited by 511
The Flexible Correction Model: The Role of Naive Theories of Bias in Bias Correction
Wegener, D. T., & Petty, R. E. (1997) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60017-9 · Cited by 471
Measuring price discrimination and steering on e-commerce web sites
Hannak, A., Soeller, G., Lazer, D., Mislove, A., & Wilson, C. (2014) · Proceedings of the 2014 Conference on Internet Measurement Conference · doi:10.1145/2663716.2663744 · Cited by 405 · cited 1×
“Algorithm audits have been used to examine the personalization of search engine rankings [53, 64], prices in online markets [17, 54, 80, 81], rating systems [37], and social media newsfeeds [33, 71].”
MethodSelf-citationUnderstanding User Beliefs About Algorithmic Curation in the Facebook News Feed
Rader, E., & Gray, R. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702174 · Cited by 386 · cited 1×
“The processes through which users become aware of and react to algorithms has been a topic of recent research interest [33–35, 37, 106].”
BackgroundFirst I "like" it, then I hide it: Folk Theories of Social Feeds
Eslami, M., Karahalios, K., Sandvig, C., Vaccaro, K., Rickman, A., Hamilton, K., & Kirlik, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858494 · Cited by 375
Accountability in a computerized society
Nissenbaum, H. (1996) · Science and Engineering Ethics · doi:10.1007/bf02639315 · Cited by 346
Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr
Hannák, A., Wagner, C., Garcia, D., Mislove, A., Strohmaier, M., & Wilson, C. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998327 · Cited by 286 · cited 2×
“As regulators and academics have noted, the unregulated use of such technologies may lead to detrimental outcomes for users [15, 27, 30, 55, 101, 113, 124], and our results suggest that deploying external design interventions could mitigate such outcomes while legislation takes shape.”
BackgroundSelf-citationQuantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media
Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998321 · Cited by 148 · cited 4×
“It terms of Kulshrestha et al’s formulas [71], we found a ranking bias of -2.”
BackgroundMethodInfluentialRest in peace? Brand-induced mortality salience and consumer behavior
Fransen, M. L., Fennis, B. M., Pruyn, A. T. H., & Das, E. (2008) · Journal of Business Research · doi:10.1016/j.jbusres.2007.09.020 · Cited by 100 · cited 2×
“For example, online systems can: (1) provide a platform for constant, large-scale, rapid experimentation [66], (2) tailor their persuasive strategies by mining detailed demographic and behavioral profiles of users [1, 6, 9, 18, 121], and (3) provide users with a sense of control over the system that enhances their sus…”
BackgroundPark, S., Ko, M., Kim, J., Liu, Y., & Song, J. (2011) · Proceedings of the ACM 2011 conference on Computer supported cooperative work · doi:10.1145/1958824.1958842 · Cited by 88
Crowd-assisted search for price discrimination in e-commerce: first results
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2013) · Proceedings of the ninth ACM conference on Emerging networking experiments and technologies · doi:10.1145/2535372.2535415 · Cited by 82
Cited by
Auditing Partisan Audience Bias within Google Search
Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274417 · Cited by 209
Self-citationProblematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
Users choose to engage with more partisan news than they are exposed to on Google Search
Robertson, R. E., Green, J., Ruck, D. J., Ognyanova, K., Wilson, C., & Lazer, D. (2023) · Nature · doi:10.1038/s41586-023-06078-5 · Cited by 126
Self-citationJiang, S., & Wilson, C. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274351 · Cited by 125
Self-citationQuantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., & Wilson, C. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317595 · Cited by 92
Self-citation
Metadata
- Authors
- Epstein, R., Robertson, R. E., Lazer, D., & Wilson, C.
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2017-12-06
- Volume
- 1
- Issue
- CSCW
- Pages
- 1-22
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W2774114807
- Cited by
- 86
- References
- 30
- FWCI
- 14.26
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.