Algorithmic transparency and assessing effects of algorithmic ranking
Open accessAbstract
1. At established platforms, algorithmic ranking and recommendation involve using many signals and are typically not aimed at simply maximizing short-run engagement.2. Quantifying the impacts of algorithmic ranking is quite difficult,even with access to proprietary data. This is not only becauseof the complexity of these technical systems, but due to people’scomplex and often strategic responses to changes in algorithms.3. We lack clear evidence about broader benefits or harms of algorithmic ranking. Nonetheless, simple rankings and recommendations (e.g., chronological, overall popularity) can make some formsof undesirable strategic behavior easier.4. Policy-makers can protect the ability of external researchers toprobe these systems, and they can provide clear paths for platforms to retain and share data in privacy-preserving ways.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Engagement, user satisfaction, and news originality are key signals used by the Facebook feed algorithm to determine how to rank content for individual users.
“Engagement, user satisfac-tion, and news originality are key signals used by the Facebook feed algorithm to determine how to rank content for individual users [more publicly available information is available in (20, 39, 40, 41)] (SM section S1.1).”
From How do social media feed algorithms affect attitudes and behavior in an election campaign? (2023) · cites this paper as [20]
The Current practice section describes ranking based on predicted user actions, such as likes and comments, and says Facebook uses feed-quality ratings that can inform its algorithm. This supports engagement and user satisfaction as ranking signals, but our copy does not identify news originality as one. (LLM, conf=0.86)
Experimental — accuracy not yet measuredA chronological feed does not exhibit the black-box, engagement-optimization, or learning features because the same rule is applied to all users at all times.
“By contrast, a chronological feed does not ex-hibit any of these features because the same rule is applied to all users at all times (20).”
From How do social media feed algorithms affect attitudes and behavior in an election campaign? (2023) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · see citing works
(2018) · Cited by 3,510 · see citing works
Exposure to ideologically diverse news and opinion on Facebook
Bakshy, E., Messing, S., & Adamic, L. A. (2015) · Science · doi:10.1126/science.aaa1160 · Cited by 3,245 · see citing works
A 61-million-person experiment in social influence and political mobilization
Bond, R. M., Fariss, C. J., Jones, J. J., Kramer, A. D. I., Marlow, C., Settle, J. E., & Fowler, J. H. (2012) · Nature · doi:10.1038/nature11421 · Cited by 2,575 · see citing works
Exposure to opposing views on social media can increase political polarization
Bail, C. A., Argyle, L. P., Brown, T. W., Bumpus, J. P., Chen, H., Hunzaker, M. B. F., Lee, J., Mann, M., Merhout, F., & Volfovsky, A. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1804840115 · Cited by 1,863 · see citing works
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364 · see citing works
Shifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155 · see citing works
Self-citationRacial Discrimination in the Sharing Economy: Evidence from a Field Experiment
Edelman, B. G., Luca, M., & Svirsky, D. (2015) · American Economic Journal Applied Economics · doi:10.1257/app.20160213 · Cited by 982 · see citing works
The welfare effects of social media
Allcott, H., Braghieri, L., Eichmeyer, S., & Gentzkow, M. (2020) · American Economic Review · doi:10.1257/aer.20190658 · Cited by 857 · see citing works
Social Media, News Consumption, and Polarization: Evidence from a Field Experiment
Levy, R. (2021) · American Economic Review · doi:10.1257/aer.20191777 · Cited by 624 · see citing works
Evaluating the fake news problem at the scale of the information ecosystem
Allen, J., Howland, B., Mobius, M., Rothschild, D., & Watts, D. J. (2020) · Science Advances · doi:10.1126/sciadv.aay3539 · Cited by 528 · see citing works
Quantifying the invisible audience in social networks
Bernstein, M. S., Bakshy, E., Burke, M., & Karrer, B. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2470658 · Cited by 357 · see citing works
Exercise contagion in a global social network
Aral, S., & Nicolaides, C. (2017) · Nature Communications · doi:10.1038/ncomms14753 · Cited by 329 · see citing works
Constrained Bayesian Optimization with Noisy Experiments
Letham, B., Karrer, B., Ottoni, G. D. L., & Bakshy, E. (2018) · Bayesian Analysis · doi:10.1214/18-ba1110 · Cited by 312 · see citing works
Auditing Algorithms: Understanding Algorithmic Systems from the Outside In
Metaxa, D., Park, J., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021) · Found. Trends Hum. Comput. Interact. · doi:10.1561/1100000083 · Cited by 218 · see citing works
Using massive online choice experiments to measure changes in well-being
Brynjolfsson, E., Collis, A., & Eggers, F. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1815663116 · Cited by 211 · see citing works
Protecting elections from social media manipulation
Aral, S., & Eckles, D. (2019) · Science · doi:10.1126/science.aaw8243 · Cited by 209 · see citing works
Self-citationInterdependence and the cost of uncoordinated responses to COVID-19
Holtz, D., Zhao, M., Benzell, S. G., Cao, C. Y., Rahimian, M. A., Yang, J., Allen, J. N. L., Collis, A., Moehring, A. V., Sowrirajan, T., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2009522117 · Cited by 191 · see citing works
Self-citationDesigning and deploying online field experiments
Bakshy, E., Eckles, D., & Bernstein, M. S. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2567967 · Cited by 181 · see citing works
Self-citationEstimating peer effects in networks with peer encouragement designs
Eckles, D., Kizilcec, R. F., & Bakshy, E. (2016) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1511201113 · Cited by 141 · see citing works
Self-citationTop Challenges from the first Practical Online Controlled Experiments Summit
Gupta, S., Kohavi, R., Tang, D., XU, Y. N., Andersen, R., Bakshy, E., Cardin, N. J., Chandran, S., Chen, N., Coey, D., et al. (2019) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3331651.3331655 · Cited by 140 · see citing works
Practical Diversified Recommendations on YouTube with Determinantal Point Processes
Wilhelm, M., Ramanathan, A., Bonomo, A., Jain, S. C., Ed, H., & Gillenwater, J. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3272018 · Cited by 130 · see citing works
Jones, J. J., Bond, R. M., Bakshy, E., Eckles, D., & Fowler, J. H. (2017) · PLoS ONE · doi:10.1371/journal.pone.0173851 · Cited by 127 · see citing works
Self-citationCuration Algorithms and Filter Bubbles in Social Networks
Berman, R., & Katona, Z. (2020) · Marketing Science · doi:10.1287/mksc.2019.1208 · Cited by 124 · see citing works
Discrimination through Optimization: How Facebook's Ad Delivery Can Lead to Biased OutcomesPreprint
Ali, M., Sapieżyński, P., Bogen, M., Korolova, A., Mislove, A. E., & Rieke, A. (2019) · arXiv (Cornell University) · Cited by 116 · see citing works
Singh, I., & Singh, S. (2021) · Business and Society Review · doi:10.1111/basr.12225 · Cited by 112 · see citing works
Images and Misinformation in Political Groups: Evidence from WhatsApp in India
Garimella, K., & Eckles, D. (2020) · Harvard Kennedy School Misinformation Review · doi:10.37016/mr-2020-030 · Cited by 111 · see citing works
Self-citationBreaking the Social Media Prism: How to Make Our Platforms Less Polarizing
Bail, C. (2021) · doi:10.1515/9780691216508 · Cited by 86 · see citing works
Neutral bots probe political bias on social media
Chen, W., Pacheco, D. F., Yang, K.-C., & Menczer, F. (2021) · RePEc: Research Papers in Economics · Cited by 68 · see citing works
Algorithmic Amplification of Politics on Twitter
Huszár, F., Ktena, S. I., O’Brien, C., Belli, L., Schlaikjer, A., & Hardt, M. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2025334119 · Cited by 65 · see citing works
Two Cheers for Corporate Experimentation: The A/B Illusion and the Virtues of Data-Driven Innovation
Meyer, M. N. (2015) · Colorado Law Scholarly Commons (University of Colorado Colorado Springs) · Cited by 63 · see citing works
Brinberg, M., Ram, N., Yang, X., Cho, M.-J., Sundar, S. S., Robinson, T. N., & Reeves, B. B. (2020) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106570 · Cited by 58 · see citing works
Breaking the Social Media Prism
Bail, C. A. (2022) · Princeton University Press eBooks · doi:10.1515/9780691246499 · Cited by 58 · see citing works
Design and Analysis of Experiments in Networks: Reducing Bias from Interference
Eckles, D., Karrer, B., & Ugander, J. (2017) · Journal of Causal Inference · doi:10.1515/jci-2015-0021 · Cited by 49 · see citing works
Self-citationAral, S. (2021) · Cited by 49 · see citing works
Curating Quality? How Twitter’s Timeline Algorithm Treats Different Types of News
Bandy, J., & Diakopoulos, N. A. (2021) · Social Media + Society · doi:10.1177/20563051211041648 · Cited by 41 · see citing works
Trend Alert: A Cross-Platform Organization Manipulated Twitter Trends in the Indian General Election
Jakesch, M., Garimella, K., Eckles, D., & Naaman, M. (2021) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3479523 · Cited by 39 · see citing works
Self-citationUsing Massive Online Choice Experiments to Measure Changes in Well-BeingPreprint
Brynjolfsson, E., Collis, A., & Eggers, F. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3163559 · Cited by 33 · see citing works
Learning Causal Effects From Many Randomized Experiments Using Regularized Instrumental Variables
Peysakhovich, A., & Eckles, D. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186151 · Cited by 22 · see citing works
Self-citationAuditing Algorithms: Understanding Algorithmic Systems from the Outside In
Metaxa, D., Park, J.-S., Robertson, R. E., Karahalios, K. G., Wilson, C., Hancock, J. T., & Sandvig, C. (2021) · doi:10.1561/9781680839173 · Cited by 21 · see citing works
What a Hybrid Legal-Technical Analysis Teaches Us About Privacy Regulation: The Case of Singling OutPreprint
Altman, M., Cohen, A., Kobbi Nissim, B., & Wood, A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3681729 · Cited by 21 · see citing works
Network Experimentation at Scale
Karrer, B., Shi, L., Bhole, M., Goldman, M., Palmer, T., Gelman, C., Konutgan, M., & Sun, F. J. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467091 · Cited by 20 · see citing works
Reducing Interference Bias in Online Marketplace Pricing ExperimentsPreprint
Holtz, D., Lobel, R., Liskovich, I., & Aral, S. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3583836 · Cited by 19 · see citing works
An Experimental Study of Structural Diversity in Social Networks
Lasky-Su, J. A., Kamath, K. Y., Sharma, A., Ugander, J. H., & Goel, S. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7332 · Cited by 19 · see citing works
Serving a Billion Personalized News Feeds
Bäckström, L. (2016) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/2835776.2835848 · Cited by 15 · see citing works
Notification Volume Control and Optimization System at Pinterest
Zhao, B., Narita, K., Orten, B., & Egan, J. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219906 · Cited by 14 · see citing works
Algorithmic Amplification of Politics on Twitter
Huszár, F., Ktena, S. I., O'Brien, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.11010
Reducing Interference Bias in Online Marketplace Pricing Experiments
Holtz, D., Lobel, R., Liskovich, I., & Aral, S. (2020) · arXiv · doi:10.48550/arxiv.2004.12489
Flagged first, then by citations
Research works printed but not linked: 5
- Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.
- Obeng, A. & Bakshy, E. (2020). Preference learning for real-world multi-objective decision making. In ICML 2020 Workshop on Real World Experiment Design and Active Learning.
- Hosanagar, K. & Miller, A. P. (2020). Who do we blame for the filter bubble? On the roles of math, data, and people in algorithmic social systems. In K. Werbach (Ed.), After the Digital Tornado: Networks, Algorithms, Humanity (pp. 103-121). Cambridge University Press.
- Messing, S. (2013). Friends that matter: How social transmission of elite discourse shapes political knowledge, attitudes, and behavior. Doctoral dissertation, Chapter 7. Stanford Uni- versity.
- Keller, D. (2021). Amplification and its discontents: Why regulating the reach of online con- tent is hard. Journal of Free Speech Law, 1, 227-268.
Cited by
Sorted by citations
How do social media feed algorithms affect attitudes and behavior in an election campaign?
Guess, A. M., Malhotra, N., Pan, J., Barberá, P., Allcott, H., Brown, T., Crespo-Tenorio, A., Dimmery, D., Freelon, D., Gentzkow, M., et al. (2023) · Science · doi:10.1126/science.abp9364 · Cited by 363 · see citing works
People think that social media platforms do (but should not) amplify divisive content
Rathje, S., Robertson, C., Brady, W. J., & Van Bavel, J. J. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190392 · Cited by 59 · see citing works
Embedding Societal Values into Social Media Algorithms
Bernstein, M., Christin, A., Hancock, J., Hashimoto, T., Jia, C., Lam, M., Meister, N., Persily, N., Piccardi, T., Saveski, M., et al. (2023) · Journal of Online Trust and Safety · doi:10.54501/jots.v2i1.148 · Cited by 31 · see citing works
Embedding Democratic Values into Social Media AIs via Societal Objective Functions
Jia, C., Lam, M. S., Mai, M. C., Hancock, J. T., & Bernstein, M. S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641002 · Cited by 31 · see citing works
Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
Lam, M. S., Ma, Z., Li, A., Freitas, I., Wang, D., Landay, J. A., & Bernstein, M. S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581290 · Cited by 25 · see citing works
Moehring, A. V. (2026) · Management Science · doi:10.1287/mnsc.2024.06298 · Cited by 4 · see citing works
A survey on secured internet of things architecture
Harita, M. U., & Sagar, K. V. D. (2018) · International Journal of Engineering & Technology · doi:10.14419/ijet.v7i2.7.10596 · Cited by 4 · see citing works
Solutions and Challenges for Addressing Misinformation
Martel, C., & Rand, D. G. (2025) · Journal of Public Policy & Marketing · doi:10.1177/07439156251352415 · Cited by 3 · see citing works
Algorithmic Audiences: Navigating Identity, Influence, and Power in the Age of Platformized Media
Rustamova, N. R. (2025) · International journal of mass communication. · doi:10.6000/2818-3401.2025.03.09 · Cited by 1 · see citing works
Value misalignment in X’s feed algorithm is a reflection of value tensions in engagement
Epstein, Z. G., Jahanbakhsh, F., Piccardi, T., Peytavin, A., Gallegos, I., Sapkota, S., Zhao, D., Ugander, J. H., & Bernstein, M. S. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2610388123 · Cited by 1 · see citing works
Designing Pandora's Box: Product Rankings and Marketplace ParticipationPreprint
Papanastasiou, Y., & Sekar, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5006666
Algorithmic systems, human agency and the future of platform research
Hosseinmardi, H., Dutta, U., Rothschild, D. M., & Watts, D. J. (2026) · Nature Computational Science · doi:10.1038/s43588-026-01038-1
- Authors
- Eckles, D.
- Corresponding author
- Eckles, Dean
- ORCID
- Eckles, Dean
- Publisher
- Center for Open Science
- Published
- 2022-04-07
- OpenAlex
- W4226353236
- Cited by
- 12
- References
- 47
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 2 · 2023: 4 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SocArXiv (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- algorithmic transparency, algorithmic ranking, recommender systems, algorithmic impact assessment, strategic behavior, privacy-preserving data sharing, platform regulation, engagement optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
