A Classification of Feedback Loops and Their Relation to Biases in Automated Decision-Making Systems
Open accessAbstract
Prediction-based decision-making systems are becoming increasingly prevalent in various domains. Previous studies have demonstrated that such systems are vulnerable to runaway feedback loops, e.g., when police are repeatedly sent back to the same neighborhoods regardless of the actual rate of criminal activity, which exacerbate existing biases. In practice, the automated decisions have dynamic feedback effects on the system itself that can perpetuate over time, making it difficult for short-sighted design choices to control the system's evolution. While researchers started proposing longer-term solutions to prevent adverse outcomes (such as bias towards certain groups), these interventions largely depend on ad hoc modeling assumptions and a rigorous theoretical understanding of the feedback dynamics in ML-based decision-making systems is currently missing. In this paper, we use the language of dynamical systems theory, a branch of applied mathematics that deals with the analysis of the interconnection of systems with dynamic behaviors, to rigorously classify the different types of feedback loops in the ML-based decision-making pipeline. By reviewing existing scholarly work, we show that this classification covers many examples discussed in the algorithmic fairness community, thereby providing a unifying and principled framework to study feedback loops. By qualitative analysis, and through a simulation example of recommender systems, we show which specific types of ML biases are affected by each type of feedback loop. We find that the existence of feedback loops in the ML-based decision-making pipeline can perpetuate, reinforce, or even reduce ML biases.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
These feedback loops, but also deliberate product design choices along with policy approaches, provide potent leverage points for aligning recommender systems with human values.
“These feedback loops, but also deliberate product design choices along with policy approaches, provide potent leverage points for aligning recommender systems with human values 146 .”
From Machine culture (2023) · cites this paper as [146]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812 · see citing works
A survey on concept drift adaptation
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014) · ACM Computing Surveys · doi:10.1145/2523813 · Cited by 3,735 · see citing works
“Clinical versus Actuarial Judgment.”
Dawes, R. M., Faust, D., & Meehl, P. E. (1989) · Science · doi:10.1126/science.2648573 · Cited by 2,298 · see citing works
Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194 · see citing works
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149 · see citing works
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · see citing works
Ramaprasad, A. (1983) · Behavioral Science · doi:10.1002/bs.3830280103 · Cited by 915 · see citing works
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019) · Frontiers in Big Data · doi:10.3389/fdata.2019.00013 · Cited by 763 · see citing works
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687 · see citing works
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
Lum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611 · see citing works
A Review on Fairness in Machine Learning
Pessach, D., & Shmueli, E. (2023) · ACM Computing Surveys · doi:10.1145/3494672 · Cited by 598 · see citing works
Predictably Unequal? The Effects of Machine Learning on Credit Markets
FUSTER, A., GOLDSMITH‐PINKHAM, P., RAMADORAI, T., & WALTHER, A. (2022) · The Journal of Finance · doi:10.1111/jofi.13090 · Cited by 548 · see citing works
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545 · see citing works
Webb, G. I., Hyde, R., Cao, H., Nguyen, H. L., & Petitjean, F. (2016) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-015-0448-4 · Cited by 541 · see citing works
The Frontiers of Fairness in Machine LearningPreprint
Chouldechova, A., & Roth, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08810 · Cited by 372 · see citing works
A snapshot of the frontiers of fairness in machine learning
Chouldechova, A., & Roth, A. (2020) · Communications of the ACM · doi:10.1145/3376898 · Cited by 362 · see citing works
How algorithmic confounding in recommendation systems increases homogeneity and decreases utility
Chaney, A. J. B., Stewart, B., & Engelhardt, B. E. (2018) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3240323.3240370 · Cited by 345 · see citing works
Learning in the Presence of Malicious Errors
Kearns, M. J., & Li, M. (1993) · SIAM Journal on Computing · doi:10.1137/0222052 · Cited by 340 · see citing works
Kleinberg, J., Ludwig, J., Mullainathan, S., & Rambachan, A. (2018) · AEA Papers and Proceedings · doi:10.1257/pandp.20181018 · Cited by 278 · see citing works
On the (im)possibility of fairness
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · arXiv · doi:10.48550/arxiv.1609.07236 · Cited by 247 · see citing works
Hardt, M., Megiddo, N., Papadimitriou, C. H., & Wootters, M. K. (2016) · Conference on Innovations in Theoretical Computer Science · doi:10.1145/2840728.2840730 · Cited by 238 · see citing works
van Giffen, B., Herhausen, D., & Fahse, T. (2022) · Journal of Business Research · doi:10.1016/j.jbusres.2022.01.076 · Cited by 230 · see citing works
Vorobeychik, Y., & Kantarcıoğlu, M. (2018) · Synthesis lectures on artificial intelligence and machine learning · doi:10.1007/978-3-031-01580-9 · Cited by 158 · see citing works
The Social Cost of Strategic Classification
Milli, S., Miller, J. P., Dragan, A. D., & Hardt, M. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287576 · Cited by 140 · see citing works
The Disparate Effects of Strategic Manipulation
Hu, L. J., Immorlica, N., & Vaughan, J. W. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287597 · Cited by 135 · see citing works
A Short-term Intervention for Long-term Fairness in the Labor Market
Hu, L. J., & Chen, Y. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186044 · Cited by 133 · see citing works
Robust Linear Regression Against Training Data Poisoning
Liu, C., Li, B., Vorobeychik, Y., & Oprea, A. (2017) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3128572.3140447 · Cited by 84 · see citing works
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82 · see citing works
From Fair Decision Making To Social Equality
Mouzannar, H., Ohannessian, M. I., & Srebro, N. (2019) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3287560.3287599 · Cited by 82 · see citing works
Optimal Transport in Systems and Control
Chen, Y., Georgiou, T. T., & Pavon, M. (2020) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-070220-100858 · Cited by 78 · see citing works
Fair Algorithms for Learning in Allocation Problems
Elzayn, H., Jabbari, S., Jung, C., Kearns, M., Neel, S., Roth, A., & Schutzman, Z. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287571 · Cited by 70 · see citing works
On the Moral Justification of Statistical Parity
Hertweck, C., Heitz, C., & Loi, M. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445936 · Cited by 49 · see citing works
The disparate equilibria of algorithmic decision making when individuals invest rationally
Liu, L. T., Wilson, A., Haghtalab, N., Kalai, A. T., Borgs, C., & Chayes, J. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372861 · Cited by 47 · see citing works
The Closed Loop Between Opinion Formation and Personalized Recommendations
Rossi, W. S., Polderman, J. W., & Frasca, P. (2021) · IEEE Transactions on Control of Network Systems · doi:10.1109/tcns.2021.3105616 · Cited by 46 · see citing works
A Broader View on Bias in Automated Decision-Making: Reflecting on Epistemology and DynamicsPreprint
Dobbe, R., Dean, S., Gilbert, T. K., & Kohli, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00553 · Cited by 36 · see citing works
Enforcing Group Fairness in Algorithmic Decision Making: Utility Maximization Under Sufficiency
Baumann, J., Hannák, A., & Heitz, C. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3534645 · Cited by 18 · see citing works
Self-citationLong-Term Impacts of Fair Machine Learning
Zhang, X., Khalili, M. M., & Liu, M. (2019) · Ergonomics in Design The Quarterly of Human Factors Applications · doi:10.1177/1064804619884160 · Cited by 14 · see citing works
Performative PowerPreprint
Hardt, M., Jagadeesan, M., & Mendler-Dünner, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.17232 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 33
- Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica, May 23, 2016 (2016), 139-159. https://www.propublica. org/article/machine-bias-risk-assessments-in-criminal-sentencing
- Karl Johan Åström and Richard M Murray. 2021. Feedback systems: an introduction for scientists and engineers. Princeton university press.
- Yahav Bechavod, Katrina Ligett, Aaron Roth, Bo Waggoner, and Zhiwei Steven Wu. 2019. Equal opportunity in online classification with partial feedback. Advances in Neural Information Processing Systems 32, NeurIPS (2019).
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A Friedler and Christo Wilson (Eds.). PMLR, 77-91. https://proceedings.mlr.press/v81/buolamwini18a.html
- Silvia Chiappa, Ray Jiang, Tom Stepleton, Aldo Pacchiano, Heinrich Jiang, and John Aslanides. 2020. A General Approach to Fairness with Optimal Transport. The 34th AAAI Conference on Artificial Intelligence (2020). https://ojs.aaai.org/index.php/AAAI/article/view/5771
- Kate Crawford. 2016. Artificial intelligence's white guy problem. The New York Times 25, 06 (2016).
- Alexander D'Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D. Sculley, and Yoni Halpern. 2020. Fairness is not static: Deeper understanding of long term fairness via simulation studies. FAT* 2020 -Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020), 525-534. https://doi.org/10.1145/3351095.3372878 · DOI
- Matthew Ellis, Helen Durand, and Panagiotis D. Christofides. 2014. A tutorial review of economic model predictive control methods. Journal of Process Control 24, 8 (2014). http://dx.doi.org/10.1016/j.jprocont.2014.03.01 · DOI
- Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, Suresh Venkatasubramanian, Mehryar Mohri, and Karthik Sridharan. 2018. Decision making with limited feedback: Error bounds for predictive policing and recidivism prediction. Proceedings of Machine Learning Research 83 (2018), 1-9.
- Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, Suresh Venkatasubramanian, and Christo Wilson. 2018. Runaway Feedback Loops in Predictive Policing. In Proceedings of Machine Learning Research, Vol. 81. 1-12. https://github.com/algofairness/
- Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NIPS'16).
- Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness Without Demographics in Repeated Loss Minimization. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 1929-1938. https://proceedings.mlr.press/v80/hashimoto18a.html
- Hoda Heidari, Vedant Nanda, and Krishna P. Gummadi. 2019. On the Long-term Impact of Algorithmic Decision Policies: Effort unfairness and feature segregation through social learning. 36th International Conference on Machine Learning, ICML 2019 2019-June (2019), 4787-4796.
- Sterman John D. 2000. Business Dynamics: Systems Thinking and Modeling for a Complex World. McGraw-Hill Education.
- Michael Kearns and Aaron Roth. 2019. The Ethical Algorithm: The Science of Socially Aware Algorithm Design. Oxford University Press, Inc., USA.
- Jon Kleinberg and Manish Raghavan. 2020. How Do Classifiers Induce Agents to Invest Effort Strategically? ACM Transactions on Economics and Computation 8, 4 (2020). https://doi.org/10.1145/3417742 · DOI
- Frank L Lewis, Draguna Vrabie, and Vassilis L Syrmos. 2012. Optimal control. John Wiley & Sons.
- Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed Impact of Fair Machine Learning. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 3150-3158. https://proceedings.mlr.press/v80/liu18c.html
- Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed Impact of Fair Machine Learning. In Proceedings of the 35th International Conference on Machine Learning. https://proceedings.mlr.press/v80/liu18c.html
- Nataša Obermajer, Ravikumar Muthuswamy, Jamie Lesnock, Robert P Edwards, and Pawel Kalinski. 1983. Positive feedback between PGE$_2$ and COX2 redirects the differentiation of human dendritic cells toward stable myeloid-derived suppressor cells. Immunobiology 119, 20 (1983). https://doi.org/10.1182/blood-2011-07-365825 · DOI
- Juan C. Perdomo, Tijana Zrnic, Celestine Mendler-Dunner, and Moritz Hardt. 2020. Performative prediction. 37th International Conference on Machine Learning, ICML 2020 PartF16814 (2020), 7555-7565.
- Nicola Perra and Luis E C Rocha. 2019. Modelling opinion dynamics in the age of algorithmic personalisation. Scientific reports 9, 1 (2019), 1-11.
- Tom Simonite. 2015. Probing the dark side of google's ad-targeting system. MIT Technology Review (2015).
- Ayan Sinha, David F Gleich, and Karthik Ramani. 2016. Deconvolving Feedback Loops in Recommender Systems. In Advances in Neural Information Processing Systems, D Lee, M Sugiyama, U Luxburg, I Guyon, and R Garnett (Eds.), Vol. 29. Curran Associates, Inc. https://proceedings.neurips.cc/ paper/2016/file/962e56a8a0b0420d87272a682bfd1e53-Paper.pdf
- Yi Sun. 2022. Algorithmic Fairness in Sequential Decision Making. Ph. D. Dissertation.
- Yi Sun, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. 2022. The Backfire Effects of Fairness Constraints. ICML 2022 Workshop on Responsible Decision Making in Dynamic Environments (2022). https://responsibledecisionmaking.github.io/assets/pdf/papers/44.pdf
- Stratis Tsirtsis, Behzad Tabibian, Moein Khajehnejad, Adish Singla, Bernhard Schölkopf, and Manuel Gomez-Rodriguez. 2019. Optimal Decision Making Under Strategic Behavior. (2019). http://arxiv.org/abs/1905.09239 · arXiv
- Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. 2021. To be Robust or to be Fair: Towards Fairness in Adversarial Training. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 11492-11501. https://proceedings.mlr.press/v139/xu21b.html
- Bernard P Zeigler, Tag Gon Kim, and Herbert Praehofer. 2000. Theory of modeling and simulation. Academic press.
- Xueru Zhang, Mohammad Mahdi Khalili, Cem Tekin, and Mingyan Liu. 2019. Group retention when using machine learning in sequential decision making: The interplay between user dynamics and fairness. Advances in Neural Information Processing Systems 32, NeurIPS (2019).
- Xueru Zhang and Mingyan Liu. 2021. Fairness in Learning-Based Sequential Decision Algorithms: A Survey. Studies in Systems, Decision and Control 325 (2021), 525-555. https://doi.org/10.1007/978-3-030-60990-0{_}18 · DOI
- Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellström, Kun Zhang, and Cheng Zhang. 2020. How do fair decisions fare in long-term qualification? Advances in Neural Information Processing Systems 2020-Decem, NeurIPS (2020), 1-13.
- Kemin Zhou and John Comstock Doyle. 1998. Essentials of robust control. Vol. 104. Prentice hall Upper Saddle River, NJ.
Cited by
Sorted by citations
Brinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132 · see citing works
- Authors
- Pagan, N., Baumann, J., Elokda, E., De Pasquale, G., Bolognani, S., & Hannák, A.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2305.06055
- OpenAlex
- W4376167270
- Cited by
- 4
- References
- 39
- Citations by year
- 2024: 2 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
- arXiv (Cornell University)
- ETH Zürich Research Collection (cc-by)
- Funders
- National Science Foundation (180545)
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (180545)
- Eidgenössische Technische Hochschule Zürich
- Universität Zürich
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Keywords
- feedback loops, algorithmic bias, automated decision-making, dynamical systems theory, algorithmic fairness, recommender systems, bias mitigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
