Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 12

The DOI matches its DataCite record; 5 of the 5 references it cites were found in a registry or an index.

Data Shapley: Equitable Valuation of Data for Machine Learning

Ghorbani, Amirata, Zou, James·2019·arXiv
DOI 10.48550/arxiv.1904.02868arXiv 1904.02868
Open this paper in CoChat — free account

Abstract

As data becomes the fuel driving technological and economic growth, a fundamental challenge is how to quantify the value of data in algorithmic predictions and decisions. For example, in healthcare and consumer markets, it has been suggested that individuals should be compensated for the data that they generate, but it is not clear what is an equitable valuation for individual data. In this work, we develop a principled framework to address data valuation in the context of supervised machine learning. Given a learning algorithm trained on $n$ data points to produce a predictor, we propose data Shapley as a metric to quantify the value of each training datum to the predictor performance. Data Shapley value uniquely satisfies several natural properties of equitable data valuation. We develop Monte Carlo and gradient-based methods to efficiently estimate data Shapley values in practical settings where complex learning algorithms, including neural networks, are trained on large datasets. In addition to being equitable, extensive experiments across biomedical, image and synthetic data demonstrate that data Shapley has several other benefits: 1) it is more powerful than the popular leave-one-out or leverage score in providing insight on what data is more valuable for a given learning task; 2) low Shapley value data effectively capture outliers and corruptions; 3) high Shapley value data inform what type of new data to acquire to improve the predictor.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1904.02868

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 25

  1. Posner, E. A. & Weyl, E. G. Radical Markets: Uprooting Capitalism and Democracy for a Just Society (Princeton University Press, 2018).
  2. Shapley, L. S., Roth, A. E. et al. The Shapley value: essays in honor of Lloyd S. Shapley (Cambridge University Press, 1988).
  3. Maleki, S., Tran-Thanh, L., Hines, G., Rahwan, T. & Rogers, A. Bounding the estimation error of sampling-based shapley value approximation. arXiv preprint arXiv:1306.4265 (2013). · arXiv
  4. Mahajan, D. et al. Exploring the limits of weakly supervised pretraining. arXiv preprint arXiv:1805.00932 (2018). · arXiv
  5. Frénay, B. & Verleysen, M. Classification in the presence of label noise: a survey. IEEE transactions on neural networks learning systems 25, 845-869 (2014).
  6. Steinhardt, J., Koh, P. W. W. & Liang, P. S. Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems, 3517-3529 (2017).
  7. Torralba, A., Efros, A. A. et al. Unbiased look at dataset bias. In CVPR, vol. 1, 7 (Citeseer, 2011).
  8. Gebru, T., Hoffman, J. & Fei-Fei, L. Fine-grained recognition in the wild: A multi-task domain adaptation approach. In Proceedings of the IEEE International Conference on Computer Vision, 1349-1358 (2017).
  9. Tschandl, P., Rosendahl, C. & Kittler, H. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. data 5, 180161 (2018).
  10. Buolamwini, J. & Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency, 77-91 (2018).
  11. Kim, M. P., Ghorbani, A. & Zou, J. Multiaccuracy: Black-box post-processing for fairness in classification. arXiv preprint arXiv:1805.12317 (2018). · arXiv
  12. Szegedy, C., Ioffe, S., Vanhoucke, V. & Alemi, A. A. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, vol. 4, 12 (2017).
  13. Liu, Z., Luo, P., Wang, X. & Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, 3730-3738 (2015).
  14. Wolf, L., Hassner, T. & Taigman, Y. Effective unconstrained face recognition by combining multiple descriptors and learned background statistics. IEEE transactions on pattern analysis machine intelligence 33, 1978-1990 (2011).
  15. Nie, A. et al. Deeptag: inferring diagnoses from veterinary clinical notes. npj Digit. Medicine 1, 60 (2018).
  16. Hull, J. J. A database for handwritten text recognition research. IEEE Transactions on pattern analysis machine intelligence 16, 550-554 (1994).
  17. Metsis, V., Androutsopoulos, I. & Paliouras, G. Spam filtering with naive bayes-which naive bayes? In CEAS, vol. 17, 28-69 (Mountain View, CA, 2006).
  18. Datta, A., Sen, S. & Zick, Y. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In Security and Privacy (SP), 2016 IEEE Symposium on, 598-617 (IEEE, 2016).
  19. Chen, J., Song, L., Wainwright, M. J. & Jordan, M. I. L-shapley and c-shapley: Efficient model interpretation for structured data. arXiv preprint arXiv:1808.02610 (2018). · arXiv
  20. Lundberg, S. M., Erion, G. G. & Lee, S.-I. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888 (2018). · arXiv
  21. Jia, R. et al. Towards efficient data valuation based on the shapley value. arXiv preprint arXiv:1902.10275 (2019). · arXiv
  22. Agarwal, A., Dahleh, M. & Sarkar, T. A marketplace for data: an algorithmic solution. arXiv preprint arXiv:1805.08125 (2018). · arXiv
  23. Ghorbani, A., Abid, A. & Zou, J. Interpretation of neural networks is fragile. arXiv preprint arXiv:1710.10547 (2017). · arXiv
  24. Arrieta Ibarra, I., Goff, L., Jiménez Hernández, D., Lanier, J. & Weyl, E. G. Should we treat data as labor? moving beyond'free'. Mov. Beyond'Free'(December 27, 2017). Am. Econ. Assoc. Pap. & Proc. 1 (2017).
  25. Xiao, H., Rasul, K. & Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017). · arXiv

Cited by

See all 12

Metadata

Authors
Ghorbani, A., & Zou, J.
Venue
arXiv
Publisher
arXiv
Published
2019
DOI
10.48550/arxiv.1904.02868
arXiv
1904.02868
References
5
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Data Shapley: Equitable Valuation of Data for Machine Learning. https://app.cochat.ai/papers/10.48550/arxiv.1904.02868
The paper
Ghorbani, A., & Zou, J. (2019). Data Shapley: Equitable Valuation of Data for Machine Learning. arXiv. https://doi.org/10.48550/arxiv.1904.02868

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.