Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 559

The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.

Advances, challenges and opportunities in creating data for trustworthy AI

Liang, Weixin, Tadesse, Girmaw Abebe, Ho, Daniel, Fei-Fei, L., Zaharia, Matei, Zhang, Ce, Zou, James·2022·Nature Machine Intelligence

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-022-00516-1

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 74 references

Research works printed but not linked: 30

  1. Abadi, M. et al. TensorFlow: a system for large-scale machine learning. In Proc. 12th USENIX Symposium on Operating Systems Design and Implementation 265–283 (USENIX Association, 2016).
  2. Koch, B., Denton, E., Hanna, A. & Foster, J. G. Reduced, reused and recycled: the life of a dataset in machine learning research. In NeurIPS 2021 Datasets and Benchmarks Track 50 (OpenReview, 2021).
  3. Coleman, C. et al. DAWNBench: An end-to-end deep learning benchmark and competition. In NeurIPS MLSys Workshop 10 (MLSys, 2017).
  4. Buolamwini, J. & Gebru, T. Gender shades: intersectional accuracy disparities in commercial gender classification. In Proc. 1st Conference on Fairness, Accountability and Transparency 77–91 (PMLR, 2018).
  5. Li, C. et al. iGibson 2.0: object-centric simulation for robot learning of everyday household tasks. In Proc. 5th Annual Conference on Robot Learning Vol. 164 455–465 (PMLR, 2022).
  6. Choi, K., Grover, A., Singh, T., Shu, R. & Ermon, S. Fair generative modeling via weak supervision. Proc. Mach. Learn. Res. 119, 1887–1898 (2020).
  7. Holland, S., Hosny, A., Newman, S., Joseph, J. & Chmielinski, K. The dataset nutrition label: a framework to drive higher data quality standards. Preprint at https://arxiv.org/abs/1805.03677 (2018). · arXiv
  8. Liang, W. & Zou, J. MetaShift: a dataset of datasets for evaluating contextual distribution shifts and training conflicts. In International Conference on Learning Representations 400 (OpenReview, 2022).
  9. Ghorbani, A. & Zou, J. Data Shapley: equitable valuation of data for machine learning. Proc. Mach. Learn. Res. 97, 2242–2251 (2019).
  10. Kwon, Y., Rivas, M. A. & Zou, J. Efficient computation and analysis of distributional Shapley values. Proc. Mach. Learn. Res. 130, 793–801 (2021).
  11. Koh, P. W. & Liang, P. Understanding black-box predictions via influence functions. Proc. Mach. Learn. Res. 70, 1885–1894 (2017).
  12. Kwon, Y. & Zou, J. Beta Shapley: a unified and noise-reduced data valuation framework for machine learning. In Proc. 25th International Conference on Artificial Intelligence and Statistics Vol. 151 8780–8802 (PMLR, 2022).
  13. Northcutt, C. G., Athalye, A. & Mueller, J. Pervasive label errors in test sets destabilize machine learning benchmarks. In NeurIPS 2021 Datasets and Benchmarks Track 172 (OpenReview, 2021).
  14. Park, J., Krishna, R., Khadpe, P., Fei-Fei, L. & Bernstein, M. AI-based request augmentation to increase crowdsourcing participation. Proc. AAAI Conf. Hum. Comput. Crowdsourcing 7, 115–124 (2019).
  15. Ratner, A. J., De, Sa,C. M., Wu, S., Selsam, D. & Ré, C. Data programming: creating large training sets, quickly. Adv. Neural Inf. Process. Syst. 29, 3567–3575 (2016).
  16. Kang, D., Raghavan, D., Bailis, P. & Zaharia, M. Model assertions for debugging machine learning. In NeurIPS MLSys Workshop 23 (MLSys, 2020).
  17. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25, 1097–1105 (2012).
  18. Sennrich, R., Haddow, B. & Birch, A. Improving neural machine translation models with monolingual data. In Proc. 54th Annual Meeting of the Association for Computational Linguistics 86–96 (ACL, 2016). · DOI · Paper page
  19. Zhang, H., Cissé, M., Dauphin, Y. N. & Lopez-Paz, D. mixup: beyond empirical risk minimization. In Proc. International Conference on Learning Representations 296 (OpenReview, 2018).
  20. Deng, Z., Zhang, L., Ghorbani, A. & Zou, J. Improving adversarial robustness via unlabeled out-of-domain. Data. Proc. Mach. Learn. Res. 130, 2845–2853 (2021).
  21. Zhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A. & Zou, J. How does mixup help with robustness and generalization? In Proc. International Conference on Learning Representations 79 (OpenReview, 2021).
  22. Azizzadenesheli, K., Liu, A., Yang, F. & Anandkumar, A. Regularized learning for domain adaptation under label shifts. In Proc. International Conference on Learning Representations 432 (OpenReview, 2019).
  23. Zaharia, M. et al. Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng Bull 41, 39–45 (2018).
  24. Mello, M. M., Triantis, G., Stanton, R., Blumenkranz, E. & Studdert, D. M. Waiting for data: barriers to executing data use agreements. Science 367, 150–152 (2020). · doi:10.1126/science.aaz7028
  25. Loh, W., Hauschke, A., Puntschuh, M. & Hallensleben, S. VDE SPEC 90012 V1.0: VCIO Based Description of Systems for AI Trustworthiness Characterisation (VDE Press, 2022).
  26. Bagdasaryan, E., Poursaeed, O. & Shmatikov, V. Differential privacy has disparate impact on model accuracy. Adv. Neural Inf. Process. Syst. 32, 15453–15462 (2019).
  27. Izzo, Z., Smart, M. A., Chaudhuri, K. & Zou, J. Approximate data deletion from machine learning models. Proc. Mach. Learn. Res. 130, 2008–2016 (2021).
  28. Wilson, D. R. Beyond probabilistic record linkage: Using neural networks and complex features to improve genealogical record linkage. In 2011 International Joint Conference on Neural Networks 9–14 (IEEE, 2011); https://doi.org/10.1109/IJCNN.2011.6033192 · doi:10.1109/ijcnn.2011.6033192
  29. Victor, D. COCO-Africa: a curation tool and dataset of common objects in the context of Africa. In 2018 Conference on Neural Information Processing, 2nd Black in AI Workshop 1 (NeurIPS, 2019).
  30. Siminyu, K. et al. AI4D—African language program. Preprint at http://arxiv.org/abs/2104.02516 (2021). · arXiv

Cited by

See all 559

Metadata

Authors
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J.
Venue
Nature Machine Intelligence
Published
2022-08-17
Volume
4
Issue
8
Pages
669-677
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4292121845
Cited by
559
References
74
FWCI
124.81
Type
journal article
Language
en
Open access
closed
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Advances, challenges and opportunities in creating data for trustworthy AI. https://app.cochat.ai/papers/10.1038/s42256-022-00516-1
The paper
Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022). Advances, challenges and opportunities in creating data for trustworthy AI. Nature Machine Intelligence. https://doi.org/10.1038/s42256-022-00516-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.