Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.

Implicit data crimes: Machine learning bias arising from misuse of public data

Open access
Shimron, Efrat, Tamir, Jonathan I., Wang, Ke, Lustig, Michael · 2022 · Proceedings of the National Academy of Sciences

Abstract

Significance: Public databases are an important resource for machine learning research, but their growing availability sometimes leads to “off-label” usage, where data published for one task are used for another. This work reveals that such off-label usage could lead to biased, overly optimistic results of machine-learning algorithms. The underlying cause is that public data are processed with hidden processing pipelines that alter the data features. Here we study three well-known algorithms developed for image reconstruction from magnetic resonance imaging measurements and show they could produce biased results with up to 48% artificial improvement when applied to public databases. We relate to the publication of such results as implicit “data crimes” to raise community awareness of this growing big data problem.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.2117203119

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Data crimes: how retrospective subsampling of processed data leads to biased results. (A) A common data processing pipeline often implemented inside commercial MRI scanners includes k-space zero padding, application of the inverse Fourier transform, and coil combination via an RSS step. The output image, which is interpolated and nonnegative, is stored in a database. In data crime I, this image is later used for synthesizing new k-space data; this yields artificial data in previously zero-padded areas. (B) A common data storage pipeline includes JPEG compression. In data crime II, the compressed image is later used for retrospective experiments. (C) Standard research pipelines commonly involve retrospective subsampling of fully sampled k-space data. In the data crimes scenarios, the fully sampled data are based on processed data; hence, image reconstruction algorithms benefit from the early processing. Moreover, because the gold standard image is based on the same processed data as the reconstructed one, error metrics become blind to the processing and, therefore, also are prone to bias.
Fig. 1. Data crimes: how retrospective subsampling of processed data leads to biased results. (A) A common data processing pipeline often implemented inside commercial MRI scanners includes k-space zero padding, application of the inverse Fourier transform, and coil combination via an RSS step. The output image, which is interpolated and nonnegative, is stored in a database. In data crime I, this image is later used for synthesizing new k-space data; this yields artificial data in previously zero-padded areas. (B) A common data storage pipeline includes JPEG compression. In data crime II, the compressed image is later used for retrospective experiments. (C) Standard research pipelines commonly involve retrospective subsampling of fully sampled k-space data. In the data crimes scenarios, the fully sampled data are based on processed data; hence, image reconstruction algorithms benefit from the early processing. Moreover, because the gold standard image is based on the same processed data as the reconstructed one, error metrics become blind to the processing and, therefore, also are prone to bias.CC BY · PubMed Central
Fig. 2. An experiment demonstrating how retrospective subsampling of k-space synthesized from processed data leads to increased effective sampling density of the true k-space data. (A) Subsampling masks were generated for different combinations of zero-padding factors (Left to Right) and subsampling schemes (Top to Bottom). The masks were generated from symmetric 2D PDFs (profiles displayed) with 17% sampling in all cases. The regions covering the original nonpadded k-space data are marked with yellow boxes. Notice that the zero padding squashes the original data to the center, so when a VD scheme is used, those data are sampled with an increased rate. (B) The effective sampling rate, which is the subsampling rate inside the original k-space area (i.e., inside the yellow boxes in A), versus the zero-padding rate. Notice that for the variable-density schemes, the effective rate is much higher than the global rate (17%) and may rise above 55%. (C) Real-world examples for k-space data generated from MR images found in public open-access databases (23, 24) show evidence of zero padding (the yellow box is our estimation). These experiments and examples indicate that training algorithms using data from public databases could lead to increased effective sampling.
Fig. 2. An experiment demonstrating how retrospective subsampling of k-space synthesized from processed data leads to increased effective sampling density of the true k-space data. (A) Subsampling masks were generated for different combinations of zero-padding factors (Left to Right) and subsampling schemes (Top to Bottom). The masks were generated from symmetric 2D PDFs (profiles displayed) with 17% sampling in all cases. The regions covering the original nonpadded k-space data are marked with yellow boxes. Notice that the zero padding squashes the original data to the center, so when a VD scheme is used, those data are sampled with an increased rate. (B) The effective sampling rate, which is the subsampling rate inside the original k-space area (i.e., inside the yellow boxes in A), versus the zero-padding rate. Notice that for the variable-density schemes, the effective rate is much higher than the global rate (17%) and may rise above 55%. (C) Real-world examples for k-space data generated from MR images found in public open-access databases (23, 24) show evidence of zero padding (the yellow box is our estimation). These experiments and examples indicate that training algorithms using data from public databases could lead to increased effective sampling.CC BY · PubMed Central
Fig. 3. Example for data crime I: CS reconstructions from retrospectively subsampled k-space of processed images. Notice how the reconstruction quality improves, both visually and in terms of NRMSE, with the zero-padding (processing) extent. This improvement is completely artificial; it stems from the coupling of early data processing and retrospective subsampling, which leads to increased sampling of true nonpadded data (as illustrated in Fig. 2). The artificial improvement is more significant when the sampling is stronger around the k-space center (Bottom; strong VD).
Fig. 3. Example for data crime I: CS reconstructions from retrospectively subsampled k-space of processed images. Notice how the reconstruction quality improves, both visually and in terms of NRMSE, with the zero-padding (processing) extent. This improvement is completely artificial; it stems from the coupling of early data processing and retrospective subsampling, which leads to increased sampling of true nonpadded data (as illustrated in Fig. 2). The artificial improvement is more significant when the sampling is stronger around the k-space center (Bottom; strong VD).CC BY · PubMed Central
Fig. 4. Data crime I. The CS, DictL, and DL algorithms were trained and tested using two versions of the same knee MRI dataset, one processed without zero padding and one with 2× zero padding. In the latter case, which represents the scenario of data crime I, the reconstructions exhibit sharper images with improved visibility of small, clinically relevant details. This illustrates that training inverse problem solvers using processed data may lead to overly optimistic results.
Fig. 4. Data crime I. The CS, DictL, and DL algorithms were trained and tested using two versions of the same knee MRI dataset, one processed without zero padding and one with 2× zero padding. In the latter case, which represents the scenario of data crime I, the reconstructions exhibit sharper images with improved visibility of small, clinically relevant details. This illustrates that training inverse problem solvers using processed data may lead to overly optimistic results.CC BY · PubMed Central
Fig. 5. Data crime I statistics. The CS, DictL, and DL algorithms were trained and evaluated using data with various processing extents. The data-processing pipeline typically implemented inside commercial scanners includes k-space zero padding (Fig. 1A). Retrospective subsampling experiments were performed with VD subsampling with R=4. The curves display the mean and SD of the NRMSE and SSIM error metrics for the test set. Notice that both metrics show an artificial improvement that is correlated with the data-processing extent. This demonstrates that algorithms evaluated on retrospectively subsampled processed data tend to yield overly optimistic evaluation.
Fig. 5. Data crime I statistics. The CS, DictL, and DL algorithms were trained and evaluated using data with various processing extents. The data-processing pipeline typically implemented inside commercial scanners includes k-space zero padding (Fig. 1A). Retrospective subsampling experiments were performed with VD subsampling with R=4. The curves display the mean and SD of the NRMSE and SSIM error metrics for the test set. Notice that both metrics show an artificial improvement that is correlated with the data-processing extent. This demonstrates that algorithms evaluated on retrospectively subsampled processed data tend to yield overly optimistic evaluation.CC BY · PubMed Central
Fig. 6. Example for data crime II. A DL algorithm was separately trained and tested on noncompressed data and different levels of JPEG-compressed data. Although the compression reduces the visual image quality, the NRMSE surprisingly reduces with increased compression, reflecting a seemingly better image quality. The reason is that in the retrospective experiments, both the gold standard and reconstructed images are based on processed data; hence, the error metric is blind to the processing and prone to bias. Strikingly, although the reconstructions from noncompressed and default-compressed data are visually similar, the NRMSE of the latter is lower by 30%. This demonstrates the implicit bias induced by training and evaluating algorithms on JPEG-compressed data.
Fig. 6. Example for data crime II. A DL algorithm was separately trained and tested on noncompressed data and different levels of JPEG-compressed data. Although the compression reduces the visual image quality, the NRMSE surprisingly reduces with increased compression, reflecting a seemingly better image quality. The reason is that in the retrospective experiments, both the gold standard and reconstructed images are based on processed data; hence, the error metric is blind to the processing and prone to bias. Strikingly, although the reconstructions from noncompressed and default-compressed data are visually similar, the NRMSE of the latter is lower by 30%. This demonstrates the implicit bias induced by training and evaluating algorithms on JPEG-compressed data.CC BY · PubMed Central
Fig. 7. Statistical results demonstrating data crime II. The CS, DictL, and DL algorithms were applied to datasets with no compression and increasing JPEG compression levels. The graphs depict the mean and SD computed for the test set. Notice that all the curves show the same trend: The error metrics improve consistently with increased JPEG compression. This improvement is artificial and stems only from the data processing, which reduces the data entropy. The results, therefore, demonstrate the subtle bias caused by training inverse problem solvers on JPEG-compressed data.
Fig. 7. Statistical results demonstrating data crime II. The CS, DictL, and DL algorithms were applied to datasets with no compression and increasing JPEG compression levels. The graphs depict the mean and SD computed for the test set. Notice that all the curves show the same trend: The error metrics improve consistently with increased JPEG compression. This improvement is artificial and stems only from the data processing, which reduces the data entropy. The results, therefore, demonstrate the subtle bias caused by training inverse problem solvers on JPEG-compressed data.CC BY · PubMed Central
Fig. 8. The potentially harmful impact of data crimes. In real-world applications, networks trained on processed data could be applied to unprocessed data. To show the negative effect, we took networks trained on processed data and tested them on both processed and unprocessed versions of the same test set. (A) Examples for knee MR images zoomed in on the clinically important meniscus area. Notice that in the real-world application (Right), the images are more blurred, and some details are missing (arrows). (B) Impact of data crime I. Notice that when networks trained on zero-padded data are applied to nonpadded data, their performance drops significantly, by up to 47%. (C) Impact of data crime II. Similarly, when networks trained using JPEG data are applied to non-JPEG data, their performance reduces.
Fig. 8. The potentially harmful impact of data crimes. In real-world applications, networks trained on processed data could be applied to unprocessed data. To show the negative effect, we took networks trained on processed data and tested them on both processed and unprocessed versions of the same test set. (A) Examples for knee MR images zoomed in on the clinically important meniscus area. Notice that in the real-world application (Right), the images are more blurred, and some details are missing (arrows). (B) Impact of data crime I. Notice that when networks trained on zero-padded data are applied to nonpadded data, their performance drops significantly, by up to 47%. (C) Impact of data crime II. Similarly, when networks trained using JPEG data are applied to non-JPEG data, their performance reduces.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 53 references

Research works printed but not linked: 3

  1. F. Ong, S. Amin, S. Vasanawala, M. Lustig, Mridata.org: An open archive for sharing MRI raw data. Proc. Intl. Soc. Mag. Reson. Med. 26, 1 (2018).
  2. E. Shimron J. I. Tamir K. Wang M. Lustig “Subtle inverse crimes: Naïvely using publicly available images could make reconstruction results seem misleadingly better!” in Proceedings of the Annual Meeting of the International Society of Magnetic Resonance in Medicine (ISMRM) (ISMRM 2021).
  3. F. Ong M. Lustig “SigPy: A python package for high performance iterative reconstruction” in Proceedings of the Annual Meeting of the International Society of Magnetic Resonance in Medicine (ISMRM 2019).

Cited by

See all 137

Metadata

Authors
Shimron, E., Tamir, J. I., Wang, K., & Lustig, M.
Affiliations
  • University of California, Berkeley (US)
  • The University of Texas at Austin (US)
Corresponding author
Shimron, Efrat
Venue
Proceedings of the National Academy of Sciences
Published
2022-03-21
Volume
119
Issue
13
Pages
e2117203119-e2117203119
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4221051856
Cited by
112
References
53
Citations by year
2026: 8 · 2025: 42 · 2024: 20 · 2023: 31 · 2022: 11
FWCI
30.10
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
Funders
HHS | National Institutes of Health (U24 EB029240-01, R01EB009690, R01HL136965)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
  • Social Sciences › Education › Academic integrity and plagiarism
Keywords
machine learning bias, data pipelines, magnetic resonance imaging, image reconstruction, big data ethics, algorithmic fairness
MeSH terms
Machine Learning, Algorithms, Crime, Image Processing, Computer-Assisted, Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Implicit data crimes: Machine learning bias arising from misuse of public data. https://app.cochat.ai/papers/10.1073/pnas.2117203119
The paper
Shimron, E., Tamir, J. I., Wang, K., & Lustig, M. (2022). Implicit data crimes: Machine learning bias arising from misuse of public data. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2117203119

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.