CoChat
Sign Up
Cited by 110Advanced Causal Inference TechniquesEthics and Social Impacts of AIMachine Learning and Data Classification

The Selective Labels Problem

Open access
Lakkaraju, Himabindu, Kleinberg, Jon, Leskovec, Jure, Ludwig, Jens, Mullainathan, Sendhil·2017·Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
DOI 10.1145/3097983.3098066PMC PMC5958915
Open this paper in CoChat — free account

Abstract

Evaluating whether machines improve on human performance is one of the central questions of machine learning. However, there are many domains where the data is selectively labeled, in the sense that the observed outcomes are themselves a consequence of the existing choices of the human decision-makers. For instance, in the context of judicial bail decisions, we observe the outcome of whether a defendant fails to return for their court appearance only if the human judge decides to release the defendant on bail. This selective labeling makes it harder to evaluate predictive models as the instances for which outcomes are observed do not represent a random sample of the population. Here we propose a novel framework for evaluating the performance of predictive models on selectively labeled data. We develop an approach called contraction which allows us to compare the performance of predictive models and human decision-makers without resorting to counterfactual inference. Our methodology harnesses the heterogeneity of human decision-makers and facilitates effective evaluation of predictive models even in the presence of unmeasured confounders (unobservables) which influence both human decisions and the resulting outcomes. Experimental results on real world datasets spanning diverse domains such as health care, insurance, and criminal justice demonstrate the utility of our evaluation metric in comparing human decisions and machine predictions.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper formalizes the selective labels problem, where human decisions determine which outcomes are observed, and proposes a contraction technique to evaluate predictive models against human decision-makers without counterfactual inference. Experiments on synthetic and real-world data show contraction closely tracks true model performance, while imputation and labeled-outcomes-only evaluations can be misleadingly optimistic.

Key findings

  • In synthetic data with 100 judges, 500 subjects each (N = 50k), contraction closely followed the true evaluation curve, whereas propensity score matching and other imputation techniques heavily under-estimated failure rates and over-estimated model performance.
  • At βZ = 1, contraction’s mean absolute error was 6.4 times smaller than the best-performing imputation technique, and the gap increased as the unobservable’s influence grew.
  • On real datasets, machine evaluation curves using contraction indicated lower failure rates than human decision-makers at all acceptance rates across bail, asthma treatment, and insurance decisions.
  • Agreement rates between the model and the most lenient decision-maker on no decisions were 0.891 for bail, 0.932 for asthma treatments, and 0.718 for insurance requests; the paper states these high rates make contraction estimates close to true values.

Methods

  • The framework takes observational data 𝒟 = {(xi, ji, ti, yi)} and a black-box predictive model ℬ, then uses the most lenient decision-maker q to contract the set of accepted instances down to a target acceptance rate r, computing failure rate only on instances with observed labels.
  • It defines acceptance rate as the fraction of yes decisions and failure rate as the fraction of undesirable outcomes among all judged subjects, then plots machine and human evaluation curves of failure rate versus acceptance rate.
  • Synthetic experiments used logistic regression and varied βZ, agreement rates, and number of subjects judged by the most lenient decision-maker; real experiments used gradient boosted trees (100 trees) on bail data (about 9k defendants, 18 judges; evaluation subset 4.8k defendants and 9 judges), medical treatment data (60k patients; evaluation subset 28k adults and 39 doctors), and insurance data (about 50k requests, 74 managers; evaluation subset 38k subjects and 41 chemical-domain managers).

Limitations

  • The paper states contraction’s error bound depends on the most lenient decision-maker’s acceptance rate, the agreement rate on no decisions between the model and that decision-maker, and the number of subjects judged by that decision-maker; estimates are only defined up to that decision-maker’s acceptance rate ψ.
  • The method requires multiple decision-makers, random assignment of subjects to decision-makers, and heterogeneity in decision thresholds; the paper tests random assignment and reports it holds on all evaluation sets, but this remains an assumption.
  • The paper describes itself as an initial attempt and leaves for future work how contraction can be used to improve training models on selectively labeled data; it also notes that experimentation to acquire missing labels is often infeasible or unethical in these domains.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3097983.3098066

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

7 of 17 cited sources checked · 2 supported · 2 flagged · accuracy not yet measured

  • Premisecitation

    Counterfactual inference techniques explicitly assume that there are no unmeasured confounders (unobservable variables Z) that could affect the outcome Y.

    “However, counterfactual inference techniques explicitly assume that there are no unmeasured confounders (that is, no unobservable variables Z) that could affect the outcome Y.”
    Quote in the paperSubstantial claim2 of 5 sources checked
    • cites [9] Chernozhukov V Fernández-Val I Melly B Inference on counterfactual distributions Econometrica 81 6 2205 2268 2013 source

      Section 2.1 (page unavailable) represents the outcome as a function of observed covariates and a disturbance U, so the paper does not rule out unobserved factors affecting Y. Its stated assumption is instead that the conditional distribution of Y given the covariates remains unchanged by the intervention (Section 2.1, page unavailable). The disturbance is independent of the covariates in this representation; the passage does not identify it as a confounder. (LLM, conf=0.82)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [12] Gelman A Hill J Data analysis using regression and multilevel/hierarchical models Cambridge university press 2006 source
    • cites [16] Johansson FD Shalit U Sontag D Learning representations for counterfactual inference arXiv preprint arXiv:1605.03661 2016 source

      The Problem setup (p. 2) defines potential outcomes in terms of context, and the discussion after Algorithm 1 (p. 3) describes treatment assignment conditional on context. Neither passage states that counterfactual inference techniques explicitly assume there are no unmeasured confounders affecting outcomes. Searches for unmeasured confounders, hidden variables, and ignorability found no passage making that claim. (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites [36] Rosenbaum PR Rubin DB The central role of the propensity score in observational studies for causal effects Biometrika 41 55 1983

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [38] Rubin DB Causal inference using potential outcomes: Design, modeling, decisions JASA 100 469 322 331 2005

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    Berk et al. evaluated their model only on the set of defendants who had been granted parole.

    “For instance, Berk et al. evaluated their model only on the set of defendants who have been granted parole.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [7] Berk R Sherman L Barnes G Kurtz E Ahlman L Forecasting murder within a population of probationers and parolees: a high stakes application of statistical learning JRSS 172 1 191 211 2009 source

      The Setting (p. unavailable) describes the supervised population as both probationers and parolees. The Data section (p. unavailable) says the test sample was drawn from APPD intake cases, not solely from people granted parole. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Premisecitation

    The assumption of no unmeasured confounders does not typically hold in cases where human decisions are providing data labels.

    “This assumption does not typically hold in cases where human decisions are providing data labels [7, 21].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [7] Berk R Sherman L Barnes G Kurtz E Ahlman L Forecasting murder within a population of probationers and parolees: a high stakes application of statistical learning JRSS 172 1 191 211 2009 source

      The Setting (section 3; page unavailable) says homicide charges are an imperfect outcome measure and that convictions introduce further error through adjudication. The Discussion (section 6; page unavailable) also notes that subjective forecasts may contain unmeasured racial bias. These passages support concerns about human decisions in the data, but the paper does not say that the no-unmeasured-confounders assumption typically fails when humans provide labels. (LLM, conf=0.77)

      Experimental — accuracy not yet measured
    • cites [21] Lakkaraju H Leskovec J Confusions over time: An interpretable bayesian model to characterize trends in decision making NIPS 3261 3269 2016

      Not checked in the free run.

  • Premisecitation

    Prior research often overlooks the selective labels problem, evaluating models by computing traditional metrics such as AUC or accuracy only on data points for which ground truth labels are available.

    “While prior research has explored applications of machine learning to domains where human judgments are part of the data labeling process (crime [7, 21, 43], medical diagnosis [6, 22]), the selective labels problem is often overlooked, with evaluation of the models typically carried out by computing traditional metrics such as AUC or accuracy only on the data points for which ground truth labels”
    Quote in the paperSubstantial claim3 of 5 sources checked
    • cites [7] Berk R Sherman L Barnes G Kurtz E Ahlman L Forecasting murder within a population of probationers and parolees: a high stakes application of statistical learning JRSS 172 1 191 211 2009 source
    • cites [21] Lakkaraju H Leskovec J Confusions over time: An interpretable bayesian model to characterize trends in decision making NIPS 3261 3269 2016

      Not checked in the free run.

    • cites [43] Zeng J Ustun B Rudin C Interpretable classification models for recidivism prediction JRSS 2016

      Not checked in the free run.

    • cites [6] Bayati M Braverman M Gillam M Mack KM Ruiz G Smith MS Horvitz E Data-driven decisions for reducing readmissions for heart failure: General methodology and case study PloS one 9 10 e109264 2014 25295524 PMC4190088 source

      The Results (Model validation; page unavailable) report evaluating the classifier with AUC, and Materials and Methods says the readmission outcomes came from CMS data. The paper does not identify a selective-labels problem or say that evaluation was restricted to patients whose outcomes were available because of human labeling decisions. Its example of clinician judgment in the Discussion concerns whether a cocaine test was administered, not whether a readmission outcome was labeled. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [22] Lakkaraju H Rudin C Learning cost-effective treatment regimes using markov decision processes CoRR abs/1610.06972 2016 source
  • Findingown data

    Experimental results on real-world datasets spanning health care, insurance, and criminal justice demonstrate the utility of the evaluation metric in comparing human decisions and machine predictions.

    “Experimental results on real world datasets spanning diverse domains such as health care, insurance, and criminal justice demonstrate the utility of our evaluation metric in comparing human decisions and machine predictions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Many domains involving selective labels have data on many decision-makers rather than just one.
    • Rests on an assumption: Different decision-makers have a similar pool of cases, so that it is as if the instances had been randomly assigned across them.
    • Rests on an assumption: Decision-makers differ in the thresholds they use for their yes-no decisions, resulting in different acceptance rates.
  • Classified as probably not a substantial claim. 10
  • The quote only partly states this claim. 3
  • May combine two claims. 13

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data

    Other predictive models gave similar behavior to logistic regression in the synthetic data experiments.

    “We also experimented with other predictive models and observed similar behavior.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The mean absolute error of contraction is 6.4 times smaller (at βZ = 1) compared to the best performing imputation technique and this gap only increases with the increase in the value of βZ.

    “The mean absolute error of contraction is 6.4 times smaller (at βZ = 1) compared to the best performing imputation technique and this gap only increases with the increase in the value of βZ.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Imputation techniques heavily overestimate the performance of the model and the degree of overestimation steadily increases with βZ.

    “On the other hand, imputation techniques heavily overestimate the performance of the model and the degree of overestimation steadily increases with βZ.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The evaluation curve plotted using contraction closely follows the true evaluation curve, demonstrating that contraction is very effective in estimating the true performance of the predictive model.

    “It can be seen that the evaluation curve plotted using contraction (blue curve) closely follows the true evaluation curve demonstrating that contraction is very effective in estimating the true performance of the predictive model.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Observations are randomly assigned to human decision-makers (there is no relationship between human decision-makers and characteristics of subjects).

  • The quote only partly states this claim.

    Findingown data

    The evaluation curve plotted using labeled outcomes only exhibits similar behavior (under-estimates failure rate).

    “Similar behavior is exhibited by the evaluation curve plotted using labeled outcomes only.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    Other imputation techniques exhibited similar behavior to propensity score matching (heavy under-estimation of failure rate).

    “We also observed similar behavior in the case of other imputation techniques.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The mean absolute error is low when the agreement rate values are high and it increases with a decrease in the agreement rate.

    “It can be seen that the mean absolute error is low when the agreement rate values are high and it increases with a decrease in the agreement rate.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Observations are randomly assigned to human decision-makers (there is no relationship between human decision-makers and characteristics of subjects).

  • May combine two claims.

    Findingown data

    The agreement rate is 0.891, 0.932, and 0.718 on bail decisions, asthma treatments, and insurance requests datasets respectively.

    “We found that the agreement rate (see Section 4) which is the ratio of the number of observations that both the predictive model at risk tolerance r′ and j′ agree on assigning to no decision (t = 0) to the total number of observations assigned to no decision by j′ is 0.891, 0.932, and 0.718 on bail decisions, asthma treatments, and insurance requests datasets respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The predictive model exhibits lower failure rates than human decision-makers at all possible values of acceptance rates on all three real-world datasets, according to both the contraction and labeled outcomes curves.

    “Figure 6 shows that both the machine evaluation curve plotted using contraction (black) as well as the one plotted only using only the labeled outcomes (blue) indicate that the predictive model is exhibiting lower failure rates than human decision-makers (red) at all possible values of acceptance rates on all the three datasets.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Observations are randomly assigned to human decision-makers (there is no relationship between human decision-makers and characteristics of subjects).

  • Classified as probably not a substantial claim.

    Interpretationnone given

    Due to the presence of unobservables, it cannot be guaranteed that either postulated model will be identical to the true models.

    “However, due to the presence of unobservables, we cannot guarantee that either of the postulated models will be identical to the true models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The assumption that released and denied defendants have matching distributions is often violated in practice.

    “This assumption is often violated in practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Contraction is ideal for judgment-based settings where unobservables may influence the outcomes.

    “This makes our contraction technique ideal for judgment-based settings where unobservables may influence the outcomes.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Unobservables available to decision-makers but not recorded in the data may influence outcomes and are independent of the features recorded in the data.

  • May combine two claims.

    Study design and conductown data

    Random assignment of subjects to decision-makers is required for fair comparison of failure rate estimates, and this requirement typically holds in practice in many decision-making settings.

    “We therefore require that observations/subjects are randomly assigned to decision-makers. We find that this requirement typically holds in practice in many decision making settings (more details provided in the experimental evaluation section).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Contraction compares a black box predictive model to a human decision-maker by forcing the model's acceptance rate to equal the judge's and measuring the corresponding failure rate; if the model has a lower failure rate at the same acceptance rate, the model is better than that judge.

    “We can compare a black box predictive model to any given human decision-maker by forcing the acceptance rate of the model to be the same as that of the judge and measuring the corresponding failure rate. If the model exhibits a lower failure rate than the judge at the same acceptance rate, then we can claim that the model is better than that judge.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Many domains involving selective labels have data on many decision-makers rather than just one.
    • Rests on an assumption: Different decision-makers have a similar pool of cases, so that it is as if the instances had been randomly assigned across them.
    • Rests on an assumption: Decision-makers differ in the thresholds they use for their yes-no decisions, resulting in different acceptance rates.

  • Classified as probably not a substantial claim.

    Premisereasoning

    Parametric and non-parametric counterfactual inference methods result in biased estimates of outcomes if their postulated models are not identical to the true models.

    “If the postulated models are not identical to the true models, then these techniques result in biased estimates of outcomes.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The extent to which the failure rate is lower for the hybrid human-algorithmic mechanism is a concrete type of performance guarantee provided by the contraction technique.

    “The extent to which the failure rate is lower for the hybrid human-algorithmic mechanism is a concrete type of performance guarantee provided by the contraction technique.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisenone given

    The effectiveness of matching techniques relies on the absence of unobservables, an assumption that does not hold in the selective labels setting.

    “The effectiveness of these techniques relies on the absence of unobservables, an assumption, which does not hold in our case.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    The methodology harnesses the heterogeneity of human decision-makers and facilitates effective evaluation of predictive models even in the presence of unmeasured confounders that influence both human decisions and outcomes.

    “Our methodology harnesses the heterogeneity of human decision-makers and facilitates effective evaluation of predictive models even in the presence of unmeasured confounders (unobservables) which influence both human decisions and the resulting outcomes.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Many domains involving selective labels have data on many decision-makers rather than just one.
    • Rests on an assumption: Different decision-makers have a similar pool of cases, so that it is as if the instances had been randomly assigned across them.
    • Rests on an assumption: Decision-makers differ in the thresholds they use for their yes-no decisions, resulting in different acceptance rates.
    • The authors state this as an assumption: Unobservables available to decision-makers but not recorded in the data may influence outcomes and are independent of the features recorded in the data.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Doubly robust methods require only one of the postulated models to be identical to the true model in order to generate unbiased estimates.

    “Doubly robust methods require only one of the postulated models to be identical to the true model in order to generate unbiased estimates.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The selective labels problem is a form of missing data problem that commonly arises where judgments of human decision-makers determine which instances have labels.

    “Recall that the selective labels problem is a form of missing data problem [1] which commonly arises in various settings where judgments of human decision-makers determine which instances have labels.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [1] Allison PD Missing data: Quantitative applications in the social sciences British Journal of Mathematical and Statistical Psychology 55 1 193 196 2002

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Imputation techniques so heavily overestimate the performance of the predictive model that it would lead us to wrongly conclude that the model is outperforming the human judges, while in fact its predictions are less accurate than those of the human judges.

    “However, notice that imputation techniques so heavily overestimate the performance of the predictive model that it would lead us to wrongly conclude that the model is outperforming the human judges, while in fact its predictions are less accurate than those of the human judges.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MNAR problems are often addressed using selection and pattern-mixture models, which require assumptions about the underlying generative process of the data.

    “MNAR problems are often addressed using approaches such as selection and pattern-mixture models [24] which require us to make assumptions about the underlying generative process of the data.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24] Little RJ Selection and pattern-mixture models Longitudinal data analysis 409 431 2008

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    We utilized a multiple hypothesis testing procedure to validate the assumption that observations are randomly assigned to human decision-makers.

    “We utilized the following multiple hypothesis testing procedure [23] to validate this assumption:”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [23] List JA Shaikh AM Xu Y Multiple hypothesis testing in experimental economics. Technical report NBER 2016

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Higher values of acceptance rate, agreement rate, and number of subjects judged by the most lenient decision-maker result in tighter error bounds and consequently accurate estimates of failure rates.

    “More specifically, higher values of each of these parameters result in tighter error bounds (Section 4) and consequently accurate estimates of failure rates.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Observations are randomly assigned to human decision-makers (there is no relationship between human decision-makers and characteristics of subjects).
24 of 24

  • The paper relies on this without support.

    Many domains involving selective labels have data on many decision-makers rather than just one.

    “We have data on many decision-makers rather than just one;”

  • The paper relies on this without support.

    Different decision-makers have a similar pool of cases, so that it is as if the instances had been randomly assigned across them.

    “These different decision-makers have a similar pool of cases, so that it is as if the instances had been randomly assigned across them;”

  • The paper relies on this without support.

    Decision-makers differ in the thresholds they use for their yes-no decisions, resulting in different acceptance rates.

    “The decision-makers differ in the thresholds they use for their yes-no decisions.”

  • The authors state this as an assumption.

    Unobservables available to decision-makers but not recorded in the data may influence outcomes and are independent of the features recorded in the data.

    “Furthermore, these unobservables may influence the outcomes and are independent of the features recorded in the data.”

  • The authors state this as an assumption.

    Observations are randomly assigned to human decision-makers (there is no relationship between human decision-makers and characteristics of subjects).

    “Recall that the correctness of our contraction technique relies on the assumption that observations are randomly assigned to human decision-makers, i.e., there is no relationship between human decision-makers and characteristics of subjects.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Data used to train models represent past decisions, and as such inherent bias(es) in these decisions are amplified.

    “[163,172,186,187,298] note that there is an additional challenge in the datasets used to train models: data represent past decisions, and as such inherent bias(es) in these decisions are ampliied.”

    From Fairness in machine learning: A survey (2024) · cites this paper as [186]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Stated error bound does not tighten with sample size

    Minor

    The stated bound is (1.0−a)|Dq−Rq|/|Dq|. Increasing the number of subjects while holding the acceptance and agreement rates fixed leaves that ratio unchanged, contrary to the claim that a higher subject count tightens the Section 4 error bound.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. J. Zeng , B. Ustun , and C. Rudin . Interpretable classification models for recidivism prediction . JRSS , 2016 .endthebibliography J. Zeng, B. Ustun, and C. Rudin. Interpretable classification models for recidivism prediction. JRSS, 2016.endthebibliography

Cited by

Sorted by citations

Related

Authors
Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S.
Affiliations
  • Stanford University (US)
  • Cornell University (US)
  • University of Chicago (US)
  • Harvard University (US)
ORCID
Venue
Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
Publisher
ACM
Published
2017-08-04
Volume
2017
Pages
275-284
DOI
10.1145/3097983.3098066
PMC
PMC5958915
OpenAlex
W2745133928
Semantic Scholar
a3ec84e1d148f770de3cadb83d7e33931e1ff998
Cited by
110
References
50
Citations by year
2026: 9 · 2025: 9 · 2024: 8 · 2023: 15 · 2022: 10 · 2021: 15 · 2020: 16 · 2019: 14 · 2018: 12 · 2017: 1 · 2015: 1
FWCI
23.84
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (submittedVersion)
  • PubMed Central (submittedVersion)
Funders
  • NSF (NSF IIS-1149837)
  • SAP
  • Lightspeed
  • Stanford Data Science Initiative
  • Chan Zuckerberg Biohub
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
selective labeling, model evaluation, human decision-makers, prediction model evaluation, contractions, unmeasured confounding, human-machine comparison, bail decisions, judicial decisions, counterfactual inference, evaluation metrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review