The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 11 of the 71 sources its claims cite were checked for support.
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Open accessAbstract
Despite impressive performance in many benchmark datasets, AI models can still make mistakes, especially among out-of-distribution examples. It remains an open question how such imperfect models can be used effectively in collaboration with humans. Prior work has focused on AI assistance that helps people make individual high-stakes decisions, which is not scalable for a large amount of relatively low-stakes decisions, e.g., moderating social media comments. Instead, we propose conditional delegation as an alternative paradigm for human-AI collaboration where humans create rules to indicate trustworthy regions of a model. Using content moderation as a testbed, we develop novel interfaces to assist humans in creating conditional delegation rules and conduct a randomized experiment with two datasets to simulate in-distribution and out-of-distribution scenarios. Our study demonstrates the promise of conditional delegation in improving model performance and provides insights into design for this novel paradigm, including the effect of AI explanations.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes conditional delegation, where humans create keyword rules defining trustworthy regions for an AI model, and tests it on content moderation. In a 240-participant randomized experiment, conditional delegation improved model precision over the model alone, but achieved complementary performance over manual rule-based moderation only for in-distribution AI, not out-of-distribution AI.
Key findings
- Conditional delegation with human-created rules improved precision over the model working alone both in-distribution and out-of-distribution (p < 0.001 for average and union precision). On WikiAttack, the model alone already outperformed the manual condition, and conditional delegation further improved precision, achieving complementary performance; on Reddit, it improved the model but did not compensate enough to outperform the manual rule-based approach.
- Distribution type had the most salient effect: two-way ANOVA found significant effects of distribution type, experimental condition, and their interaction (p < 0.001). For average precision, condition was significant on WikiAttack (p < 0.001) but not Reddit (p = 0.864); for union precision, condition was significant on both (p < 0.001), and on Reddit the global explanation condition was worse than manual (p = 0.004) and predicted labels (p = 0.028).
- Explanations did not significantly improve conditional delegation performance; global explanations slightly hurt union precision and reward out-of-distribution. Explanations improved efficiency: rules per minute had a significant condition effect (p = 0.008), with global explanations vs predicted labels significant (p = 0.031), and global explanations led to higher overlap with rationale words.
- Participants were more engaged with delegation support than manual (actions p < 0.001; manual vs predicted labels/global p < 0.001, local p = 0.009). Subjective measures were generally positive with no strong condition differences; local explanations trended toward the best experience on WikiAttack.
Methods
- Between-subject randomized experiment with 240 Mechanical Turkers (30 per condition per distribution; 120 in-distribution, 120 out-of-distribution; 6 of 240 filtered from analysis). Four conditions: predicted labels; predicted labels + local explanations; predicted labels + local explanations + global explanations; manual. Two datasets: WikiAttack (in-distribution) and Reddit hate speech (out-of-distribution). Participants had to submit at least 10 rules; bonus was $0.10 per 100 toxic comments correctly reported and −$0.10 per 100 nontoxic comments mistakenly reported, bounded $0–$2.
- Model: rationale-style neural architecture with BERT encoders, trained on WikiAttack (70k training, 23K test comments) and applied to Reddit (22K comments); recall weight 0.5 chosen. Measures: average precision, union precision, reward, engagement actions, elapsed time, rules per minute, and Likert subjective measures. Analysis: t-tests, two-way/one-way ANOVA, and Tukey’s HSD.
Limitations
- Stated: the work is one instantiation of conditional delegation; precision/coverage tradeoffs may vary by application; participants are not representative of content moderators; evaluations are limited by the roughly 10 rules participants created in about 10 minutes; model, dataset, and explanation choices affect outcomes; the study does not account for future rule updates or existing moderator rules, and uses a narrow toxicity component of content moderation.
- Evident: the in-distribution vs out-of-distribution contrast is confounded with model performance and dataset/community differences, which the paper itself notes as a complexity; reward is highly volatile because a small number of keywords can dominate, especially out-of-distribution, as the paper states.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
46 Own result · 38 Own interpretation · 27 Not checked yet · 15 Own method · 8 No basis given · 3 Supported · 1 Supported so far · 1 Partly supported · 1 Not supported · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“For example, moderators on Reddit use a tool called AutoModerator, with which they manually customize a rule-based system to automatically identify comments for deleting or reporting for further review [18,44].”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
What the sources say
- Supportedcites [18] source
Section 3.3 says moderators write and maintain AutoModerator’s rules, which filter content for removal or send it to a moderation queue for human review. Section 3.3.1 gives examples of rules that detect and remove submissions or comments. (LLM, conf=0.94)
- Supportedcites [44] source
Section 6.3 says moderators directly control AutoModerator by editing its configuration, and Section 5.2.2 says they configure rules for submissions and comments. Another passage explains that AutoModerator can remove content directly or flag it for human review. Together, these support the paraphrase. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there exist diverse rules and preferences of austerity and value in diferent communities [20,31,74,77]”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
- Not checked yetcites [20] source
- Not checked yetcites [31]
- Not checked yetcites [74]
- Partly supportedcites [77] sourceExperimental — accuracy not yet measured
The Introduction identifies values held by Wikipedia stakeholders, and “Valuing Experimentation vs. Serving CCVs” discusses community norms and values. But “The Scope of the ‘Convergent Community Values’” cautions against generalizing across communities and does not establish that different communities have diverse rules or preferences of austerity. The claim extends the paper’s findings beyond what it shows. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that model explanations can improve efciency in identifying delegation conditions and, with weak evidence, improve user experiences.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Through a randomized experiment with 240 mechanical turkers, we show that even crowdworkers are able to create high-quality rules that lead to higher precision with conditional delegation than the model working alone.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The study assumes the same workflow as the manual rule-based approach: comments falling outside keyword rules are ignored without taking an action.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also found that model explanations can improve efciency in identifying delegation conditions and, with weak evidence, improve user experiences.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“which already outperforms the manual rule-based approach for content moderation”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The study assumes the same workflow as the manual rule-based approach: comments falling outside keyword rules are ignored without taking an action.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Anecdotally, we deployed a version of our model on a subreddit to report comments that are predicted as toxic, and the moderators asked us to shut it down due to high false positive rates (i.e., low precision).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We fnd qualitatively that the model produces sensible rationales in this application.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.1 Model Development
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it retains only 2% of tokens on average for predicted-nontoxic comments, versus 15% for predicted-toxic comments.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.1 Model Development
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Two-way ANOVA does not show any statistically signifcant efects of distribution types and experimental conditions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.2 Efciency and Engagement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 56
- It may state two separate claims as one. 55
- The quote only partly states this claim. 2
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“For instance, McCoy et al. [61] fnd that state-of-the-art models in natural language inference adopt three fallible syntactic heuristics and perform around random chance when tested on examples where these heuristics fail.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.3 Distribution Shift and Experimental Studies on Out-of-distribution Examples
What the sources say
- Disputedcites [61] sourceExperimental — accuracy not yet measured
The Abstract identifies three fallible syntactic heuristics, and the Results (section 5) say the models behaved consistently with using them. But on examples where those heuristics made incorrect predictions, accuracy was below 10% in most cases—not around the 50% chance level claimed. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Disputed
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Many studies have investigated the role of AI in assisting humans in making individual decisions [9,16,36, 37, 50, 51, 55, 58, 69, 83, 84, 89], e.g., predicting whether a person will recidivate in the near future.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 1 INTRODUCTION
What the sources say
- Supportedcites [9] source
The AI-Assisted Eye Screening section describes a deep-learning system intended to help nurses make referral recommendations for individual patients. The Prospective Study Protocol describes the system returning patient-specific assessments and referral recommendations. This paper therefore fits the cited group of studies on AI assisting human decisions; the recidivism example is not attributed specifically to it. (LLM, conf=0.90)
- Not checked yetcites [16]
- Not checked yetcites [36] source
- Not checked yetcites [37] source
- Supportedcites [50] source
The Introduction (p. 1) describes machine-learning models assisting human decision makers, and the Results (p. 8) report an experiment measuring human performance with real-time AI assistance. This paper is therefore an example of a study of AI-assisted individual decisions. The citing sentence’s recidivism example need not describe this particular study, whose task was deceptive-review detection. (LLM, conf=0.91)
- Supportedcites [51] source
The Introduction (p. 1) says the study investigates machine-learning assistance in human decision-making and focuses on individual predictions about whether hotel reviews are deceptive. The Concluding Discussion (p. 8) reports the effects of machine predictions and explanations on human performance. This makes the paper an example of the studies cited in the sentence; recidivism is an example in that sentence, not the task this paper studied. (LLM, conf=0.94)
- Not checked yetcites [55]
- Not checked yetcites [58]
- Supportedcites [69] source
The Introduction describes models aiding people’s decisions, including in judicial sentencing and bail. Related Work discusses research on people’s use of computational decision-making aids, and the paper’s own experiments examine how model predictions affect individual decisions. The recidivism example is consistent with its discussion of judicial applications, though it is not the subject of its experiments. (LLM, conf=0.91)
- Not checked yetcites [83] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Notably, the task for humans to create conditional delegation rules share some similarity with what many social media moderators are already doing by writing manual automation rules to deal with the massive amount of comments (Fig. 1(C)).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
What the sources say
- Supportedcites [18] source
Section 3.3.1 describes Reddit moderators using hard-coded AutoModerator rules to automate routine moderation. Section 4.4 says moderators configure such rules and describes Crossmod's similar if-condition-then-action format; Section 1.2 links automation to the volume of content. (LLM, conf=0.90)
- Supportedcites [44] source
Section 5.2.1 says Reddit moderators define Automod rules themselves, including rules that automatically remove comments. Section 5.2.2 says moderators use extensive rules to handle moderation work at scale. This supports the citing sentence’s comparison to humans writing conditional delegation rules. (LLM, conf=0.92)
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, there exists a need for audit tools to monitor the performance of the keyword rules.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
- Supportedcites [44] source
Section 6.2 explicitly recommends audit tools that let moderators see how each Automod rule affects moderation and identify mistakes. Section 5.4.2 identifies the need for data on which keyword-based rules were triggered and how often. Together, these support the claimed need to monitor keyword-rule performance. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Supported
In the paper
“This model can achieve competitive accuracy while having built-in interpretability because the prediction is made exclusively based on rationales.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
- Not checked yetcites [52] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In short, this model achieves competitive accuracy while having the ability to provide explanations directly by showing the rationales on which the prediction is based on.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.1 Model Development
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe a clear performance drop out-of-distribution (e.g., F1 drops from about ∼0.8 to ∼0.6), which validated our choice of Reddit as an out-of-distribution scenario.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.1 Model Development
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Overall, participants report a good global understanding of the model (M=3.37,SD=0.97) and local understanding of individual predictions (M=3.56,SD=1.05) on Wiki-Attack than Reddit, possibly related to the diference in model performance between distribution types.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > Summary.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In particular, on WikiAttack, the model working alone already outperforms the manual condition, and conditional delegation further improves the precision.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Two-way ANOVA fnds a statistically signifcant efect of distribution type and interaction between distribution type and experimental condition (p <0.001).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The reward mechanism assumes that the cost of wrongly reported non-toxic comments (false positive) equals the benefit of correctly reported toxic comments (true positive).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Fig. 14a shows the time spent on the interactive interface in each condition: participants with predicted labels only spent the most time on this task.This result is consistent with the number of actions because it likely requires more time to take more actions.However, the diference is relatively weak: one-way ANOVA shows that the efect of experimental conditions is only borderline signifcant (p”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.2 Efciency and Engagement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Reddit, we found the global explanation condition is worse than the manual condition (p =0.004), and only showing prediction labels (p =0.028).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We are able to achieve complementary performance in-distribution on WikiAttack, but not out-ofdistribution on Reddit.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The efect of distribution type is the most salient, suggesting a clear diference between in-distribution and out-of-distribution.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The word does not show up in top 10 for the manual condition, but shows up in the other conditions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > What rules do people make?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, adding the global explanation feature seems to unexpectedly hurt people's ability in choosing rules with both high coverage and high precision, and lead to slightly lower human-AI performance in union precision and reward.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Global explanations lead to much higher overlap between most frequent words in rationales (Fig. 14c).”
- Also stated in: § 5.2 Efciency and Engagement · plain
“Fig. 14c shows that global explanations lead to much higher overlap than the other conditions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.2 Efciency and Engagement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Also stated in: § 5.2 Efciency and Engagement · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One-way ANOVA also fnds a statistically signifcant efect in experimental condition (p < 0.001).Post-hoc Tukey's HSD shows the negative diference between the manual condition and other experimental conditions are all statistically signifcant (p <0.001 for predicted labels and global explanations, p =0.009 for local explanations).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.2 Efciency and Engagement
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On Reddit, the experimental condition also has a statistically signifcant efect (p =0.018).Post-hoc Tukey's HSD shows that only the diference between the manual condition and global explanations is signifcant (p =0.018).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The reward mechanism assumes that the cost of wrongly reported non-toxic comments (false positive) equals the benefit of correctly reported toxic comments (true positive).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conditional delegation based on these words leads to greater precision than the model working alone (dashed lines), suggesting that users can improve the precision of the model by identifying these words for conditional delegation.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.2 Performance of Individual Words
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all evaluation measures of engagement and efciency, we only fnd statistically signifcant efects of experiment conditions, suggesting that patterns with two distribution types are comparable.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.2 Efciency and Engagement
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For WikiAttack, we observe a weak trend that local explanations lead to less subjective workload (lower mental demand, more feeling of success, and less negative emotion) while adding global explanation has the opposite efect.These patterns, however, do not hold for Reddit.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.2 Efciency and Engagement
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For average precision (Fig. 10), experimental condition has a signifcant efect in Wiki-Attack (p <0.001), but not in Reddit (p =0.864).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On WikiAttack (Fig. 11c), we fnd a statistically signifcant efect of experimental condition (p =0.01), and post-hoc Tukey's HSD shows that the manual condition is signifcantly worse than other experimental conditions with delegation support features (p <0.001 for global explanations, p =0.007 for local explanations, and p =0.004 for predicted labels).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The reward mechanism assumes that the cost of wrongly reported non-toxic comments (false positive) equals the benefit of correctly reported toxic comments (true positive).
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is a non-signifcant trend that conditions with explanation, especially local explanation, result in better confdence for Wiki-Attack, but not for Reddit, where the model performs relatively poorly.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > Summary.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, participants were neutral about whether the task was mentally demanding (M=3.15,SD=1.08), agreed that they felt relatively successful in accomplishing the task (M=3.95,SD=0.91), and disagreed that they felt negative emotions (M=1.93,SD=1.03).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.2 Efciency and Engagement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, participants report a better understanding of the model as a whole and individual predictions on WikiAttack than Reddit, although the diferences are not statistically signifcant.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can immediately observe diferences between Reddit and WikiAttack:”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants spent more time working on conditional delegation than the manual condition, but the efciency is improved with explanations, especially global explanations.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Second, the nature of comments on Wikipedia and Reddit difers substantially.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is impressive that crowdworkers can already create keyword-based rules that achieve greater precision than the model working alone.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conditional delegation with all delegation support features leads to much better engagement and more submitted rules than the manual condition.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is worth noting that local explanations seem to have less of a priming efect than global explanations but still improves efciency.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Having an appropriate mental model allows people to accurately anticipate a system's behaviors and interact more efectively.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.2 AI explanations for human-AI interaction
What the sources say
- Not checked yetcites [65]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“One such example is content moderation, where moderator decisions on individual comments for further actions (e.g., hiding the content or prompting further review, depending on the community policy) are of limited consequence; instead the key challenge lies in dealing with the massive scale of comments.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 1 INTRODUCTION
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Coverage warrants that the AI model can identify as many toxic comments as possible to alleviate the scalability issues.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“HCI studies on XAI have found explanations to improve user understanding of AI systems [12,22,32], and somewhat mixed results on enhancing user trust [22], satisfaction [32] and willingness to adopt AI systems [80].”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.2 AI explanations for human-AI interaction
What the sources say
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach employs a greater level of automation than human-AI collaboration on every single decision and provides human with active control on when to use an AI model and in what ways.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“This approach, however, misses out the benefit of AI especially since rigid rules often do not work on informal languages such as social media posts (e.g., containing swear words without being toxic).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
What the sources say
- Not checked yetcites [18] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, human-AI collaboration on every single decision is not scalable and is thus less appropriate for tasks involving a large amount of relatively low-stakes decisions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Precision ensures that AI behavior is indeed trustworthy in the delegation mode and avoids unnecessary actions, whether it is mistaken deletion or extra work for further review.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > No action
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, to the best of our knowledge, our work is the frst study with controlled experiments to examine the efect of conditional delegation.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2 RELATED WORK 2.1 Human-AI Collaboration
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“This approach tends to focus on high-stakes decisions and are not scalable in the number of decisions because humans are required to make each decision.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2 RELATED WORK 2.1 Human-AI Collaboration
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“HCI studies on XAI have found explanations to improve user understanding of AI systems [12,22,32], and somewhat mixed results on enhancing user trust [22], satisfaction [32] and willingness to adopt AI systems [80].”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.2 AI explanations for human-AI interaction
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“People's mental model can be refned by explanations of how the system works.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.2 AI explanations for human-AI interaction
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“The empirical results, however, are mixed at best, because there was either insufcient complementarity in human and AI's domain knowledge or a lack of ability for people to judge the reliability of AI recommendations.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2 RELATED WORK 2.1 Human-AI Collaboration
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such rules can be created for any model of choice, so it is orthogonal to the research on improving the capability of AI.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Chiang and Yin [23] examined human reliance on the model in human-AI decision making and found that surprisingly humans rely on AI more out-of-distribution, where the AI performance is worse.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.3 Distribution Shift and Experimental Studies on Out-of-distribution Examples
What the sources say
- Not checked yetcites [23] source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“They suggested that complementary performance is more plausible for out-of-distribution examples because of AI's performance drop.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.3 Distribution Shift and Experimental Studies on Out-of-distribution Examples
What the sources say
- Not checked yetcites [56]
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Chandrasekharan et al. [18] also found that hard-coded rules are prone to mistakes.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
- Not checked yetcites [18] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“models are trained with historical data and can present issues such as gender bias and racial bias in AI models [68,73], potentially exacerbating structural inequalities [10]”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Through interviews with 16 moderators, Jhaver et al. [44] found that AutoModerator improves the efciency of moderation.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
- Not checked yetcites [44] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, the datasets that participants explore to create rules are of comparable size between Wikipedia (in-distribution) and Reddit (out-of-distribution).”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.1 Model Development
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The existence of distribution shift is a strong motivation for some form of conditional delegation so that humans can identify the trustworthy regions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.3 Distribution Shift and Experimental Studies on Out-of-distribution Examples
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“They also highlight the fact that AutoModerator fundamentally changes the work of moderators and may introduce additional unnecessary work.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
- Not checked yetcites [44] source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Note that this explanation method has attracted some criticism for producing rationales that don't necessarily align with human reasoning [90], but it has the advantage of producing rationales that are, by construction, sufcient”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.1 Model Development
What the sources say
- Not checked yetcites [90]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The diference between the training dataset and the deployment data is called distribution shift, which often results in a performance drop [24,46,61].”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.3 Distribution Shift and Experimental Studies on Out-of-distribution Examples
What the sources say
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our setup, however, as we conditionally delegate decisions to AI, strong AI performance in-distribution is likely more critical for the human-AI collaborative performance.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.3 Distribution Shift and Experimental Studies on Out-of-distribution Examples
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The key advantage of this method is that it is entirely under the control of moderators.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Conditional delegation combines traditional rule-based approaches and AI models by providing moderators with the ability to decide when to trust or distrust the AI model.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It can also be used to tailor diferent requirements of precision and tune the tradeofs between false positives and false negatives.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“For instance, Reddit moderators can confgure an AutoModerator bot to set rules for reporting or deleting all comments that contain certain words.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
- Not checked yetcites [44] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“The scale of content is the most important argument for some form of automation in content moderation [33,35].”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 2.4 Content Moderation
What the sources say
- Not checked yetcites [33]
- Not checked yetcites [35] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In short, our results demonstrate that conditional delegation is more efective than the model working alone, and that even laypeople are able to create high-quality conditional delegation rules for content moderation.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > What rules do people make?
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Turkers are a sufficiently good sample to compare whether conditional delegation improves content moderation outcomes over the baseline condition.
- Rests on an assumption: Social media content moderation is often performed by part-time volunteers whose expertise varies.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to the manual rule-based approach currently used for content moderation, advantage of our human-AI collaborative approach via conditional delegation may depend on the base performance of the AI, and may not be sufcient if the AI signifcantly under-performs, e.g., when used on out-ofdistribution comments.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > What rules do people make?
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: In practice, an AI model rarely faces exactly the same distribution in deployment.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We employ a between-subject design with the four conditions described above.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 4.2 Study Design and Procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, subjective measures show a relatively positive experience across board, but not strong diference between conditions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > Summary.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, conditional delegation can potentially alleviate AI bias as we give users the freedom to choose trustworthy regions based on their domain knowledge or notion of fairness.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we are able to recognize such words with the beneft of a fully-labeled dataset (i.e., oracle access) -discovering them in a real-world setting could be very challenging.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.2 Performance of Individual Words
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As such, we recruited 30 participants for each condition, summing up to 120 participants each for in-distribution and out-of-distribution and 240 participants in total.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 4.2 Study Design and Procedure
plainn = 240- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is some evidence that when the model performs well (WikiAttack), local explanations provide the best experience: strong performance with relatively high efciency, less subjective workload and more confdence in the outcomes.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > Summary.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We conducted pilot studies with lab mates at the same university and mechanical turkers to gather feedback and ensured that interactions were intuitive and the interface was easy to use.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These observations demonstrate that humans, in our case turkers who are not experts in content moderation, are able to create rules that improve model precision, suggesting that conditional delegation can be a promising direction to pursue.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.1 Efcacy
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: Turkers are a sufficiently good sample to compare whether conditional delegation improves content moderation outcomes over the baseline condition.
- Rests on an assumption: Social media content moderation is often performed by part-time volunteers whose expertise varies.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Providing explanations can improve efciency by hinting on rules to consider, but can also mislead people to use high-frequency but not necessarily high-precision rules.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > Summary.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We aimed to obtain 0.95 power to detect an efect size of 0.1 in precision at the standard 0.01 alpha error probability using F-tests.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 4.2 Study Design and Procedure
plain- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Taken together, our results show that people are more engaged when performing conditional delegation than working on creating manual rules, with more actions and a tendency to spend more time on the task.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 5.2 Efciency and Engagement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This observation suggests that the delegation support features can help users identify good rules when the model performs well, but may mislead people when the model performs poorly.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > What rules do people make?
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The study is approved by the University IRB.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 4.2 Study Design and Procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, even a very conservative application of this model is still producing 2 false positives for every 3 true positives-impractical for use by real moderators, and something we would like to be able to improve on via conditional delegation.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 3.1 Model Development
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure high quality responses, all participants satisfy the following criteria: (1) performed at least 1000 HITs; (2) approval of 99% performed HITs in previous requesters; (3) reside in the United States; 4) has the adult content qualifcation since our task shows toxic comments.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > Participant information.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The experiment follows a between-subject design therefore we do not allow any repeated participants.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > Participant information.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“After the instruction, participants are asked three attentioncheck questions to allow us to flter out participants who did not pay attention to the instructions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > 4.2 Study Design and Procedure
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, crowdworkers are not representative of moderators, who have way more experience with their platform's data.”
- Also stated in: § The promise of conditional delegation. · plain
“Second, our participants are not representative of content moderators.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: Moderators are more familiar with the moderation process and more knowledgeable about important words.
- Also stated in: § The promise of conditional delegation. · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the fip side is that this process could introduce human bias, if for example one's notion of fairness is ill conceived.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“First, our work represents one instantiation of conditional delegation.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, prior work has shown that moderators often update the rules used by AutoModerator [18,44] and our work does not take into account any future updates.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Third, prior work has shown that moderators often update the rules used by AutoModerator [18,44] and our work does not take into account any future updates.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Depending on the workflow, avoiding false negatives could be important in other instantiations.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, our choice of model, datasets, and explanations afect the experimental outcome.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Last but not least, content moderation involves a wide range of diferent rules beyond toxicity, and even the policies under the umbrella term of toxicity can vary, so the AI model that we uses represents a narrow component in content moderation.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Content moderation involves a wide range of different rules beyond toxicity.
- Rests on an assumption: Even the policies under the umbrella term of toxicity can vary.
None of the sources cited for this claim have been checked yet.
In the paper
“These observations echo concerns of unintended consequences with the use of explanations in human-AI collaboration [8,37,51].”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
What the sources say
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our case study shows the promise of conditional delegation, but further study is required in each application domain of interest to develop the best design for human-AI collaboration in identifying delegation conditions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We choose the model with 0.5 because it achieves competitive performance both in-distribution and out-of-distribution.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This tradeof between true positives, true negatives, false positives, and false negatives can vary in practice depending on the application and the actions taken according to AI predictions.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Neither do we leverage any existing rules that moderators have created.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we recognize that our experiment setup is only a frst step towards using conditional delegation in content moderation.”
§ Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation > The promise of conditional delegation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
The study assumes the same workflow as the manual rule-based approach: comments falling outside keyword rules are ignored without taking an action.
“We assume the same workflow in our study and only focus on the precision and coverage related metrics for comments within the scope of keywords rules.”
The paper relies on this without support.
Content moderators often do not have labeled comments to quantify model performance.
“When given an AI model, content moderators often do not have labeled comments to quantify model performance.”
The paper relies on this without support.
In content moderation, recall is often less of a priority given limited time for content moderators, who are often volunteers, to deal with massive incoming comments.
“In the context of content moderation, recall (identifying all toxic comments) is often less of a priority given the limited time for content moderators, who are often volunteers, to deal with a massive amount of incoming comments.”
The paper relies on this without support.
The current workflow using manual rule-based approach ignores comments falling outside the rules without taking an action.
“This is reflected in the current workflow using the manual rule-based approach (Fig. 1(C)), where comments falling outside the rules are ignored without taking an action.”
The paper relies on this without support.
In a real-world scenario, it is unlikely that models are used to classify data that is exactly the same as the training dataset.
“In a real-world scenario, it is unlikely that models are used to classify data that is exactly the same as the training dataset.”
The paper relies on this without support.
Current AI models rely on identifying patterns in training datasets.
“Current AI models rely on identifying patterns in training datasets.”
The authors state this as an assumption.
The reward mechanism assumes that the cost of wrongly reported non-toxic comments (false positive) equals the benefit of correctly reported toxic comments (true positive).
“To make the calculation easy to understand for participants, this reward makes a simplifed assumption that the cost of wrongly reported non-toxic comments (false positive) equals the beneft of correctly reported toxic comments (true positive).”
The authors state this as an assumption.
Turkers are a sufficiently good sample to compare whether conditional delegation improves content moderation outcomes over the baseline condition.
“Furthermore, we believe turkers are a sufciently good sample for us to compare whether conditional delegation improves the content moderation outcomes over the baseline condition, and expert users are likely to further enhance the improvement pattern, if any.”
The paper relies on this without support.
Social media content moderation is often performed by part-time volunteers whose expertise varies.
“We note that while this recruiting choice may limit the generalizability of our results, social media content moderation is often performed by part-time volunteers whose expertise varies.”
The paper relies on this without support.
In practice, an AI model rarely faces exactly the same distribution in deployment.
“In practice, it is rarely the case that an AI model faces exactly the same distribution in deployment.”
The paper relies on this without support.
Keyword-based rules are a reasonable approach in content moderation given that rule-based methods are already used in AutoModerator.
“Keyword-based rules are a reasonable approach in content moderation given that rule-based methods are already used in AutoModerator.”
The paper relies on this without support.
In practice, moderators usually have historical data on which moderation decisions were made.
“Second, in practice, moderators usually have historical data on which moderation decisions were made.”
The paper relies on this without support.
Content moderation involves a wide range of different rules beyond toxicity.
“Last but not least, content moderation involves a wide range of diferent rules beyond toxicity, and even the policies under the umbrella term of toxicity can vary, so the AI model that we uses represents a narrow component in content moderation.”
The paper relies on this without support.
Even the policies under the umbrella term of toxicity can vary.
“Last but not least, content moderation involves a wide range of diferent rules beyond toxicity, and even the policies under the umbrella term of toxicity can vary, so the AI model that we uses represents a narrow component in content moderation.”
The paper relies on this without support.
Moderators are more familiar with the moderation process and more knowledgeable about important words.
“As moderators are more familiar with the moderation process and more knowledgeable about important words, experts might fnd the interface more useful than crowdworkers.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“the empirical results from these studies are mixed at best [6,52,82,106].”
From Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making (2023) · cites this paper as [52]
Citations
Cites
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
A model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · cited 2×
“Extending existing models of human agency and automation [59, 67], we introduce proactive human agency, with which human can act and exercise control prior to model deployment, instead of reacting to model outputs.”
BackgroundMethodCategorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507
Explaining collaborative filtering recommendations
Herlocker, J. L., Konstan, J. A., & Riedl, J. (2000) · Proceedings of the 2000 ACM conference on Computer supported cooperative work · doi:10.1145/358916.358995 · Cited by 1,731
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Research works printed but not linked: 22
- Gagan Bansal , Besmira Nushi , Ece Kamar , Walter S Lasecki , Daniel S Weld , and Eric Horvitz . 2019 . Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. 2–11 . Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. 2–11.
- Noam Brown and Tuomas Sandholm . 2019. Superhuman AI for multiplayer poker. Science 365, 6456 ( 2019 ), 885–890. Noam Brown and Tuomas Sandholm. 2019. Superhuman AI for multiplayer poker. Science 365, 6456 (2019), 885–890.
- Samuel Carton , Qiaozhu Mei , and Paul Resnick . 2020 . Feature-Based Explanations Don’t Help People Detect Misclassifications of Online Toxicity . In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14 . 95–106. Samuel Carton, Qiaozhu Mei, and Paul Resnick. 2020. Feature-Based Explanations Don’t Help People Detect Misclassifications of Online Toxicity. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14. 95–106.
- Jonathan P. Chang and Cristian Danescu-Niculescu-Mizil . 2019 . Trajectories of Blocked Community Members: Redemption, Recidivism and Departure. In Proceedings of WWW. 184–195. Jonathan P. Chang and Cristian Danescu-Niculescu-Mizil. 2019. Trajectories of Blocked Community Members: Redemption, Recidivism and Departure. In Proceedings of WWW. 184–195.
- Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2019 . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding . In Proceedings of NAACL. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL.
- Casey Fiesler , Jialun Jiang , Joshua McCann , Kyle Frye , and Jed Brubaker . 2018 . Reddit rules! Characterizing an Ecosystem of Governance . In Proceedings of International AAAI Conference on Web and Social Media(ICWSM). Casey Fiesler, Jialun Jiang, Joshua McCann, Kyle Frye, and Jed Brubaker. 2018. Reddit rules! Characterizing an Ecosystem of Governance. In Proceedings of International AAAI Conference on Web and Social Media(ICWSM).
- Tarleton Gillespie . 2018. Custodians of the Internet: Platforms , Content Moderation, and the Hidden Decisions That Shape Social Media . Yale University Press , New Haven . Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media. Yale University Press, New Haven.
- Riccardo Guidotti , Anna Monreale , Salvatore Ruggieri , Franco Turini , Fosca Giannotti , and Dino Pedreschi . 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 ( 2018 ), 1–42. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2018), 1–42.
- Shagun Jhaver , Amy Bruckman , and Eric Gilbert . 2019 . Does transparency in moderation really matter? User behavior after content removal explanations on reddit . Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1–27. Shagun Jhaver, Amy Bruckman, and Eric Gilbert. 2019. Does transparency in moderation really matter? User behavior after content removal explanations on reddit. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–27.
- Sara Kiesler , Robert Kraut , Paul Resnick , and Aniket Kittur . 2012. Regulating behavior in online communities. Building successful online communities: Evidence-based social design ( 2012 ), 125–178. Sara Kiesler, Robert Kraut, Paul Resnick, and Aniket Kittur. 2012. Regulating behavior in online communities. Building successful online communities: Evidence-based social design (2012), 125–178.
- Jon Kleinberg , Himabindu Lakkaraju , Jure Leskovec , Jens Ludwig , and Sendhil Mullainathan . 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 ( 2018 ), 237–293. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The quarterly journal of economics 133, 1 (2018), 237–293.
- Zhiyuan “ Jerry” Lin, Jongbin Jung , Sharad Goel , Jennifer Skeem , 2020 . The limits of human predictions of recidivism. Science advances 6, 7 (2020), eaaz0652. Zhiyuan “Jerry” Lin, Jongbin Jung, Sharad Goel, Jennifer Skeem, 2020. The limits of human predictions of recidivism. Science advances 6, 7 (2020), eaaz0652.
- Han Liu Vivian Lai and Chenhao Tan. 2021. Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making. arXiv preprint arXiv:2101.05303(2021). Han Liu Vivian Lai and Chenhao Tan. 2021. Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Making. arXiv preprint arXiv:2101.05303(2021). · arXiv
- Brian Lubars and Chenhao Tan . 2019 . Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability . In Proceedings of NeurIPS. Brian Lubars and Chenhao Tan. 2019. Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability. In Proceedings of NeurIPS.
- Scott M Lundberg , Bala Nair , Monica S Vavilala , Mayumi Horibe , Michael J Eisses , Trevor Adams , David E Liston , Daniel King-Wai Low , Shu-Fang Newman , Jerry Kim, 2018 . Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering 2, 10 (2018), 749–760. Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering 2, 10 (2018), 749–760.
- J Nathan Matias . 2016 . The civic labor of online moderators . In Internet Politics and Policy conference . Oxford, United Kingdom. J Nathan Matias. 2016. The civic labor of online moderators. In Internet Politics and Policy conference. Oxford, United Kingdom.
- Scott Mayer McKinney , Marcin Sieniek , Varun Godbole , Jonathan Godwin , Natasha Antropova , Hutan Ashrafian , Trevor Back , Mary Chesus , Greg C Corrado , Ara Darzi , 2020. International evaluation of an AI system for breast cancer screening. Nature 577, 7788 ( 2020 ), 89–94. Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg C Corrado, Ara Darzi, 2020. International evaluation of an AI system for breast cancer screening. Nature 577, 7788 (2020), 89–94.
- Don Norman. 2013. The design of everyday things: Revised and expanded edition. Basic books. Don Norman. 2013. The design of everyday things: Revised and expanded edition. Basic books.
- Thomas O’Neill , Nathan McNeese , Amy Barron , and Beau Schelble . 2020. Human–autonomy teaming: A review and analysis of the empirical literature. Human Factors ( 2020 ), 0018720820960865. Thomas O’Neill, Nathan McNeese, Amy Barron, and Beau Schelble. 2020. Human–autonomy teaming: A review and analysis of the empirical literature. Human Factors (2020), 0018720820960865.
- Morgan Klaus Scheuerman Jialun Aaron Jiang Casey Fiesler and Jed R Brubaker. 2021. A Framework of Severity for Harmful Content Online. arXiv preprint arXiv:2108.04401(2021). Morgan Klaus Scheuerman Jialun Aaron Jiang Casey Fiesler and Jed R Brubaker. 2021. A Framework of Severity for Harmful Content Online. arXiv preprint arXiv:2108.04401(2021). · arXiv
- David Silver , Thomas Hubert , Julian Schrittwieser , Ioannis Antonoglou , Matthew Lai , Arthur Guez , Marc Lanctot , Laurent Sifre , Dharshan Kumaran , Thore Graepel , 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 ( 2018 ), 1140–1144. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140–1144.
- Michael Wiegand , Josef Ruppenhofer , and Thomas Kleinbauer . 2019 . Detection of Abusive Language: the Problem of Biased Datasets . In Proceedings of NAACL. Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In Proceedings of NAACL.
Cited by
Sorted by citations
Harnessing the power of synthetic data in healthcare: innovation, application, and privacy
Giuffrè, M., & Shung, D. L. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00927-3 · Cited by 392
Three Challenges for AI-Assisted Decision-Making
Steyvers, M., & Kumar, A. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231181102 · Cited by 217
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208
“Third, as modern AI is not only more performant and offers more application areas, but also complements humans [27, 28, 40], there is an increasing potential to achieve complementary team performance (CTP), i.”
BackgroundHelp Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
Defining human-AI teaming the human-centered way: a scoping review and network analysis
Berretta, S., Tausch, A., Ontrup, G., Gilles, B. T., Peifer, C., & Kluge, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1250725 · Cited by 234
Metadata
- Authors
- Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C.
- Affiliations
- University of Colorado Boulder (US)
- Amazon (United States) (US)
- Microsoft (Canada) (CA)
- Twitter (United States) (US)
- University of Chicago (US)
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-27
- Pages
- 1-18
- OpenAlex
- W4224940987
- Semantic Scholar
- 29c29d19a1d6fc62b19251547ce9df351e49763e
- Cited by
- 145
- References
- 63
- Citations by year
- 2026: 32 · 2025: 40 · 2024: 40 · 2023: 31 · 2022: 2
- FWCI
- 54.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nd)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National science foundation (2125116, 2125113, 1837986)
- IBM faculty award
- Topics
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- human-AI collaboration, content moderation, social media comments, AI explanations, human-AI interface
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
