Exploring the psychology of LLMs’ moral and legal reasoning
Open accessAbstract
Large language models (LLMs) exhibit expert-level performance in tasks across a wide range of different domains. Ethical issues raised by LLMs and the need to align future versions makes it important to know how state of the art models reason about moral and legal issues. In this paper, we employ the methods of experimental psychology to probe into this question. We replicate eight studies from the experimental literature with instances of Google's Gemini Pro, Anthropic's Claude 2.1, OpenAI's GPT-4, and Meta's Llama 2 Chat 70b. We find that alignment with human responses shifts from one experiment to another, and that models differ amongst themselves as to their overall alignment, with GPT-4 taking a clear lead over all other models we tested. Nonetheless, even when LLM-generated responses are highly correlated to human responses, there are still systematic differences, with a tendency for models to exaggerate effects that are present among humans, in part by reducing variance. This recommends caution with regards to proposals of replacing human participants with current state-of-the-art LLMs in psychological research and highlights the need for further research about the distinctive aspects of machine psychology.
Verified 2026-10-08 · 2 references not checked yet
AI snapshot
Written from the paper’s full text.
TL;DR
This paper uses experimental-psychology methods to compare four state-of-the-art LLMs—Gemini Pro, Claude 2.1, GPT-4, and Llama 2 Chat 70b—against human participants across eight moral and legal reasoning studies. It finds that alignment varies by study and model, with GPT-4 generally leading, but that even highly correlated LLM responses show systematic differences, often exaggerating human effects by reducing variance.
Key findings
- Alignment with human responses shifted across experiments and models; GPT-4 took a clear overall lead, but correlations varied widely, e.g., GPT-4 reached r = .96 for deception and r = .92 for rule violation, while its consent correlation was only r = .58.
- LLMs often amplified effects already present among humans, partly through reduced variance: in Study 1, mean LLM SDs for blame/praise were 0.30 (harm) and 0.77 (help), versus human SDs of 1.32 and 1.41.
- The “correct answer” effect appeared across tasks: GPT-4, Gemini Pro, and Claude 2.1 gave invariant intentionality answers in Study 1 (SDs = 0); GPT-4 rated objective probability as 50% in all 99 within-subjects hindsight trials; GPT-4, Claude 2.1, and Llama 2 Chat 70b always identified as politically “moderate” (SDs = 0).
- Consent judgments showed substantial mismatch: no model adequately followed the human pattern, with Llama 2 giving very low consent ratings (M = 2.22) and GPT-4 slightly higher ratings than humans (M = 6.18 vs. 5.70).
Methods
- The authors replicated eight vignette-based studies from experimental moral and legal psychology, using original stimuli and data where available, and generated 100 responses per model per study (200 for the consent study) via API calls to Gemini Pro, Claude 2.1, GPT-4, and Llama 2 Chat 70b.
- Model settings included temperatures of 0.95 for Gemini Pro, 0.85 for Claude 2.1, 1.2 for GPT-4, and 0.85 for Llama 2 Chat 70b, with a 600-token output limit and system prompts to elicit questionnaire-format answers.
- Analyses computed by-condition pairwise correlations between agents, per-condition standard deviations, and reproduced the original studies’ analyses on joint human–LLM data while adding agent main effects and interactions; materials, code, and data are available on OSF.
Limitations
- The paper states that the eight studies were not selected by an objective, systematic criterion, though the authors say they were not cherry-picked; the work is framed as exploratory.
- The authors state that temperature was not systematically manipulated and that temperature values are not interchangeable across models, so model differences may partly reflect different temperature settings; they call for future systematic manipulation.
- The paper reports frequent invalid or verbose responses, especially from Claude 2.1 and Llama 2 Chat 70b, reducing valid sample sizes and excluding some models from some analyses; it also notes that well-known older stimuli may appear in training data, though alignment was also observed for recent or unpublished studies.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 60 cited sources checked · 0 supported · 0 flagged
- Findingown data
Alignment between LLM-generated responses and human responses shifts from one experiment to another.
“We find that alignment with human responses shifts from one experiment to another, and that models differ amongst themselves as to their overall alignment, with GPT-4 taking a clear lead over all other models we tested.”
Quote in the paperSubstantial claim - Findingnone given
Claude 2.1, GPT-4, and Gemini Pro gave invariant answers to the intentionality question.
“Finally, Claude 2.1, GPT-4, and Gemini Pro gave invariant answers to the intentionality question.”
Quote in the paperSubstantial claim - Findingown data
On average, the standard deviation for blame or praise ratings in the harm and help conditions were lower for LLMs than for humans, with the exception of Gemini Pro in the help condition.
“On average, the standard deviation for blame or praise ratings in the harm and help conditions were lower for LLMs (Mean SD harm = 0.30, Mean SD help = 0.77) than for humans (SD harm = 1.32,SD help = 1.41), with the exception of Gemini Pro in the help condition (SD help = 1.65).”
Quote in the paperNot a substantial claim - Findingown data
Our preliminary tests confirmed that GPT-4 responses varied very little at the default temperature of 1.
“Our preliminary tests confirmed this.”
Quote in the paperNot a substantial claim - Findingown data
Exploratory testing has shown that the temperature parameter does not seem to be interchangeable between models.
“Exploratory testing has shown that the temperature parameter does not seem to be interchangeable between models.”
Quote in the paperNot a substantial claim
- Findingown data
GPT-4, Gemini Pro, and Llama 2 Chat 70b assigned overall more blame and praise than humans.
“Second, GPT-4, Gemini Pro, and Llama 2 Chat 70b assigned overall more blame and praise than humans.”
Quote in the paperSubstantial claim - Findingnone given
Claude 2.1 showed a significantly lessened sensitivity to condition in rendering blame or praise judgments when compared to all other LLMs and to humans.
“First, Claude 2.1 showed a significantly lessened sensitivity to condition in rendering blame or praise judgments when compared to all other LLMs and to humans.”
Quote in the paperSubstantial claim - Findingnone given
The same surprising patterns which are significant among ordinary humans are also significant with regards to responses generated by all the LLMs tested.
“The same surprising patterns which are significant among ordinary humans are also significant with regards to responses generated by all the LLMs we have tested.”
Quote in the paperSubstantial claim - Findingcitation
Across the board, LLMs' responses showed less variance than human-generated responses.
“In fact, across the board, LLMs' responses showed less variance than human-generated responses (Park et al., 2023).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Park et al., 2023
Not checked in the free run.
- Findingown data
The condition*agent interaction in intentionality judgments was driven by a drastically increased effect of condition for the intentionality ratings of Gemini Pro, Claude 2.1, and GPT-4.
“Individual χ² tests for each agent make it clear that this interaction was driven by a drastically increased effect of condition for the intentionality ratings of Gemini Pro, Claude 2.1, and GPT-4 (χ²s between 38 and 50, Vs = 1.00, ps < .001).”
Quote in the paperSubstantial claim - Findingown data
For GPT-4, subjective moral evaluation and the manipulation of a purpose's moral valence seem to independently affect rule violation judgments.
“In GPT-4's case, subjective moral evaluation and the manipulation of a purpose's moral valence seem to independently affect rule violation judgments.”
Quote in the paperSubstantial claim - Findingown data
Just as with human responses, the rule violation judgments of LLMs were sensitive to violations of both text and purpose.
“Just as with human responses, the rule violation judgments of LLMs were sensitive to violations of both text and purpose.”
Quote in the paperSubstantial claim - Findingown data
This two-way interaction was qualified by a three-way interaction in the case of GPT-4.
“This two-way interaction was qualified by a three-way interaction in the case of GPT-4.”
Quote in the paperNot a substantial claim - Findingown data
The only model which revealed the exact same significance patterns as humans was Gemini Pro.
“Thus, the only model which revealed the exact same significance patterns as humans was Gemini Pro.”
Quote in the paperSubstantial claim - Findingown data
Claude 2.1, GPT-4, and Llama 2 Chat 70b, but not Gemini Pro, showed smaller variance than humans in per-item analysis.
“Per-item analysis also revealed that Claude 2.1 (Mean SD = 0.74), GPT-4 (Mean SD = 0.79), and Llama 2 Chat 70b (Mean SD = 0.92), but not Gemini Pro (Mean SD = 1.59), showed smaller variance than humans (Mean SD = 1.23).”
Quote in the paperNot a substantial claim - Findingown data
Unlike Park et al. (2023)'s finding, our results were robust to order effects.
“Moreover, unlike their finding, our results were robust to order effects.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Park et al., 2023
Not checked in the free run.
- Findingnone given
Compared to humans, GPT-4 gave answers that were less favorable to deception.
“Furthermore, compared to humans, GPT-4 gave answers that were less favorable to deception.”
Quote in the paperNot a substantial claim - Findingown data
All models displayed strong correlation to the overall mean human answer considering the mean score of all 15 questions, with Gemini being the lowest at 0.57.
“Turning to MFQ scores, we can see that all models displayed strong correlation to the overall mean human answer considering the mean score of all 15 questions, with Gemini being the lowest at 0.57.”
Quote in the paperSubstantial claim - Findingown data
The purpose * condition interaction remained significant for Claude 2.1 and GPT-4.
“On the other hand, the purpose * condition interaction remained significant for Claude 2.1 and GPT-4.”
Quote in the paperSubstantial claim - Findingown data
LLMs differed significantly from humans in a considerable number of the MFQ items.
“Looking at the per-item contrasts between agents, we find that LLMs differed significantly from humans in a considerable amount of the items.”
Quote in the paperSubstantial claim - Findingown data
Gemini Pro was more likely to rate deception as permissible in some cases where humans clearly see it as forbidden.
“On the one hand, it was more likely to rate deception as permissible in some cases where humans clearly see it as forbidden.”
Quote in the paperSubstantial claim - Findingown data
LLM responses align consistently with the expected deontic status, similar to human participants.
“Investigating individual patterns reveals that, similar to human participants, LLM responses align consistently with the expected deontic status.”
Quote in the paperSubstantial claim - Findingown data
Our models also reveal several systematic differences between human responses and LLM-generated responses.
“However, our models also reveal several systematic differences between human responses and LLM-generated responses.”
Quote in the paperSubstantial claim - Findingown data
Gemini Pro was less consistent than GPT-4.
“Gemini Pro, on the other hand, was less consistent.”
Quote in the paperNot a substantial claim - Findingown data
On average, per-condition standard deviations were lower for LLMs than for humans.
“On average, per-condition standard deviations were lower for LLMs (Mean SD GPT-4 = 0.16, Mean SD Gemini Pro = 0.30) than for humans (Mean SD = 0.53).”
Quote in the paperNot a substantial claim - Findingown data
After Bonferroni corrections, there was no significant difference between conditions for any MFQ item across LLMs.
“After Bonferroni corrections for multiple comparisons, we found no significant difference between conditions for none of the MFQ items across LLMs.”
Quote in the paperSubstantial claim - Findingown data
With the exception of Gemini Pro, all LLMs correlated highly with per-item human responses.
“With the exception of Gemini Pro (r df = 13 = .57[.08,.84], p = .027),all LLMs correlated highly with per-item human responses (rs between .82 and .86,all ps < .001).”
Quote in the paperNot a substantial claim - Findingown data
Like GPT-4, Gemini Pro was less likely to judge deception to be obligatory, even when humans do.
“On the other hand, like GPT-4, it was less likely to judge deception to be obligatory, even when humans do.”
Quote in the paperNot a substantial claim - Findingown data
LLMs also tended to correlate highly with each other.
“Moreover, LLMs also tended to correlate highly with each other (see Supplementary Table 2).”
Quote in the paperNot a substantial claim - Findingown data
The purpose * condition interaction term reached statistical significance for all models.
“The purpose * condition interaction term, described in the original study as the crucial effect, reached statistical significance for all models (all Fs > 15, all ps < .001;see Supplementary Table 4).”
Quote in the paperSubstantial claim - Findingown data
For Gemini Pro, the vast majority (42 out of 47) of responses reproduced the same political orientation value.
“For Gemini Pro, although there was some variation, the vast majority (42) out of the 47 responses also reproduced the same value.”
Quote in the paperNot a substantial claim - Findingown data
There were moderate to high correlations between human-and AI-generated responses.
“In fact, there were moderate to high correlations between human-and AI-generated responses.”
Quote in the paperSubstantial claim - Findingown data
Humans and LLMs alike gave more weight to morally good purposes than to morally bad ones.
“Moreover, humans and LLMs alike gave more weight to morally good purposes than to morally bad ones.”
Quote in the paperSubstantial claim - Findingown data
Per-foundation contrasts show no consistent behavior on Human vs AI for Care, Fairness, or Authority.
“Per-foundation contrasts show no consistent behavior on Human vs AI for Care, Fairness, or Authority.”
Quote in the paperSubstantial claim - Findingown data
Adding subjective moral evaluation ratings to models of LLM-generated results revealed interesting differences between models.
“Adding subjective moral evaluation ratings to models of LLM-generated results revealed interesting differences between models.”
Quote in the paperSubstantial claim - Findingown data
The addition of the new term absorbed the variance accounted for by the condition term for all but Llama 2 Chat 70b.
“The addition of the new term absorbed the variance accounted for by the condition term for all but Llama 2 Chat 70b (F = 6.92, p = .010).”
Quote in the paperSubstantial claim - Findingown data
In our data, the models consistently identified as moderate, without left or right leaning tendency.
“However, in our data they consistently identified as moderate, without left or right leaning tendency in this question.”
Quote in the paperSubstantial claim - Findingown data
The difference between conditions for objective probability, subjective probability, recklessness and negligence variables were much reduced in a within-subjects design for GPT-4, Gemini Pro, and Claude 2.1.
“The difference between conditions for the objective probability, subjective probability, recklessness and negligence variables were much reduced in a within-subjects design for GPT-4, Gemini Pro, and, Claude 2.1 (see Figure 6).”
Quote in the paperSubstantial claim - Findingown data
For Claude 2.1, the effects of condition remained significant and large (ts > 12.1, ps < .001, ds between 1.73 and 6.37) for all but the objective probability measure (t = 1.00, p = .32, d = 0.22).
“For Claude 2.1,the effects of condition remained significant and large (ts > 12.1, ps < .001,ds between 1.73 and 6.37) for all but the objective probability measure (t = 1.00, p = .32,d = 0.22).”
Quote in the paperSubstantial claim - Findingown data
In the within-subjects design, at least three distinct patterns of significance and departure from human participants were observed among the four LLMs tested.
“On the other hand, in the within-subjects design, we observed at least three distinct patterns of significance and departure from human participants among the four LLMs we tested.”
Quote in the paperSubstantial claim - Findingown data
GPT-4 showed no variation at all for objective probability, rating it as 50% for both conditions in all 99 trials.
“In yet another demonstration of the”
Quote in the paperNot a substantial claim - Findingown data
The significant effects detected were much larger than those observed among humans.
“The significant effects detected were much larger than those observed among humans (for whom the largest effect had d = 0.83).”
Quote in the paperSubstantial claim - Findingown data
Overall, there were very strong correlations between the per cell means of each of the LLMs and human responses (rs > .82, ps < .001).
“Overall, we found very strong correlations between the per cell means of each of the LLMs and human responses (rs > .82,ps < .001;see Supplementary Table 7).”
Quote in the paperSubstantial claim - Findingown data
In the LLM causation study, only responses generated by GPT-4 correlated significantly with those produced by humans (r df=14 = .68 [.28, .88], p = .004).
“Only responses generated by GPT-4 correlated significantly with those produced by humans (r df=14 = .68[.28, .88],p = .004).”
Quote in the paperSubstantial claim - Findingown data
Model-generated answers varied less than those of humans.
“Moreover, analyzing the per-cell variance revealed that model-generated answers varied less (Mean SD = 0.88) than those of humans (Mean SD = 1.62).”
Quote in the paperNot a substantial claim - Findingown data
For GPT-4 and Gemini Pro, none of the differences for objective probability, subjective probability, recklessness and negligence reached statistical significance (all |t|s < 1.21, all ps > .23).
“In fact, for GPT-4 and Gemini Pro, none of those differences reached statistical significance (all |t|s < 1.21, all ps > .23).”
Quote in the paperSubstantial claim - Findingown data
For Claude 2.1, the difference between judgments of blame (d between = 4.47, d within = 6.37) and punishment (d between = 2.86, d within = 5.62) was amplified, instead of dampened, in the within-subjects design.
“The difference between judgments of blame (d between = 4.47, d within = 6.37) and punishment (d between = 2.86, d within = 5.62) was amplified, instead of dampened, in the within-subjects design.”
Quote in the paperNot a substantial claim - Findingnone given
Llama 2 Chat 70b showed the opposite pattern, with effects for all but the subjective probability variable increasing in a within-subjects design.
“Finally, Llama 2 Chat 70b showed the opposite pattern, with effects for all but the subjective probability variable increasing in a within-subjects design.”
Quote in the paperSubstantial claim - Findingown data
With the exception of Llama 2 Chat 70b, ex ante probability judgments did not differ between conditions for the LLMs tested.
“First, with the exception of Llama 2 Chat 70b (t = 3.35, p = .002,d = 0.60), ex ante probability judgments did not differ between conditions for the LLMs we tested (|t|s < 2.01, ps > .057).”
Quote in the paperSubstantial claim - Findingown data
Claude 2.1 performed similarly in that the effects of condition over the four variables (objective probability, subjective probability, recklessness, negligence) was dampened in the within-subjects design.
“Claude 2.1 performed similarly in that the effects of condition over those four variables was dampened in the within-subjects design.”
Quote in the paperNot a substantial claim - Findingown data
The only other significant pairwise correlation in the causation study was between GPT-4 and Gemini Pro (r df=14 = .60 [.15, .84], p = .014).
“Moreover, the only other significant pairwise correlation was that between GPT-4 and Gemini Pro (r df=14 = .60[.15, .84],p = .014,;see Supplementary Table 10 for all pairwise correlations).”
Quote in the paperSubstantial claim - Findingown data
With the exception of Gemini Pro, which produced very few valid responses, all other LLMs showed an increased sensitivity to the outcome when compared to humans.
“With the exception of Gemini Pro, which produced very few valid responses, all other LLMs showed an increased sensitivity to the outcome when compared to humans.”
Quote in the paperSubstantial claim - Findingown data
LLM answers varied slightly less (Mean SD = 7.19) than human answers (Mean SD = 9.99) in the within-subjects study.
“Comparing average per-condition standard deviations revealed that LLM answers varied slightly less (Mean SD = 7.19) than human answers (Mean SD = 9.99).”
Quote in the paperNot a substantial claim - Findingown data
The only significant correlation between agents in the consent study was between Llama 2 Chat 70b and Claude 2.1 (r = 0.76 [0.20, 0.95], p = .017).
“The only significant correlation was between Llama 2 Chat 70b and Claude 2.1 (r = 0.76 [0.20, 0.95], p = .017;see Supplementary Table 9).”
Quote in the paperSubstantial claim - Findingown data
LLM responses were often correlated with each other.
“as LLM responses were often correlated with each other”
Quote in the paperNot a substantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The correlation between per-condition means is a valid measure of similarity between human and LLM responses.
- Findingreasoning
For most of the studies where increased effect sizes were found, there were also reductions in variance among LLM responses compared to human-generated responses.
“for most of the studies where we found increased effect sizes, we also found reductions in variance among LLM-responses when compared to human-generated responses.”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The reduced variance of LLM-generated responses is not an artifact of the measurement or prompting.
- Findingown data
GPT-4 again demonstrated the 'correct answers' effect: for all situations where there wasn't a norm violation, GPT-4 selected '4' as an answer.
“Turning to variance, GPT-4 again demonstrated the”
Quote in the paperNot a substantial claim - Findingown data
Whenever the protagonist's behavior was a necessary but not sufficient contributor to the outcome, all models were more likely to agree that the protagonist caused the outcome more when they violated a norm (0.24 < d < 3.26).
“Whenever the protagonist's behavior was a necessary but not sufficient contributor to the outcome, all models were more likely to agree that the protagonist caused the outcome more when they violated a norm (0.24 < d < 3.26).”
Quote in the paperSubstantial claim - Findingnone given
With the exception of Claude 2.1, the 95% confidence interval of the Cohen's D estimate did not contain 0 for any model.
“With the exception of Claude 2.1, the 95% confidence interval of the Cohen's D estimate did not contain 0 for any model.”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The reduced variance of LLM-generated responses is not an artifact of the measurement or prompting.
- Findingown data
Attenuated versions of this trend of increasing effect sizes partly by reducing variance were also clearly present for Studies 2 (Deception), 5, and 6 (Hindsight Bias).
“Attenuated versions of this trend of increasing effect sizes partly by reducing variance were also clearly present for Studies 2 (Deception, reported in Section 2.2), 5, and 6 (Hindsight Bias, reported in Sections 2.5.1 and 2.5.2).”
Quote in the paperSubstantial claim - Findingown data
GPT-4 and Gemini Pro were more likely to agree that the norm-violating protagonist was the cause even in disjunctive structures (d GPT-4 = 1.2 [0.58,1.82], d Gemini Pro = 0.71 [0.08,1.35]).
“On the other hand, GPT-4 and Gemini Pro were more likely to agree that the norm-violating protagonist was the cause even in disjunctive structures (d GPT-4 = 1.2 [0.58,1.82],d Gemini Pro = 0.71 [0.08, 1.35]).”
Quote in the paperSubstantial claim - Findingown data
The exact opposite ordering of consent judgments was observed in Gemini Pro (M Police =4.66, M Medical =4.58, M Sexual =3.44).
“The exact opposite ordering was observed in Gemini Pro (M Police =4.66, M Medical =4.58, M Sexual =3.44).”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Findingown data
The per-study ranking for other models varied substantially across different studies.
“The per-study ranking for other models varied substantially across different studies (see Table 1).”
Quote in the paperNot a substantial claim - Findingown data
This same order of consent judgments was observed in Llama 2, albeit with much lower means (M Sexual =2.68, M Medical =2.03, M Police =1.85).
“This same order was observed in Llama 2, albeit with much lower means (M Sexual =2.68, M Medical =2.03, M Police =1.85).”
Quote in the paperNot a substantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Findingown data
Despite similarities in cross-study variation, GPT-4 was consistently more well-aligned with human responses than other models (with the exception of Claude 2.1 in Study 3).
“Despite those similarities in cross-study variation, GPT-4 was consistently more well-aligned with human responses than other models (with the exception of Claude 2.1 in Study 3).”
Quote in the paperSubstantial claim - Findingown data
Significance patterns for the crucial prescriptive abnormality and prescriptive abnormality * causal structure terms varied substantially across agents.
“Significance patterns for the crucial prescriptive abnormality and prescriptive abnormality * causal structure terms varied substantially across agents.”
Quote in the paperNot a substantial claim - Findingown data
LLM responses had on average less variance than human responses (Mean SD conjunctive/no violation = 1.20, Mean SD conjunctive/violation = 1.47, Mean SD disjunctive/no violation = 1.35, Mean SD disjunctive/violation = 1.70).
“LLM responses had on average less variance (Mean SD conjunctive/no violation = 1.20,Mean SD conjunctive/violation = 1.47,Mean SD disjunctive/no violation = 1.35,Mean SD disjunctive/violation = 1.70).”
Quote in the paperSubstantial claim - Findingown data
GPT-4's responses correlated much more strongly with human responses in study 2 (deception) and study 4 (rule violation) than in study 7 (Consent).
“GPT-4's responses correlated much more strongly with human responses in studies 2 (deception, r n = 7 = .96[.82, .88],see Section 2.6) and 4 (rule violation, r n = 31 = .92[.83, .96],see Section 2.4) than in study 7 (Consent, r n = 9 = .58[-.14, .90],see Section 2.6).”
Quote in the paperNot a substantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The correlation between per-condition means is a valid measure of similarity between human and LLM responses.
- Findingreasoning
The effects of the experimental manipulations tested were often more pronounced for LLMs than for humans.
“the effects of the experimental manipulations we have tested were often more pronounced for LLMs than for humans”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
- Findingown data
The highest standard deviation for every unique combination of causal structure and prescriptive abnormality was produced by humans (SD conjunctive/no violation = 2.11, SD conjunctive/violation = 1.78, SD disjunctive/no violation = 1.93, SD disjunctive/violation = 2.05).
“More broadly, the highest standard deviation for every unique combination of causal structure and prescriptive abnormality was produced by humans (SD conjunctive/no violation = 2.11, SD conjunctive/violation = 1.78,SD disjunctive/no violation = 1.93,SD disjunctive/violation = 2.05).”
Quote in the paperSubstantial claim - Findingreasoning
Some tasks generated higher overall agreement between humans and LLMs than others.
“some tasks generated higher overall agreement between humans and LLMs than others.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The correlation between per-condition means is a valid measure of similarity between human and LLM responses.
- Findingown data
Claude 2.1, GPT-4, and Gemini Pro gave invariant responses in Study 1, always expressing that the chairman acted intentionally when he harmed the environment and unintentionally when he helped it.
“Although the effect is heightened for all models, it is striking that Claude 2.1, GPT-4, and Gemini Pro gave invariant responses, always expressing that the chairman acted intentionally when he harmed the environment and unintentionally when he helped it.”
Quote in the paperSubstantial claim - Findingown data
As with humans, GPT-4 ratings for 'Mere Capacity' were significantly higher than in the 'Lacks Capacity' condition (b=-0.667, t=5.88, p < .001).
“As with humans, GPT-4 ratings for”
Quote in the paperNot a substantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
- Findingown data
GPT-4's judgments of consent followed a distinct ranking, with the highest judgments for the 'Police' scenario (M=6.47), followed by the 'Sexual' (M=6.12) and 'Medical' (M=6.05) scenarios.
“Finally, GPT-4's judgments of consent followed a distinct ranking, with the highest judgments for the”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Findingown data
For Claude 2.1 and Llama 2 there was no significant difference between the 'Mere Capacity' and 'Lacks Capacity' conditions.
“For Claude 2.1 and Llama 2 there was no significant difference between these conditions.”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
- Findingown data
As with humans, Claude 2.1 and Llama 2 Chat 70b were more likely to disagree that the protagonist was the cause when they violated a norm compared to cases where no norm was violated (d Claude 2.1 = -0.48 [-1.44,0.19], d Llama 2 Chat 70b = -0.33 [-0.91,0.26]).
“As with humans, Claude 2.1 and Llama 2 Chat 70b were more likely to disagree that the protagonist was the cause when they violated a norm when compared to cases where no norm was violated (d Claude 2.1 = -0.48[-1.44,0.19],d Llama 2 Chat 70b = -0.33 [-0.91, 0.26]).”
Quote in the paperSubstantial claim - Findingown data
There was substantial variation in the strength of pairwise correlations between human- and machine-generated answers across different models and, for each model, across different studies.
“there was substantial variation in the strength of pairwise correlations between human-and machine-generated answers across different models and, for each model, across different studies.”
- Also stated in: § Additional information · plain
“For all models, there was substantial variation in the degree to which LLM-generated responses correlated with human responses across different studies.”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The correlation between per-condition means is a valid measure of similarity between human and LLM responses.
- Also stated in: § Additional information · plain
- Findingown data
Some LLMs (GPT-4 and Gemini Pro) differed from humans insofar as they were not as sensitive to differences in the causal structure that dictates how several actions combine to bring about their effects.
“Some LLMs (GPT-4 and Gemini Pro) differed from humans insofar as they were not as sensitive to differences in the causal structure that dictates how several actions combine to bring about their effects.”
Quote in the paperSubstantial claim - Findingown data
In most studies, the same factors that successfully explained human decision-making were also successful in explaining LLM-generated responses.
“While, in most studies, the same factors which successfully explained human decision-making were also successful in explaining LLM-generated responses”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The eight experiments are representative of moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
- Findingown data
In contrast, Claude 2.1 and Llama 2 Chat 70b responded in ways that more closely mimic the interaction between prescriptive abnormality and causal structure observed among humans.
“In contrast, Claude 2.1 and Llama 2 Chat 70b responded in ways that more closely mimic the interaction between prescriptive abnormality and causal structure observed among humans.”
Quote in the paperSubstantial claim - Findingown data
This same kind of variation in correlation strength across studies was present for all other models.
“This same kind of variation was also present for all other models (see Table 1).”
Quote in the paperNot a substantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The correlation between per-condition means is a valid measure of similarity between human and LLM responses.
- Findingnone given
Across the board the variance present in LLM responses was substantially smaller than that present in human responses.
“Finally, across the board the variance present in LLM responses was substantially smaller than that present in human responses.”
Quote in the paperNot a substantial claim - Findingnone given
This reduced-variance tendency is less pronounced in Gemini Pro in comparison to other models.
“This reduced-variance tendency is less pronounced in Gemini Pro in comparison to other models.”
Quote in the paperNot a substantial claim - Premisecitation
Recent publications have shown differences between the way humans and various LLMs respond to vignettes about causation and moral responsibility.
“Recent publications have also shown differences between the way that humans and various LLMs respond to vignettes about causation and moral responsibility (Nie et al., 2023).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Nie et al., 2023
Not checked in the free run.
- Premisecitation
Humans have little access to the mechanisms underlying some of their psychology, such as perception.
“Similarly, humans are unreliable guides to their own inner workings: they are often overconfident in their explanatory knowledge (Rozenblit & Keil, 2002) and have little access to the mechanisms underlying some of their psychology (e.g., perception, see Firestone & Scholl, 2016).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Firestone & Scholl, 2016 source
Not checked in the free run.
- Premisecitation
Knobe (2003) found that 82% of participants in the harm condition said the chairman acted intentionally, compared to only 23% in the help condition.
“Strikingly, 82% of participants in the harm condition said that the chairman acted intentionally, compared to only 23% in the help condition.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Knobe, 2003 source
Not checked in the free run.
- Premisecitation
Previous research found that models in the GPT-3 family produced responses to psychological stimuli that varied very little with the default temperature of 1.
“Previous research has found that models in the GPT-3 family produced responses to psychological stimuli that varied very little with the default temperature of 1 (Park et al., 2023).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Park et al., 2023
Not checked in the free run.
- Interpretationreasoning
LLMs' psychology closely mimics human psychology.
“All of this suggests that LLMs' psychology closely mimics human psychology.”
Quote in the paperSubstantial claim - Interpretationown data
The condition*agent interaction in responsibility judgments is mainly driven by the reduced effect of condition over Claude 2.1's responses.
“As is clear from Figure 1's top row, this interaction is mainly driven by the reduced effect of condition over Claude 2.1's responses.”
Quote in the paperNot a substantial claim - Premisecitation
For all three cases, the majority of human participants selected the hypothesized option.
“For all three cases, the majority of human participants selected the hypothesized option (Engelmann, forthcoming).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Engelmann, forthcoming
Not checked in the free run.
- Premisecitation
Humans are often overconfident in their explanatory knowledge.
“Similarly, humans are unreliable guides to their own inner workings: they are often overconfident in their explanatory knowledge (Rozenblit & Keil, 2002) and have little access to the mechanisms underlying some of their psychology (e.g., perception, see Firestone & Scholl, 2016).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Rozenblit & Keil, 2002 source
Not checked in the free run.
- Interpretationcitation
Different models, while sharing much in terms of architecture, might come to have very different intuitions.
“These results suggest that different models, while sharing much in terms of architecture, might come to have very different intuitions.”
Quote in the paperSubstantial claim - Premisecitation
Among humans, different cultures and social groups differ in their moral foundations.
“Among humans, different cultures and social groups differ in their moral foundations.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Graham et al., 2013
Not checked in the free run.
- cites Klein et al., 2018 source
Not checked in the free run.
- Premiseown data
Participants were much more likely to blame the chairman in the harm condition than to praise him in the help condition.
“Moreover, participants were much more likely to blame the chairman in the harm condition than to praise him in the help condition.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Knobe, 2003 source
Not checked in the free run.
- Premisecitation
ChatGPT (GPT-3.5-turbo) provided different responses when prompted in different languages in ways predicted by pre-existing cross-cultural studies with humans.
“Not only that, but ChatGPT (GPT-3.5-turbo)provided different responses when prompted in different languages in ways predicted by pre-existing cross-cultural studies with humans (Goli & Singh, 2023).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Goli & Singh, 2023 source
Not checked in the free run.
- Interpretationown data
Overall, GPT-4, Llama 2 Chat 70b, and Gemini Pro tended to assign both more blame in the Harm condition and praise in the Help condition to the vignette's protagonist than human participants.
“which indicates that, overall, GPT-4, Llama 2 Chat 70b, and Gemini Pro tended to assign both more blame in the Harm condition and praise in the Help condition to the vignette's protagonist than human participants.”
Quote in the paperSubstantial claim - Premisecitation
Current state-of-the-art LLMs exhibit expert-level performance in tasks across a wide range of different domains.
“Even if current LLMs are not quite AGI, they nonetheless exhibit expert-level performance in tasks across a wide range of different domains.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Bubeck et al., 2023
Not checked in the free run.
- Premisecitation
Recent research has shown that both text and purpose matter in rule violation judgments, even in different countries.
“Recent research has shown that both of these elements do matter (Bregant et al., 2019;Garcia et al., 2014;LaCosse & Quintanilla, 2021;Struchiner et al., 2020), even in different countries (Hannikainen et al., 2022).”
Quote in the paperSubstantial claim0 of 3 sources checked - Interpretationown data
The results of this study support the hypothesis that LLMs roughly reproduces ordinary intuitions.
“In that respect, the results of this study support the hypothesis that LLMs roughly reproduces ordinary intuitions.”
Quote in the paperSubstantial claim - Interpretationown data
The data lends some support to the sui generis AI hypothesis.
“Thus, the data also lends some support to the sui generis AI hypothesis.”
Quote in the paperSubstantial claim - Interpretationown data
There is evidence to suggest that the cognitive processes involved in generating responses are somehow different for GPT-4 when compared to humans.
“Although GPT-4 was by far the model which best correlated to human responses, there was also some evidence to suggest that the cognitive processes involved in generating those responses are somehow different for GPT-4 when compared to humans.”
Quote in the paperSubstantial claim - Study design and conductown data
Only GPT-4 and Gemini Pro produced a sufficient number of valid responses in Study 2.
“Only GPT-4 and Gemini Pro produced a sufficient number of valid responses.”
Quote in the paperSubstantial claim - Premisecitation
In within-subjects designs where participants have access to all outcomes, the difference between conditions with bad and neutral outcomes is greatly diminished.
“Moreover, in within-subjects designs where participants have access to all outcomes, the difference between conditions with bad and neutral outcomes is greatly diminished (see Kneer & Machery, 2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Kneer & Machery, 2019
Not checked in the free run.
- Premisecitation
The typical 'abnormal selection' effect (preferential selection of causes that are abnormal, for example morally bad) is found in conjunctive causal structures.
“The typical”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
- Premisecitation
Icard et al. (2017) demonstrated an interaction between the moral status of actions and the causal structure with which actions combine to bring about their effects.
“Icard et al. (2017) additionally demonstrated an interaction between the moral status of actions and the causal structure with which actions combine to bring about their effects.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
- Premisecitation
In conjunctive cases, the salient counterfactual reveals that the outcome wouldn't have occurred if not for the norm-violating agent, thus leading to increased attribution of causality.
“In conjunctive cases, this salient counterfactual reveals that the outcome wouldn't have occurred if not for the norm-violating agent, thus leading to increased attribution of causality.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
- Premisecitation
In Demaree-Cotton and Sommers (2022), compared to the Exercises Capacity baseline (M = 5.98, SD = 1.08), the 'Lacks Capacity' condition yielded significantly lower agreement that the agent gave valid consent (M = 4.78, SD = 1.41), b = -1.21, SE = 0.14, t = -8.81, p < .001, 95% CI [-1.48, -0.94].
“Compared to the Exercises Capacity baseline (M = 5.98, SD = 1.08) the”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Demaree-Cotton & Sommers, 2022
Not checked in the free run.
- Premisecitation
In conjunctive scenarios, when asked for agreement to the claim that 'Billy caused the motion detector to go off', people's ratings are typically higher compared to a control version where Billy is also allowed to be in the room at 9am (abnormal inflation).
“When asked for their agreement to the claim that”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
- Premisecitation
In disjunctive structures, considering the same salient counterfactual reveals that the outcome would still have occurred, which deflates causal attribution.
“In contrast, considering the same salient counterfactual in disjunctive structures reveals that the outcome would still have occurred.This deflates causal attribution.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
- Interpretationown data
In a between-subjects design, legal judgments made by both humans and LLMs are affected by the hindsight bias.
“In a between-subjects design, legal judgments made by both humans and LLMs are affected by the hindsight bias.”
Quote in the paperSubstantial claim - Premisecitation
When the two outcomes are simultaneously presented to humans in a within-subjects design, this leads to a substantial decrease in outcome-sensitivity for most kinds of judgments.
“When the two outcomes are simultaneously presented to humans in a within-subjects design, this leads to a substantial decrease in outcome-sensitivity for most kinds of judgments.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Kneer & Skoczeń, 2023
Not checked in the free run.
- Premisecitation
Research on human participants has consistently shown that at least some types of causal judgments are reliably affected by moral considerations.
“While causation is often taken to be a purely descriptive concept, research on human participants has consistently shown that at least some types of causal judgments are reliably affected by moral considerations.”
Quote in the paperSubstantial claim - Premisecitation
In disjunctive causal structures, Billy's action would be rated as less causal when he is forbidden to be in the room at 9am compared to when he is allowed to do so (abnormal deflation).
“However, when the description is changed to a so-called disjunctive causal structure where either Billy's or Suzy's being in the room at 9am is sufficient for the motion detector to go off, this changes.In this version of the scenario, Billy's action would be rated as less causal when he is forbidden to be in the room at 9am compared to when he is allowed to do so.This effect has been dubbed ab”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
- Interpretationown data
Among LLMs, the contrast between within- and between-subjects designs produces varied effects.
“Among LLMs, the contrast between within-and between-subjects designs produces varied effects.”
Quote in the paperSubstantial claim - Premisecitation
In Kneer and Skoczeń's Experiment 2, the largest effect size for dependent variables other than blame and punishment was a Cohen's d of .26 for negligence.
“In fact, the largest effect size reported for the other dependent variables was a Cohen's d of .26 for negligence.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Kneer & Skoczeń, 2023
Not checked in the free run.
- Interpretationown data
Put together, these results show how model performance is still task-dependent to a large extent.
“Put together, these results show how model performance is still task-dependent to a large extent.”
Quote in the paperSubstantial claim - Interpretationreasoning
Existential risks may eventually arise from sufficiently capable models that are perfectly aligned with (misguided) human intuitions rather than from misaligned models.
“Thus, existential risks may eventually arise from sufficiently capable models that are perfectly aligned with (misguided) human intuitions rather than from misaligned models.”
Quote in the paperNot a substantial claim - Interpretationown data
The view that LLMs closely match human intuitions in the moral and legal domains should be viewed with caution.
“Thus, the view that LLMs closely match human intuitions in the moral and legal domains should be viewed with caution.”
Quote in the paperSubstantial claim - Interpretationown data
This highlights that there were various systematic differences between humans and LLMs when it comes to judgments of causation.
“This highlights that there were various systematic differences between humans and LLMs when it comes to judgments of causation.”
Quote in the paperSubstantial claim - Interpretationown data
The degree to which experimental stimuli appears in the training set for each model, while perhaps a sufficient condition for LLMs to produce very similar patterns to those generated by humans, doesn't seem to be a necessary condition for alignment to occur.
“Thus, the degree to which experimental stimuli appears in the training set for each model, while perhaps a sufficient condition for LLMs to produce very similar patterns to those generated by humans, doesn't seem to be a necessary condition for alignment to occur.”
Quote in the paperSubstantial claim - Interpretationown data
The significant differences observed between different LLMs suggest that there are also ways in which each of the models tested departs from human intuitions in unique and currently unpredictable ways.
“Moreover, the significant differences we have observed between different LLMs suggest that there are also ways in which each of the models we have tested departs from human intuitions in unique and currently unpredictable ways.”
Quote in the paperSubstantial claim - Interpretationreasoning
The increased effect sizes for LLMs are due in part to the reduced variance of LLM-generated responses.
“due in part to the reduced variance of LLM-generated responses.”
Quote in the paperSubstantial claim- Rests on an assumption: The reduced variance of LLM-generated responses is not an artifact of the measurement or prompting.
- Interpretationreasoning
The more ambitious proposals of using AI to substitute for human participants or as surrogates for collective decision-making are clearly off the mark, at least right now.
“As a result, the more ambitious proposals of using AI to substitute for human participants or as surrogates for collective decision-making (Dillion et al., 2023) are clearly off the mark, at least right now.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Dillion et al., 2023 source
Not checked in the free run.
- Interpretationreasoning
These are important differences which suggest that GPT-4's and Gemini Pro's cognition might systematically deviate from that of human beings in ways that cannot be reduced to the 'correct answers' effect.
“In any event, these are important differences which suggest that GPT-4's and Gemini Pro's cognition might systematically deviate from that of human beings in ways that cannot be reduced to the”
Quote in the paperSubstantial claim - Interpretationown data
LLMs show the same tendency to moralize the concept of a cause as human beings, consistently inflating causal judgment when the norm was violated in conjunctive chains.
“LLMs show the same tendency to moralize the concept of a cause as human beings, consistently inflating causal judgment when the norm was violated in conjunctive chains.”
Quote in the paperSubstantial claim - Interpretationcitation
Some of the systematic differences observed amplify what many philosophers have considered to be biases, as is the case with the outcome-sensitivity of probability judgments in Study 5.
“Some of the systematic differences observed amplify what many philosophers have considered to be biases, as is the case with the outcome-sensitivity of probability judgments in Study 5 (Kneer & Skoczeń, 2023).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Kneer & Skoczeń, 2023
Not checked in the free run.
- Interpretationown data
Inspecting the marginal means revealed that the condition effect was caused by different patterns for different agents.
“However, inspecting the marginal means revealed that this effect was caused by different patterns for different agents.”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
- Interpretationreasoning
This finding shows that there are uniform ways in which all current state-of-the-art LLMs depart from human reasoning.
“This finding shows that there are uniform ways in which all current state-of-the-art LLMs depart from human reasoning.”
Quote in the paperSubstantial claim - Interpretationreasoning
LLM psychology diverges from human psychology in significant and partially systematic ways.
“We concluded that this suggests that LLM psychology diverges from human psychology in significant and partially systematic ways.”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The eight experiments are representative of moral and legal reasoning.
- Rests on an assumption: The four models tested are state-of-the-art and representative of current LLMs.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
- Interpretationown data
All tested models show the same abnormal inflation effect that Icard et al. found for conjunctive structures.
“In other words, all tested models show the same abnormal inflation effect that Icard et al. found for conjunctive structures.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
- Interpretationown data
The significant two-way interaction between agent and scenario reflects systematic differences in the way that GPT-4 and humans rated consent in the different scenarios explored.
“The significant two-way interaction between agent and scenario reflects systematic differences in the way that GPT-4 and humans rated consent in the different scenarios explored.”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- May combine two claims. 81
- Classified as probably not a substantial claim. 31
- The quote only partly states this claim. 4
May combine two claims.
Findingown dataIn the linear model of responsibility judgments on the joint dataset, there were significant main effects of condition and agent, qualified by a significant condition*agent interaction.
“A linear model of responsibility (blame/praise) judgments on the joint dataset revealed significant main effects of condition (F (1, 478) = 1498.60,p < .001,η² = 0.76) and agent (F (4, 478) = 40.95,p < .001,η² = 0.25) qualified by a significant condition*agent interaction (F (4, 478) = 60.04,p < .001,η² = 0.33).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataA model excluding Claude 2.1's responses still reveals a significant condition*agent interaction, but one that accounts for a much smaller portion of the model's explanatory power compared to the main effects of agent.
“A model excluding Claude 2.1's response still reveals a significant condition*agent interaction (F (3, 392) = 4.74, p = .003)but one that accounts for a much smaller portion of the model's explanatory power (η² = 0.035) when compared to the main effects of agent (F (3, 392) = 44.82,p < .001,η² = 0.26)”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataModels differ amongst themselves as to their overall alignment with human responses, with GPT-4 taking a clear lead over all other models tested.
“We find that alignment with human responses shifts from one experiment to another, and that models differ amongst themselves as to their overall alignment, with GPT-4 taking a clear lead over all other models we tested.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn a logistic model of intentionality judgments, there were significant main effects of blame/praise ratings, condition, and agent, qualified by a significant two-way interaction between condition and agent.
“Turning to intentionality judgments, a logistic model revealed significant main effects of blame/praise ratings (χ² df=1 = 6.69, p < .001),condition (χ² df=1 = 72.51,p < .001)and agent (χ² df=4 = 32.70,p < .001),which were qualified by a significant two-way interaction between condition and agent (χ² df=4 = 31.14,p < .001).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataEach LLM's judgments about the protagonist's moral responsibility differed significantly between conditions, mimicking the pattern observed among humans.
“Despite these differences in effect sizes and variance, each LLMs' judgments about the protagonist's moral responsibility (which meant praise in the help and blame in the harm condition; 6.41 < ts < 55.28, ps < .001,1.58 < ds < 11.06)) differed significantly between conditions, mimicking the pattern observed among humans.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataGPT-4 and Gemini Pro almost always complied with the request to answer only with the numerical scale provided, while Claude 2.1 and Llama 2 Chat 70b frequently included much more information in their responses.
“While GPT-4 and Gemini Pro almost always complied with the request to answer only with the numerical scale provided, Claude 2.1 and Llama 2 Chat 70b frequently included much more information in their responses.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataEven when LLM-generated responses are highly correlated with human responses, there are still systematic differences, with a tendency for models to exaggerate effects present among humans, in part by reducing variance.
“Nonetheless, even when LLM-generated responses are highly correlated to human responses, there are still systematic differences, with a tendency for models to exaggerate effects that are present among humans, in part by reducing variance.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe increased effect was caused by an invariance in responses from GPT-4, Gemini Pro and Claude 2.1, where all answers in the harm condition indicated that the chairman acted intentionally while all answers in the help condition indicated that he did not.
“This increased effect was caused by an invariance in responses from GPT-4, Gemini Pro and Claude 2.1, where all answers in the harm condition indicated that the chairman acted intentionally while all answers in the help condition indicated that he did not (SDs = 0).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn a hierarchical model with agent, foundation, and their interaction, there were significant main effects of agent and foundation, qualified by a significant agent*foundation interaction.
“Similarly, a hierarchical model with agent, foundation, and the agent*foundation interaction as fixed effects with random intercepts for each participant ID revealed significant main effects of agent (F (4, 7128) = 21.57,p < .0001,η² = .006),foundation (F (4, 28423) = 3697.08,p < .0001,η² = .197)and the agent*foundation interaction (F (16, 28398) = 36.3,p < .0001,η² = .009).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn a mixed effect model for relevancy ratings, there were significant main effects of agent and item, qualified by a significant agent*item interaction.
“This model revealed significant main effects of agent (F (4, 7135) = 21.56,p < .001,η² = .004)and item (F (14, 99520) = 2058.36,p < .001,η² = .166).These main effects were qualified by a significant interaction between agent and item (F (56, 99501) = 18.03, p < .001,η² = .007).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataLLM's and participants' moral evaluations of deceptions were surprisingly similar and highly correlated, despite some subtle differences.
“LLM's and participants' moral evaluations of deceptions were surprisingly similar and highly correlated, despite some subtle differences.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe correlation for the nine means between humans and each LLM was high and significant, as was the correlation between the LLMs.
“The correlation for the nine means between humans and each LLM was high and significant (Humans vs. GPT-4: r df = 7 = 0.96, 95% CI: 0.82 -0.99, p <.001, Humans vs. Gemini Pro: r df = 7 = 0.82, 95% CI: 0.34 -0.96, p = .007),as was the correlation between the LLMs ( r df = 7 = 0.80, 95% CI: 0.30 -0.96, p = .009).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-4's proportions in the forbidden and permissible versions perfectly matched the anticipated deontic status (SDs = 0), while participants' and Gemini Pro's judgments exhibit some degree of variation.
“A noteworthy difference is that GPT-4's proportions in the forbidden and permissible versions perfectly matched the anticipated deontic status (SDs = 0), while participants' and Gemini Pro's judgments exhibit some degree of variation.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIncluding the interaction between deontic status and agent improved model fit in all three analyses, indicating that deontic status affected responses somewhat differently for humans and the two LLMs.
“Including the interaction improved fit in all three analyses (obligatory DV: 𝝌 2 df = 4 = 14.4,p = .006,permissible DV: 𝝌 2 df = 4 = 109.83,p <.001, forbidden DV: 𝝌 2 df = 4 = 71.77,p <.001), indicating that deontic status affected responses somewhat differently for humans and the two LLMs (see also Figure 4).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAll models attributed significantly lower relevance to the Loyalty foundation, and all but Llama 2 attributed significantly lower relevance to the Purity foundation.
“However, all models attributed significantly lower relevance to the Loyalty foundation (ps < .04),and all but Llama 2 (b = .35,p = .37)attributed significantly lower relevance to the Purity foundation (ps<.001).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-4's evaluations were uniform (in six out of nine cases), while participants deviated from the expected status in at least 10% and up to 40% of their responses.
“Most notably, GPT-4's evaluations were uniform (in six out of nine cases), while participants deviated from the expected status in at least 10% and up to 40% of their responses.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAll LLMs presented moderate to high correlation with human responses, with GPT-4 reaching a correlation coefficient of 0.92.
“All LLMs presented moderate to high correlation with human responses, with GPT-4 reaching a correlation coefficient of 0.92 ([0.83, 0.96].p < .001;see Supplementary Table 3).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataClaude 2.1 deviated from humans in 11 out of 15 items, Gemini Pro in 10, GPT-4 in 7, and Llama 2 Chat 70b in 4.
“Claude 2.1 deviated from humans in 11 out of the 15 items (all p's Kenward-Roger < .01),Gemini Pro in 10, GPT-4 in 7, and Llama 2 Chat 70b in 4.”
Quote in the paperSubstantial claimMay combine two claims.
FindingcitationLlama 2 Chat 70b did not show the same pattern; with the exception of subjective probability, all other variables showed even larger effects in the within-subjects design than in the between-subjects design (ts > 23.0, ps < .001, ds between 3.32 and 6.01).
“Llama 2 Chat 70b did not show the same pattern.With the exception of subjective probability, all other variables showed even larger effects in the within subjects design than in the between subjects design (ts > 23.0, ps < .001,ds between 3.32 and 6.01).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor Llama 2, ratings were systematically much lower than for humans; on aggregate, not even fully capable adults who exercised their capacities for autonomous decision-making are able to give valid consent.
“The opposite effect occurs for Llama 2, for whom ratings were systematically much lower than for humans.In fact, for Llama 2 on aggregate, not even fully capable adults who exercised their capacities for autonomous decision-making are able to give valid consent.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataEstimates for all of the 6 DVs differed significantly between conditions for all models, with the exception of Gemini Pro, for which only objective and subjective probability were significantly different between conditions.
“In contrast, estimates for all of the 6 DVs differed significantly between conditions for all models (all t's > 7.48, p's < .001,all d's between 1.43 and 5.97), with the exception of Gemini Pro, for which only objective (t = 7.94, p < .001,d = 3.00) and subjective probability (t = 7.14, p < .001,d = 2.70) were significantly different between conditions (all other |t|s < 1.22, ps > .10).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataFor Claude 2.1, the effects were smaller than those observed in the between-subjects design for subjective probability (d between = 5.97, d within = 1.76), recklessness (d between = 5.21, d within = 1.92), and negligence (d between = 5.77, d within = 1.73).
“Nonetheless, the effects were smaller than those observed in the between-subjects design for subjective probability (d between = 5.97, d within = 1.76), recklessness (d between = 5.21, d within = 1.92), and negligence (d between = 5.77, d within = 1.73).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataWhereas these effects became small for humans and non-significant for GPT-4 and Gemini Pro, they remained large for Claude 2.1.
“However, whereas these effects became small for humans and non-significant for GPT-4 and Gemini Pro, they remained large for Claude 2.1.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe significant main effects of agent reflect a tendency of Llama 2 (M=2.22) to give very low consent ratings; GPT-4 (M=6.18) gave slightly higher consent ratings than Human (M=5.70), while Claude 2.1 (M=4.20) stood at the middle but still closer to Humans and GPT-4 (all comparisons significant with ps<.001).
“The significant main effects of agent reflect a tendency of Llama 2 (M=2.22) to give very low consent ratings.In addition, GPT-4 (M=6.18)gave slightly higher consent ratings than Human (M=5.70), while Claude 2.1 (M=4.20)stood at the middle but still closer to Humans and GPT-4 (all comparisons are significant with ps<.001).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe linear model of consent judgments revealed a significant main effect of agent (F(2,794)=643.74, p<.001, η2=0.71) qualified by significant two-way interactions with condition (F(6,794)=21.52, p<.001, η2=0.14) and scenario (F(6,794)=11.35, p<.001, η2=0.08).
“This model revealed a significant main effect of agent (F (2, 794) = 643.74,p < .001,η² = 0.71) which was qualified by significant two-way interactions with condition (F (6, 794) = 21.52,p < .001,η² = 0.14) and scenario (F (6, 794) = 11.35,p < .001,η² = 0.08).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGemini Pro showed more sensitivity to vitiating factors, giving significantly lower ratings than humans to the 'Mere capacity' and 'Lacks capacity' conditions, but not to the 'Exercises capacity' condition.
“Finally, Gemini Pro showed more sensitivity to vitiating factors, giving significantly lower ratings than humans to the”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAll models revealed significant interactions between condition and agent, indicating that the experimental manipulation had different effects over different agents.
“All models revealed significant interactions between condition and agent, which indicates that the experimental manipulation had different effects over different agents (all p's < .001;see Supplementary Table 6).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataFor GPT-4 and Gemini Pro, there was an even larger difference between the two designs, with no significant differences between conditions for the objective probability, subjective probability, recklessness, and negligence variables.
“For GPT-4 and Gemini Pro, we observed an even larger difference between the two designs, with no significant differences between conditions for the objective probability, subjective probability, recklessness, and negligence variables.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAll two-way interactions between agent and condition were statistically significant, revealing that the effects of condition varied from one agent to another (Fs > 40.15, ps < .001).
“All two-way interactions between agent and condition were statistically significant, revealing that the effects of condition varied from one agent to the other (Fs > 40.15, ps < .001;see Supplementary Table 8).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataBlame and punishment judgments remained significantly higher for the 'Bad' outcome condition when generated by GPT-4 (ts > 35.9, ps < .001), but not by Gemini Pro (ts < 2.1, ps > .11).
“Just as among humans, blame and punishment judgments remained significantly higher for the”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-4's responses (Mean SD = 13.11) had higher variance compared to humans (SD = 10.85), while Llama 2 Chat 70b's responses showed considerably less variation (SD = 5.28); Gemini Pro showed a similar level of variance (SD = 9.81) to human responses.
“Considering the mean SD per cell for each agent, we found that GPT-4's responses (Mean SD = 13.11) had higher variance when compared to humans (SD = 10.85), while Llama 2 Chat 70b's responses showed considerably less variation (SD = 5.28).The answers provided by Gemini Pro showed a similar level of variance (SD = 9.81) to human responses.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataWhile ordinary people think that having a capacity that the protagonist failed to exercise renders their consent more valid when compared to the Exercises Capacity condition, all LLMs generated answers that it makes it less so.
“While ordinary people surprisingly think that having a capacity that the protagonist failed to exercise renders their consent more valid when compared to the Exercises Capacity condition, all LLMs generated answered that it makes it less so.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataAlthough the same experimental manipulations which are significant among humans are also usually significant among LLMs, there were almost always significant main effects and interactions involving the agent term.
“A fourth implication of our findings is that, although the same experimental manipulations which are significant among humans are also usually significant among LLMs, there were almost always significant main effects and interactions involving the agent term.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataWe found a significant effect for condition in all models (chi2 values: Human = 122.44; Gemini Pro = 87.44; GPT-4 = 79.84; Llama 2 Chat 70b = 370.48; all ps < .001).
“We found a significant effect for condition in all models (𝜒 2 (2) -Human = 122.44;𝜒 2 (2) -Gemini Pro = 87.44,𝜒 2 (2) -GPT-4 = 79.84;𝜒 2 (2) -Llama 2 Chat 70b = 370.48;all ps < .001).”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
May combine two claims.
Findingown dataLLMs have produced low-variance outputs that closely match human responses for Studies 1 and 3.
“And, indeed, LLMs have produced low-variance outputs that closely match human responses for each of these two studies.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataLlama 2 was across the board less likely to consider that valid consent was given in Study 6 than humans, and the effects of norm violation over causation judgments depend on causal structure on humans, but not for GPT-4 and Gemini Pro (Study 7).
“For instance, Llama 2 was across the board less likely to consider that valid consent was given in Study 6 than humans, and the effects of norm violation over causation judgments depend on causal structure on humans, but not for GPT-4 and Gemini Pro (Study 7, Section 2.6).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThat same pattern was also observed in Studies 2 (Deception) and (for most models) 5 (Hindsight bias, between subjects), regardless of the fact that these two experiments were reported in papers that are either extremely recent or still in manuscript form.
“However, that same pattern was also observed in Studies 2 (Deception) and (for most models) 5 (Hindsight bias, between subjects), regardless of the fact that these two experiments were reported in papers that are either extremely recent (Kneer & Skoczeń, 2023) or still in manuscript form (Engelmann, forthcoming).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Kneer & Skoczeń, 2023
Not checked in the free run.
- cites Engelmann, forthcoming
Not checked in the free run.
May combine two claims.
Findingown dataIn Study 1, while humans assigned to the condition where the chairman of the board's actions harmed the environment tend to say that he acted intentionally and those assigned to the condition where his actions helped the environment tend to say that he acted unintentionally, that tendency is much more pronounced among LLMs.
“For instance, in Study 1 (Section 2.1) while humans assigned to the condition where the chairman of the board's actions harmed the environment tend to say that he acted intentionally and those assigned to the condition where his actions helped the environment tend to say that he acted unintentionally, that tendency is much more pronounced among LLMs.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe model of causation ratings revealed a significant main effect of agent (F(4,746)=18.05, p<.001, η2=.09) qualified by significant interactions between causal structure and agent, scenario and agent, prescriptive abnormality, causal structure, and agent, and between causal structure, scenario, and agent.
“These differences were reflected in a model of causation ratings encompassing data generated by humans and each of the LLMs as main effects of agent (F (4, 746) = 18.05, p < .001,η² = .09)qualified by significant interactions between causal structure and agent (F (1, 746) = 2.46, p = .044,η² = 0.01), scenario and agent (F (12, 746) = 4.55, p < .001,η² = .07),prescriptive abnormality, causal struct”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone givenThe highest correlation out of any study for all models was in Study 1, while Studies 8 (Causation) and 7 (Consent) were amongst the lowest for all LLMs.
“highest correlation out of any study for all models, while Studies 8 (Causation, Section 2.7) and 7 (Consent, Section 2.6) were amongst the lowest for all LLMs.”
Quote in the paperNot a substantial claimMay combine two claims.
FindingcitationModel-generated responses showed on average less variance than human-generated responses for all but Study 4 (Rule violation), displaying the 'correct answers' effect in several different conditions across different studies.
“Finally, model-generated responses showed on average less variance than human-generated responses for all but Study 4 (Rule violation), displaying the”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Park et al., 2023
Not checked in the free run.
May combine two claims.
Findingown dataAmong humans, judgments of consent were highest for the 'Sexual' scenario (M = 5.90), followed by the 'Medical' (M=5.80) and 'Police' (M=5.23) scenarios.
“Among humans, judgments of consent were highest for the”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
May combine two claims.
Findingown dataIn GPT-4's case, this resulted in a significant prescriptive abnormality * causal structure interaction, due to the fact that the effects of abnormal inflation were significantly larger in the conjunctive compared to the disjunctive condition.
“In GPT-4's case, this also resulted in a significant prescriptive abnormality * causal structure interaction, due to the fact that the effects of abnormal inflation were significantly larger in the conjunctive when compared to the disjunctive condition.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataWhile variations in causal structure changed the degree to which morally bad actions received higher causal ratings than morally good actions, the same effect was present in both conjunctive and disjunctive causal scenarios.
“While variations in causal structure changed the degree to which morally bad actions received higher causal ratings than morally good actions, the same effect was present in both conjunctive and disjunctive causal scenarios.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataEven though some of the main significance patterns were the same, their strength varied a lot between agents.
“Even though some of the main significance patterns were the same, their strength varied a lot between agents.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
FindingreasoningThat is especially true for Studies 1 and 3, which reproduce relatively older studies that are very well-cited.
“That is especially true for Studies 1 and 3, which reproduce relatively older studies that are very well-cited.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
Findingown dataFor each of the three AI models in the experiment, 'Mere Capacity' condition answers were significantly lower than those in the 'Exercises Capacity' condition (ps < 0.04).
“For each of the three AI models in this experiment,”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationHuman beings inflate causes which are morally abnormal.
“For instance: cognitive scientists have observed that human beings inflate causes which are morally abnormal (Willemsen & Kirfel, 2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Willemsen & Kirfel, 2019
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationA study designed to test the counterfactual explanation found support for it.
“In this case, a study designed to achieve exactly this goal found support for the proposed explanation, as we will see in Section 2.7 (Icard et al., 2017).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
PremisecitationExperiment 1 by Engelmann (forthcoming) employed three different vignettes, each of which came in three versions distinguished by the anticipated deontic status of the contained deception.
“Experiment 1 by Engelmann (forthcoming) employed three different vignettes, each of which came in three versions.These versions were distinguished by the anticipated deontic status of the contained deception.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Engelmann, forthcoming
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationRLHF aims to make LLMs less harmful, and more helpful and useful, by fine-tuning the model based on a reward signal gathered from explicit human feedback.
“RLHF aims to make LLMs less harmful, and more helpful and useful, by fine-tuning the model based on a reward signal gathered from explicit human feedback (Bai, Jones, et al., 2022).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Bai, Jones, et al., 2022
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable).
PremisereasoningResearch into the beliefs and cognitive processes underlying LLMs needs to resort to more indirect and interdisciplinary methods.
“This means that research into the beliefs and cognitive processes underlying LLMs need to resort to more indirect and interdisciplinary methods (Rahwan et al., 2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Rahwan et al., 2019 source
Not checked in the free run.
- Rests on an assumption: LLMs have no special insight into their inner workings.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationQuerying text-davinci-002 multiple times with the same prompt tends to elicit very similar responses, even in scenarios where we wouldn't necessarily expect this to happen.
“This dovetails well with other research showing that querying text-davinci-002 multiple times with the same prompt tends to elicit very similar responses, even in scenarios where we wouldn't necessarily expect this to happen (Araujo et al., 2022).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Araujo et al., 2022
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone givenThere are important differences between the processes by which LLMs and humans produce behavior.
“On the other hand, it's trivially true that there are some important differences between the processes by which LLMs and humans produce behavior.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationFlanagan and colleagues (2023) manipulated whether the rule's text and intended goal were violated, and whether the intended goal was morally good or bad.
“To find out precisely what goes into the purpose or spirit of the law, Flanagan and colleagues (2023) manipulated not only whether the rule's text and the rule's intended goal were violated, but also whether the intended goal was morally good or morally bad (their Study 1).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Flanagan et al., 2023 source
Not checked in the free run.
May combine two claims.
Interpretationown dataEven though the overall significance patterns that occur among humans also occur among LLMs, they are not exactly the same.
“This shows that, even though the overall significance patterns that occur among humans also occur among LLMs, they are not exactly the same.”
Quote in the paperSubstantial claimMay combine two claims.
Premisenone givenOther work has used Moral Foundation Theory and the MFQ to explore the moral reasoning of LLMs, but none has used any of the state of the art models investigated here.
“Other work has used Moral Foundation Theory and the MFQ to explore the moral reasoning of LLMs (Abdulhai et al., 2023;Park et al., 2023;Simmons, 2022), but none has used any of the state of the art models we investigate here .”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Abdulhai et al., 2023
Not checked in the free run.
- cites Park et al., 2023
Not checked in the free run.
- cites Simmons, 2022
Not checked in the free run.
May combine two claims.
Interpretationown dataOverall, these results indicate that, just as humans, LLMs not only consider text and purpose in rendering rule violation judgments, but also that purpose matters much more when it's morally good.
“Overall, these results indicate that, just as humans, LLMs not only consider text and purpose in rendering rule violation judgments, but also that purpose matters much more when it's morally good.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataThe number of valid outputs varied substantially among models, with invalid responses returning the wrong number of items or providing numerical values larger than the provided scale.
“The number of valid outputs varied substantially among models, with invalid responses returning the wrong number of items in the response, or providing numerical values larger than the provided scale.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataOnly 19 model instances provided valid responses to all questions.
“Only 19 model instances provided valid responses to all questions.”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductown dataLlama 2 presented 23 valid responses for the MFQ and 50 for the political ideology question.
“Llama 2 presented 23 valid responses for the MFQ and 50 for the political ideology question.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.
Study design and conductown dataOther models yielded substantially lower success rates than GPT-4 in Study 4.
“Unfortunately, other models yielded substantially lower success rates.”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductown dataLlama 2 Chat 70b and Claude 2.1 did not stick to the required response format in most cases (73% for Claude 2.1, 100% for Llama 2 Chat 70b).
“Llama 2 Chat 70b and Claude 2.1 did not stick to the required response format in most cases (producing long-form text instead of picking a response option, this happened in 73% of all answers for Claude 2.1, and in all answers for Llama 2 Chat 70b).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationEarlier studies with GPT-3 (text-davinci-003) have shown the model's moral foundations to be closest to that of political conservatives.
“Earlier studies with GPT-3 (text-davinci-003) have shown the model's moral foundations to be closest to that of political conservatives (Park et al., 2023).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Park et al., 2023
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenWe needed to discard 23 responses from Llama 2 Chat 70b, 33 responses from Gemini Pro, and 147 responses from Claude 2.1.
“We needed to discard 23 responses from Llama 2 Chat 70b, 33 responses from Gemini Pro, and 147 responses from Claude 2.1.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationMFQ scores have been shown to vary between cultures and between social groups within a single culture.
“MFQ scores have been shown to vary between cultures (Graham et al., 2011) and between social groups within a single culture (Graham et al., 2009).”
Quote in the paperNot a substantial claim0 of 2 sources checkedMay combine two claims.
Study design and conductown dataGPT-4 produced valid answers in all cases, while Gemini Pro did so in 67% of all cases.
“GPT-4 produced valid answers in all cases, while Gemini Pro did so in 67% of all cases.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataAfter exclusions, there were 25 valid answers from Claude 2.1, 47 for Gemini Pro, and 100 responses for GPT-4.
“After exclusions, we ended up with 25 valid answers from Claude 2.1, 47 for Gemini Profoot_2 , and 100 responses for GPT-4.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationTheir results showed that ordinary judgments of rule violation are much more sensitive to morally good than morally bad purposes, which suggests an influence of moral appraisals over the folk concept of rule.
“Their results showed that ordinary judgments of rule violation are much more sensitive to morally good than morally bad purposes, which suggests an influence of moral appraisals over the folk concept of rule.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Flanagan et al., 2023 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataOut of 100 trials, there were 99 valid responses from GPT-4, 77 from Claude 2.1, 54 from Llama 2 Chat 70b, and 28 from Gemini Pro.
“In the end, out of 100 trials, we were left with 99 valid responses from GPT-4, 77 valid responses from Claude 2.1, 54 valid responses from Llama 2 Chat 70b, and 28 valid responses from Gemini Pro.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationThe finding that morally bad or forbidden actions are more likely to be selected as the cause has been replicated many times and has proven robust.
“While there are competing explanations as to why this effect occurs, the finding itself was replicated many times and has proven robust (for a recent overview, see Willemsen & Kirfel, 2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Willemsen & Kirfel, 2019
Not checked in the free run.
May combine two claims.
Study design and conductown dataIn the LLM consent study, Claude 2.1 did not produce any valid answer for the sexual consent domain, and it was dropped from the analysis.
“Claude 2.1 did not produce any valid answer for the sexual consent domain.To prevent skewing the results, we decided to drop it from the analysis.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataThe LLM consent study received 199 valid answers from GPT-4, 122 from Gemini Pro, and 145 from Llama 2 Chat 70b.
“We received 199 valid answers from GPT-4, 122 from Gemini Pro, and 145 from Llama 2 Chat 70b.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataAmong humans, the three questions directly relating to consent showed good internal reliability (alpha = 0.74), but the same was not the case with the models, which presented significantly lower alphas: Gemini Pro (alpha = 0.53) and GPT-4 (alpha = 0.63).
“However, whereas among humans these three questions showed good internal reliability (alpha = 0.74), the same was not the case with the models, which presented significantly lower alphas: Gemini Pro (alpha = 0.53) and GPT-4 (alpha = 0.63).”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataThis suggests that LLMs differ from both humans and each other in the consent task.
“This suggests that LLMs differ from both humans and each other in this task.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationKneer and Skoczeń conceptually replicated these findings and tested several strategies for alleviating the effects of the hindsight bias.
“Kneer and Skoczeń conceptually replicated these findings and tested several strategies for alleviating the effects of the hindsight bias (2023).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Kneer & Skoczeń, 2023
Not checked in the free run.
May combine two claims.
Interpretationown dataIn the between-subjects design, most LLMs (a) showed the same significance patterns as humans and (b) departed from humans in the same direction, i.e., by showing heightened outcome-sensitivity.
“In the between-subjects design, most LLMs (a) showed the same significance patterns as humans and (b) departed from humans in the same direction, i.e., by showing heightened outcome-sensitivity.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims.
PremisecitationIn Demaree-Cotton and Sommers (2022), the 'Mere Capacity' condition failed to yield lower agreement that the agent gave valid consent; participants gave higher ratings of valid consent in the 'Mere Capacity' condition (M = 6.38, SD = 0.84) than in the 'Exercises Capacity' condition, b = 0.36, SE = 0.15, t = 2.35, p = .019, CI [0.06, 0.65].
“The”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Demaree-Cotton & Sommers, 2022
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataModels often failed to produce valid responses in Study 5.
“Still, models often failed to produce valid responses, e.g., by producing either more or less numeric values than required, or by failing to produce any numeric values.”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductown dataDropping the first question would increase Gemini's alpha to 0.6 and GPT-4 to 0.76, indicating both models provided answers to the first question inconsistent with the remaining evaluations.
“Dropping the first question would increase Gemini's alpha to 0.6 and GPT-4 to 0.76.This indicates both models provided answers to the first question inconsistent with the remaining evaluations.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductown dataThe final dataset for the within-subjects LLM study contained 99 valid responses from GPT-4, 90 from Llama 2 Chat 70b, 86 from Claude 2.1, and 27 from Gemini Pro.
“Our final dataset contained 99 valid responses from GPT-4, 90 responses from Llama 2 Chat 70b, 86 responses from Claude 2.1, and 27 responses from Gemini Pro.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataThe results of the consent study do not support the view according to which LLMs closely match ordinary human judgments, since no models adequately followed the pattern observed in humans.
“The results of this study do not support the view according to which LLMs closely match ordinary human judgments, since no models adequately followed the pattern observed in humans.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationKneer and Skoczeń found that, although the ex ante estimates for objective and subjective probability did not differ significantly between conditions, ex post judgments of objective probability, subjective probability, negligence, blame, and punishment all did.
“Kneer and Skoczeń found that, although the ex ante estimates for objective and subjective probability did not differ significantly between conditions, ex post judgments of objective probability, subjective probability, negligence, blame, and punishment all did (all p's < .004,all Cohen's d's > .46).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Kneer & Skoczeń, 2023
Not checked in the free run.
May combine two claims.
Study design and conductown dataLlama 2 presented a different behavior, with an even lower alpha (0.23) and dropping the first question would worsen the alpha, while dropping the third would increase it to over 0.9.
“Llama 2 presented a different behavior, with an even lower alpha (0.23) and dropping the first question would worsen the alpha, while dropping the third would increase it to over 0.9.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationAccording to Demaree-Cotton and Sommers (2022), the received view in the academic world is that consent is valid only if it is autonomous.
“According to Demaree-Cotton and Sommers (2022), the received view in the academic world is that consent is valid only if it is autonomous.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Demaree-Cotton & Sommers, 2022
Not checked in the free run.
May combine two claims.
Interpretationown dataEven if per-cell correlations remain large across the board, LLMs might differ systematically from humans and from each other.
“This shows that, even if per-cell correlations remain large across the board, LLMs might differ systematically from humans and from each other.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationIcard and colleagues proposed that abnormal deflation in disjunctive scenarios can be explained by counterfactual reasoning: norm violation affects the salience of counterfactual cases such that we're more likely to consider counterfactuals in which the norm wasn't violated.
“Icard and colleagues have proposed that this can be explained by counterfactual reasoning.Crucially, they claim that norm violation affects the salience of counterfactual cases such that we're more likely to consider counterfactuals in which the norm wasn't violated.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Icard et al., 2017 source
Not checked in the free run.
May combine two claims.
Interpretationown dataGPT-4's ratings were not only significantly and substantially above the midpoint for all conditions, but also systematically higher than human consent ratings, which might show an excessive focus on the fact that the protagonist indicated explicit agreement.
“The fact that GPT-4's ratings were not only significantly and substantially above the midpoint for all conditions, but also systematically higher than human consent ratings might show an excessive focus on the fact that the protagonist indicated explicit agreement.”
Quote in the paperSubstantial claimMay combine two claims. The quote only partly states this claim.
PremisecitationIn Kneer and Skoczeń's Experiment 2 (within-subjects), all dependent variables differed significantly between conditions, but effect sizes were much lower than in a between-subjects design and small for all variables except blame and punishment.
“In this study, Kneer and Skoczeń found that, although all dependent variables differed significantly between conditions,”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Kneer & Skoczeń, 2023
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationMorally bad or forbidden actions are more likely to be selected as 'the' cause of harmful outcomes, even when the descriptive causal contribution of other actions has been identical.
“Most famously, morally bad or forbidden actions are more likely to be selected as”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Hitchcock & Knobe, 2009
Not checked in the free run.
- cites Icard et al., 2017 source
Not checked in the free run.
- cites Kirfel & Lagnado, 2021
Not checked in the free run.
May combine two claims.
Interpretationown dataThe significant two-way interaction between agent and condition reflects that the 'Exercises Capacity' hypothesis provides a better explanation for differences between conditions for LLMs than the 'Mere Capacity' hypothesis, while the opposite obtains among humans.
“The significant two-way interaction between agent and condition reflects the fact that the”
Quote in the paperSubstantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningAs we work on debiasing LLMs with regards to the hindsight bias, correlations between human and LLM decision-making will increase until they reach a maximum at the point in which LLMs and humans are equally biased.
“As we work on debiasing LLMs with regards to the hindsight bias, correlations between humanand LLM-decision-making will increase until they reach a maximum at the point in which LMMs and humans are equally biased.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataThe fact that the magnitude of the correlations between the responses of each model and human subjects varied widely from one study to the other suggests that alignment should be evaluated in a domain-specific fashion.
“First, the fact that the magnitude of the correlations between the responses of each model and human subjects varied widely from one study to the other suggests that alignment should be evaluated in a domain-specific fashion.”
Quote in the paperNot a substantial claimThe quote only partly states this claim.
InterpretationcitationThere is no doubt that some of the stimuli we have used are subject to the criticism that well-established stimuli occur frequently in training datasets, meaning models might be simply memorizing appropriate answers instead of actually going through the processes which humans who don't know the stimuli would engage with.
“There is no doubt that some of the stimuli we have used are subject to that criticism.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Marcus & Davis, 2023
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenGemini Pro's low number of valid responses might mean that the model is not answering appropriately to the stimuli in this experiment.
“Gemini Pro's low number of valid responses might mean that the model is not answering appropriately to the stimuli in this experiment.”
Quote in the paperNot a substantial claim- Rests on an assumption: Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
- Rests on an assumption: The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
May combine two claims.
Interpretationown dataThis means that Claude 2.1 and Llama 2 Chat 70b showed the same abnormal deflation present in humans, reflected in significant prescriptive abnormality * causal structure interactions.
“This means that these two models showed the same abnormal deflation present in humans, which is reflected in significant prescriptive abnormality * causal structure interactions.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenIt was not possible to extract token probabilities directly for any of the models used in this paper.
“However, it's not possible for us to extract the token probabilities directly for any of the models we have used in this paper.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationHuman beings tend to be 'groupish' and usually favor intra-group cooperation over inter-group cooperation.
“Human beings tend to be”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Haidt, 2012
Not checked in the free run.
May combine two claims.
Interpretationown dataThis suggests significant differences not only between LLMs and humans, but also among LLMs when it comes to reasoning about causation.
“This suggests significant differences not only between LLMs and humans, but also among LLMs when it comes to reasoning about causation.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
InterpretationreasoningOur findings suggest that each specific area merits its own investigation.
“In this paper, we have looked at the way LLMs respond in some of those specific areas, but our findings suggest that each specific area merits its own investigation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenWe discarded 9 answers generated by Gemini Pro, while 11 API calls resulted in an 'error due to the content' message.
“We discarded 9 answers generated by Gemini Pro, while 11 API calls resulted in an”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe data for the scenario where neither text nor purpose was violated with good valence under the 'No shooting' scenario was missing from the original dataset due to a survey error.
“The data for the scenario where neither text nor purpose was violated with good valence under the”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningMisalignment through debiasing might well be what's happening in other situations, such as in Study 2, where GPT-4 and Gemini Pro were more likely to match anticipated deontic statuses than humans.
“In fact, misalignment through debiasing might well be what's happening in other situations, such as in Study 2, where GPT-4 and Gemini Pro were more likely to match anticipated deontic statuses than humans.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationPsychologists have long argued that human intuitions are systematically biased, including in ways that have led philosophers to cast doubt on the reliability of human moral intuitions.
“Psychologists have also long argued that human intuitions are systematically biased (Tversky & Kahneman, 1974), including in ways that have led philosophers to cast doubt on the reliability of human moral intuitions (Sinnott-Armstrong, 2008).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Tversky & Kahneman, 1974 source
Not checked in the free run.
- cites Sinnott-Armstrong, 2008
Not checked in the free run.
The paper relies on this without support.
LLMs have no special insight into their inner workings.
“the fact that the models themselves have no special insight into their inner workings”
The paper relies on this without support.
LLM-generated responses can be treated as analogous to the distribution of answers from a single human population.
“We can see this as akin to the distribution of different answers that people belonging to a single population would produce.”
The paper relies on this without support.
The Moral Foundations Questionnaire is a valid instrument for measuring the weight people assign to different fundamental values.
“The Moral Foundations Questionnaire (MFQ) is a well established psychological instrument designed to measure the weight people assign to different fundamental values.”
The paper relies on this without support.
LLM responses can be validly compared to human responses as if they were from a similar population.
“Just as human subjects, each LLM began by answering the question about blame or praise and then answered whether the chairman acted intentionally.”
The paper relies on this without support.
The human responses from the original studies provide a valid benchmark for evaluating LLM alignment in moral and legal reasoning.
“To compute the correlation between responses generated by humans and LLMs, we averaged causality ratings across all 16 unique combinations of causal structure, norm violation, and scenario for each agent.”
The paper relies on this without support.
The stimuli and questions from the original human studies are equally valid and interpretable for LLMs as they are for humans.
“Using the stimuli, data, and analysis code for Demaree-Cotton and Sommers (2022), which are available at: https://osf.io/z5cdh/,we created 200 responses for each model following a 3 (autonomy: Exercises Capacity; Mere Capacity; Lacks Capacity) x 3 (domain: medical consent; sexual consent; consent to police entry) between-subjects design.”
The paper relies on this without support.
The models' outputs can be meaningfully compared to human responses despite differences in training and architecture.
“To further explore the differences between human-and machine-generated responses, we built linear models of each of the DVs with condition, agent, and their interaction as independent variables.”
The paper relies on this without support.
Human responses in the original studies provide a valid benchmark for evaluating LLM moral and legal reasoning.
“We compared the answers of LLMs with those produced by human participants in the original studies or replications.”
The paper relies on this without support.
The eight experiments are representative of moral and legal reasoning.
“we carried out an exploratory analysis of current state-of-the-art LLMs' capacity for moral and legal reasoning by assessing its behavior across eight experiments.”
The paper relies on this without support.
The four models tested are state-of-the-art and representative of current LLMs.
“we tested how four state-of-the-art models (Gemini Pro, Claude 2.1, GPT-4, and Llama 2 Chat 70b) performed on eight different survey-based experiments”
The paper relies on this without support.
The experimental manipulations in the original studies are valid for probing moral and legal reasoning in LLMs.
“the effects of the experimental manipulations we have tested”
The paper relies on this without support.
The correlation between per-condition means is a valid measure of similarity between human and LLM responses.
“Correlation between per-condition means between each model's responses and human generated responses for each study.”
The paper relies on this without support.
The reduced variance of LLM-generated responses is not an artifact of the measurement or prompting.
“due in part to the reduced variance of LLM-generated responses.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current LLM benchmarks may provide misleading assessments of LLM capabilities due to superficial optimization strategies.
“Consequently, current benchmarks may provide misleading assessments of LLM capabilities, due to superficial optimization strategies [28].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [28]
Predominantly English-centric benchmarks overlook sophisticated reasoning patterns embedded within languages, when language structure can shape distinct cognitive processes.
“Predominantly English-centric benchmarks overlook the sophisticated reasoning patterns embedded within languages, when language structure can shape distinct cognitive processes [28], [57].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [28]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Inconsistent p-value across alpha = .05: F (1, 746) = 2.46, p = .044
MajorThe paper reports p = .044 here; recomputing from the test statistic and its degrees of freedom gives p = .117. The two fall on opposite sides of alpha = .05. The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .044 · recomputed p = .117
“F (1, 746) = 2.46, p = .044”
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upF(1, 746) = 2.46reported p = .044 · recomputed p = .117The two fall on opposite sides of .05
“F (1, 746) = 2.46, p = .044”
- ConsistentF(1, 478) = 1498.6reported p < .001 · recomputed p < .001
“F (1, 478) = 1498.60,p < .001”
- ConsistentF(4, 478) = 40.95reported p < .001 · recomputed p < .001
“F (4, 478) = 40.95,p < .001”
- ConsistentF(4, 478) = 60.04reported p < .001 · recomputed p < .001
“F (4, 478) = 60.04,p < .001”
- ConsistentF(3, 392) = 4.74reported p = .003 · recomputed p = .003
“F (3, 392) = 4.74, p = .003”
- ConsistentF(3, 392) = 44.82reported p < .001 · recomputed p < .001
“F (3, 392) = 44.82,p < .001”
- ConsistentF(4, 7135) = 21.56reported p < .001 · recomputed p < .001
“F (4, 7135) = 21.56,p < .001”
- ConsistentF(14, 99520) = 2058.36reported p < .001 · recomputed p < .001
“F (14, 99520) = 2058.36,p < .001”
- ConsistentF(56, 99501) = 18.03reported p < .001 · recomputed p < .001
“F (56, 99501) = 18.03, p < .001”
- ConsistentF(4, 7128) = 21.57reported p < .0001 · recomputed p < .001
“F (4, 7128) = 21.57,p < .0001”
- ConsistentF(4, 28423) = 3697.08reported p < .0001 · recomputed p < .001
“F (4, 28423) = 3697.08,p < .0001”
- ConsistentF(16, 28398) = 36.3reported p < .0001 · recomputed p < .001
“F (16, 28398) = 36.3,p < .0001”
- ConsistentF(1, 1670) = 733.36reported p < .001 · recomputed p < .001
“F (1, 1670) = 733.36,p < .001”
- ConsistentF(1, 1672) = 344.8reported p < .001 · recomputed p < .001
“F (1, 1672) = 344.80,p < .001”
- ConsistentF(1, 1670) = 51.18reported p < .001 · recomputed p < .001
“F (1, 1670) = 51.18,p < .001”
- ConsistentF(1, 1671) = 126.52reported p < .001 · recomputed p < .001
“F (1, 1671) = 126.52,p < .001”
- ConsistentF(1, 1671) = 10.56reported p = .001 · recomputed p = .001
“F (1, 1671) = 10.56,p = .001”
- ConsistentF(4, 1265) = 19.95reported p < .001 · recomputed p < .001
“F (4, 1265) = 19.95,p < .001”
- ConsistentF(4, 1670) = 48.71reported p < .001 · recomputed p < .001
“F (4, 1670) = 48.71,p < .001”
- ConsistentF(4, 1336) = 3.53reported p = .007 · recomputed p = .007
“F (4, 1336) = 3.53, p = .007”
- ConsistentF(4, 1672) = 6.86reported p < .001 · recomputed p < .001
“F (4, 1672) = 6.86, p < .001”
- ConsistentF(4, 1670) = 4.32reported p = .002 · recomputed p = .002
“F (4, 1670) = 4.32, p = .002”
- ConsistentF(4, 1675) = 5.78reported p < .001 · recomputed p < .001
“F (4, 1675) = 5.78, p < .001”
- ConsistentF(1, 435) = 138.91reported p < .001 · recomputed p < .001
“F (1, 435) = 138.91,p < .001”
- ConsistentF(2, 794) = 643.74reported p < .001 · recomputed p < .001
“F (2, 794) = 643.74,p < .001”
- ConsistentF(6, 794) = 21.52reported p < .001 · recomputed p < .001
“F (6, 794) = 21.52,p < .001”
- ConsistentF(6, 794) = 11.35reported p < .001 · recomputed p < .001
“F (6, 794) = 11.35,p < .001”
- ConsistentF(4, 746) = 18.05reported p < .001 · recomputed p < .001
“F (4, 746) = 18.05, p < .001”
- ConsistentF(12, 746) = 4.55reported p < .001 · recomputed p < .001
“F (12, 746) = 4.55, p < .001”
- ConsistentF(4, 746) = 5.72reported p < .001 · recomputed p < .001
“F (4, 746) = 5.72, p < .001”
- ConsistentF(12, 746) = 2.17reported p = .011 · recomputed p = .012
“F (12, 746) = 2.17, p = .011”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Consent results include Claude after excluding it
MajorThe methods say Claude 2.1 was dropped from the consent analysis because it had no valid sexual-consent responses. The results nevertheless report a correlation involving Claude across the nine condition-by-domain combinations, which include sexual consent.
“Claude 2.1 did not produce any valid answer for the sexual consent domain.To prevent skewing the results, we decided to drop it from the analysis.”
“The only significant correlation was between Llama 2 Chat 70b and Claude 2.1 (r = 0.76 [0.20, 0.95], p = .017;see Supplementary Table 9).”
GPT-4 was not the top human correlate in Study 3
MinorThe summary names Claude 2.1 as the sole exception to GPT-4's lead. The Moral Foundations table also places Llama 2 above GPT-4 in correlation with humans (.86 versus .82).
“GPT-4 was consistently more well-aligned with human responses than other models (with the exception of Claude 2.1 in Study 3).”
“| 4. GPT-4 | | 15 | | .82** | .81** | .97** | | | | | | | [.54, .94] | [.50, .93] | [.90, .99] | | | | | | | p < .001 | p < .001 | p < .001 | | | 5. Llama 2 Chat 70b | 15 | | .86** | .60* | .88** | .80** | |”
Consent model reports incompatible agent degrees of freedom
MinorThe reported six degrees of freedom for the agent-by-condition interaction, with three consent conditions, imply three degrees of freedom for agent. The same model reports only two degrees of freedom for its agent main effect.
“This model revealed a significant main effect of agent (F (2, 794) = 643.74,p < .001,η² = 0.71)”
“significant two-way interactions with condition (F (6, 794) = 21.52,p < .001,η² = 0.14)”
GPT-4 deception correlation has two different confidence intervals
MinorThe results give the human-GPT-4 deception correlation as .96 with a 95% confidence interval ending at .99. The later summary gives the same correlation an interval ending at .88, below the reported correlation itself.
“Humans vs. GPT-4: r df = 7 = 0.96, 95% CI: 0.82 -0.99, p <.001”
“studies 2 (deception, r n = 7 = .96[.82, .88],see Section 2.6)”
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · see citing works
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Liberals and conservatives rely on different sets of moral foundations.
Graham, J., Haidt, J., & Nosek, B. A. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015141 · Cited by 4,560 · see citing works
The righteous mind: why good people are divided by politics and religion
(2012) · Choice Reviews Online · doi:10.5860/choice.49-7185 · Cited by 4,347 · see citing works
Graham, J., Nosek, B. A., Haidt, J., Iyer, R., Koleva, S., & Ditto, P. H. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021847 · Cited by 2,666 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Superintelligence: paths, dangers, strategies
(2015) · Choice Reviews Online · doi:10.5860/choice.187536 · Cited by 1,977 · see citing works
Graham, J., Haidt, J., Koleva, S. P., Motyl, M. S., Iyer, R., Wojcik, S. P., & Ditto, P. H. (2013) · Advances in experimental social psychology · doi:10.1016/b978-0-12-407236-7.00002-4 · Cited by 1,815 · see citing works
Proceedings of the 29th Annual Conference of the Cognitive Science Society
Rohde, H., Kehler, A., & Elman, J. L. (2007) · Cognitive Science · Cited by 1,782 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364 · see citing works
Cognition does not affect perception: Evaluating the evidence for “top-down” effects
Firestone, C., & Scholl, B. J. (2016) · Behavioral and Brain Sciences · doi:10.1017/s0140525x15000965 · Cited by 1,200 · see citing works
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
Intentional action and side effects in ordinary language
Knobe, J. (2003) · Analysis · doi:10.1093/analys/63.3.190 · Cited by 1,046 · see citing works
The misunderstood limits of folk science: An illusion of explanatory depth
Rozenblit, L., & Keil, F. (2002) · Cognitive Science · doi:10.1207/s15516709cog2605_1 · Cited by 1,032 · see citing works
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Many Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · see citing works
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · see citing works
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Xu, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.01219 · Cited by 544 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · see citing works
Persson, I., & Savulescu, J. (2012) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199653645.001.0001 · Cited by 434 · see citing works
Can AI language models replace human participants?
Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.04.008 · Cited by 405 · see citing works
Person as scientist, person as moralist
Knobe, J. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000907 · Cited by 376 · see citing works
Hitchcock, C., & Knobe, J. (2009) · The Journal of Philosophy · doi:10.5840/jphil20091061128 · Cited by 371 · see citing works
A Companion to Experimental Philosophy
(2016) · Wiley eBooks · doi:10.1002/9781118661666 · Cited by 363 · see citing works
Evaluating Large Language Models in Theory of Mind Tasks
Kosinski, M. (2023) · arXiv · doi:10.48550/arxiv.2302.02083 · Cited by 339 · see citing works
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · see citing works
Philosophers’ biased judgments persist despite training, expertise and reflection
Schwitzgebel, E., & Cushman, F. A. (2015) · Cognition · doi:10.1016/j.cognition.2015.04.015 · Cited by 318 · see citing works
Moral psychology, Vol 2: The cognitive science of morality: Intuition and diversity.
Sinnott-Armstrong, W. P. (2008) · Cited by 232 · see citing works
Estimating the Reproducibility of Experimental Philosophy
Cova, F., Strickland, B., Abatista, A., Allard, A., Andow, J., Attie, M., Beebe, J. R., Berniûnas, R., Boudesseul, J., Colombo, M., et al. (2018) · Review of Philosophy and Psychology · doi:10.1007/s13164-018-0400-9 · Cited by 184 · see citing works
Normality and actual causal strength
Icard, T. F., Kominsky, J. F., & Knobe, J. (2017) · Cognition · doi:10.1016/j.cognition.2017.01.010 · Cited by 155 · see citing works
The Artificial Moral Advisor. The “Ideal Observer” Meets Artificial Intelligence
Giubilini, A., & Savulescu, J. (2017) · Philosophy & Technology · doi:10.1007/s13347-017-0285-z · Cited by 136 · see citing works
Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political IdentityPreprint
Simmons, G. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2209.12106 · Cited by 122 · see citing works
Kominsky, J. F., Phillips, J., Gerstenberg, T., Lagnado, D. A., & Knobe, J. (2015) · Cognition · doi:10.1016/j.cognition.2015.01.013 · Cited by 117 · see citing works
Kneer, M., & Machery, É. (2018) · Cognition · doi:10.1016/j.cognition.2018.09.003 · Cited by 114 · see citing works
Whose Opinions Do Language Models Reflect?
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2303.17548 · Cited by 114 · see citing works
How prescriptive norms influence causal inferences
Samland, J., & Waldmann, M. R. (2016) · Cognition · doi:10.1016/j.cognition.2016.07.007 · Cited by 97 · see citing works
Diminished diversity-of-thought in a standard large language model
Park, P. S., Schoenegger, P., & Zhu, C. (2024) · Behavior Research Methods · doi:10.3758/s13428-023-02307-x · Cited by 83 · see citing works
Experimental Philosophy and the Philosophical Tradition
Stich, S., & Tobia, K. P. (2016) · doi:10.1002/9781118661666.ch1 · Cited by 82 · see citing works
Kahan, D. M., Hoffman, D. A., Evans, D., et al. (2016) · Penn Carey Law Legal Scholarship Repository (University of Pennsylvania) · Cited by 78 · see citing works
Mens rea ascription, expertise and outcome effects: Professional judges surveyed
Kneer, M., & Bourgeois‐Gironde, S. (2017) · Cognition · doi:10.1016/j.cognition.2017.08.008 · Cited by 72 · see citing works
Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68 · see citing works
Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity
Simmons, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.40 · Cited by 60 · see citing works
The letter versus the spirit of the law: a lay perspective on culpability
Garcia, S. M., Chen, P., & Gordon, M. T. (2014) · Judgment and Decision Making · doi:10.1017/s1930297500006835 · Cited by 54 · see citing works
An experimental guide to vehicles in the park
Struchiner, N., Hannikainen, I. R., & Almeida, G. (2020) · Judgment and Decision Making · doi:10.1017/s1930297500007130 · Cited by 53 · see citing works
Self-citationOutcome effects, moral luck and the hindsight bias
Kneer, M., & Skoczeń, I. (2022) · Cognition · doi:10.1016/j.cognition.2022.105258 · Cited by 44 · see citing works
Should large language models replace human participants?Preprint
Crockett, M. J., & Messeri, L. (2023) · doi:10.31234/osf.io/4zdx9 · Cited by 43 · see citing works
Coordination and expertise foster legal textualism
Hannikainen, I. R., Tobia, K. P., Almeida, G., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2206531119 · Cited by 40 · see citing works
Self-citationIntuitive Expertise in Moral Judgments
Horvath, J., & Wiegmann, A. (2021) · Australasian Journal of Philosophy · doi:10.1080/00048402.2021.1890162 · Cited by 39 · see citing works
Self-citationRecent empirical work on the relationship between causal judgements and norms
Willemsen, P., & Kirfel, L. (2018) · Philosophy Compass · doi:10.1111/phc3.12562 · Cited by 38 · see citing works
Causal judgments about atypical actions are influenced by agents' epistemic states
Kirfel, L., & Lagnado, D. A. (2021) · Cognition · doi:10.1016/j.cognition.2021.104721 · Cited by 37 · see citing works
Bregant, J., Wellbery, I., & Shaw, A. (2019) · Journal of Experimental Child Psychology · doi:10.1016/j.jecp.2018.09.019 · Cited by 36 · see citing works
Advances in Experimental Philosophy of Law
(2023) · Bloomsbury Academic eBooks · doi:10.5040/9781350278301 · Cited by 25 · see citing works
Autonomy and the folk concept of valid consent
Demaree‐Cotton, J., & Sommers, R. (2022) · Cognition · doi:10.1016/j.cognition.2022.105065 · Cited by 23 · see citing works
Moral appraisals guide intuitive legal determinations
Flanagan, B., de Almeida, G. F. C. F., Struchiner, N., & Hannikainen, I. R. (2023) · Law and Human Behavior · doi:10.1037/lhb0000527 · Cited by 16 · see citing works
Empathy influences the interpretation of whether others have violated everyday indeterminate rules.
LaCosse, J., & Quintanilla, V. D. (2021) · Law and Human Behavior · doi:10.1037/lhb0000456 · Cited by 9 · see citing works
MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment TasksPreprint
Nie, A., Zhang, Y., Amdekar, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19677 · Cited by 9 · see citing works
Diminished Diversity-of-Thought in a Standard Large Language Model
Park, P. S., Schoenegger, P., & Zhu, C. (2023) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.48550/arxiv.2302.07267 · Cited by 9 · see citing works
Specific versus General Principles for Constitutional AI
Kundu, S., Bai, Y., Kadavath, S., Askell, A., Callahan, A., Chen, A., Goldie, A., Balwit, A., Mirhoseini, A., McLean, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.13798 · Cited by 7 · see citing works
Experiments in Moral and Political Philosophy
Viciana, H., Gaitán, A., & Aguiar, F. (2023) · Routledge eBooks · doi:10.4324/9781003301424 · Cited by 6 · see citing works
Goli (2023) · doi:10.2139/ssrn.4437617 · Cited by 3 · see citing works
de Araújo, M., Almeida, G., & Nunes, J. L. (2024) · Minds and Machines · doi:10.1007/s11023-024-09660-6 · Cited by 3 · see citing works
Self-citationde Araújo, M., Almeida, G., & Nunes, J. L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4204178 · Cited by 2 · see citing works
Self-citationMurderer at the door! To lie or to mislead?Preprint
Engelmann, N. (2023) · doi:10.31234/osf.io/habrm · Cited by 1 · see citing works
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
3 of the 69 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Bricken, 2023, Towards monosemanticity: decompsoing language models with dictionary learning, Transform. Circ. Threads
- Sommers, 2020, Commonsense consent, Yale Law J., 129, 2232
- Marcus, 2023, How not to test GPT-3, Commun. ACM Blog
Cited by
Sorted by citations
Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., Wang, W., Li, R., Xu, J., Tang, X., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768165 · Cited by 106 · see citing works
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
“For instance, an LLM used in a customer service chatbot might produce inappropriate or biased responses based on hidden internal biases, without any easy way to diagnose the underlying issue or improve the LLM’s behavior [29].”
MethodExploring the frontiers of LLMs in psychological applications: a comprehensive review
Ke, L., Tong, S., Cheng, P., & Peng, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11297-5 · Cited by 92 · see citing works
AI language model rivals expert ethicist in perceived moral expertise
Dillion, D., Mondal, D., Tandon, N., & Gray, K. J. (2025) · Scientific Reports · doi:10.1038/s41598-025-86510-0 · Cited by 54 · see citing works
Large language models show amplified cognitive biases in moral decision-making
Cheung, V., Maier, M., & Lieder, F. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2412015122 · Cited by 53 · see citing works
A Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 51 · see citing works
Surprising gender biases in GPT
Fulgu, R. A., & Capraro, V. (2024) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100533 · Cited by 24 · see citing works
Beyond WEIRD: Can synthetic survey participants substitute for humans in global policy research?
Shrestha, P., Krpan, D., Koaik, F., Schnider, R., Sayess, D., & Binbaz, M. S. (2024) · Behavioral Science & Policy · doi:10.1177/23794607241311793 · Cited by 21 · see citing works
MoralBench: Moral Evaluation of LLMs
Ji, J., Chen, Y., Jin, M., Xu, W., Hua, W., & Zhang, Y. (2025) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3748239.3748246 · Cited by 25 · see citing works
“These studies often provide valuable insights, offering a basic idea of the similarities between human and LLM choices [38; 39].”
Metaheuristics and Large Language Models Join Forces: Toward an Integrated Optimization Approach
Sartori, C. C., Blum, C., Bistaffa, F., & Corominas, G. R. (2024) · IEEE Access · doi:10.1109/access.2024.3524176 · Cited by 16 · see citing works
From humans to machines: Researching entrepreneurial AI agents built on large language models
Obschonka, M., & Fisch, C. O. (2025) · Journal of Business Venturing Insights · doi:10.1016/j.jbvi.2025.e00581 · Cited by 13 · see citing works
Large language models for mental health diagnosis and treatment: a survey
Ghorbian, M., & Ghobaei‐Arani, M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11418-0 · Cited by 15 · see citing works
“In addition, accurate diagnosis of mental disorders is often complex and time-consuming, requiring in-depth analysis of patients’ behavioral and emotional data, which in many cases is fraught with human error or a lack of sufficient data (Almeida et al. 2024; Huang et al. 2023).”
Dillion, D., Mondal, D., Tandon, N., & Gray, K. J. (2024) · doi:10.31234/osf.io/w7236 · Cited by 14 · see citing works
Simulating Human Opinions with Large Language Models
Kaiser, C., Kaiser, J., Manewitsch, V., Rau, L., & Schallner, R. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3708319.3733685 · Cited by 12 · see citing works
Hila, A. (2026) · AI & SOCIETY · doi:10.1007/s00146-025-02426-3 · Cited by 14 · see citing works
“LLMs are being adopted in education (Q. Li et al. 2023; Ng 2024), medicine (Sayin et al. 2024; Thirunavukarasu et al. 2023; Wang et al. 2024a, b), law (Almeida et al. 2024; Cheong et al. 2024; Lai et al. 2024), finance, to name a few yet consequential domains of adoption.”
Li, L., Kong, S., Zhao, H., Li, C., Teng, Y., & Wang, Y. (2025) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13781 · Cited by 10 · see citing works
Large Language Models meet moral values: A comprehensive assessment of moral abilities
Bulla, L., Giorgis, S. D., Mongiovı̀, M., & Aldo Gangemi, A. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100609 · Cited by 14 · see citing works
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · doi:10.31219/osf.io/tg79n · Cited by 8 · see citing works
OntoTune: Ontology-Driven Self-training for Aligning Large Language Models
Liu, Z., Gan, C. G., Wang, J., Zhang, Y., Bo, Z., Sun, M., Chen, H., & Zhang, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714816 · Cited by 11 · see citing works
“To further enhance the capabilities of LLMs in specific domain, such as medical, financial, and science, the research and industry community have begun to focus on developing domain-specific LLMs [2, 7, 31].”
Interacting with AI at Work: Perceptions and Opportunities from the UK Judiciary
Solovey, E., Flanagan, B., & Chen, D. L. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729192 · Cited by 8 · see citing works
“There is even evidence that in hard cases chatbots simulate people’s tendency to equivocate between the law’s letter and the law’s spirit [2].”
Xu, Z., Sengar, N., Chen, T., Chung, H., & Oviedo-Trespalacios, Ó. (2025) · Travel Behaviour and Society · doi:10.1016/j.tbs.2025.101039 · Cited by 8 · see citing works
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · doi:10.31234/osf.io/d695y · Cited by 8 · see citing works
Ayash, L., Alhuzali, H., Alasmari, A., & Aloufi, S. A. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00137-9 · Cited by 10 · see citing works
GPT-4V shows human-like social perceptual capabilities at phenomenological and neural levels
Santavirta, S., Wu, Y., Suominen, L., & Nummenmaa, L. (2025) · Imaging Neuroscience · doi:10.1162/imag.a.134 · Cited by 7 · see citing works
“…who are liberal, high income, well-educated, and not religious in one study ( Santurkar et al., 2023) and the opinions were interpreted as right-leaning in another ( Park et al., 2024) while another study with GPT-4 model described the political views as moderate ( Almeida et al., 2024).”
Cyber Threat Intelligence meets the Analytic Tradecraft
Bjurling, B., & Raza, S. (2024) · ACM Transactions on Privacy and Security · doi:10.1145/3701299 · Cited by 6 · see citing works
“However, although LLMs have been applied for reasoning in domains such as medical diagnosis [ 62 ] and in general problem solving as well as in complex reasoning tasks [ 3 , 38 ], they perform poorly on these complex reasoning tasks.”
Surprising gender biases in GPTPreprint
Fulgu, R. A., & Capraro, V. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4888393 · Cited by 5 · see citing works
Learning Human Feedback from Large Language Models for Content Quality-aware Recommendation
Wang, H., Wu, C., Huang, Y., & Qi, T. (2025) · ACM Transactions on Information Systems · doi:10.1145/3727144 · Cited by 5 · see citing works
“However, there remains ongoing debate about whether LLMs can fully align with human expectations, particularly with respect to the phenomenon of hallucinations, where models generate fabricated or incorrect information [4, 18, 19, 28, 94].”
Exploring large language models’ responses to moral reasoning dilemmas
Nabizadeh, D., Walker, D., Han, H., & Laird, E. (2025) · Ethics & Behavior · doi:10.1080/10508422.2025.2601022 · Cited by 5 · see citing works
“…integrated into decision-making processes in healthcare (Armitage, 2025; Hager et al., 2024), education (Zhui et al., 2024), legal systems (Almeida et al., 2024; Marcos, 2024), art (Ivanova, 2025), and self-driving cars (Ahmad & Takemoto, 2024; Takemoto, 2024; Xu et al., 2025), their…”
Grizzard, M., Frazer, R., Luttrell, A., Monge, C. K., Matthews, N. L., Francemone, C. J., & Frazer, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-24700-6 · Cited by 5 · see citing works
“Other researchers have been less optimistic, pointing to the need for additional research (particularly pre-registered experiments 5 , such as the one presented here) and more in-depth consideration of the possible limitations of AI 6–12 .”
Elkins, K. (2024) · Poetics Today · doi:10.1215/03335372-11092884 · Cited by 4 · see citing works
Evaluating the Explainability of Large Language Models for Ethical Decision Making
Miguel, G. S. S., Griffith, H., Silva, J., & Rathore, H. (2025) · IEEE International Conference on Consumer Electronics · doi:10.1109/icce63647.2025.10929891 · Cited by 4 · see citing works
Zhang, J., Han, S., Weng, Y., Hu, Y., Zhang, M., Cheng, M., & Zhang, G. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106594 · Cited by 4 · see citing works
Large Language Models Amplify Human Biases in Moral Decision-MakingPreprint
Cheung, V., Maier, M., & Lieder, F. (2024) · doi:10.31234/osf.io/aj46b · Cited by 4 · see citing works
Lin, Q., Hu, Z., & Ma, J. (2024) · Information · doi:10.3390/info15110679 · Cited by 3 · see citing works
Road Rage Against the Machine: Humans and LLMs Share a Blame Bias Against Driverless Cars
Chu, Y., Liu, P., Savulescu, J., & Earp, B. D. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2526593 · Cited by 3 · see citing works
“The current evidence on human-LLM alignment in moral judgments is limited but growing (e.g., Almeida et al., 2024; Hendrycks et al., 2021; Scherrer et al., 2023), with apparently conflicting results.”
InfluentialLarge Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3 · see citing works
The ethics mirror? Comparing LLM and human responses to ethical dilemmas of varying complexity
Tzini, K., Illia, L., & Zyglidopoulos, S. C. (2025) · European Management Journal · doi:10.1016/j.emj.2025.12.002 · Cited by 2 · see citing works
Comparing AI and human moral reasoning: context-sensitive patterns beyond utilitarian bias
Barabadi, E., Fotuhabadi, Z., Arghavan, A., & Booth, J. R. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1710410 · Cited by 4 · see citing works
Linguistic profile of LLMs: an overview of linguistic performance and jailbreak vulnerability
Abdelzaher, E. M., & Rafaee, E. (2026) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-025-00192-8 · Cited by 2 · see citing works
Evaluating Large Language Models in Psychological Research: A Guide for ReviewersPreprint
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2024) · doi:10.31234/osf.io/ag7hy · Cited by 2 · see citing works
GPT-4 accurately predicts human emotions and their neural correlatesPreprint
Santavirta, S., Suominen, L., Wu, Y., Sander, D., & Nummenmaa, L. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.09.18.677029 · Cited by 2 · see citing works
“However, research on these biases is inconclusive (Santurkar et al. 2023; Park, Schoenegger, and Zhu 2024; Almeida et al. 2024), and the rapid evolution of models makes it diFicult to keep up with investigating the properties of any specific version.”
Kwon, B., & Yu, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-06508-6 · Cited by 1 · see citing works
Karamanou, A., Kalampokis, E., Fitsilis, F., Theodorakopoulos, G., & Tarabanis, K. A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-01589-1_24 · Cited by 1 · see citing works
Zhang, Z., Zeng, W., Tang, J., Wang, J., & Zhao, X. H. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761350 · Cited by 1 · see citing works
Using Human Cumulative Prospect Theory to Understand Large Language Models Decision-Making
Zhang, G., Wu, J., Wang, P., et al. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3636502 · Cited by 1 · see citing works
“While LLMs and humans operate on fundamentally different architectures, deep neural networks versus biological neurons, there are surprising parallels between the two in decision-making [31], [32].”
Tao, Y., Viberg, O., Dennison, D. V., Wu, Z., & Kizilcec, R. F. (2026) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3774398.3811616 · Cited by 1 · see citing works
Ferreira, G., Amidei, J., Nieto, R., & Kaltenbrunner, A. (2025) · ACM Transactions on Computing for Healthcare · doi:10.1145/3712301 · Cited by 1 · see citing works
“…considering that personality tests are tailored for humans, and there is no guarantee beforehand that they will yield valid and reliable results for LLMs. Alongside personality traits also value orientations [52], moral beliefs [3, 50, 52, 60], and legal reasoning [3] are been evaluated.”
The Rule of Law or the Rule of Robots? Nationally Representative Survey Evidence from KenyaPreprint
Flanagan, B., Almeida, G., Chen, D., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4865782 · Cited by 1 · see citing works
Self-citationZenergy: Designing Taoist-Inspired Transformative Nature Imagery for Everyday Empowerment
Li, Z., Wang, Y., Wang, Y., & Sun, X. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790421
GPT-4V shows human-like social perceptual capabilities at phenomenological and neural levelsPreprint
Santavirta, S., Wu, Y., Suominen, L., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.20.608741
Evidence of conceptual mastery in the application of rules by Large Language ModelsPreprint
Nunes, J. L., Almeida, G., & Flanagan, B. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5161877
Self-citationLin, Q., Hu, Z., Liu, Z., & Ma, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93230-4_19
Exploring Large Language Models' Responses to Moral Reasoning DilemmasPreprint
Nabizadeh, D., Walker, D. I., Han, H., & Laird, E. (2025) · Research Square · doi:10.21203/rs.3.rs-6823916/v1
Technical and inspirational philosophy of law: Lawyers' reasoning and artificial intelligence
Spaić, B. (2025) · Glasnik Advokatske komore Vojvodine · doi:10.5937/gakv97-53818
Risks of Large Language Models Misalignment: Multi-Stakeholder Obligations and Governance
Wu, L., Zou, W., Li, J., & Qi, J. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-0880-8_22
Assessing the Ethical Alignment of Large Language Models Using Deep Embedding Models
Silva, J., Griffith, H., & Rathore, H. (2025) · International Conference on Artificial Intelligence Testing · doi:10.1109/aitest66680.2025.00029
“As large language models (LLMs) become increasingly integrated into decision-making systems, ensuring their alignment with human ethics has become a pressing concern [1].”
Pan, T., Yang, Q., Cole, A., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5436577
Iyer, N., & Pradhan, A. (2025) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3715070.3749272
Karami, B., Zandi, F., & Hatami, J. (2025) · Research Square · doi:10.21203/rs.3.rs-8270539/v1
Artificial research participants in behavioral science
Medina, V. A., & Mohan, M. (2025) · Journal of Ethics in Entrepreneurship and Technology · doi:10.1108/jeet-03-2025-0014
Pan, T., Yang, Q., Cole, A. J., Wilson, R., & Woodard, D. L. (2026) · Expert systems with applications · doi:10.1016/j.eswa.2026.131507 · Cited by 1 · see citing works
Structured Definitions and Segmentations for Legal Reasoning in LLMs: A Study on Indian Legal Data
Khatri, M., Yusuf, M., Shah, R. R., & Kumaraguru, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15134-6_21
Yu, F. X., Adam, G. C., Bastian, N. D., & Lan, T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6568443
Exploring attribution bias in LLMs: Social influences and prompt-based mitigation
Jiang, L., Cao, J., Zhu, G., & Wang, Y. (2026) · Technology in Society · doi:10.1016/j.techsoc.2026.103361
Feroz, F., & Kunst, J. R. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101285
GPT-4’s Alignment with Human Lie and Falsity Attribution in Cases of Deceptive Implicatures
Shurakov, N., Kolbe, M., Engelmann, N., et al. (2026) · Synthese Library/Synthese library · doi:10.1007/978-3-032-10073-3_25
Self-citationExploiting Emotional and Moral Knowledge for Moral Values Detection in Text
Nogales, A. D. A., Martín, C., & Araque, Ó. (2026) · Expert Systems · doi:10.1111/exsy.70294
Institutions as cached computation for resource-rational negotiation
Huang, M., & Evans, J. A. (2026) · Behavioral and Brain Sciences · doi:10.1017/s0140525x2510174x
Synthetic survey data in renewable energy acceptance research: A validity audit using PLS-SEM
Bhatia, T., Bharathy, G. K., & Prasad, M. (2026) · Energy Sustainable Development/Energy for sustainable development · doi:10.1016/j.esd.2026.102075
Sridharan, K., & Sivaramakrishnan, G. (2026) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/15562646261488630
Abduction In Silico: Large Language Models Replicate Human Inference to the Best Explanation
Douven, I. (2026) · Minds and Machines · doi:10.1007/s11023-026-09803-x
“While some studies suggest that LLMs exhibit human-like biases and heuristics in decision-making and moral reasoning (Dillion et al., 2023; Almeida et al., 2024), it has also been argued that these models often rely on superficial pattern matching rather than robust reasoning structures and…”
- Authors
- Almeida, G. F. C. F., Nunes, J. L., Engelmann, N., Wiegmann, A., & Araújo, M. D.
- Affiliations
- Insper (BR)
- Pontifícia Universidade Católica do Rio de Janeiro (BR)
- Max Planck Society (DE)
- Ruhr University Bochum (DE)
- Universidade Federal do Rio de Janeiro (BR)
- Corresponding author
- Nunes, José Luiz
- Venue
- Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2024-08
- Volume
- 333
- Pages
- 104145
- ISSN
- 0004-3702 (ISSN-L)
- OpenAlex
- W4396608701
- Semantic Scholar
- 2c4df0a828b2227d70dc255600623e89bf53dd70
- Cited by
- 70
- References
- 69
- Citations by year
- 2026: 17 · 2025: 41 · 2024: 10 · 2023: 2
- FWCI
- 28.31
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Keywords
- large language models, moral reasoning, legal reasoning, GPT-4, Llama 2, Claude 2, Gemini Pro, variance reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
