Attitudes Surrounding an Imperfect AI Autograder
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Deployment of AI assessment tools in education is widespread, but work on students’ interactions and attitudes towards imperfect autograders is comparatively lacking. This paper presents students’ perceptions surrounding a ∼ 90% accurate automated short-answer grader that determined homework and exam credit in a college-level computer science course. Using surveys and interviews, we investigated students’ knowledge about the autograder and their attitudes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 1×
“Due to the risk of Type I errors, we used the BenjaminiHochberg procedure [7] to control the false discovery rate, or the expected proportion of signifcant results that are false positives.”
MethodHow Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
Fereday, J., & Muir-Cochrane, E. (2006) · International Journal of Qualitative Methods · doi:10.1177/160940690600500107 · Cited by 11,829
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1×
“individual’s goals in a situation characterized by uncertainty and vulnerability” [35].”
BackgroundHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 2×
“For example, Eastern Flight 401 crashed because the crew failed to notice the autopilot was disengaged [44].”
BackgroundAlgorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Information bias in health research: definition, pitfalls, and adjustment methods
Althubaiti, A. (2016) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s104807 · Cited by 3,018
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 3×
“Setting proper expectations can help counter algorithm aversion, and providing transparency around algorithmic processes before use has shown promise in the literature [19, 33].”
BackgroundThe influence of technology anxiety on consumer use and experiences with self-service technologies
Meuter, M. L., Ostrom, A. L., Bitner, M. J., & Roundtree, R. (2003) · Journal of Business Research · doi:10.1016/s0148-2963(01)00276-4 · Cited by 1,247
Cited by
State of the art and practice in AI in education
Holmes, W., & Tuomi, I. (2022) · European Journal of Education · doi:10.1111/ejed.12533 · Cited by 1,093
Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature
Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022) · Big Data & Society · doi:10.1177/20539517221115189 · Cited by 316
“While some studies defined procedural fairness more narrowly as the inclusion of selected (sensitive) input features, others conceptualized it in broader terms, also addressing such criteria as the consistency or the revocability of decisions (Hsu et al., 2021; Schlicker et al., 2021).”
BackgroundResultInfluentialLarge-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189
Rodway, P., & Schepman, A. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100150 · Cited by 144
Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers
Schneider, J., Richner, R., & Riser, M. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00289-z · Cited by 48
Metadata
- Authors
- Hsu, S., Li, T. W., Zhang, Z., Fowler, M., Zilles, C., & Karahalios, K.
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-15
- OpenAlex
- W3161516024
- Cited by
- 61
- References
- 25
- FWCI
- 7.65
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Attitudes Surrounding an Imperfect AI Autograder. https://app.cochat.ai/papers/10.1145/3411764.3445424
- The paper
- Hsu, S., Li, T. W., Zhang, Z., Fowler, M., Zilles, C., & Karahalios, K. (2021). Attitudes Surrounding an Imperfect AI Autograder. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445424
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.