Attitudes Surrounding an Imperfect AI Autograder
The DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
Deployment of AI assessment tools in education is widespread, but work on students’ interactions and attitudes towards imperfect autograders is comparatively lacking. This paper presents students’ perceptions surrounding a ∼ 90% accurate automated short-answer grader that determined homework and exam credit in a college-level computer science course. Using surveys and interviews, we investigated students’ knowledge about the autograder and their attitudes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 1×
“Due to the risk of Type I errors, we used the BenjaminiHochberg procedure [7] to control the false discovery rate, or the expected proportion of signifcant results that are false positives.”
MethodHow Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
Fereday, J., & Muir-Cochrane, E. (2006) · International Journal of Qualitative Methods · doi:10.1177/160940690600500107 · Cited by 11,829
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1×
“individual’s goals in a situation characterized by uncertainty and vulnerability” [35].”
BackgroundHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 2×
“For example, Eastern Flight 401 crashed because the crew failed to notice the autopilot was disengaged [44].”
BackgroundAlgorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232
Information bias in health research: definition, pitfalls, and adjustment methods
Althubaiti, A. (2016) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s104807 · Cited by 3,018
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · cited 1×
“may result in trust and acceptance issues [17, 33, 36], but we know less about how students interact with and perceive imperfect AI autograders.”
BackgroundThe role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 3×
“Setting proper expectations can help counter algorithm aversion, and providing transparency around algorithmic processes before use has shown promise in the literature [19, 33].”
BackgroundThe influence of technology anxiety on consumer use and experiences with self-service technologies
Meuter, M. L., Ostrom, A. L., Bitner, M. J., & Roundtree, R. (2003) · Journal of Business Research · doi:10.1016/s0148-2963(01)00276-4 · Cited by 1,247 · cited 1×
“We assessed tech-savviness with a subset of the technology anxiety instrument validated by Meuter et al [41], and then constructed a tech-savviness score by coding the Likert responses as integers and summing.”
MethodFriedman, B., & Nissenbaum, H. (1996) · ACM Transactions on Information Systems · doi:10.1145/230538.230561 · Cited by 1,203
Lepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709 · cited 1×
“We had two other measures of fairness [37].”
BackgroundHow Much Information?: Effects of Transparency on Trust in an Algorithmic Interface
Kizilcec, R. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858402 · Cited by 615
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1041624.1041673 · Cited by 575 · cited 1×
“While a full discussion of the theory and evidence behind the teaching efectiveness of EiPE problems is beyond the scope of this paper, code reading is believed to be a developmental skill that precedes code writing [38] and previous research has found that performance on EiPE problems predicts performance on code wri…”
ResultUnderstanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“found that trust is more tied to a users’ personal experiences with a system [58].”
BackgroundFirst I "like" it, then I hide it: Folk Theories of Social Feeds
Eslami, M., Karahalios, K., Sandvig, C., Vaccaro, K., Rickman, A., Hamilton, K., & Kirlik, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858494 · Cited by 375
Self-citationRelationships between reading, tracing and writing skills in introductory programming
Lopez, M., Whalley, J., Robbins, P., & Lister, R. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404531 · Cited by 363
DeVito, M. A., Gergle, D., & Birnholtz, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025659 · Cited by 311
Wilson, J., & Czik, A. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.05.004 · Cited by 297
Kocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277
The effects of computer-generated feedback on the quality of writing
Stevenson, M., & Phakiti, A. (2014) · Assessing Writing · doi:10.1016/j.asw.2013.11.007 · Cited by 275 · cited 1×
“AWE has been deployed in primary, secondary, and higher education settings to help students improve their general essay-writing skills [27, 43, 51, 56].”
BackgroundA Qualitative Exploration of Perceptions of Algorithmic Fairness
Woodruff, A., Fox, S. E., Rousso-Schindler, S., & Warshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174230 · Cited by 256
User attitudes towards algorithmic opacity and transparency in online reviewing platforms
Eslami, M., Vaccaro, K., Lee, M. K., Elazari Bar On, A., Gilbert, E., & Karahalios, K. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300724 · Cited by 184 · cited 1×
“found folk theories drove action on Yelp, where users wrote reviews to cater to how they thought Yelp’s review fltering algorithm worked [21].”
BackgroundSelf-citationPalermo, C., & Thomson, M. M. (2018) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2018.07.002 · Cited by 107 · cited 1×
“AWE has been deployed in primary, secondary, and higher education settings to help students improve their general essay-writing skills [27, 43, 51, 56].”
BackgroundRoscoe, R. D., Wilson, J., Johnson, A. C., & Mayra, C. R. (2017) · Computers in Human Behavior · doi:10.1016/j.chb.2016.12.076 · Cited by 98 · cited 2×
“trust the system would give accurate scores) [45].”
Background
Cited by
State of the art and practice in AI in education
Holmes, W., & Tuomi, I. (2022) · European Journal of Education · doi:10.1111/ejed.12533 · Cited by 1,093
Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature
Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022) · Big Data & Society · doi:10.1177/20539517221115189 · Cited by 316
“While some studies defined procedural fairness more narrowly as the inclusion of selected (sensitive) input features, others conceptualized it in broader terms, also addressing such criteria as the consistency or the revocability of decisions (Hsu et al., 2021; Schlicker et al., 2021).”
BackgroundResultInfluentialLarge-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189
Rodway, P., & Schepman, A. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100150 · Cited by 144
Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers
Schneider, J., Richner, R., & Riser, M. (2023) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00289-z · Cited by 48
Metadata
- Authors
- Hsu, S., Li, T. W., Zhang, Z., Fowler, M., Zilles, C., & Karahalios, K.
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-15
- OpenAlex
- W3161516024
- Cited by
- 61
- References
- 25
- FWCI
- 7.65
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.