TY - GEN
T1 - Atitudes surrounding an imperfect ai autograder
AU - Hsu, Silas
AU - Wentin, Tifany
AU - Zhang, Zhilin
AU - Fowler, Max
N1 - Publisher Copyright:
© 2021 ACM.
PY - 2021/5/6
Y1 - 2021/5/6
N2 - Deployment of AI assessment tools in education is widespread, but work on students' interactions and attitudes towards imperfect autograders is comparatively lacking. This paper presents students' perceptions surrounding a ~90% accurate automated short-answer grader that determined homework and exam credit in a college-level computer science course. Using surveys and interviews, we investigated students' knowledge about the autograder and their attitudes. We observed that misalignment between folk theories about how the autograder worked and how it actually worked could lead to suboptimal answer construction strategies. Students overestimated the autograder's probability of marking correct answers as wrong, and estimates of this probability were associated with dissatisfaction and perceptions of unfairness. Many participants expressed a need for additional instruction on how to cater to the autograder. From these fndings, we propose guidelines for incorporating imperfect short answer autograders into classroom in a manner that is considerate of students' needs.
AB - Deployment of AI assessment tools in education is widespread, but work on students' interactions and attitudes towards imperfect autograders is comparatively lacking. This paper presents students' perceptions surrounding a ~90% accurate automated short-answer grader that determined homework and exam credit in a college-level computer science course. Using surveys and interviews, we investigated students' knowledge about the autograder and their attitudes. We observed that misalignment between folk theories about how the autograder worked and how it actually worked could lead to suboptimal answer construction strategies. Students overestimated the autograder's probability of marking correct answers as wrong, and estimates of this probability were associated with dissatisfaction and perceptions of unfairness. Many participants expressed a need for additional instruction on how to cater to the autograder. From these fndings, we propose guidelines for incorporating imperfect short answer autograders into classroom in a manner that is considerate of students' needs.
KW - Asag
KW - Autograder
KW - Code reading
KW - Computer science education
KW - Eipe
KW - Folk theories
KW - Human-ai interaction
KW - Imperfect ai
KW - Perception and acceptance of ai
UR - https://www.scopus.com/pages/publications/85106684874
UR - https://www.scopus.com/pages/publications/85106684874#tab=citedBy
U2 - 10.1145/3411764.3445424
DO - 10.1145/3411764.3445424
M3 - Conference contribution
AN - SCOPUS:85106684874
T3 - Conference on Human Factors in Computing Systems - Proceedings
BT - CHI 2021 - Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
T2 - 2021 CHI Conference on Human Factors in Computing Systems: Making Waves, Combining Strengths, CHI 2021
Y2 - 8 May 2021 through 13 May 2021
ER -