AI Versus A University Professor: A Grading Interrater Reliability Study

Session Abstract: This session explores the findings of a study comparing the interrater reliability between an AI system and a university professor in grading academic work. As AI becomes increasingly integrated into education, understanding how its assessments align with human evaluators is crucial. The study examines the consistency of grades assigned by both AI and a professor in an undergraduate research methods course, highlighting areas of agreement and discrepancy. Key factors such as grading rubrics, subjectivity in evaluation, and biases are analyzed to determine the strengths and limitations of AI in replicating or complementing human judgment. Attendees will gain insights into the potential for AI to enhance grading efficiency and fairness, while also addressing challenges in ensuring academic integrity and rigor. The session will provoke discussion on the future role of AI in education and its implications for teaching, learning, and assessment practices.

Keywords: artificial intelligence grading, interrater reliability, technology, assessment fairness