arxiv

Authors: Sabina Elkins, Ekaterina Kochmar, Jackie C.K. Cheung, Iulian Serban

Abstract: Question generation (QG) is a natural language processing task with an abundance of potential benefits and use cases in the educational domain. In order for this potential to be realized, QG systems must be designed and validated with pedagogical needs in mind. However, little research has assessed or designed QG approaches with the input from real teachers or students. This paper applies a large language model-based QG approach where questions are generated with learning goals derived from Bloom’s taxonomy. The automatically generated questions are used in multiple experiments designed to assess how teachers use them in practice. The results demonstrate that teachers prefer to write quizzes with automatically generated questions, and that such quizzes have no loss in quality compared to handwritten versions. Further, several metrics indicate that automatically generated questions can even improve the quality of the quizzes created, showing the promise for large scale use of QG in the classroom setting.

Link: https://arxiv.org/abs/2401.05914

Authors: Haowei Hua(1), Hong Jiao(2), Xinyi Wang(3) ((1) Princeton University, (2) University of Maryland, College Park, (3) University of Maryland, College Park & Beijing Normal University)

Abstract: BERT and its variants are extensively explored for automated scoring. However, a limit of 512 tokens for these encoder-based models showed the deficiency in automated scoring of long essays. Thus, this research explores generative language models for automated scoring of long essays via summarization and prompting. The results revealed great improvement of scoring accuracy with QWK increased from 0.822 to 0.8878 for the Learning Agency Lab Automated Essay Scoring 2.0 dataset.

Link: https://arxiv.org/abs/2510.22830

Authors: Hamed Mahdavi(Pennsylvania State University), Pouria Mahdavinia(Pennsylvania State University), Alireza Farhadi(Amirkabir University of Technology), Pegah Mohammadipour(Pennsylvania State University), Samira Malek(Pennsylvania State University), Majid Daliri(New York University), Pedram Mohammadipour(Amirkabir University of Technology), Alireza Hashemi(City University of New York), Amir Khasahmadi(Autodesk), Vasant Honavar(Pennsylvania State University)

Abstract: State-of-the-art (SOTA) LLMs have progressed from struggling on proof-based Olympiad problems to solving most of the IMO 2025 problems, with leading systems reportedly handling 5 of 6 problems. Given this progress, we assess how well these models can grade proofs: detecting errors, judging their severity, and assigning fair scores beyond binary correctness. We study proof-analysis capabilities using a corpus of 90 Gemini 2.5 Pro-generated solutions that we grade on a 1-4 scale with detailed error annotations, and on MathArena solution sets for IMO/USAMO 2025 scored on a 0-7 scale. Our analysis shows that models can reliably flag incorrect (including subtly incorrect) solutions but exhibit calibration gaps in how partial credit is assigned. To address this, we introduce agentic workflows that extract and analyze reference solutions and automatically derive problem-specific rubrics for a multi-step grading process. We instantiate and compare different design choices for the grading workflows, and evaluate their trade-offs. Across our annotated corpus and MathArena, our proposed workflows achieve higher agreement with human grades and more consistent handling of partial credit across metrics. We release all code, data, and prompts/logs to facilitate future research.

Link: https://arxiv.org/abs/2510.27094

Authors: Hong Jiao, Hanna Choi, Haowei Hua

Abstract: This study explored the utilities of rationales generated by GPT-4.1 and GPT-5 in automated scoring using Prompt 6 essays from the 2012 Kaggle ASAP data. Essay-based scoring was compared with rationale-based scoring. The study found in general essay-based scoring performed better than rationale-based scoring with higher Quadratic Weighted Kappa (QWK). However, rationale-based scoring led to higher scoring accuracy in terms of F1 scores for score 0 which had less representation due to class imbalance issues. The ensemble modeling of essay-based scoring models increased the scoring accuracy at both specific score levels and across all score levels. The ensemble modeling of essay-based scoring and each of the rationale-based scoring performed about the same. Further ensemble of essay-based scoring and both rationale-based scoring yielded the best scoring accuracy with QWK of 0.870 compared with 0.848 reported in literature.

Link: https://arxiv.org/abs/2510.27131

Authors: Hugo Roger Paz

Abstract: The emergence of Generative Artificial Intelligence (GenAI) has reshaped higher education, presenting both opportunities and ethical-pedagogical challenges. This article presents an empirical case study on the complete cycle (design, initial failure, redesign, and re-evaluation) of an intervention using an AI Tutor (ChatGPT) in the “Hydrology and Hydraulic Works” course (Civil Engineering, UTN-FRT, Argentina). The study documents two interventions in the same cohort (n=23). The first resulted in widespread failure (0% pass rate) due to superficial use and serious academic integrity issues (65% similarity, copies > 80%). This failure forced a comprehensive methodological redesign. The second intervention, based on a redesigned prompt (Prompt V2) with strict evidence controls (mandatory Appendix A with exported chat, minimum time $\geq$ 120 minutes, verifiable numerical exercise) and a refined rubric (Rubric V2), showed significantly better results: a median score of 88/100 and verifiable compliance with genuine interaction processes. Using a mixed-methods approach (reproducible document analysis and rubric analysis), the impact of the redesign on integrity and technical performance is evaluated. The results demonstrate that, without explicit process controls, students prioritize efficiency over deep learning, submitting documents without real traceability. A transferable assessment protocol for STEM courses is proposed, centered on “auditable personal zones,” to foster higher-order thinking. The study provides key empirical evidence from the context of a public Latin American university.

Link: https://arxiv.org/abs/2510.22279