AVALIAÇÃO AUTOMÁTICA DE RESPOSTAS CURTAS EM PORTUGUÊS BRASILEIRO: UM ESTUDO SOBRE MODELOS DE LINGUAGEM, ENGENHARIA DE PROMPT E CARACTERÍSTICAS TEXTUAIS

Abstract

Automatic Short Answer Grading (ASAG) has emerged as a promising approach to reducing human effort in large-scale educational assessments, but studies focused on Brazilian Portuguese remain limited. This work evaluates the performance of three Large Language Models (GPT-4o-mini, Sabiazinho-3, and Gemini 2.0-Flash) in ASAG, testing all 128 possible combinations of seven prompt engineering components and examining how textual characteristics—such as word count and lexical richness—affect model accuracy. Results show that combining few-shot examples with explicit rubrics was the most effective strategy, while step-by-step reasoning particularly benefited GPT-4o-mini. Sabiazinho-3 achieved the highest agreement with human evaluators, Gemini 2.0-Flash obtained the lowest mean absolute error but exhibited a high hallucination rate, and GPT-4o-mini produced the cleanest and most consistent numeric outputs. Furthermore, the lexical profile of student responses significantly influenced model performance, with medium levels of lexical richness posing the greatest challenge across all models.