After the release of ChatGPT at the end of 2022, L2 language learning has entered a new era of human and generative AI coexistence. Its widespread use has brought about significant changes in the research, teaching, and learning of foreign languages. The authors had been developing an automatic evaluation system to measure the “quality of content” of writing by machine learning for two years (Mita & Shimoda, 2023a, 2024). The developed automated evaluation system has a 59.7% agreement rate with the teacher's assessment and a correlation coefficient of .433**. When the old model was used by students for writing preparation, it was found to contribute significantly to the quality of the content (Mita & Shimoda, 2023b). One drawback, however, is that our study collected less data than is desirable for machine learning (1611 essays), limiting its ability to increase the degree of agreement. ChatGPT was released during our development of the automatic evaluation system, and we introduced it into our model, resulting in a higher agreement rate (69.8%) and correlation coefficient (0.684**) than the previous model. In the presentation, we will compare the old model using machine learning and the new model introducing ChatGPT, and discuss new possibilities for English writing evaluation using generative AI.

Authors:
Kaoru Mita, Jissen Women's University, Japan
Atsuko Shimoda, Jissen Women's University, Japan

About the Presenter(s)
Ms Kaoru Mita is a University Professor/Principal Lecturer at Jissen Women's Junior College in Japan

See this presentation on the full schedule – Saturday Schedule

Bookmark