Ingeborg Krange, Meerita Segaran, Siv Gamlem, Synnøve Moltudal, Irina Engeness
pp. 39 – 63, download
(https://doi.org/10.55612/s-5002-059-001)
Abstract
The aim of this study was to investigate eighth-grade students’ assessment literacy and writing skills in English as a foreign language using an AI-based automated essay assessment tool (EAT). Data were gathered from a design-based research initiative where the EAT was designed, developed, and tested in naturalistic school settings. Fifty-six eighth-grade students wrote individual essays, for which they received automatic feedback. The feedback was discussed with their teachers and peers. Both the writing process and teacher and peer interactions were video recorded. The video data were analyzed using an interaction analysis. The improvements made on the essay based on the feedback logs registered by the EAT for each student’s writing trajectory and the different versions of the essay were examined using frequency analyses. The findings demonstrate that automated essay assessment might be useful for fostering students’ writing skills if teachers help students get started, identify errors, and share interpretations.
Keywords: assessment for learning, sociocultural interpretation of learning and teaching, design-based research, interaction analysis, frequency analysis, artificial intelligence (AI), automatic essay assessment, junior high school
References
1. Black, P. Wiliam, D.: Classroom assessment and pedagogy, Assessment in Education: Principles, Policy & Practice, 25(6), pp. 551–575 (2018). https://doi.org/10.1080/0969594X.2018.1441807
2. Black, P., Wiliam, D.: Assessment and classroom learning. Assessment in Education, 5, pp. 7–74 (1998). https://doi.org/10.1080/0969595980050102
3. Hattie, J., Timperley, H.: The power of feedback, Review of Educational Research, 77(1), pp. 81–112 (2007). https://doi.org/10.3102/003465430298487
4. Hattie, J. A.: Visible learning: a synthesis of over 800 meta-analyses relating to achievement. London and New York: Routledge (2009). https://doi.org/10.4324/9780203887332
5. Gamlem, S.M., Smith, K.: Student perceptions of classroom feedback, Assessment in Education: Principles, Policy & Practice, 20(2), pp. 150–169 (2013). https://doi.org/10.1080/0969594X.2012.749212
6. Schildkamp, K., van der Kleij, F.M., Heitink, M.C., Kippers, W.B., Veldkamp, B.P.: Formative assessment: a systematic review of critical teacher prerequisites for classroom practice, International Journal of Educational Research, 103, p. 101602 (2020). https://doi.org/https://doi.org/10.1016/j.ijer.2020.101602
7. Gamlem, S. M., & Vattøy, K.-D.: “Feedback and classroom practice” in International encyclopedia of education. eds. R. J. Tierney, F. Rizvi, and K. Ercikan, vol. 13. 4th ed (Elsevier), 89–95 (2023). https://doi.org/10.1016/j.tate.2019.06.024
8. Black, P., Wiliam, D.: Developing the theory of formative assessment, Educational Assessment, Evaluation and Accountability, 21(1), pp. 5–31 (2009). https://doi.org/10.1007/s11092-008-9068-5
9. Gamlem, S.M.: Feedback to support learning: changes in teachers’ practice and beliefs, Teacher Development, 19(4), pp. 461–482 (2015). https://doi.org/10.1080/13664530.2015.1060254
10. Wilson, J., Olinghouse, N.G., Andrada, G.N.: Does automated feedback improve writing quality?, Learning Disabilities: A Contemporary Journal, 12, pp. 93–118 (2014)
11. Hopfenbeck, T. N., Zhang, Z., Sun, S. Z., Robertson, P., & McGrane, J. A.: Challenges and opportunities for classroom-based formative assessment and AI: a perspective article [Perspective]. Frontiers in Education, 8. (2023). https://doi.org/10.3389/feduc.2023.1270700
12. Hegelheimer, V., Dursun, A., Li, Z.: Automated writing evaluation in language teaching: theory, development, and application, Computer Assisted Language Instruction Consortium, 33(1), pp. i–v (2016). https://doi.org/10.1558/cj.v33i1.29251
13. Hannigan, C., Alonzo, D., Oo, C.Z.: Student assessment literacy: indicators and domains from the literature, Assessment in Education: Principles, Policy & Practice, 29(4), pp. 482–504 (2022). https://doi.org/10.1080/0969594X.2022.2121911
14. Engeness, I.: What teachers do: facilitating the writing process with feedback from EssayCritic and collaborating peers, Technology, Pedagogy and Education, 27(3), pp. 297–311 (2018). https://doi.org/10.1080/1475939X.2017.1421259
15. Engeness, I., Mørch, A.: Developing writing skills in English using content-specific computer-generated feedback with EssayCritic, Nordic Journal of Digital Literacy, 10(02), pp. 118–135 (2016). https://doi.org/10.18261/issn.1891-943x-2016-02-
16. Brown, A.L.: Design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings, Journal of the Learning Sciences, 2, pp. 141–178 (1992). brown-1992.pdf (uio.no)
17. The Design-Based Research Collective: Design-based research: an emerging paradigm for educational inquiry, Educational Researcher, 32(1), pp. 5–8 (2003). https://doi.org/10.3102/0013189X032001005
18. Ludvigsen, S., Rasmussen, I., Krange, I., Moen, A., Middleton, D.: Intersecting trajectories of participation; temporality and learning, in Learning Across Sites; New Tools, Infrastructures and Practices, pp. 105–121, Pergamon, (2010). https://doi.org/10.4324/9780203847817-13
19. Jordan, B., Henderson, A.: Interaction analysis: foundations and practice. Journal of the Learning Sciences, 4, pp. 39–103 (1995). https://doi.org/10.1207/s15327809jls0401_2
20. Black, P., Wiliam, D.: Assessment and Learning (2 edition), SAGE Publications Ltd., (2012). https://doi.org/10.4135/9781446250808
21. Vattøy, K.-D., Gamlem, S.M., Kobberstad, L.R., Rogne, W.M.: Students’ experiences of assessment and feedback engagement in digital contexts: a mixed-methods case study in upper secondary school, Education Inquiry, pp. 1-22 (2022). https://doi.org/10.1080/20004508.2022.2122202
22. Smith, C.D., Worsfold, K., Davies, L., Fisher, R., McPhail, R. Assessment literacy and student learning: the case for explicitly developing students ‘assessment literacy’, Assessment and Evaluation in Higher Education, 38(1), pp. 44–60 (2013). https://doi.org/10.1080/02602938.2011.598636
23. Price, M., Rust, C., O’Donovan, B., Handley, K., Bryant, R. Assessment literacy: the foundation for improving student learning, ASKe, Oxford Centre for Staff and Learning Development. (2012).
24. Torkildsen, L.G., Erickson, G.: ‘If they’d written more…’—On students’ perceptions of assessment and assessment practices, Education Inquiry, 7(2), p. 27416 (2016). https://doi.org/10.3402/edui.v7.27416
25. Gamlem, S.M., Munthe, E.: Mapping the quality of feedback to support students’ learning in lower secondary classrooms, Cambridge Journal of Education, 44(1), pp. 75–92 (2014). https://doi.org/10.1080/0305764X.2013.855171
26. Graham, S.: Changing how writing is taught. Review of Educational Research, 43, pp. 277–303 (2019). https://doi.org/10.3102/0091732X18821125
27. Graham, S.: Instructional feedback in writing, in The Cambridge Handbook of Instructional Feedback, pp. 145–168, Cambridge University Press, (2018). https://doi.org/10.1017/9781316832134.009
28. Graham, S., McKeown, D., Kiuhara, S., Harris, K.: A meta-analysis of writing instruction for students in the elementary grades, Journal of Educational Psychology, 104(4), pp. 879–896 (2012). https://doi.org/10.1037/a0029185
29. Moltudal, S., Høydal, K., & Krumsvik, R. J. Glimpses Into Real-Life Introduction of Adaptive Learning Technology: A Mixed Methods Research Approach to Personalised Pupil Learning, Designs for Learning, 12(1), 13–28. https://doi.org/10.16993/dfl.138
30. Moltudal, S. H., Krumsvik, R. J., & Høydal, K. L. Adaptive Learning Technology in Primary Education: Implications for Professional Teacher Knowledge and Classroom Management. Frontiers in Education. https://doi.org/10.3389/feduc.2022.830536
31. Munthe, E., Erstad, O., Njå, M.B., Forsström, S., Gilje, Ø., Amdam, S., Moltudal, S., Hagen, S.B. (2022). Digitalisering i grunnopplæring; kunnskap, trender og framtidig forskningsbehov. Kunnskapssenter for utdanning: Universitetet i Stavanger. https://www.uis.no/sites/default/files/2022-12/13767200%20Rapport%20GrunDig_0.pdf
32. Danielsen, A.G., Samdal, O., Hetland, J., Wold, B.: School-related social support and students’ perceived life satisfaction. The Journal of Educational Research, 102(4), pp. 303–318 (2009). https://doi.org/10.3200/JOER.102.4.303-320
33. Vattøy, K.-D., Gamlem, S. Teacher-student interactions and feedback in English as a foreign language classroom. Cambridge Journal of Education, 50(3), pp. 371–389 (2020). https://doi.org/10.1080/0305764X.2019.1707512
34. Ranalli, J., Link, S., Chukharev-Hudilainen, E.: Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation, Educational Psychology (Dorchester-on-Thames), 37(1), pp. 8–25 (2017). https://doi.org/10.1080/01443410.2015.1136407
35. Collins, A., Joseph, D., Bielaczyc, K.: Design research: theoretical and methodological issues, Journal of the Learning Sciences, 13, pp. 15–42 (2004). https://doi.org/10.1207/s15327809jls1301_2
36. Dasgupta, J.: Imparting hands-on industry 4.0 education at low cost using open-source tools and python eco-system, in New Paradigm of Industry 4.0: Internet of Things, Big Data & Cyber Physical Systems, pp. 37–47, Springer, (2019)
37. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., Manning, C.D.: Stanza: a Python natural language processing toolkit for many human languages, arXiv preprint arXiv:2003.07082 (2020). https://doi.org/10.48550/arXiv.2003.07082
38. Mercer, N.: The guided construction of knowledge. Talk Amongst Teachers and Learners, Multilingual Matters Ltd., (1995)
39. Han, T., & Sari, E. An investigation on the use of automated feedback in Turkish EFL students’ writing classes. Computer assisted language learning, ahead-of-print(ahead-of-print), 1-24. (2022). https://doi.org/10.1080/09588221.2022.2067179.
40. Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer assisted language learning, 35(4), 605-634. https://doi.org/10.1080/09588221.2020.1743323
41. Gao, J. Exploring the Feedback Quality of an Automated Writing Evaluation System Pigai. International journal of emerging technologies in learning. (2021). https://doi.org/10.3991/ijet.v16i11.19657
42. Geng, J., & Razali, A. B. (2020). Tapping the Potential of Pigai Automated Writing Evaluation (AWE) Program to Give Feedback on EFL Writing. Universal Journal of Educational Research, 8, 8334-8343. https://doi.org/10.13189/ujer.2020.082638