Ingeborg Krange , Meerita Segaran , Siv Gamlem , Synnøve Moltudal , Irina Engeness
pp. 37 - 61, view paper, download
(, Google Scholar

Submitted on 15 Sep 2023 - Accepted on 23 Jan 2024

Interaction Design and Architecture(s) IxD&A Journal
Issue N. 59, Winter 2023


The aim of this study was to investigate eighth-grade students’ assessment literacy and writing skills in English as a foreign language using an AI-based automated essay assessment tool (EAT). Data were gathered from a design-based research initiative where the EAT was designed, developed, and tested in naturalistic school settings. Fifty-six eighth-grade students wrote individual essays, for which they received automatic feedback. The feedback was discussed with their teachers and peers. Both the writing process and teacher and peer interactions were video recorded. The video data were analyzed using an interaction analysis. The improvements made on the essay based on the feedback logs registered by the EAT for each student’s writing trajectory and the different versions of the essay were examined using frequency analyses. The findings demonstrate that automated essay assessment might be useful for fostering students’ writing skills if teachers help students get started, identify errors, and share interpretations.

Keywords: assessment for learning, sociocultural interpretation of learning and teaching, design-based research, interaction analysis, frequency analysis, artificial intelligence (AI), automatic essay assessment, junior high school

CRediT author statement: Ingeborg Krange: Data collection, Conceptualization, Methodology, Analyses, Writing- Original draft presentation. Meerita Segeran: Data collection, Writing- Methodology, analyses, and development of figures. Siv Gamlem: Data collection, Writing- Review. Synnøve Moltudal: Writing- Methodology, analyses, and presentation of overall argument. Irina Engeness: Data collection, Writing- Review.

Cite this article as:
Krange I., Segaran M., Gamlem S., Moltudal S., Engeness I.: A Triple Challenge: Students’ Identification, Interpretation, and Use of Individualized Automated Feedback in Learning to Write English as a Foreign Language, Interaction Design & Architecture(s) – IxD&A Journal, N.59, 2023, pp. 37–61, DOI:


1. Black, P. Wiliam, D.: Classroom assessment and pedagogy, Assessment in Education: Principles, Policy & Practice, 25(6), pp. 551–575 (2018).
2. Black, P., Wiliam, D.: Assessment and classroom learning. Assessment in Education, 5, pp. 7–74 (1998).
3. Hattie, J., Timperley, H.: The power of feedback, Review of Educational Research, 77(1), pp. 81–112 (2007).
4. Hattie, J. A.: Visible learning: a synthesis of over 800 meta-analyses relating to achievement. London and New York: Routledge (2009).
5. Gamlem, S.M., Smith, K.: Student perceptions of classroom feedback, Assessment in Education: Principles, Policy & Practice, 20(2), pp. 150–169 (2013).
6. Schildkamp, K., van der Kleij, F.M., Heitink, M.C., Kippers, W.B., Veldkamp, B.P.: Formative assessment: a systematic review of critical teacher prerequisites for classroom practice, International Journal of Educational Research, 103, p. 101602 (2020).
7. Gamlem, S. M., & Vattøy, K.-D.: “Feedback and classroom practice” in International encyclopedia of education. eds. R. J. Tierney, F. Rizvi, and K. Ercikan, vol. 13. 4th ed (Elsevier), 89–95 (2023).
8. Black, P., Wiliam, D.: Developing the theory of formative assessment, Educational Assessment, Evaluation and Accountability, 21(1), pp. 5–31 (2009).
9. Gamlem, S.M.: Feedback to support learning: changes in teachers’ practice and beliefs, Teacher Development, 19(4), pp. 461–482 (2015).
10. Wilson, J., Olinghouse, N.G., Andrada, G.N.: Does automated feedback improve writing quality?, Learning Disabilities: A Contemporary Journal, 12, pp. 93–118 (2014)
11. Hopfenbeck, T. N., Zhang, Z., Sun, S. Z., Robertson, P., & McGrane, J. A.: Challenges and opportunities for classroom-based formative assessment and AI: a perspective article [Perspective]. Frontiers in Education, 8. (2023).
12. Hegelheimer, V., Dursun, A., Li, Z.: Automated writing evaluation in language teaching: theory, development, and application, Computer Assisted Language Instruction Consortium, 33(1), pp. i–v (2016).
13. Hannigan, C., Alonzo, D., Oo, C.Z.: Student assessment literacy: indicators and domains from the literature, Assessment in Education: Principles, Policy & Practice, 29(4), pp. 482–504 (2022).
14. Engeness, I.: What teachers do: facilitating the writing process with feedback from EssayCritic and collaborating peers, Technology, Pedagogy and Education, 27(3), pp. 297–311 (2018).
15. Engeness, I., Mørch, A.: Developing writing skills in English using content-specific computer-generated feedback with EssayCritic, Nordic Journal of Digital Literacy, 10(02), pp. 118–135 (2016).
16. Brown, A.L.: Design experiments: theoretical and methodological challenges in creating complex interventions in classroom settings, Journal of the Learning Sciences, 2, pp. 141–178 (1992). brown-1992.pdf (
17. The Design-Based Research Collective: Design-based research: an emerging paradigm for educational inquiry, Educational Researcher, 32(1), pp. 5–8 (2003).
18. Ludvigsen, S., Rasmussen, I., Krange, I., Moen, A., Middleton, D.: Intersecting trajectories of participation; temporality and learning, in Learning Across Sites; New Tools, Infrastructures and Practices, pp. 105–121, Pergamon, (2010).
19. Jordan, B., Henderson, A.: Interaction analysis: foundations and practice. Journal of the Learning Sciences, 4, pp. 39–103 (1995).
20. Black, P., Wiliam, D.: Assessment and Learning (2 edition), SAGE Publications Ltd., (2012).
21. Vattøy, K.-D., Gamlem, S.M., Kobberstad, L.R., Rogne, W.M.: Students’ experiences of assessment and feedback engagement in digital contexts: a mixed-methods case study in upper secondary school, Education Inquiry, pp. 1-22 (2022).
22. Smith, C.D., Worsfold, K., Davies, L., Fisher, R., McPhail, R. Assessment literacy and student learning: the case for explicitly developing students ‘assessment literacy’, Assessment and Evaluation in Higher Education, 38(1), pp. 44–60 (2013).
23. Price, M., Rust, C., O’Donovan, B., Handley, K., Bryant, R. Assessment literacy: the foundation for improving student learning, ASKe, Oxford Centre for Staff and Learning Development. (2012).
24. Torkildsen, L.G., Erickson, G.: ‘If they’d written more…’—On students’ perceptions of assessment and assessment practices, Education Inquiry, 7(2), p. 27416 (2016).
25. Gamlem, S.M., Munthe, E.: Mapping the quality of feedback to support students’ learning in lower secondary classrooms, Cambridge Journal of Education, 44(1), pp. 75–92 (2014).
26. Graham, S.: Changing how writing is taught. Review of Educational Research, 43, pp. 277–303 (2019).
27. Graham, S.: Instructional feedback in writing, in The Cambridge Handbook of Instructional Feedback, pp. 145–168, Cambridge University Press, (2018).
28. Graham, S., McKeown, D., Kiuhara, S., Harris, K.: A meta-analysis of writing instruction for students in the elementary grades, Journal of Educational Psychology, 104(4), pp. 879–896 (2012).
29. Moltudal, S., Høydal, K., & Krumsvik, R. J. Glimpses Into Real-Life Introduction of Adaptive Learning Technology: A Mixed Methods Research Approach to Personalised Pupil Learning, Designs for Learning, 12(1), 13–28.
30. Moltudal, S. H., Krumsvik, R. J., & Høydal, K. L. Adaptive Learning Technology in Primary Education: Implications for Professional Teacher Knowledge and Classroom Management. Frontiers in Education.
31. Munthe, E., Erstad, O., Njå, M.B., Forsström, S., Gilje, Ø., Amdam, S., Moltudal, S., Hagen, S.B. (2022). Digitalisering i grunnopplæring; kunnskap, trender og framtidig forskningsbehov. Kunnskapssenter for utdanning: Universitetet i Stavanger.
32. Danielsen, A.G., Samdal, O., Hetland, J., Wold, B.: School-related social support and students’ perceived life satisfaction. The Journal of Educational Research, 102(4), pp. 303–318 (2009).
33. Vattøy, K.-D., Gamlem, S. Teacher-student interactions and feedback in English as a foreign language classroom. Cambridge Journal of Education, 50(3), pp. 371–389 (2020).
34. Ranalli, J., Link, S., Chukharev-Hudilainen, E.: Automated writing evaluation for formative assessment of second language writing: investigating the accuracy and usefulness of feedback as part of argument-based validation, Educational Psychology (Dorchester-on-Thames), 37(1), pp. 8–25 (2017).
35. Collins, A., Joseph, D., Bielaczyc, K.: Design research: theoretical and methodological issues, Journal of the Learning Sciences, 13, pp. 15–42 (2004).
36. Dasgupta, J.: Imparting hands-on industry 4.0 education at low cost using open-source tools and python eco-system, in New Paradigm of Industry 4.0: Internet of Things, Big Data & Cyber Physical Systems, pp. 37–47, Springer, (2019)
37. Qi, P., Zhang, Y., Zhang, Y., Bolton, J., Manning, C.D.: Stanza: a Python natural language processing toolkit for many human languages, arXiv preprint arXiv:2003.07082 (2020).
38. Mercer, N.: The guided construction of knowledge. Talk Amongst Teachers and Learners, Multilingual Matters Ltd., (1995)
39. Han, T., & Sari, E. An investigation on the use of automated feedback in Turkish EFL students’ writing classes. Computer assisted language learning, ahead-of-print(ahead-of-print), 1-24. (2022).
40. Link, S., Mehrzad, M., & Rahimi, M. (2022). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer assisted language learning, 35(4), 605-634.
41. Gao, J. Exploring the Feedback Quality of an Automated Writing Evaluation System Pigai. International journal of emerging technologies in learning. (2021).
42. Geng, J., & Razali, A. B. (2020). Tapping the Potential of Pigai Automated Writing Evaluation (AWE) Program to Give Feedback on EFL Writing. Universal Journal of Educational Research, 8, 8334-8343.

back to Table of Contents