Summer 2015

Table of contents  –   Authors index

Technology-enhanced assessment: Agency change in the educational eco-system
(https://doi.org/10.55612/s-5002-025-001psi)

Today, learning often occurs collaboratively in learner networks, formal learning is combined with informal learning, and learners use, for example, personalized and personal learning environments adapted to their needs and preferences. While our learning environments have progressed with the help of technology, assessment practices are often reproducing traditional power relations and do not provide more control for the learner enabling self-directed learning. While in the literature often an ‘agency change’ in the educational eco-system is proposed, in practice this agency change is often not happening. While many e-assessment technologies are still rooted in an old testing paradigm triggered by the institution or the teacher, new approaches need to strife for an agency change towards the learners as the trigger of feedback and assessment processes need to enable self-regulated learning. This special issue of the Interaction Design and Architecture(s) Journal (IxD&A) focuses on innovative technologies with the potential to contribute to the agency change in the educational eco-system. The issue brings together contributions in TEL that deal with approaches and innovative assessment technologies that support the transition from current assessment scenarios towards the development of novel forms of e-assessment through which different types of knowledge and skills are evaluated, continuous feedback is provided, and students are more engaged in the learning process.
The special issue has attracted 10 submissions. These submissions have been evaluated by an international expert panel of 14 reviewers via a double-blind review. At least two reviews per paper have been conducted. As a result of this review process three papers have been accepted for the special issue.

In the paper by Gierl & Lai the authors present a method to automatically generate a large number of diverse test items along with their solution and rationales to support formative feedback. Today, assessment item authors spend a tremendous amount of effort to author high quality items. In particular in the domain of formative assessment hundreds or even thousands of items are necessary to guarantee a sufficient variety of items, which address also different levels of difficulty. Gierl and Lai point out that first results of the generation method are very promising. Nevertheless, future research should, for example, investigate how typical errors made by students can be used to generate distractors in MCQ items. The paper shows that a templates-based approach can automatize the creation of test items and simply represent a very innovation e-assessment technology to support items authors on the one hand and teachers on the other hand, because they can over a more continuous, even on-demand  formative assessment experience to students.

The second contribution by Luckner and Purgathofer proposes a system to implement formative and summative assessment with personal feedback in the context of large classes with several hundreds of students. An e-assessment system is proposed which is based on double-blind peer review. To lower the risk for students to learn a new topic and at the same time assess learning outcomes and produced works of others, a complex task is split into more simple tasks. After each task the e-assessment system requires that the students assess and review the work of three other colleagues. The system checks the status of the reviews and only if these preconditions are fulfilled, the student can proceed to the next respectively final task. The teaching staff only gets involved when students do not agree the received reviews, summative feedback or the grading. In addition, the system can show hidden tasks upon completion of the final task – the student can earn additional points. During a large-scale study with 350 students, more than 11.000 reviews were submitted. The results were used to derive design implication for the reviews, the review process, and feedback.  In addition, the authors conclude that the approach is sustainable enough to build and improve upon. With this e-assessment system the teachers are given a more supportive role and hence, the students receive more agency in the learning process through peer assessment and the grading of others work.

The paper by Adams and Clough introduces the concept of the assessment burger and focuses on enabling teachers to write effective multiple-choice items for formative purposes. The authors report about a co-design process under participation of teachers who represent end-users of the system. The system is based on the idea of threshold concepts that signal critical issues for understanding and addressing problems. These concepts are underused for the purpose of formative assessment today according to the authors. The system guides the teacher through a set of components to define the threshold concepts, to break it down in smaller chunks and to develop a set of diagnostic quiz questions. With a mixed-method evaluation methodology the authors show that the system supported teachers reflection upon and accurate identification of gaps in students understanding. The work presented by the authors opens new pathways for realizing agency-change not by giving the learner directly more control but to increase the understanding of reoccurring difficulties on the side of the teachers.

In a recently published book, the guest editors of this special issue have summarized the state of the art of e-assessment for learning (Ras, Whitelock, Kalz, 2015). They have derived six aspects of innovative e-assessment for learning approaches. Besides the overall goal of agency change, flexible timing of e-assessment, automation in the assessment lifecycle, adaptivity/adaptability of the e-assessment system to individuals, triangulation of data from different sources, continuity of assessment and feedback beyond a course, and the dialogue between teachers, learners, peers and system are essential to develop e-assessment systems of the future. According to Ras, Whitelock and Kalz (2015) “Technology can be an enabler to deliver data as a basis to facilitate self-, peer- or teacher-driven assessment and feedback or technology can create completely new assessment practices or allow the assessment of skills for which no assessment means exist before.” (Ras, Whitelock, Kalz)

Marco Kalz, Eric Ras, Denise Whitelock 

Ras, E., Whitelock, D., & Kalz, M. (2015). The promise and potential of e-assessment for learning. In P. Reimann, S. Bull, M. Kickmeier-Rust, R. Vatrapu & B. Wasson (Eds.), Measuring and visualising learning in the information-rich classroom: Routledge.