Summer 2017

Table of contents  –   Authors index

Connecting Learning Design and Learning Analytics
(https://doi.org/10.55612/s-5002-033-001psi)
download

Learning Design (LD) and Learning Analytics (LA) are both domains of research and action that aim to improve learning effectiveness. The fact of sharing a common aim raises the potential of the synergies that may emerge between them.
Learning Design or, as some prefer, Design for Learning [1][2][3], is an emerging field of educational research and practice. Its practitioners are interested in understanding how the intuitive processes undertaken by teachers and trainers can be made visible, shared, exposed to scrutiny, and consequently made more effective and efficient. Mor and Craft [4] define learning design as “the creative and deliberate act of devising new practices, plans of activity, resources and tools aimed at achieving particular educational aims in a given context”. The emphasis on this activity as both “creative and deliberate” highlights the dual nature of design, and in particular learning design, as both a creative practice and a rigorous inquiry.
Arguably, most of the work in the field of LD has focused on the creative processes, on practices, tools and representations to support it, and on mechanisms for sharing its outputs between practitioners. Very little has been done in terms of the practices, tools and representations used for evaluating the effects of the designs. Several approaches emphasise top-down quality enhancement, which help designers to base their work on sound pedagogical principles. What is missing is the trajectory that would complete the feedback loop: the built-in evaluation of designs to see whether they achieved the expected outcomes.
Learning Analytics are defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” [5]. LA typically employ large datasets to provide real-time or retrospective insights about the effect and effectiveness of various elements and features of learning environments. Learning analytics are rooted in data science, artificial intelligence, and practices of recommender systems, online marketing and business intelligence. The tools and techniques developed in these domains make it possible to identify trends and patterns, and then benchmark individuals or groups against these trends. LA can help to identify at-risk learners and provide interventions, transform pedagogical approaches, and help students gain insight into their own learning.
Thus, taking into consideration the purpose, strengths and weak points of learning design and analytics, there is a natural and synergistic relationship that emerges between these two domains, which has led to a growing interest and some initial effort in bringing them together [6][7].

How Learning Design may help Learning Analytics? According to situational approaches, one of the prerequisites to obtain relevant outputs is not to isolate the analysis of educational data from the context in which it is embedded [8]. Following this perspective, some authors state that this tandem between LD and LA offers the opportunity to better understand student behaviour and provide pedagogical recommendations when deviations from the original pedagogical intention emerge [9][10][11][12][13], addressing one of the challenges posed by LA, i.e.: interpreting the resulting data against the original pedagogical intent and the local context, to evaluate the success of a particular learning activity [14]. This approach of linking LD and LA has been already applied to support learning e.g., using e-portfolios [15] and on-line simulators [16], monitoring in CSCL [17], at different abstraction levels e.g., connecting the analysis with the accomplishment of the curriculum objectives defined in a course [18]. Recently, some authors have suggested design patterns to mediate LA and LD [19][20][21].

How Learning Analytics may support Learning Design? Reciprocally, well-formulated learning analytics can be helpful to inform teachers on the success and outcomes of their learning designs [11][22][23]. Learning analytics can provide evidences of the impact of a design in one or several learning situations in aspects such as engagement patterns in the activities proposed by the learning design, learning paths followed by the students, time consumed to complete the activities, etc (e.g., [24][25]).  Examples of how LA has been used to better understand the design impact can be found in the context of MOOCs [26][27], where the authors found that learning design decisions made by teachers were related to learning behavior of students in blended and online environments [26]. Apart from analysing the impact of the design, LA can also support the regulation and redesign processes, by facilitating the identification of design elements that need to be revised before reuse [28].
In summary, LD offers LA a domain vocabulary, representing the elements of a learning system to which analytics can be applied. LA in turn, offers LD a higher degree of rigor by validating or refuting assumptions about the effects of various designs in diverse contexts. However, making these links operational and coherent is still an open challenge. Thus, this special issue brings together different approaches that illustrate the benefits and challenges of the alignment between learning design and analytics.

What does this special issue offer to the reader? This special issue solicited original contributions connecting learning design with learning analytics. Out of 25 submissions to the special issue, 1 position paper and 6 research papers were accepted for inclusion in the journal (28% acceptance rate). The topics covered by the papers go from models, frameworks and tools that benefit from the connection between LD and LA to critical perspectives and lessons learned, including practical opportunities, principles and challenges for synergies between LD and LA.
Several contributions to the special issue focused on models, frameworks, and tools for developing data-enriched learning design and/or design-aware learning analytics:

• The position paper of Inventado, P.S. & Scupelli, P. presents “An Online Learning Collaboratory to Address Multidisciplinary Learning Challenges at Scale” that proposes a framework to encourage collaboration among existing communities of stakeholders and bridge the gap between research and practice. The authors argue that existing research are currently varied and fragmented, but an online learning collaboratory may help to connect stakeholders and, as a result, optimize current research efforts through existing tools and methodologies. LA and LD are crucial in the framework as they guide research and inform design decisions to promote effective student learning.

• Reflective Analytics for Interactive e-books” by Mavrikis, M and Karkalas, S., presents an analytics platform that offers awareness regarding the use of educational e-books. Analytics can inform decisions on the redesign of e-books taking into account the learning design and data from their usage. The paper presents architectural decisions of the dashboard and the evaluation of a prototype.

• Alhadad, S. & Thompson, K, in “Understanding the mediating role of teacher inquiry when connecting learning analytics with design for learning” argue that a valuable connection between learning analytics and design for learning is only realised through the mediation of effective teacher inquiry processes. Their paper proposes a working model and studies educators inquiry processes in structured professional learning workshops.

• The authors Eradze, M., Rodríguez-Triana, M. J., & Laanpere, M., presented the paper titled “Semantically Annotated Lesson observation Data in Learning Analytics Datasets: a Reference Model”, in which they offer a theoretical proposal towards the creation of a tool to integrate lesson observations into MMLA. The tool is agnostic of the pedagogical approach and has been validated by a group of teachers-researchers. Targeted stakeholders are observers and those consuming the analytics, but the contribution also affects researchers and developers in order to create compliant solutions.

• Law, N., Li, L., Farias Herrera, F., Chan, A., Pong, T.C., in “A Pattern Language Based Learning Design Studio for an Analytics Informed Inter-Professional Design Community” describe an online tool and a pattern language to support learning design. The tool provides analytics of the designs to facilitate inter-professional collaboration.

Finally, two papers in the special issue provide critical perspectives derived from theoretical discussions and lessons derived from prior studies:

• The paper by Griffiths, D., “The Use of Models in Learning Design and Learning Analytics” provides critical perspective about learning design, learning analytics and their connection. After analysing their impact from various perspectives, including their influence on the management of educational processes, the paper proposes a set of design principles can help LD and LA to be implemented.

• Finally, Rienties, B., Nguyen, Q, Holmes, W., & Reedy, K. in “A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK” articulate the lessons learnt in eight studies conducted over ten years in their institution that use learning analytics to understand the used learning design approaches and their impact on student behaviour, satisfaction and performance. Four themes are identified for future research.

The collection of papers in this special issue shows that there are multiple and mutually-beneficial synergies between the LD and LA domains. They offer practical examples focused on particular application cases, such as eBooks, online learning systems, or teacher inquiry and provide frameworks that enable rich LA approaches aligned with LD. Moreover, the papers also contribute valuable and critical insights about the practical implementation of LA and LD as well as their resulting challenges. This special issue hopes to encourage more discussions and collaboration among stakeholders from both domains including practitioners, researchers, educational technology providers and policy makers; and that the opportunities and challenges triggered by the call for papers of this SI (as introduced in this Preface) and discussed in the selected papers can, in the near future, lead to additional research advancing the state of art around the connection between LD and LA.

Davinia Hernandez Leo, María Jesús Rodríguez-Triana, Paul Salvador Inventado, Yishay Mor

Acknowledgements

We would like to thank the multiple reviewers contributing to this special issues with their valuable help with selecting and enhancing the papers in the special issue.  This editorial work has been possible thanks to the support of the Spanish Ministry (TIN2014-53199-C3-3-R, MDM-2015-0502), RecerCaixa (CoT), National Science Foundation under DRL-1252297, and the European Union in the context of CEITER and Next-Lab (Horizon 2020 Research and Innovation Programme, grant agreements no. 669074 and 731685).

References

[1] Beetham, H., Sharpe, R.: Rethinking pedagogy for a digital age: Designing for 21st century learning, Routledge, (2013)
[2] Laurillard, D.: Rethinking university teaching: A conversational framework for the effective use of learning technologies, Routledge, (2013)
[3] Mor, Y., Craft, B., Hernández-Leo, D:. The art and science of learning design: Editorial. Research in Learning Technology, 21 (2013)
[4] Mor, Y., Craft, B.: Learning design: reflections upon the current landscape, Research in learning technology, 20 (2012)
[5] Ferguson, R.: Learning analytics: drivers, developments and challenges, International Journal of Technology Enhanced Learning, 4(5-6), pp. 304–317 (2012)
[6] Mor, Y., Ferguson, R., Wasson, B.: Editorial: Learning design, teacher inquiry into student learning and learning analytics: A call for action, British Journal of Educational Technology 46(2), pp. 221–229  (2015)
[7] Rodríguez-Triana, M.J., Hernández-Leo, D., Mor, Y., Inventado, P.S., Warburton, S., Rienties, B., Prieto, L.P., Scupelli, P.: Connecting Learning Analytics and Learning Design Workshop (CLAD 2016) collocated at  EC-TEL. URL: http://clad2016.ld-grid.org/ (2016)
[8] Crook, C.: Motivation and the Ecology of Collaborative Learning, in R. Joiner, K. Littleton, D. Faulkner, and D. Miell (Eds.), Rethinking Collaborative Learning, pp. 161–178. Free Association Books, (2000)
[9] Looney, J., Siemens, G.: Assessment competency: Knowing what you know and Learning Analytics. It is time for a breakthrough. Promethean Thinking Deeper Research Paper No. 3 (2011)
[10] Lockyer, L. Dawson, S: Learning designs and learning analytics. In: International Conference on Learning Analytics and Knowledge, pp. 153-156. ACM (2011)
[11] Lockyer, L., Heathcote, E. Dawson, S.: Informing pedagogical action: Aligning Learning Analytics with Learning Design, American Behavioral Scientist, 57(10), pp. 1439–1459  (2013)
[12] Griffiths, D.: The implications of Analytics for teaching practice in Higher Education, JISC CETIS Analytics Series, 1(10), pp. 1–23 (2013)
[13] Wise, A.F.: Designing pedagogical interventions to support student use of learning analytics. In: Proceedings of the 4th International Conference on Learning Analytics And Knowledge, pp. 203-–211. ACM (2014)
[14] Sutherland, R; Eagle, S., Joubert, M.: A vision and strategy for Technology Enhanced Learning, Report from the STELLAR Network of Excellence (2012)
[15] Reimann, P., Kickmeier-Rust, M., Meissl-Egghart, G., Utz, W.: Specification of ECAAD methodology v3, Technical report, NEXT-TELL Project (2013)
[16] Lejeune, A., Gueraud, V. Embedding observation means into the learning scenario: Authoring approach and environment for simulations-based learning. In: Proceedings of the 12th International Conference on Advanced Learning Technologies, pp. 273-275. IEEE (2012)
[17] Rodríguez-Triana, M. J.;Martínez-Monés, A.; Asensio-Pérez, J. I., Dimitriadis, Y.: Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations, British Journal of Educational Technologies, 46(2), pp. 330-–343 (2015)
[18] Gluga, R., Kay, K., Lister, R., Charleston, S.M., Harland, J., Teague, D.: A conceptual model for reflecting on expected learning vs. demonstrated student performance. In: Proceedings of the 15th Australasian Computing Education Conference. Australian Computer Society (2013)
[19] Inventado, P.S., Scupelli, P.: Towards an open, collaborative repository for online learning system design patterns, eLearning Papers, 42, pp. 14–27 (2015)
[20] Rodríguez-Triana, M., Martínez-Monés, A., Asensio-Pérez, J., Jorrín-Abellán, I., Dimitriadis, Y.: Monitoring pattern-based CSCL scripts: a case study, in Towards Ubiquitous Learning, LNCS, vol. 6964, pp. 313–326, Springer, Heidelberg, (2011)
[21] Michos, K., Manathunga, K., Hernández Leo, D.: Connecting pattern-based learning designs with analytics: The case of the PyramidApp. In: Proceedings of the Workshop on Connecting Learning Analytics and Learning Design (CLAD 2016), co-located with 11th European Conference on Technology Enhanced Learning (EC-TEL 2016), available online at https://repositori.upf.edu/handle/10230/32136. (2016)
[22] Melero, J., Hernández-Leo, D., Sun, J., Santos, P., Blat, J.: How was the activity? A visualization support for a case of location-based learning design, British Journal of Educational Technology, 46(2), pp. 317-329 (2015)
[23] Giovannella C., Carcone S., Camusi A.: What and how to monitor complex educative experiences. Toward the definition of a general framework, IxD&A Journal, N. 11-12, pp. 7–23 (2011)
[24] Giovannella C., Spadavecchia C., Camusi A.: Educational complexity: centrality of design and monitoring of the experience, in G. Leitner, M. Hitz, and A. Holzinger (Eds.): USAB 2010, LNCS, vol. 6389, pp. 353–372, Springer, Heidelberg, (2010)
[25] Wang, Y.E., Baker, R., Paquette, L.: Behavioral Predictors of MOOC Post-Course Development. In: Proceedings of the Workshop on Integrated Learning Analytics of MOOC Post-Course Development. (2017)
[26] Rienties, B., Toetenel, L. & Bryan, A.: ‘Scaling Up’ Learning Design: Impact of Learning Design Activities on LMS Behavior and Performance. In: Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, pp. 315–319. Australian Computer Society (2015)
[27] Toetenel, L., Rienties, B.: Analysing 157 Learning Designs using Learning Analytic approaches as a means to evaluate the impact of pedagogical decision-making, British Journal of Educational Technology, 47(5), pp. 981–992 (2016)
[28] Chacón, J., Hernández-Leo, D.: Learning design family tree to back reuse and cooperation, In: Proceedings of the 9th International Conference on Networked Learning, Edinburg, pp. 510–517, Lancaster University (2014)