Summer 2018

Table of contents  –   Authors index

Future Directions of UX Studies: Enriching Experience Evaluation through Diverse Perspectives
(https://doi.org/10.55612/s-5002-037-001fsi)
download

1. Introduction

This special issue is situated within the ongoing discussion about the applicability of different methodological approaches in user experience (UX) evaluation. Evaluation represents the core of user-centered design and thereby also of the process how to best design UX [33]. Nevertheless, there is still no common ground how to best study and evaluate UX.
As UX has evolved in the multidisciplinary intersection of fields such as cognitive science, design, psychology and engineering it is not surprising that UX as research are experienced influences from the various perspectives and disciplines on how to best gather, analyze, and present user feedback. The core of the resulting discussing about suitable methodological approaches is primarily based on the researchers’ differing foci on the analysis of what users do or say as well as what users feel or think, and on the complexity of an experience which demands new approaches for evaluation [12]. Thereby, the applicability of either quantitative methods or qualitative methods [23] is strongly discussed. As a recent literature study about UX evaluation[28] shows, researchers rely more and more on so called mixed-method approaches, i.e., the combination or triangulation of different methods, to derive reliable user insights by regarding UX from different angles to gather the full complexity of users’ experiences.
Hence, this special issue juxtaposes different case studies that apply and discuss mixed-method approaches in UX evaluation. The overall goal is to derive insights from experiences about how to combine and apply different methods for UX evaluation. The editors of this issue conduct research in the area of UX in different domains, ranging from automotive to cross-cultural Human-Computer Interaction (HCI) research and a common focus on UX methods. We would like to foster an ongoing discourse about the suitability of different UX methods in distinct contexts to eventually derive guidelines for UX method triangulation.

1.1 The History of Human-Computer Interaction Research

The field of HCI experienced several paradigmatic changes since its early days. In general, the development of the research field can be described as a process that was pushed forward in three waves [7]. The first waves represents the impact of cognitive scientists and ergonomics researchers around the 1980s. As a consequence of the diffusion of personal computers at that time, a more human perspective on product development has been established. Initial efforts mainly centered on the analysis of a pleasant usability [25]. Between the 1990s and 2000s, the second wave pushed the perspective of HCI researchers from the investigation of single users to the study of collaborative systems [6]. With the beginning of this century, HCI researchers and practitioners alike started to postulate a focus on experience-related product characteristics to serve the needs of a post-materialistic experience society [18]. The resulting third wave generally considers additional contextual factors as well as users’ experiences as part of the design process [12, 13]. Today, the question arises if we are currently experiencing a fourth wave (i.e., an interdisciplinary wave) with more and more disciplines entering the field of UX and the associated discussion about the applicability of methodological approaches [7]. Although it is arguable that the multidisciplinarity of current UX research leads to a fourth paradigmatic wave, it certainly further influences how UX is seen as a research field and what it means to properly evaluate a product’s UX.  In general, marketing and other business-related disciplines that take part in the discussion about UX often focus on numbers to quantify experiences and eventually derive reliable business decisions. Social sciences, ethnographic as well as design-focused disciplines, in contrast, ask for more qualitative insights and end-user involvement [31]. However, the field of UX still lacks the knowledge on how to combine the best different methods to counterbalance the downsides of single approaches despite the known benefit of evaluation approaches that are based on several methods [3].

1.2 User Experience Evaluation

“To understand the users’’ is widely seen as a substantial factor for good design [33]. Consequently, evaluation represents a major task of both academic and industrial UX research [2, 22].
To be able to evaluate the UX of a product researchers have first to know and understand what they have to evaluate. However, although scholars have developed a common understanding that experience-related product characteristics exceed traditional usability and functionality-focused product characteristics, there is still no consensus on a terminological definition. The word user experience is often used as an umbrella term for diverse experience-related product factors, such as hedonic quality [14], emotions [9], joy [15], pleasure [20] beside pure usability and functionality. Various definitions exist and some examples are given in Table 1. This lack of common ground and theory seems to be the core of current challenge in UX research [29].

Reference	Definition
McNamara & Kirakowski [24]	“the wider relationship between the product and the user in order to investigate the individual’s personal experience of using it’’
Hassenzahl [16]	“momentary, primarily evaluative feeling (good-bad) while interacting with a product or service”
ISO 2010 [1]	“a person’s perceptions and responses that result from the use or anticipated use of a product, system or service”
Norman & Nielsen [27]	“all aspects of the end-user’s interaction with the company, its services, and its products’’

Table 1: Selection of existing UX definitions

On this basis, there are different opinions on how best to rate experiences. UX evaluation, in general, ranges from the investigation of task-oriented user goals to psychological user needs. The focus on more task-related user goals emerges from a rather micro perspective that studies distinct product characteristics, such as visual elements. The consideration of psychological user needs, in contrast, is applicable for a macro perspective, e.g., to study a products general purpose [16, 26]. The major difference between the analysis of a product’s usability and the study of UX-related aspects such as need fulfillment and emotions [17] is that an experience is shaped already before, during, and after the actual use of a product. The creation of an experience generally evolves over time along the pattern of anticipation, orientation, incorporation, and identification [21]. As part of this cognitive process, diverse aspects need to be studied to “understand the users’’, ranging from learnability over social interaction to personal reflection [5, 10]. Thus, it is not surprising that different UX methods are currently being used for different use cases. In fact, until today, we do not see a joint opinion about how users’ experience should and can be best evaluated [2].

Several researchers aimed to shed light on the field of UX evaluation in the past and identified patterns, challenges, and differences in methodological UX evaluation approaches. The results and conclusions of the associated studies, however, report differing insights. Vermeeren et al. [32], e.g., present a study of 96 UX methods derived from literature and workshops. They mention that the majority of studies is based on quantitative methods although one third combines both qualitative and quantitative methods. In their study, methods for early design phases or anticipated use are rather rare. Bargas-Avila & Hornbæk [4] report similar results regarding methods for anticipated UX in their meta-review on UX methods. However, they differ in the notion about qualitative and quantitative methods. Although questionnaires represent the most frequently applied method for UX studies in their analysis, the majority of the analyzed evaluations is based on qualitative methods. Interestingly, most analyzed studies do not specify their understanding of UX but rather focus on the evaluation of “generic UX’’. At the turn of the century, when these meta-reviews were conducted, the discussion about mixed-method approaches or UX method triangulation began to rise. However, the benefits of rich insights due to comprehensive data collections usually faces the challenge of analyzing the vast amount of data that the application of multiple methods entails. More guidance was mentioned as a requirement to better understand how to combine different methods to effectively and efficiently plan, execute, and analyze mixed-method UX studies [32].
Nowadays, the landscape of UX evaluation has not changed drastically [28]. Most researchers still focus on “generic UX’’ without defining what they are investigating. This comes along with only a small share of papers, which refer to a UX definition of established UX theory. If authors specify UX, most study usability and pragmatic aspects of a product applying traditional methods like questionnaires and interviews. Specific UX evaluation methods developed by academic UX research in the last years^[1]are found to be only rarely re-used in academic papers.

1.3 Method Triangulation

The consideration of method triangulation emerged similarly to the advent of HCI during the 1980s. To balance diverse epistemological standpoints in social, behavioral and human sciences, triangulation served as mean to bridge differences in knowledge production approaches [19]. On the one hand, qualitative research was seen as a way to understand the subjective, on the other hand, quantitative research allowed the identification of statistical trends. According to Denzin [8], triangulation can be employed on different levels, i.e., data triangulation, investigator triangulation, theory triangulation, and method triangulation. Triangulation, in general, can help to generate holistic and more reliable study insights. In the UX context, method triangulation represents a vital part for the field to better grasp different layers of an experience, i.e., what users express as well as what users do, think, or feel [30]. In the field of HCI, particularly UX evaluation, the combination of different methods is widely adopted. However, the field does not yet benefit from distinct guidelines for different use cases or contexts but rather experiences the application of unspoken patterns. Until today, in fact, method triangulation in UX evaluation is often based on the combination of a few established methods, particularly combinations of two or more of the methods self-developed questionnaires, activity logging, semi-structured interviews, or observation [28]. To develop guidelines to enrich experience evaluation practice, practical examples from diverse perspectives for effective combinations of different methods i.e., triangulation strategies in UX evaluation need to be discussed.

2. This Special Issue

Through a single blinded peer-review process with at least two reviewers per paper, four papers were selected for publication. The common denominator of all papers is the focus on UX method application. The authors showcase the applicability of different methods diverse use cases and, as a whole, provide insights in current opportunities and challenges for UX method application. Submissions ranged from the application domain of automated driving research, mobile-self assessment, urban public spaces, ambient displays, UX in agile software development, service design, appropriation and application of specific methods in practice. In the following, we summarize the accepted articles and discuss their contribution to the field of future direction in UX studies:

The first article, Building Collaborative Test Practices: Design Ethnography and WOz in Autonomous Driving ResearchbyOsz et al.,argues the need for new methods in order to better understand the user experience in automated driving. The authors propose a mixed-method approach for investigating user experience of different levels of self-driving modes. They combine an experimental and ethnographic research in a “Wizard of Oz” (WoZ) setup to face criticism of the investigation of real-life behavior in an experimental setting and to get a deeper understanding of humans’ expectation on automation. While there is already existing work suggesting ethnographic methods for research regarding autonomous driving, the specific application and combination with WoZ is a new perspective in the field of an application area in which products can not be tested yet in the real-life context. Therefore, methodological approaches that are able to contribute with both quantitative and qualitative data for UX evaluation in early phases of product development are necessary.

The second accepted article,Exploring Intended and Unintended Uses of (e)Books as Design Inspiration for Ambient Displays in the Home by Moser et al., utilizes a mixed-method approach to evaluate the intended and unintended use of books to receive inspiration how to design ambient displays in the home. The aim of the research was to investigate users’ needs regarding printed books (e.g., decoration and expression of personality) to identify the value of eBooks as ambient displays. Therefore, they combined visits (probing) of the home of their participants and semi-structured interviews with a large-scale online survey. Their developed concept of an ambient display representing a digital bookshelf was afterwards evaluated in a long-term field study with three households. The presented approach faces thereby the challenge of designing the shift from traditional analogue to digital products with another meaning which might differ from users’ expectations and basic needs. Further on, this article shows a good example of UX evaluation in the whole process of product development, starting from the analysis of users’ needs and evaluation of existing products to design a valuable concept which can be tested to prove if users’ needs are met.

The third presented article, Adding UX in the Service Design Loop: The Case of Crisis Management Servicesby Touloum et al., refers to the growing trend that UX and service design are merging on the example of a crisis management service. However, how to include effective UX design in the service design lifecycle is still missing. Therefore, the authors propose a methodological framework that combines UX characterization and touchpoints analysis. In particular, the proposed framework connects workflow modeling, personas, UX attributes, and touchpoints in a user journey to guide the UX prototyping of coherent services. The validation proves the framework as a tool for early phases of the product development process to derive user needs and requirements but also as tool for UX evaluation of existing concepts. Bringing together different, however, synergistic perspectives, the paper contributes to the discussions how to enrich UX evaluation.

The last article in this focus section, Situated Evaluation as an Approach to Understand the Appropriation of a Communication Tool in Special Education by Borjesson et al., regards the challenge of UX evaluation that technology is not always used in the way designers intended it. So called “appropriation” shows that also design can change the environment, which impact on UX needs to be understood. Therefore, authors utilize a situated evaluation approach to understand the appropriation process of a tool used for communication between teachers, children and parents in the special education. They thereby triangulate in a field study observation, interviews and logging data. While this article does not introduce a new method, it reports how situated evaluation was conducted in a specific and also challenging project in the field. It is a useful guidance for projects which have to deal with similar challenges of UX evaluation.

3. Conclusions

The multi-disciplinary “fourth wave” of UX will continue to shape the field, with continuous introductions and developments of methods that probe into user experiences from various perspectives. The current trend towards increasing use of method triangulation may help to understand user experiences better, especially as users may experience and adapt technologies in ways that may be challenging to anticipate by designers or to detect in usability-style studies. The papers in this special issue address these challenges. The first paper showcase innovative WoZ studies of autonomous vehicle concepts, offering a way to tap into contexts well before the technology is ready to test with users. The second paper highlights an example of how to effectively combine survey data with qualitative methods, in order to understand current user experiences, also the unintended. The fourth paper studies the over-time the appropriation of technology by situated evaluations containing both qualitative and quantitative data. The third paper offers a methodological framework combining best practices from UX and Service Design, a framework which is very useful as the user centered design field is under transition from a strong focus on users and physical products to perspectives that may include services and focus on several stakeholders [11]. All papers thus provide useful method triangulation patterns useful for researchers and practitioners, certainly reaching beyond the specific fields of vehicle interaction design, education tools, (e)books, and crisis management.
The papers all pointed towards how to grapple with the important task of understanding and evaluating the every-day UX of designs under development, by effectively using methodological combinations. We look forward to further research in this vein.

by Anna-Katharina Frison, Florian Lachner, Andreas Riener, Ingrid Pettersson

Acknowledgement. We want to thank all authors for sharing their work in this special issue and all reviewers for their time and valuable feedback as part of the selection process for this special issue.

References

[1] ISO 9241-210:2010. Ergonomics of human system interaction-Part 210: Human-centred design for interactive systems.International Standardization Organization (ISO), 2010
[2] Alves R., Valente P., and Nunes N.j.: The state of user experience evaluation practice. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. ACM, 93–102 (2014)
[3] Arhippainen L., Pakanen M., and Hickey S.: Mixed UX Methods Can Help to Achieve Triumphs. In Proceedings of CHI 2013 Workshop ’Made for Sharing: HCI Stories for Transfer, Triumph and Tragedy’. ACM (2013)
[4] Bargas-Avila J. A. and Hornbæk K:. Old Wine in New Bottles or Novel Challenges: A Critical Analysis of Empirical Studies of User Experience. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’11. ACM, pp. 2689–-2698 (2011)
[5] Battarbee K.: Defining Co-experience. In: Proceedings of the 2003 International Conference on Designing Pleasurable Products and Interfaces. DPPI ’03. ACM, pp. 109–113 (2003)
[6] Bannon L. J.: From human factors to human actors: The role of psychology and human- computer interaction studies in system design. In: Readings in Human–Computer Interaction. Elsevier, pp. 205–214 (1995)
[7] Bødker S.: Third-wave HCI, 10 Years Later – Participation and Sharing. In: Interactions 22.5, pp. 24–31 (2015)
[8] Norman K. Denzin: Triangulation: A case for methodological evaluation and combination. Sociological methods pp. 339–-357 (1978)
[9] Desmet P. and Paul H., Framework of product experience, International journal of design 1.1, pp. 57–66 (2007)
[10] Forlizzi J. and Battarbee K.: Understanding experience in interactive systems, in Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques. ACM, pp. 261–268  (2004)
[11] Forlizzi J.: Moving beyond user-centered design. Interactions 25, 5, pp. 22-23 (2018)
[12] Giovannella C.: Is complexity tameable? Toward a design for the experience in a complex world. IxD&A, 15, pp. 18–30 (2012)
[13] Harrison S., Tatar D., Sengers P.: The three paradigms of HCI, in Alt. Chi. Session at the SIGCHI Conference on Human Factors in Computing Systems San Jose, California, USA.  pp. 1–18 (2007)
[14] Hassenzahl M., Platz A., Burmester M., Lehner K.: Hedonic and ergonomic quality aspects determine a software’s appeal, in Proceedings of the SIGCHI conference on Human Factors in Computing Systems, ACM, pp. 201–208 (2000)
[15] Hassenzahl M.: User experience (UX): Towards an experiential perspective on product quality, in Proc. IHM 2008, ACM Press, pp. 11–-15 (2008)
[16] Hassenzahl M.: Experiences Before Things: A Primer for the (Yet) Unconvinced. in CHI ’13 Extended Abstracts on Human Factors in Computing Systems. CHI EA ’13. ACM,  pp. 2059–-2068 (2013)
[17] Hassenzahl M., Andreas B., and Burmester M.: Engineering joy’ Ieee Software 18.1, pp. 70–76 (2001)
[18] Hassenzahl M., Diefenbach S., Göritz A.: Needs, affect, and interactive products–Facets of user experience, Interacting with computers 22, 5, pp. 353–-362 (2010),
[19] Johnson R. B., Onwuegbuzie A. J., Turner L. A.: Toward a definition of mixed methods research, Journal of mixed methods research 1, 2, pp. 112-–133 (2007)
[20] Jordan P.: Designing pleasurable products: An introduction to the new human factors. CRC (2002)
[21] Karapanos E., Zimmerman J., Forlizzi J., Martens J.B.: User Experience over Time: An Initial Framework, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’09. ACM, pp. 729–-738 (2009)
[22] Law E.  The measurability and predictability of user experience. In Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems. ACM (2011)
[23] Law E., van Schaik P., Roto V.: Attitudes towards user experience (UX) measurement, in International Journal of Human-Computer Studies 72.6  pp. 526–-541 (2014)
[24] McNamara N. and Kirakowski J.: Functionality, usability, and user experience: three areas of concern, interactions 13.6, pp. 26–28 (2006)
[25] Myers B. A.: A Brief History of Human-computer Interaction Technology. in interactions 5.2, pp. 44–-54 (1998)
[26] Norman D. A.: The Design of Everyday Things, Basic Books (2013)
[27] Norman D.A. and Nielsen J.: The Definition of User Experience (UX) retrieved on Dec. 2018 on  URL: https: //www.nngroup.com/articles/definition-user-experience/
[28] Pettersson I., Lachner F.,  Frison A.K., Riener A., Butz A.: A Bermuda Triangle? – A Review of Method Application and Triangulation in User Experience Evaluation, in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI ’18. ACM, 461, pp 1–16 (2018)
[29] Roto V.: Commentary: Actionable Constructs for Practitioners. Human-Computer Interaction, 33(2), 198–200 (2018)
[30] Sanders E.: Converging Perspectives: Product Development Research for the 1990s, Design Management Journal, 3.4, 49–54 (1992)
[31] Vaanannen-Vainio-Mattila K., Roto V., Hassenzahl M.: Towards practical user experience evaluation methods, in Meaningful measures: Valid useful user experience measurement (VUUM), pp. 19–-22 (2008)
[32] POS Vermeeren A., Law E., Roto V., Obrist M., Hoonhout J., Vaananen-Vainio-Mattila K.: User experience evaluation methods: current state and development needs, in Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries. ACM, pp. 521–-530 (2010)
[33] Wright P. and McCarthy J.: Empathy and Experience in HCI, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’08. ACM, pp. 637–- 646 (2008)