Mark Scholten, Saskia M. Kelders, Julia E. W. C. Van Gemert-Pijnen
pp. 130 – 155, download
(https://doi.org/10.55612/s-5002-056-007)
Abstract
Self-guided eHealth has the benefit of providing autonomy to patients. However, the autonomy comes with a cost; elevated attrition rates. Embodied Conversational Agents (‘robots on screen’), have technological capabilities to build rapport with eHealth users and to support them, but are costly to realize and their evidence is inconclusive. We investigated a novel and low-technological method to build rapport. eHealth users synchronized their speech with a monologue-style ECA, a method for which there exists evidence within the human-to-human communication domain. User experiences were investigated using predominantly qualitative methods. As our study results show, users are fairly positive about speaking synchronously with the ECA. However, the experimental task needs refinements. Users need to priorly hear, at least once, the pace of their artificial interlocutor in order to synchronize. Future studies can further examine the refined synchronous speech task and its potential for more widely accessible rapport-building ECA’s aiming for eHealth adherence.
Keywords: Human computer interaction, embodied, conversational agent, virtual human, chatbot, synchrony, eHealth, E-learning, dynamical systems, rapport
References
1. van Gemert-Pijnen L. (J. E.W.C.), Kip H., Kelders S.M., Sanderman R.: Introducing eHealth eHealth Research, Theory and Development. pp. 3-26. Routledge (2018) https://doi.org/10.4324/9781315385907-1
2. Barak A., Klein B., Proudfoot J.G.: Defining Internet-Supported Therapeutic Interventions Annals of Behavioral Medicine, 38, pp. 4-17 (2009) https://doi.org/10.1007/s12160-009-9130-7 PMid:19787305
3. Melville K.M., Casey L.M., Kavanagh D.J.: Dropout from Internet-based treatment for psychological disorders British Journal of Clinical Psychology, 49, pp. 455-471 (2010) https://doi.org/10.1348/014466509X472138 PMid:19799804
4. Buhrman M., Gordh T., Andersson G.: Internet interventions for chronic pain including headache: A systematic review Internet Interventions, 4, pp. 17-34 (2016) https://doi.org/10.1016/j.invent.2015.12.001 PMid:30135787 PMCid:PMC6096254
5. Scholten, M. R., Kelders, S. M., & Van Gemert-Pijnen, J. E. (2017). Self-guided web-based interventions: scoping review on user needs and the potential of embodied conversational agents to address them. Journal of medical Internet research, 19(11), e383. https://doi.org/10.2196/jmir.7351 PMid:29146567 PMCid:PMC5709656
6. Scholten, M. R., Kelders, S. M., & Van Gemert-Pijnen, J. E. (2019). An empirical study of a pedagogical agent as an adjunct to an eHealth self-management intervention: what modalities does it need to successfully support and motivate users?. Frontiers in psychology, 10, 1063. https://doi.org/10.3389/fpsyg.2019.01063 PMid:31143152 PMCid:PMC6520839
7. Picard R.W.: Affective computing: challenges International Journal of Human-Computer Studies, 59, pp. 55-64 (2003) https://doi.org/10.1016/S1071-5819(03)00052-1
8. Potdevin D., Clavel C., Sabouret N.: Virtual intimacy in human-embodied conversational agent interactions: the influence of multimodality on its perception Journal on Multimodal User Interfaces, 15, pp. 25-43 (2021) https://doi.org/10.1007/s12193-020-00337-9
9. Abdulrahman, A., & Richards, D. (2021). In Search of Embodied Conversational and Explainable Agents for Health Behaviour Change and Adherence. Multimodal Technologies and Interaction, 5(9), 56. https://doi.org/10.3390/mti5090056
10. DEHN D.M., van MULKEN S.: The impact of animated interface agents: a review of empirical research International Journal of Human-Computer Studies, 52, pp. 1-22 (2000) https://doi.org/10.1006/ijhc.1999.0325
11. Veletsianos G., Russell G.S.: Pedagogical Agents Handbook of Research on Educational Communications and Technology. pp. 759-769. Springer New York, New York, NY (2014) https://doi.org/10.1007/978-1-4614-3185-5_61
12. Baylor A.L., Ryu J.: The Effects of Image and Animation in Enhancing Pedagogical Agent Persona Journal of Educational Computing Research, 28, pp. 373-394 (2003) https://doi.org/10.2190/V0WQ-NWGN-JB54-FAT4
13. Shamekhi A., Czerwinski M., Mark G., Novotny M., Bennett G.A.: An Exploratory Study Toward the Preferred Conversational Style for Compatible Virtual Agents Presented at the (2016) https://doi.org/10.1007/978-3-319-47665-0_4
14. Byrne D.: An Overview (and Underview) of Research and Theory within the Attraction Paradigm Journal of Social and Personal Relationships, 14, pp. 417-431 (1997) https://doi.org/10.1177/0265407597143008
15. Tajfel H.: Social identity and intergroup behaviour Social Science Information, 13, pp. 65-93 (1974) https://doi.org/10.1177/053901847401300204
16. Turner J.C., Reynolds K.J.: Self-Categorization Theory Handbook of Theories of Social Psychology. pp. 399-417. SAGE Publications Ltd, 1 Oliver’s Yard, 55 City Road, London EC1Y 1SP United Kingdom (2011) https://doi.org/10.4135/9781446249222.n46
17. Heylen D., van Es I., Nijholt A., van Dijk B.: Controlling the Gaze of Conversational Agents Presented at the (2005) https://doi.org/10.1007/1-4020-3933-6_11
18. Louwerse M.M., Graesser A.C., McNamara D.S., Lu S.: Embodied conversational agents as conversational partners Applied Cognitive Psychology, 23, pp. 1244-1255 (2009) https://doi.org/10.1002/acp.1527
19. Abbe, A., & Brandon, S. E. (2014). Building and maintaining rapport in investigative interviews. Police practice and research, 15(3), 207-220. https://doi.org/10.1080/15614263.2013.827835
20. Tickle-Degnen L., Rosenthal R.: The Nature of Rapport and Its Nonverbal Correlates Psychological Inquiry, 1, pp. 285-293 (1990) https://doi.org/10.1207/s15327965pli0104_1
21. Tatar D.: Social and personal consequences of a preoccupied listener Unpublished doctoral dissertation, Department of Psychology, Stanford University, Stanford, CA, (1997)
22. Bernieri F.J.: Coordinated movement and rapport in teacher-student interactions Journal of Nonverbal Behavior, 12, pp. 120-138 (1988) https://doi.org/10.1007/BF00986930
23. Drolet A.L., Morris M.W.: Rapport in Conflict Resolution: Accounting for How Face-to-Face Contact Fosters Mutual Cooperation in Mixed-Motive Conflicts Journal of Experimental Social Psychology, 36, pp. 26-50 (2000) https://doi.org/10.1006/jesp.1999.1395
24. Tsui P., Schultz G.L.: Failure of rapport: Why psychotherapeutic engagement fails in the treatment of Asian clients. American Journal of Orthopsychiatry, 55, pp. 561-569 (1985) https://doi.org/10.1111/j.1939-0025.1985.tb02706.x PMid:4073227
25. Burns M.: Rapport and relationships: The basis of childcare. Journal of Child Care, 2, pp. 47-57 (1984)
26. Morbini F., DeVault D., Georgila K., Artstein R., Traum D., Morency L.-P.: A Demonstration of Dialogue Processing in SimSensei Kiosk Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL). pp. 254-256. Association for Computational Linguistics, Stroudsburg, PA, USA (2014) https://doi.org/10.3115/v1/W14-4334
27. Huang L., Morency L.-P., Gratch J.: Virtual Rapport 2.0 Presented at the (2011) https://doi.org/10.1007/978-3-642-23974-8_8
28. Kenny P., Hartholt A., Gratch J., Swartout W., Traum D., Marsella S., Piepol D.: Building interactive virtual humans for training environments Proceedings of i/itsec. vol. 174. pp. 911-916 (2007)
29. Gratch J., Kang S.-H., Wang N.: Using Social Agents to Explore Theories of Rapport and Emotional Resonance Social Emotions in Nature and Artifact. pp. 181-197. Oxford University Press (2013) https://doi.org/10.1093/acprof:oso/9780195387643.003.0012
30. Novick D., Gris I.: Building Rapport between Human and ECA: A Pilot Study Presented at the (2014) https://doi.org/10.1007/978-3-319-07230-2_45
31. Weiss B., Wechsung I., Kühnel C., Möller S.: Evaluating embodied conversational agents in multimodal interfaces Computational Cognitive Science, 1, pp. 6 (2015) https://doi.org/10.1186/s40469-015-0006-9
32. Bickmore T., Picard R.: Subtle expressivity by relational agents Proceedings of the CHI 2003 Workshop on Subtle Expressivity for Characters and Robots (2003)
33. Bickmore T., Schulman D., Yin L.: MAINTAINING ENGAGEMENT IN LONG-TERM INTERVENTIONS WITH RELATIONAL AGENTS Applied Artificial Intelligence, 24, pp. 648-666 (2010) https://doi.org/10.1080/08839514.2010.492259 PMid:21318052 PMCid:PMC3035950
34. ter Stal S., Kramer L.L., Tabak M., op den Akker H., Hermens H.: Design Features of Embodied Conversational Agents in eHealth: a Literature Review International Journal of Human-Computer Studies, 138, pp. 102409 (2020) https://doi.org/10.1016/j.ijhcs.2020.102409
35. Bickmore T., Schulman D., Yin L. (2010). Maintaining engagement in long-term interventions with relational agents. Appl. Artif. Intell. 24 648-666. 10.1080/08839514.2010.492259 [PMC free article] [PubMed] [CrossRef] [Google Scholar] https://doi.org/10.1080/08839514.2010.492259 PMid:21318052 PMCid:PMC3035950
36. Fitton, I. S., Finnegan, D. J., & Proulx, M. J. (2020). Immersive virtual environments and embodied agents for e-learning applications. PeerJ Computer Science, 6, e315. https://doi.org/10.7717/peerj-cs.315 PMid:33816966 PMCid:PMC7924662
37. Schroeder N.L., Gotch C.M.: Persisting Issues in Pedagogical Agent Research Journal of Educational Computing Research, 53, pp. 183-204 (2015) https://doi.org/10.1177/0735633115597625
38. Johnson W.L., Lester J.C.: Face-to-Face Interaction with Pedagogical Agents, Twenty Years Later International Journal of Artificial Intelligence in Education, 26, pp. 25-36 (2016) https://doi.org/10.1007/s40593-015-0065-9
39. Luger E., Sellen A.: “Like Having a Really Bad PA” Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. pp. 5286-5297. ACM, New York, NY, USA (2016) https://doi.org/10.1145/2858036.2858288
40. Huang H.-H., Masuda T., Cerekovic A., Tarasenko K., Pandzic I.S., Nakano Y., Nishida T.: Toward a Universal Platform for Integrating Embodied Conversational Agent Components Presented at the (2006) https://doi.org/10.1007/11893004_28
41. Takahashi T., Takeda H.: Proposal of a script language for embodied conversational agents as asynchronous conversational media Proceedings of the first international joint conference on Autonomous agents and multiagent systems part 3 – AAMAS ’02. p. 1387. ACM Press, New York, New York, USA (2002) https://doi.org/10.1145/545056.545136
42. Rojc M., Mlakar I., Kačič Z.: The TTS-driven affective embodied conversational agent EVA, based on a novel conversational-behavior generation algorithm Engineering Applications of Artificial Intelligence, 57, pp. 80-104 (2017) https://doi.org/10.1016/j.engappai.2016.10.006
43. Masche J., Le N.-T.: A review of technologies for conversational systems International conference on computer science, applied mathematics and applications. pp. 212-225. Springer (2017) https://doi.org/10.1007/978-3-319-61911-8_19
44. Allouch M., Azaria A., Azoulay R.: Conversational Agents: Goals, Technologies, Vision and Challenges Sensors, 21, pp. 8448 (2021) https://doi.org/10.3390/s21248448 PMid:34960538 PMCid:PMC8704682
45. McTear M.: Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots Synthesis Lectures on Human Language Technologies, 13, pp. 1-251 (2020) https://doi.org/10.2200/S01060ED1V01Y202010HLT048
46. Milne-Ives M., de Cock C., Lim E., Shehadeh M.H., de Pennington N., Mole G., Normando E., Meinert E.: The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review Journal of Medical Internet Research, 22, pp. e20346 (2020) https://doi.org/10.2196/20346 PMid:33090118 PMCid:PMC7644372
47. Schachner T., Keller R., v Wangenheim F.: Artificial Intelligence-Based Conversational Agents for Chronic Conditions: Systematic Literature Review Journal of Medical Internet Research, 22, pp. e20701 (2020) https://doi.org/10.2196/20701 PMid:32924957 PMCid:PMC7522733
48. Provoost S., Lau H.M., Ruwaard J., Riper H.: Embodied Conversational Agents in Clinical Psychology: A Scoping Review Journal of Medical Internet Research, 19, pp. e151 (2017) https://doi.org/10.2196/jmir.6553 PMid:28487267 PMCid:PMC5442350
49. Heidig, S., & Clarebout, G. (2011). Do pedagogical agents make a difference to student motivation and learning?. Educational Research Review, 6(1), 27-54. https://doi.org/10.1016/j.edurev.2010.07.004
50. Wiltermuth, S. S., & Heath, C. (2009). Synchrony and cooperation. Psychological science, 20(1), 1-5. https://doi.org/10.1111/j.1467-9280.2008.02253.x PMid:19152536
51. Marsh K.L., Johnston L., Richardson M.J., Schmidt R.C.: Toward a radically embodied, embedded social psychology European Journal of Social Psychology, 39, pp. 1217-1225 (2009) https://doi.org/10.1002/ejsp.666
52. Yzerbyt V., Estrada C., Corneille O., Seron E., Demoulin S.: Subjective Essentialism in Action The psychology of group perception: Perceived variability, entitativity, and essentialism, pp. 101-124 (2004)
53. Overy K., Molnar-Szakacs I.: Being Together in Time: Musical Experience and the Mirror Neuron System Music Perception, 26, pp. 489-504 (2009) https://doi.org/10.1525/mp.2009.26.5.489
54. Rennung M., Göritz A.S.: Prosocial Consequences of Interpersonal Synchrony Zeitschrift für Psychologie, 224, pp. 168-189 (2016) https://doi.org/10.1027/2151-2604/a000252 PMid:28105388 PMCid:PMC5137339
55. Mogan R., Fischer R., Bulbulia J.A.: To be in synchrony or not? A meta-analysis of synchrony’s effects on behavior, perception, cognition and affect Journal of Experimental Social Psychology, 72, pp. 13-20 (2017) https://doi.org/10.1016/j.jesp.2017.03.009
56. Butler E.A.: Temporal Interpersonal Emotion Systems Personality and Social Psychology Review, 15, pp. 367-393 (2011) https://doi.org/10.1177/1088868311411164 PMid:21693670
57. Miles L.K., Nind L.K., Macrae C.N.: The rhythm of rapport: Interpersonal synchrony and social perception Journal of Experimental Social Psychology, 45, pp. 585-589 (2009) https://doi.org/10.1016/j.jesp.2009.02.002
58. Wiggins S., Wiggins S., Golubitsky M.: Introduction to applied nonlinear dynamical systems and chaos, Springer, (2003)
59. de Jonge-Hoekstra L., van der Steen S., van Geert P., Cox R.F.A.: Asymmetric Dynamic Attunement of Speech and Gestures in the Construction of Children’s Understanding Frontiers in Psychology, 7, (2016) https://doi.org/10.3389/fpsyg.2016.00473 PMid:27065933 PMCid:PMC4814764
60. Vallacher R.R., van Geert P., Nowak A.: The Intrinsic Dynamics of Psychological Process Current Directions in Psychological Science, 24, pp. 58-64 (2015) https://doi.org/10.1177/0963721414551571
61. Valdesolo P., DeSteno D.: Synchrony and the social tuning of compassion. Emotion, 11, pp. 262-266 (2011) https://doi.org/10.1037/a0021302 PMid:21500895
62. Mörtl A., Lorenz T., Hirche S.: Rhythm Patterns Interaction – Synchronization Behavior for Human-Robot Joint Action PLoS ONE, 9, pp. e95195 (2014) https://doi.org/10.1371/journal.pone.0095195 PMid:24752212 PMCid:PMC3994045
63. Lehmann H., Saez-Pons J., Syrdal D.S., Dautenhahn K.: In Good Company? Perception of Movement Synchrony of a Non-Anthropomorphic Robot PLOS ONE, 10, pp. e0127747 (2015) https://doi.org/10.1371/journal.pone.0127747 PMid:26001025 PMCid:PMC4441426
64. Cummins F.: Practice and performance in speech produced synchronously Journal of Phonetics, 31, pp. 139-148 (2003) https://doi.org/10.1016/S0095-4470(02)00082-7
65. Cummins F.: Measuring synchronization among speakers reading together Presented at the November 1 (2019) https://doi.org/10.36505/ExLing-2006/01/0020/000020
66. Cummins F.: Periodic and Aperiodic Synchronization in Skilled Action Frontiers in Human Neuroscience, 5, (2011) https://doi.org/10.3389/fnhum.2011.00170 PMid:22232583 PMCid:PMC3248675
67. Reddish P., Fischer R., Bulbulia J.: Let’s Dance Together: Synchrony, Shared Intentionality and Cooperation PLoS ONE, 8, pp. e71182 (2013) https://doi.org/10.1371/journal.pone.0071182 PMid:23951106 PMCid:PMC3737148
68. Delaherche E., Chetouani M., Mahdhaoui A., Saint-Georges C., Viaux S., Cohen D.: Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines IEEE Transactions on Affective Computing, 3, pp. 349-365 (2012) https://doi.org/10.1109/T-AFFC.2012.12
69. Harmon-Jones C.K.: Does musical behavior promote affiliation?, Texas A&M University, (2011)
70. Whittemore, R., Jaser, S. S., Faulkner, M. S., Murphy, K., Delamater, A., Grey, M., & TEENCOPE Research Group. (2013). Type 1 diabetes eHealth psychoeducation: youth recruitment, participation, and satisfaction. Journal of medical Internet research, 15(1), e2170. https://doi.org/10.2196/jmir.2170 PMid:23360729 PMCid:PMC3636279
71. Lally, R. M., Kupzyk, K. A., Bellavia, G., Hydeman, J., Gallo, S., Helgeson, V. S., … & Brown, J. K. (2020). Caring Guidance™ after breast cancer diagnosis eHealth psychoeducational intervention to reduce early post-diagnosis distress. Supportive Care in Cancer, 28, 2163-2174. https://doi.org/10.1007/s00520-019-05028-0 PMid:31414245 PMCid:PMC7083810
72. Hayes S.C., Strosahl K.D., Wilson K.G.: Acceptance and commitment therapy: The process and practice of mindful change, Guilford press, (2011)
73. Renner F., Schwarz P., Peters M.L., Huibers M.J.H.: Effects of a best-possible-self mental imagery exercise on mood and dysfunctional attitudes Psychiatry Research, 215, pp. 105-110 (2014) https://doi.org/10.1016/j.psychres.2013.10.033 PMid:24252218
74. Scholten, M. R., Kelders, S. M., & Van Gemert-Pijnen, J. E. (2021). Can a monologue-style ECA more effectively motivate eHealth users in initial distress than textual guidance?. Heliyon, 7(3), e06509. https://doi.org/10.1016/j.heliyon.2021.e06509 PMid:33842700 PMCid:PMC8020434
75. Cerekovic A., Aran O., Gatica-Perez D.: Rapport with Virtual Agents: What Do Human Social Cues and Personality Explain? IEEE Transactions on Affective Computing, 8, pp. 382-395 (2017) https://doi.org/10.1109/TAFFC.2016.2545650
76. Wechsung, I., Weiss, B., Kühnel, C., Ehrenbrink, P., & Möller, S. (2013, August). Development and validation of the conversational agents scale (CAS). In INTERSPEECH (pp. 1106-1110). https://doi.org/10.21437/Interspeech.2013-298
77. Schnall, R., Cho, H., & Liu, J. (2018). Health Information Technology Usability Evaluation Scale (Health-ITUES) for usability assessment of mobile health technology: validation study. JMIR mHealth and uHealth, 6(1), e8851. https://doi.org/10.2196/mhealth.8851 PMid:29305343 PMCid:PMC5775483
78. Birk, M. V., Atkins, C., Bowey, J. T., & Mandryk, R. L. (2016, May). Fostering intrinsic motivation through avatar identification in digital games. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 2982-2995). https://doi.org/10.1145/2858036.2858062
79. Fujiwara, K., Kimura, M., & Daibo, I. (2020). Rhythmic features of movement synchrony for bonding individuals in dyadic interaction. Journal of Nonverbal Behavior, 44(1), 173-193. https://doi.org/10.1007/s10919-019-00315-0
80. Andrusyna, T. P., Tang, T. Z., DeRubeis, R. J., & Luborsky, L. (2001). The factor structure of the Working Alliance Inventory in cognitive-behavioral therapy. The Journal of psychotherapy practice and research, 10(3), 173.
81. Vaismoradi, M., Turunen, H., & Bondas, T. (2013). Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study. Nursing & health sciences, 15(3), 398-405. https://doi.org/10.1111/nhs.12048 PMid:23480423
82. Braun V., Clarke V.: Using thematic analysis in psychology Qualitative Research in Psychology, 3, pp. 77-101 (2006) https://doi.org/10.1191/1478088706qp063oa
83. Konvalinka I., Vuust P., Roepstorff A., Frith C.D.: Follow you, Follow me: Continuous Mutual Prediction and Adaptation in Joint Tapping Quarterly Journal of Experimental Psychology, 63, pp. 2220-2230 (2010) https://doi.org/10.1080/17470218.2010.497843 PMid:20694920
84. Klein G., Moon B., Hoffman R.R.: Making Sense of Sensemaking 1: Alternative Perspectives IEEE Intelligent Systems, 21, pp. 70-73 (2006) https://doi.org/10.1109/MIS.2006.75
85. Xie B., Zhou J., Wang H.: How Influential Are Mental Models on Interaction Performance? Exploring the Gap between Users’ and Designers’ Mental Models through a New Quantitative Method Advances in Human-Computer Interaction, 2017, pp. 1-14 (2017) https://doi.org/10.1155/2017/3683546
86. Ziefle M., Bay S.: Mental Models of a Cellular Phone Menu. Comparing Older and Younger Novice Users Presented at the (2004) https://doi.org/10.1007/978-3-540-28637-0_3
87. Prinz W.: A Common Coding Approach to Perception and Action Relationships Between Perception and Action. pp. 167-201. Springer Berlin Heidelberg, Berlin, Heidelberg (1990) https://doi.org/10.1007/978-3-642-75348-0_7
88. Bickmore, T., & Cassell, J. (2005). Social dialogue with embodied conversational agents. Advances in natural multimodal dialogue systems, 23-54. https://doi.org/10.1007/1-4020-3933-6_2