Keisuke Sato
pp. 233-257, download
(https://doi.org/10.55612/s-5002-069-011)
Abstract
Conversational AI systems increasingly employ role/persona prompting to elicit domain-specific responses, yet roles are typically assigned only to the AI side while the user remains a neutral questioner. This study empirically investigates how dialogue content transforms when professional roles are assigned not only to the AI but also to the user, and when both parties share the same role. We conducted controlled dialogues about an identical line-drawing stimulus under four role-assignment conditions (no-role, AI-only role, user-only role, and both roles) across five professional roles (detective, psychologist, artist, architect, natural scientist) and four contemporary LLMs (ChatGPT-4o, OpenAI o1, Claude 3.5 Sonnet, Gemini 1.5 Pro), yielding 64 sessions and 320 question–response pairs. Quantitative term-frequency analysis of the detective role—presented as a representative case with the clearest lexical shifts—revealed that the shared-role condition increased domain-specific terminology compared to baseline and single-role conditions, a pattern also observed in the other four roles. Qualitative close readings of the detective role suggested that shared roles were associated with stronger expertise-oriented reasoning, multiperspectivity, and characteristics resembling professional peer exchange. Notably, user-side role assignment alone altered model responses even without an explicit AI role instruction. Cross-model comparisons under the shared detective-role condition showed that each LLM expressed the same role frame with distinct stylistic tendencies, supporting a two-layer interaction model in which externally imposed role frames set the discourse direction while model-specific traits shape expression style. These findings offer practical implications for prompt engineering, educational dialogue design, and expert-support systems. The full dataset is publicly available on Zenodo (DOI: 10.5281/zenodo.14567800).
keywords: Prompt engineering, Role prompting, Dual role assignment, Human–AI dialogue, Large language models, Interaction design.
References
1. Zhao, W.X., Zhou, K., Li, J., et al.: A Survey of Large Language Models. arXiv:2303.18223 (2023). doi:10.48550/arXiv.2303.18223.
2. Schulhoff, S., Ilie, M., Balepur, N., et al.: The Prompt Report: A Systematic Survey of Prompt Engineering Techniques. arXiv:2406.06608 (2024). doi:10.48550/arXiv.2406.06608.
3. Sahoo, P., Singh, A.K., Saha, S., et al.: A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv:2402.07927 (2024). doi:10.48550/arXiv.2402.07927.
4. Bozkurt, A., Sharma, R.C.: Generative AI and Prompt Engineering: The Art of Whispering to Let the Genie Out of the Algorithmic World. Asian Journal of Distance Education, 18(2), i–vii (2023). doi:10.5281/zenodo.8174941.
5. Giray, L.: Prompt Engineering with ChatGPT: A Guide for Academic Writers. Annals of Biomedical Engineering, 51(12), 2629–2633 (2023). doi:10.1007/s10439-023-03272-4.
6. Kong, A., Zhao, S., Chen, H., et al.: Better Zero-Shot Reasoning with Role-Play Prompting. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4099–4113 (2024). doi:10.18653/v1/2024.naacl-long.228.
7. Zheng, M., Pei, J., Logeswaran, L., Lee, M., Jurgens, D.: When “A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models. In: Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 15126–15154 (2024). doi:10.18653/v1/2024.findings-emnlp.888.
8. Anglin, A.H., Kincaid, P.A., Short, J.C., Allen, D.G.: Role Theory Perspectives: Past, Present, and Future Applications of Role Theories in Management Research. Journal of Management, 48(6), 1469–1502 (2022). doi:10.1177/01492063221081442.
9. Entman, R.M.: Framing: Toward Clarification of a Fractured Paradigm. Journal of Communication, 43(4), 51–58 (1993). doi:10.1111/j.1460-2466.1993.tb01304.x.
10. Nistor, N., Trăușan-Matu, S., Dascălu, M., Duttweiler, H., Chiru, C., Baltes, B., Smeaton, G.: Finding student-centered open learning environments on the internet: Automated dialogue assessment in academic virtual communities of practice. Computers in Human Behavior, 47, 119–127 (2015). doi:10.1016/j.chb.2014.07.029.
11. Fernandes, U. da S., Barbosa, G.A.R., Chagas, B.A., Barbosa, G.D.J., Barbosa, S.D.J., Prates, R.O.: Lessons Learned from Modeling the Interaction with Conversational Agents. IxD&A, No. 55, 139–173 (2022). doi:10.55612/s-5002-055-007.
12. Scholten, M.R., Kelders, S.M., Van Gemert-Pijnen, J.E.W.C.: Similarity in Action with an Embodied Conversational Agent: Can Synchronous Speech Yield Higher Levels of Rapport? IxD&A, No. 56, 130–155 (2023). doi:10.55612/s-5002-056-007.
13. Yin, S., Fu, C., Zhao, S., et al.: A survey on multimodal large language models. National Science Review, 11(12), nwae403 (2024). doi:10.1093/nsr/nwae403.
14. Sato, K.: PROSPECT: Professional Role Effects on Specialized Perspective Enhancement in Conversational Task. Zenodo (2024). doi:10.5281/zenodo.14567800.
15. Chen, B., Zhang, Z., Langrené, N., Zhu, S.: Unleashing the potential of prompt engineering for large language models. Patterns, 6(6), 101260 (2025). doi:10.1016/j.patter.2025.101260.
16. Aslan, I., Neu, D., Neupert, D., Grafberger, S., Weise, N., Pfeil, P., Kuschewski, M.: How to Compliment a Human – Designing Affective and Well-being Promoting Conversational Things. IxD&A, No. 58, 157–184 (2023). doi:10.55612/s-5002-058-007.
17. Lin, J., Tomlin, N., Andreas, J., Eisner, J.: Decision-Oriented Dialogue for Human-AI Collaboration. Transactions of the Association for Computational Linguistics, 12, 892–911 (2024). doi:10.1162/tacl_a_00679.
18. Cain, W.: Prompting Change: Exploring Prompt Engineering in Large Language Model AI and Its Potential to Transform Education. TechTrends, 68, 47–57 (2024). doi:10.1007/s11528-023-00896-0.