Understanding user experience in generative AI application: Evidence from topic modelling and sentiment analysis of user reviews
DOI:
https://doi.org/10.47989/ir31iConf64160Keywords:
Generative AI, User experience, Topic modelling, Sentiment analysis, Five elements of user experienceAbstract
Introduction. With the rapid surge of users in generative AI (GenAI) applications, user experience research is moving beyond surveys and experiments toward large-scale analyses of online reviews. Integrating topic and sentiment patterns from these reviews with user experience theory enables a clearer view of users’ concerns and experience gaps.
Method. This study analysed Chinese user reviews of DeepSeek using BERTopic for topic modelling combined with sentiment analysis and mapped the findings onto Jesse James Garrett’s five elements of user experience to reveal layered perceptions.
Results. Thirty-four topics were identified and grouped into nine categories covering technical stability, device adaptation, AI interaction and functional features. Negative sentiments centred on server instability, system errors and functional deficiencies, whereas positive sentiments highlighted AI performance, emotional support, and distinctive functions. The five-element mapping revealed strategic instability, scope plane functional gaps, structural interaction barriers and framework plane issues with multi-device adaptation.
Conclusion(s). Joint topic–sentiment analysis not only uncovers core concerns of GenAI application users but also offers actionable insights for improving technical stability, device adaptation and interaction design, providing a new empirical view for optimising similar applications and advancing user experience research.
References
Ahmed, A., Aziz, S., Khalifa, M., Shah, U., Hassan, A., Abd-Alrazaq, A., & Househ, M. (2022). Thematic analysis on user reviews for depression and anxiety chatbot apps: Machine learning approach. JMIR Formative Research, 6(3), e27654. https://doi.org/10.2196/27654
Alabduljabbar, R. (2024). User-centric AI: Evaluating the usability of generative AI applications through user reviews on app stores. PeerJ Computer Science, 10, e2421. https://doi.org/10.7717/peerj-cs.2421
Alhejji, S., Albesher, A., Wahsheh, H., & Albarrak, A. (2022). Evaluating and comparing the usability of mobile banking applications in Saudi Arabia. Information, 13(12), 559. https://doi.org/10.3390/info13120559
Alshammare, H., Alshayeb, M., & Baslyman, M. (2025). Revealing the mobile UX horizon: Exploring user experience aspects, attributes, and measurement methods-A systematic mapping study. Computer Standards & Interfaces, 94, 103999. https://doi.org/10.1016/j.csi.2025.103999
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human–AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–13). Association for Computing Machinery. https://doi.org/10.1145/3290605.3300233
Baj-Rogowska, A., & Sikorski, M. (2023). Exploring the usability and user experience of social media apps through a text mining approach. Engineering Management in Production and Services, 15(1), 86-105. https://doi.org/10.2478/emj-2023-0007
Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good. Review of General Psychology, 5(4), 323–370. https://doi.org/10.1037/1089-2680.5.4.323
Bubaš, G., Čižmešija, A., & Kovačić, A. (2024). Development of an assessment scale for measurement of usability and user experience characteristics of Bing Chat conversational AI. Future Internet, 16(1), 4. https://doi.org/10.3390/fi16010004
Castro, A., Pinto, J., Reino, L., Pipek, P., & Capinha, C. (2024). Large language models overcome the challenges of unstructured text data in ecology. Ecological Informatics, 82, 102742. https://doi.org/10.1016/j.ecoinf.2024.102742
Chen, X., Gao, C., Chen, C., Zhang, G., & Liu, Y. (2025). An empirical study on challenges for LLM application developers. ACM Transactions on Software Engineering and Methodology, 34(7), 205. https://doi.org/10.1145/3715007
Choung, H., David, P., & Ross, A. (2023). Trust in AI and Its Role in the Acceptance of AI Technologies. International Journal of Human–Computer Interaction, 39(9), 1727–1739. https://doi.org/10.1080/10447318.2022.2050543
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
Garrett, J. J. (2010). The elements of user experience: User-centered design for the Web and beyond (2nd ed.). New Riders Publishing.
Golding, J. M., Lippert, A., Neuschatz, J. S., Salomon, I., & Burke, K. (2024). Generative AI and college students: Use and perceptions. Teaching of Psychology, 52(3), 369–380. https://doi.org/10.1177/00986283241280350
Grootendorst, M. (2022). BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv. https://doi.org/10.48550/arXiv.2203.05794
Gu, Z., Zhu, Q., He, H., Lan, T., & Yu, Z. (2024). Analysis of emotional cognitive capacity of artificial intelligence. In W. Shen, J. Barthes, J. Luo, T. Qiu, X. Zhou, J. Zhang, H. Zhu, K. Peng, T. Xu, & N. Chen (Eds.), Proceedings of the 2024 27th International Conference on Computer Supported Cooperative Work in Design (CSCWD) (pp. 3182–3187). IEEE. https://doi.org/10.1109/CSCWD61410.2024.10580073
Hadwan, M., Al-Sarem, M., Saeed, F., & Al-Hagery, M. A. (2022). An Improved Sentiment Classification Approach for Measuring User Satisfaction toward Governmental Services’ Mobile Apps Using Machine Learning Methods with Feature Engineering and SMOTE Technique. Applied Sciences, 12(11), 5547. https://doi.org/10.3390/app12115547
Kim, J., Klopfer, M., Grohs, J. R., Eldardiry, H., Weichert, J., Cox, L. A., & Pike, D. (2025). Examining faculty and student perceptions of generative AI in university courses. Innovative Higher Education, 50, 1281–1313. https://doi.org/10.1007/s10755-024-09774-w
Li, B., Jiang, G., Li, N., & Song, C. (2024). Research on large-scale structured and unstructured data processing based on large language model. In Proceedings of the International Conference on Machine Learning, Pattern Recognition and Automation Engineering (MLPRAE '24) (pp. 111–116). Association for Computing Machinery. https://doi.org/10.1145/3696687.3696707
Lu, G., Qu, S., & Chen, Y. (2025). Understanding user experience for mobile applications: A systematic literature review. Discover Applied Sciences, 7, 587. https://doi.org/10.1007/s42452-025-07170-3
Nahar, N., Kästner, C., Butler, J., Parnin, C., Zimmermann, T., & Bird, C. (2024). Beyond the comfort zone: Emerging solutions to overcome challenges in integrating LLMs into software products. arXiv. https://doi.org/10.48550/arXiv.2410.12071
Ossai, C. I., & Wickramasinghe, N. (2023). Sentiments prediction and thematic analysis for diabetes mobile apps using Embedded Deep Neural Networks and Latent Dirichlet Allocation. Artificial Intelligence in Medicine, 138, 102509. https://doi.org/10.1016/j.artmed.2023.102509
Law, E. L.-C., Roto, V., Hassenzahl, M., Vermeeren, A. P., & Kort, J. (2009). Understanding, scoping, and defining user experience: A survey approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 719–728). https://doi.org/10.1145/1518701.1518813
Schoenmueller, V., Netzer, O., & Stahl, F. (2020). The polarity of online reviews: Prevalence, drivers, and implications. Journal of Marketing Research, 57(5), 853–877. https://doi.org/10.1177/0022243720941832
Shao, Y., Huang, Y., Shen, J., Ma, L., Su, T., & Wan, C. (2025). Are LLMs correctly integrated into software systems? In Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) (pp. 1178–1190). IEEE. https://doi.org/10.1109/ICSE55347.2025.00204
Shata, A., & Hartley, K. (2025). Artificial intelligence and communication technologies in academia: Faculty perceptions and the adoption of generative AI. International Journal of Educational Technology in Higher Education, 22, 14. https://doi.org/10.1186/s41239-025-00511-7
Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023). In ChatGPT We Trust? Measuring and Characterising the Reliability of ChatGPT. arXiv. https://doi.org/10.48550/ARXIV.2304.08979
Thorne, S. (2024). Understanding the interplay between trust, reliability, and human factors in the age of generative AI. International Journal of Simulation: Systems, Science and Technology, 25(1), 10. https://doi.org/10.5013/IJSSST.a.25.01.10
Wang, J., & Liu, Y. (2023). Deep learning-based social media mining for user experience analysis: A case study of smart home products. Technology in Society, 73, 102220. https://doi.org/10.1016/j.techsoc.2023.102220
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024). Design principles for generative AI applications. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24). Association for Computing Machinery, 378, 1–22. https://doi.org/10.1145/3613904.3642466
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Chenxin Zhou, Lihong Zhou
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
