A transdisciplinary AI-based framework for usability evaluation in digital health systems
DOI:
https://doi.org/10.47989/ir31354701Keywords:
Usability Evaluation, Transdisciplinary AI, Digital Health Systems, Artificial Intelligence, Human-Computer Interaction, User ExperienceAbstract
Introduction. As healthcare systems undergo rapid digital transformation, the usability of clinical and patient-facing technologies has become critical to adoption, user satisfaction, and patient safety. Traditional usability evaluation methods are limited in capturing the real-time cognitive, emotional, and behavioural complexities of interaction in high-stakes environments.
Method. This study presents a transdisciplinary evaluation of an AI-augmented usability framework for healthcare interfaces. The framework integrates methodologies from artificial intelligence, cognitive science, usability engineering, and clinical practice, incorporating eye-tracking analytics, machine learning-based confusion prediction, and natural language processing (NLP) sentiment analysis. Forty participants, including healthcare professionals and patients, completed standardized tasks using either traditional or AI-enhanced evaluation methods.
Analysis. Comparative performance across both groups was assessed using usability metrics including System Usability Scale (SUS) scores, task completion time, error rate, and task success rate. AI-based analytics—eye-tracking data, NLP sentiment scoring, and confusion prediction models—were used to generate additional insights. Statistical analysis confirmed significant differences in usability outcomes between groups.
Results. The AI-enhanced evaluation significantly outperformed traditional methods in all core usability metrics. The machine learning model achieved 89% accuracy and an F1-score of 0.88. Sentiment analysis correlated strongly with self-reported satisfaction, and eye-tracking highlighted areas of cognitive friction.
Conclusion. Transdisciplinary collaboration produced a novel, adaptive usability evaluation framework that offers real-time, emotionally aware insights. This replicable model enhances user-centred design and supports safer, more effective digital health technologies.
References
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014). Big data in health care: Using analytics to identify and manage high-risk and high-cost patients. Health Affairs, 33(7), 1123–1131. https://doi.org/10.1377/hlthaff.2014.0041
Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python: Analyzing text with the natural language toolkit. O’Reilly Media.
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189–194). London: Taylor & Francis.
Dervin, B. (1983). An overview of sense-making research: concepts, methods and results to date Presented at International Communication Association annual meeting, Dallas, May 1983 https://www.ideals.illinois.edu/items/2438/files/Dervin83a.htm
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 (pp. 4171–4186). https://doi.org/10.48550/arXiv.1810.04805
Duchowski, A. T. (2007). Eye tracking methodology: Theory and practice (2nd ed.). Springer.
Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). Sage Publications.
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford University Press.
Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B. (2017). What do we need to build explainable AI systems for the medical domain? arXiv 1712.00923, https://doi.org/10.48550/arXiv.1712.09923
Hunter, J. D. (2007). Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3), 90–95. https://doi.org/10.1109/MCSE.2007.55
Kocaballi, A. B., Laranjo, L., & Coiera, E. (2020). The influence of conversational agents on the quality of health information: Mixed-methods study. Journal of Medical Internet Research, 22(8), e16984. https://doi.org/10.2196/16984
Kushniruk, A. W., & Borycki, E. M. (2006). Low-cost rapid usability engineering: Designing and customizing usable healthcare information systems. Healthcare Quarterly, 9(4), 98–102. https://doi.org/10.12927/hcq.2006.18261
McKinney, W. (2010). Data structures for statistical computing in Python. In Proceedings of the 9th Python in Science Conference (pp. 51–56). https://doi.org/10.25080/Majora-92bf1922-00a
Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1–2), 1–135. https://doi.org/10.1561/1500000011
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830. https://jmlr.org/papers/volume12/pedregosa11a/pedregosa11a.pdf
Rajagopal, D., Pingali, P., & Varma, V. (2018). Analyzing mouse movement to predict user confusion in web interfaces. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval (CHIIR) (pp. 31–39). https://doi.org/10.1145/3176349.3176883
Ratwani, R. M., Reider, J., & Singh, H. (2018). A decade of health information technology usability challenges and the path forward. JAMA, 320(8), 755–756. https://doi.org/10.1001/jama.2018.8572
Shortliffe, E. H., & Sepúlveda, M. J. (2018). Clinical decision support in the era of artificial intelligence. JAMA, 320(21), 2199–2200. https://doi.org/10.1001/jama.2018.17163
Sittig, D. F., & Singh, H. (2010). A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Quality & Safety in Health Care, 19(Suppl 3), i68–i74. https://doi.org/10.1136/qshc.2010.042085
Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann.
Tullis, T., & Albert, B. (2013). Measuring the user experience: Collecting, analyzing, and presenting usability metrics (2nd ed.). Morgan Kaufmann.
Topol, E. (2019). Deep medicine: How artificial intelligence can make healthcare human again. Basic Books.
Wilson, T. (1999). Models in information seeking behaviour. Journal of Documention 55 (3) pp.249-270 https://doi.org/10.1108/EUM0000000007145
Zahabi, M., Kaber, D.B., & Swangnetr, M. (2015). Usability and safety in electronic medical records interface design: a review of recent literature and guideline formation. Human factors: the journal of the Human Factors and Ergonomics Society, 57 (5). https://doi.org/10.1177/0018720815576827
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Akshay Madhav Deshmukh, Ricardo Chalmeta
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
