The credibility of information generated by AI chatbots: an analysis of online discussions in Reddit
DOI:
https://doi.org/10.47989/ir31262998Keywords:
AI Chatbots, Credibility, Credibility Assessment, Online discussionAbstract
Introduction. The present investigation elaborates how participants of online discussion in Reddit assess the credibility of information generated by AI chatbots.
Method. The empirical findings draw on the descriptive quantitative analysis and qualitative content analysis of 1464 posts submitted by 919 individual participants to 30 Reddit discussion threads during the period of October 2023 – September 2025.
Analysis. The analysis focuses on the criteria by which the online participants assessed the credibility of information generated by AI chatbots. Second, it was examined how the credibility assessments were grounded by making use of various answer types.
Results. The most frequently used credibility criteria were correctness, trustworthiness and usefulness of information, as well as chatbot´s capability to generate relevant information. Accuracy, coverage, currency and verifiability of information were employed less frequently. Overall, the findings indicate that online participants adopted a critical stance on the credibility of information generated by AI chatbots. In particular, the occurrence of hallucinations undermined the beliefs that chatbots offer correct and trustworthy information. The credibility assessments were mainly grounded by drawing on personal opinions and experiences obtained from the use of chatbots. To a lesser extent, the assessments were also grounded by drawing on comparison and explanation.
Conclusion. Although the credibility of information generated chatbots is evaluated critically, people are not expecting that AI tools would offer fully credible answers.
References
Barbosa-Silva, J., Driusso, P., Ferreira, E.A., & de Abreu, R.M. (2024). Exploring the efficacy of artificial intelligence: a comprehensive analysis of CHAT-GPT's accuracy and completeness in addressing urinary incontinence queries. Neurourology and Urodynamics, 44(1), 153-164. https://doi.org/10.1002/nau.25603
Behesti, M., Toubal, I.E., Alaboud, K., Almalaysha, M., Ogundele, O.B., Turabieh, H., Abdalnabi, N., Boren, S.A., Scott, G.J., & Dahu, B.M. (2025). Evaluating the reliability of ChatGPT for health-related questions: a systematic review. Informatics, 12(9), 9. https://doi.org/10.3390/informatics12010009
Choi, W., Bak, H., & An, J. (2025). College students' credibility assessments of GenAI‐generated information for academic tasks: an interview study. Journal of the Association for Information Science & Technology, 76(6), 867-883. https://doi.org/10.1002/asi.24978
Citrin, J., & Stoker, L. (2018). Political trust in a cynical age. Annual Review of Political Science, 21(1), 49-70. https://doi.org/10.1146/annurev-polisci-050316-092550
Eldridge, A. (2025). Reddit. American social media forum website. Britannica Money. https://archive.org/details/httpswww.britannica.commoneyreddit (Internet Archive)
Grimes, M., von Krogh, G., Feuerriegel, S., Rink, F., & Gruber, M. (2023). From scarcity to abundance: scholars and scholarship in an age of generative artificial intelligence. Academy of Management Journal, 66(6), 1617-1624. https://doi.org/10.5465/amj.2023.4006
Hanss, K., Sarma, K.V., Glowinski, A.L., Krystal, A., Saunders, R., Halls, A., Gorrell, S., & Reilly, E. (2025). Assessing the accuracy and reliability of large language models in psychiatry using standardized multiple-choice questions: cross-sectional study. Journal of Medical Internet Research, 27, e69910. https://doi.org/10.2196/69910
Hilligoss, B. & Rieh, S.Y. (2008). Developing a unifying framework of credibility assessment: construct, heuristics, and interaction in context. Information Processing & Management, 44(4), 1467-1484. https://doi.org/10.1016/j.ipm.2007.10.001
Johnson, S.B., King, A.J., Warner, E.L., Aneja, S., Kann, B.H., & Bylund, C.L. (2023). Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. NCI Cancer Spectrum, 7(2), pkad015. https://doi.org/10.1093/jncics/pkad015
Kelly, A., Noctor, E., Ryan, L., & van de Ven, P. (2025). The effectiveness of a custom AI chatbot for type 2 diabetes mellitus health literacy: development and evaluation study. Journal of Medical Internet Research, 27, e70131. https://doi.org/doi:10.2196/70131
Kim, Y.B., Vilches, S.L., & Shapiro, S. (2025). Testing the capability of generative artificial intelligence for parent and caregiver information seeking. Family Relations, 74(3), 1266-1284. https://doi.org/10.1111/fare.13167
Kumar, N. (2025). Reddit statistics of 2025: users & revenue data. Demandsage. https://archive.org/details/httpswww.demandsage.comreddit-statistics (Internet Archive)
Lim, J.S., & Hong, N. (2025). Perceived search overload, generative AI credibility, and comparative usefulness: a channel complementarity approach to health information seeking. Health & New Media Research, 91, 124-145. https://doi.org/10.22720/hnmr.2025.00115
Lim, J.S., Shin, D., Lee, C., Kim, J., & Zhang, J. (2025). The role of user empowerment, AI hallucination, and privacy concerns in continued use and premium subscription intentions: an extended technology acceptance model for generative AI. Journal of Broadcasting & Electronic Media, 69(3), 183-199. https://doi.org/10.1080/08838151.2025.2487679
Lincoln, Y.S., & Guba, E. (1985). Naturalistic inquiry. Sage.
Liu, S., Hu, Y., Tian, Z., Jin, Z., Ruan, S., & Mao, J. (2024). Investigating users' search behavior and outcome with ChatGPT in learning-oriented search tasks. In SIGIR-AP 2024: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, Tokyo, December 9-12, 2024 (pp. 103-113). Association for Computing Machinery. https://doi.org/10.1145/3673791.3698406
Lo, L.S. (2023). The art and science of prompt engineering: a new literacy in the information age. Internet Reference Services Quarterly, 27(4), 203-210. https://doi.org/10.1080/10875301.2023.2227621
Lund, B.D., Mannuru, N.R., Katta, M., Hota, S.S.L.M., Pamukuntla, A., Uppala, S., Kola, S.M., & Mannuru, A. (2025). Bringing artificial intelligence (AI) into health information seeking behavior: a study of AI and information seeking research. Journal of Health Communication, 30(10-12), 330-335. https://doi.org/10.1080/10810730.2025.2533820
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C.D., & Ho, D.E. (2025). Hallucination-free? Assessing the reliability of leading AI legal research tools. Journal of Empirical Legal Studies, 22(2), 216-242. https://doi.org/10.1111/jels.12413
Mendel, T., Singh, N., Mann, D.M., Wiesenfeld, B., & Nov, O. (2025). Laypeople’s use of and attitudes toward large language models and search engines for health queries: survey study. Journal of Medical Internet Research, 27, e64290. https://doi.org/10.2196/64290
Metzger, M.J., Flanagin, A.J., Eyal, K., Lemus, D.R. & McCann, R.M. (2003). Credibility for the 21st century: integrating perspectives on source, message, and media credibility in the contemporary media environment. In P.J. Kalbfleisch (Ed.), Communication Yearbook, 27 (pp. 293-335). Lawrence Erlbaum Associates.
Miles, M.B. & Huberman, A.M. (1994). Qualitative data analysis: an expanded sourcebook (2nd ed.). Sage.
Murashko, N. (2025). Communicative AI, trust and the stories of war: an ethnographic exploration of user evaluation of ChatGPT 3.5’s responses on the Russian-Ukrainian war. In P. Åker & A. Kaun (Eds.), Gendering media: framing of AI, interacting with ChatGPT, and anti-fandom (pp. 43-70). Södertörns högskola, Sweden. https://archive.org/details/httpswww.diva-portal.orgsmashgetdiva21947890fulltext01.pdf-page43 (Internet Archive)
Ray, P.P. (2023). ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3, 121-154. https://doi.org/10.1016/j.iotcps.2023.04.003
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018). Saturation in qualitative research: exploring its conceptualization and operationalization. Quality and Quantity, 52(4), 1893-1907. https://doi.org/10.1007/s11135-017-0574-8
Savolainen, R. (2023). Assessing the credibility of information sources in times of uncertainty: online debate about Finland's NATO membership. Journal of Documentation, 79(7), 30-50. https://doi.org/10.1108/JD-08-2022-0172
Savolainen, R. (2025). Seeking and sharing information about the threat of nuclear war. Journal of Librarianship and Information Science, 57(1), 265-278. https://doi.org/10.1177/096100062312192
Schuetzler, R.M., Giboney, J.S., Wells, T.M., Richardson, B., Meservy, T., & Sutton, C. (2024). Student interaction with generative AI: an exploration of an emergent information-search process. In Proceedings of the 57th Hawaii International Conference on System Sciences (pp. 7500-7509). https://archive.org/details/httpsscholarspace.manoa.hawaii.eduserverapicorebitstreams4bf9154e-7067-4f7f-b60d-36946fd62dd9content (Internet Archive)
Sezgin, E., Jackson, D. I., Kocaballi, A. B., Bibart, M., Zupanec, S., Landier, W., Audino, A., Ranalli, M., & Skeens, M. (2025). Can large language models aid caregivers of pediatric cancer patients in information seeking? A cross-sectional investigation. Cancer Medicine, 14(1), e70554. https://doi.org/10.1002/cam4.70554
Shen, X., Chen, Z., Backes, M., & Zhang, Y. (2023). In ChatGPT we trust? Measuring and characterizing the reliability of ChatGPT. arXiv:2304.08979. https://arxiv.org/pdf/2304.08979
Sidhu, R.S., & Selvamogan, A. (2025). Assessing the quality and readability of AI chatbot responses to frequently asked questions about basal cell carcinoma. Clinical Surgical Oncology, 4(3), 1000094. https://doi.org/10.1016/j.cson.2025.100094
Tseng, S., & Fogg, B.J. (1999). Credibility and computing technology. Communications of the ACM, 42(5), 39-44. https://doi.org/10.1145/301353.301402
Wang, L., Li, J., Zhuang, B., Huang, S., Fang, M., Wang, C., Li, W., Zhang, M., & Gong, S. (2025). Accuracy of large language models when answering clinical research questions: systematic review and network meta-analysis. Journal of Medical Internet Research, 27, e64486. https://doi.org/10.2196/64486
Westbrook, L. (2015). Intimate partner violence online: expectations and agency in question and answer websites. Journal of the Association for Information Science and Technology, 66(3), 599-615. https://doi.org/10.1002/asi.23195
Yun, H.S., & Bickmore, T. (2025). Online health information-seeking in the era of large language models: cross-sectional web- based survey study. Journal of Medical Internet Research, 27, e68560. https://doi.org/10.2196/68560
Zhang, S., Li, J., Cagiltay, B., Kirkorian, H., Mutlu, B., & Fawaz, K. (2025). A qualitative exploration of parents and their children's uses and gratifications of ChatGPT. Family Relations, 74(3), 1056-1071. https://doi.org/10.1111/fare.13171
Zhong, T., & Fang, X. (2026). Understanding user trust in AI-generated content: an elaboration likelihood model perspective. Online Information Review, 50(1), 171-188. https://doi.org/10.1108/OIR-01-2025-0041
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Reijo Savolainen
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
