Verification of AI-generated content

Ines Hocenski; Tomislav Jakopec; Josipa Selthofer

doi:10.47989/ir31261629

Authors

Ines Hocenski University J.J.Strossmayer Osijek (Croatia), Faculty of Humanities and Social Sciences, Department of Information Sciences
Tomislav Jakopec University J.J.Strossmayer Osijek, Faculty of Humanities and Social Sciences, Department of Information Sciences
Josipa Selthofer University J.J.Strossmayer Osijek, Faculty of Humanities and Social Sciences, Department of Information Sciences

DOI:

https://doi.org/10.47989/ir31261629

Keywords:

Artificial Intelligence (AI), Content Analysis, Natural Language Processing

Abstract

Introduction. This study investigates the effectiveness of Artificial Intelligence (AI)-generated content detection tools in distinguishing between human-written and AI-produced texts in English and Croatian. As AI models are evolving rapidly, identifying generated content is becoming increasingly challenging, particularly in multilingual contexts.

Method. The research analysed twenty-four texts (both human and AI-generated) using four detection tools: Winston AI, Originality.ai, ZeroGPT and Smodin. Three expert authors produced human-written and AI-generated texts across both languages. Qualitative assessment by experts focused on analytical depth and factual accuracy, while detection and readability results were obtained quantitatively from each tool.

Analysis. Comparative analysis was conducted to evaluate accuracy, consistency, and language sensitivity of the tools. Readability metrics were also examined to identify stylistic differences between human and AI writing.

Results. The findings show high detection accuracy for English texts and inconsistent or failed detection for Croatian texts. Originality.ai demonstrated the highest overall reliability. Winston AI and Originality.ai provided readability scores, which indicated that human-written texts were more complex and suited to highly educated readers, while AI-generated texts were simpler in structure.

Conclusion. The study highlights the linguistic limitations of existing detection tools and the need for multilingual calibration. Despite its pilot scope, the research provides valuable insight into tool accuracy, readability patterns and cross-linguistic challenges in AI detection.

References

Akshay, L. C. (2018). McCulloch – Pitts Neuron – Mankind's first mathematical model of a biological neuron. TDS Archive. https://medium.com/data-science/mcculloch-pitts-model-5fdf65ac5dd1 (Archived at https://web.archive.org/web/20251031115553/https://medium.com/data-science/mcculloch-pitts-model-5fdf65ac5dd1)

Anderson, N., Belavy, D. L., Perle, S. M., Hendricks, S., Hespanhol, L., Verhagen, E., & Memon, A. R. (2023). AI did not write this manuscript, or did it? Can we trick the AI text detector into generated texts? The potential future of ChatGPT and AI in sports & exercise medicine manuscript generation. BMJ Open Sport & Exercise Medicine, 9(1), e001568. https://doi.org/10.1136/bmjsem-2023-001568

Awan, A. A. (2023, May 24). What is text generation? DataCamp. https://www.datacamp.com/blog/what-is-text-generation (Archived at https://web.archive.org/web/20251104161758/https://www.datacamp.com/blog/what-is-text-generation)

Amalia, A. (2023). How to identify AI-generated text: With or without software. https://www.contentgrip.com/how-to-spot-aigenerated-text/ (Archived at https://web.archive.org/web/20251104161045/https://www.contentgrip.com/how-to-spot-aigenerated-text/)

Bini, S. A. (2018). Artificial Intelligence, machine learning, deep learning, and cognitive computing: What do these terms mean and how will they impact health care? AAHKS Symposium, 33 (8), 2358-2361. https://doi.org/10.1016/j.arth.2018.02.067

Bellini, V., Semeraro, F., Montomoli, J., Cascella, M., & Bignami, E. (2024). Between human and AI: Assessing the reliability of AI text detection tools. Current Medical Research and Opinion, 40(3), 353–358. https://doi.org/10.1080/03007995.2024.2310086

Cao, L. (2025). A practical synthesis of detecting AI-generated textual, visual, and audio content. arXiv. https://doi.org/10.48550/arXiv.2504.02898

Chan-Olmsted, S. M. (2019). A review of Artificial Intelligence adoptions in the media industry. International Journal on Media Management, 21(3-4), 1-23. http://dx.doi.org/10.1080/14241277.2019.1695619

European Parliament. (2024, March 13). European Parliament legislative resolution of 13 March 2024 on the proposal for a regulation on artificial intelligence (Artificial Intelligence Act) (P9_TA(2024)0138). https://www.europarl.europa.eu/doceo/document/TA-9-2024-0138_EN.html (Archived at https://web.archive.org/web/20260322073421/https://www.europarl.europa.eu/doceo/document/TA-9-2024-0138_EN.html)

Gehrmann, S., Strobelt, H., & Rush, A. (2019). GLTR: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 111–116). Association for Computational Linguistics. https://doi.org/10.18653/v1/p19-3019

Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023). A large-scale comparison of human-written versus ChatGPT-generated essays. Scientific Reports, 13(1). https://doi.org/10.1038/s41598-023-45644-9

Hua, H., & Yao, C.-J. (2024). Investigating generative AI models and detection techniques: Impacts of tokenization and dataset size on identification of AI-generated text. Frontiers in Artificial Intelligence, 7. https://doi.org/10.3389/frai.2024.1469197

Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7), 100779. https://doi.org/10.1016/j.patter.2023.100779

Mao, K., Deng, C., Chen, H., Mo, F., Liu, Z., Tetsuya Sakai, & Dou, Z. (2024). ChatRetriever: Adapting large language models for generalized and robust conversational dense retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 1227–1240). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.71

McLean, D. (2025, February 10). How to detect AI writing in 2025 (Expert tips). Elegant Themes. https://www.elegantthemes.com/blog/business/how-to-detect-ai-writing(Archived at https://web.archive.org/web/20251104095622/https://www.elegantthemes.com/blog/business/how-to-detect-ai-writing)

Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025). A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology, 16(5), 1–72. https://doi.org/10.1145/3744746

Originality.ai. (2026). Home page. https://originality.ai/ (Archived at https://web.archive.org/web/20251104100248/https://originality.ai/)

Russell, S. J., &Norvig, P. (2003). Artificial Intelligence: A modern approach (2nd edition). Pearson.

Russell, S. (2022). Kao čovjek: Umjetna inteligencija - napredak ili prijetnja? Planetopija, Zagreb.

Solomonoff, R. J. (1985). The time scale of artificial intelligence: Reflections on social effects. Human Systems Management, 5, 149-152. https://doi.org/10.3233/HSM-1985-5207

Smodin. (2026). Home page. https://smodin.io/ (Archived at https://web.archive.org/web/20260323091418/https://smodin.io/)

Smrekar, M. (2024). Raider, lektor koji prepoznaje AI generirane tekstove. https://www.bug.hr/umjetna-inteligencija/raidar-lektor-koji-prepoznaje-ai-generirane-tekstove-39480 (Archived at https://web.archive.org/web/20251104123135/https://www.bug.hr/umjetna-inteligencija/raidar-lektor-koji-prepoznaje-ai-generirane-tekstove-39480)

Šuman, S. (2021). Pregled metoda obrade prirodnih jezika i strojnog prevođenja. Zbornik Veleučilišta u Rijeci, 9(1), 371-384. https://doi.org/10.31784/zvr.9.1.23

ZeroGPT. (2026). Home page. https://www.zerogpt.com/ (Archived athttps://web.archive.org/web/20251104135037/https://www.zerogpt.com/)

Žejn, A. (2020). Stilometrijska analiza slovenske pripovjedne književnosti do prve izvorne slovenske priče. Umjetnost riječi: Časopis za znanost o književnosti, izvedbenoj umjetnosti i filmu, 64(3–4), 263–283. https://doi.org/10.22210/ur.2020.064.3_4/04

Ward, J. (2023). How to detect AI content: Proven methods for distinguishing human from machine writing. https://www.linkedin.com/posts/johnathanward_how-to-detect-ai-content-proven-methods-activity-7155346352015642624-2E-x(Archived at https://web.archive.org/web/20251104143308/https://www.linkedin.com/posts/johnathanward_how-to-detect-ai-content-proven-methods-activity-7155346352015642624-2E-x)

Winston AI. (2025). Interpreting our AI detection scores. https://gowinston.ai/interpreting-our-ai-detection-scores/ (Archived at https://web.archive.org/web/20251104144024/https://gowinston.ai/interpreting-our-ai-detection-scores/)