Leveraging search engines and LLMs for creative thinking: effects of prompting skills, and domain
DOI:
https://doi.org/10.47989/ir31iConf64171Keywords:
Creative thinking, Curatorial planning in calligraphy, Domain knowledge, Information seeking process, Prompting skillsAbstract
Introduction. We investigated the effects of domain knowledge (DK) and prompting skills by leveraging a traditional search engine and a prompt-based LLM chatbot for a learning-oriented creative task.
Method. On a simulated curated calligraphy exhibition task for which one must integrate DK and creativity to write summaries of curatorial planning, we observed behavior when using the NPM Collection, Google Search, and ChatGPT. We assessed the creative thinking demonstrated in the final summaries. User perception of tools supporting creative thinking was evaluated by the creativity support index.
Analysis. We analysed information source usage behavior using descriptive statistical analysis and significance testing and used regression analysis to examine the predictive power of tool usage behavior, domain knowledge, and prompting skills for summary writing.
Results. We discovered a positive predictive effect of DK and ChatGPT usage on the quality of curatorial planning summaries. Better prompting skills enhanced task performance and perceived creativity. For creative thinking, Google seemed to better foster exploration than ChatGPT.
Conclusions. Prompt-based LLM chatbots support learning-oriented creative tasks. As prompting skills are essential, prompting should be integrated into information literacy education. The study provides implications for the design of search systems in human–information retrieval–LLM interaction to better support creative tasks.
References
Ai, Q., Bai, T. Cao Z., Chang, Y., Chen, J., Chen, Z., Cheng, Z., Dong, S., Dou, Z., Feng, F., Gao, S., Guo, J., He, X., Lan ,Y., Li C., Liu, Y., Lyu, Z., Ma, W. Ma, J., Ren, Z., Ren, P., Wang, Z., Wang, M., Wen, J.R., Wu, L., Xin, X., Xu, J., Yin, D. Zhang, P., Zhang, F., Zhang, W., Zhang, M., & Zhu X. (2023). Information retrieval meets large language models: A strategic report from Chinese IR community. AI Open, 4(3), 80-90. https://doi.org/10.1016/j.aiopen.2023.08.001
Capra, R. & Arguello J. (2023). How does AI chat change search behaviors? ArXiv. https://doi.org/10.48550/arXiv.2307.03826
Chavula, C., Choi, Y., & Rieh, S.Y. (2023). SearchIdea: An idea generation tool to support creativity in academic search. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval, 161-171. https://doi.org/10.1145/3576840.3578294
Cherry E., & Latulipe, C. (2014). Quantifying the creativity support of digital tools through the creativity support index. ACM Transactions on Computer-Human Interaction, 21(4), Article 21. https://doi.org/10.1145/2617588
Kaufman, J.C., & Beghetto, R.A. (2009). Beyond big and little: The four c model of creativity, Review of General Psychology, 13(1), 1–12. https://doi.org/10.1037/a0013688
Kules, B. (2005）. Supporting creativity with search tools, Creativity Support Tools, 50, 53–64.
Liu, Z., Zhou, Y., Zhu, Y., Lian, J., Li, C., Dou, Z., Lian, D., & Nie, J.Y. et al. (2024). Information retrieval meets large language models. In Proceedings of the ACM Web Conference 2024, 1586 –1589. https://doi.org/10.1145/3589335.3641299.
Noy, S., & Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654), 187-192. https://doi.org/10.1126/science.adh2586
Palani, S., Ding, Z., MacNeil, S., & Dow, S.P. (2021). The "active search" hypothesis: How search strategies relate to creative learning. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, 325-329. https://doi.org/10.1145/3406522.34460
Spatharioti, S. E., Rothschild, D. M., Goldstein, D. G., & Hofman J. M. (2023). Comparing traditional and LLM-based search for consumer choice: A randomized experiment, arXiv:2307.03744. https://doi.org/10.48550/arXiv.2307.03744.
Thudt, A., Hinrichs, U., & Carpendale, S. (2015). A modular approach to promote creativity and inspiration in search. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition, 245–254 https://doi.org/10.1145/ 2757226.2757253.
Vakkari, P. (2016). Searching as learning: A systematization based on literature. Journal of Information Science, 42(1), 7-18. https://doi.org/10.1177/0165551515615833
Walter, Y. (2024). Embracing the future of Artificial Intelligence in the classroom: the relevance of AI literacy, prompt engineering, and critical thinking in modern education. International Journal of Education Technology Higher Education, 21(15). https://doi.org/10.1186/s41239-024-00448-
Wazzan, A., MacNeil, S., & Souvenir, R. (2024). Comparing traditional and LLM-based search for image geolocation. In Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, 291-302. https://doi.org/10.1145/3627508.3638305.
Wilson, M.J., & Wilson, M.L. (2013). A comparison of techniques for measuring sensemaking and learning within participant-generated summaries. Journal of the American Society for Information Science and Technology, 64(2), 291-306. https://doi.org/https://doi.org/10.1002/asi.22758
Yang, Y., Urgo, K. Arguello J. & Capra R. (2025) Search+Chat: Integrating search and GenAI to Support users with learning-oriented search tasks. In Proceedings of the 2025 ACM SIGIR Conference on Human Information Interaction and Retrieval, pp. 57-70. https://doi.org/10.1145/3698204.3716446
Yilmaz, R. & Yilmaz. K.F.G. (2023). The effect of generative artificial intelligence (AI)-based tool use on students' computational thinking skills, programming self-efficacy and motivation. Computers and Education: Artificial Intelligence, 4, 100147. https://doi.org/https://doi.org/10.1016/j.caeai.2023.100147
Zhai, C. X. (2024). Large language models and future of information retrieval: Opportunities and challenges. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '24). Association for Computing Machinery, New York, NY, USA, pp. 481–490. https://doi.org/10.1145/3626772.3657848
Zhang, Y., & Capra, R. (2019). Understanding how people use search to support their everyday creative tasks. In Proceedings of the CHIIR '19: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, 153 – 162. https://doi.org/10.1145/3295750.3298936.
Zhu., C., Hu, X., Wu, H., Qin, C., Zhu H., & Xiong, H. (2025). Enhancing job recommendations with LLM-based resume completion: A behavior-denoised alignment approach. Information Processing and Management, 62(6), 104261. https://doi.org/10.1016/j.ipm.2025.104261
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 I-Chin Wu, Hsuan-Yu Lin, Pertti Vakkari
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
