Understanding user behaviors and satisfaction in LLM-assisted tasks: prompting strategies and user knowledge states among college students

Abstract

Introduction. This study examines how users interact with large language models (LLMs), focusing on the interplay between users’ knowledge states and prompt strategies in shaping satisfaction.

Method. Data were collected from 39 students, yielding 187 valid task records. Participants reported knowledge states and satisfaction and submitted dialogues for analysis. Prompt strategies and the character of LLM were coded, and a strategy matching rate was proposed as a novel indicator of LLM application ability.

Analysis. Analyses combined descriptive statistics, correlation analyses, Mann–Whitney U tests, and mixed-effects logistic regression, with intercoder reliability established through iterative agreement.

Results. Directive, contextual, and iterative refinement strategies were most frequently observed, with notable discrepancies between self-reported and coded strategy use. Regression analyses showed that non-delegability and creativity increased the likelihood of satisfaction exceeding expectations, while topic familiarity and knowledge status effects reduced it. Chain-of-thought and self-consistency strategies positively predicted satisfaction, whereas directive and role prompting strategies had negative effects.

Conclusion(s). Satisfaction in LLM-assisted tasks emerges from the interaction of users’ knowledge states, task characteristics, and prompting strategies. By introducing strategy matching rate as an empirical indicator of LLM application ability, this study contributes a practical approach to assessing AI literacy and advances understanding of human–LLM collaboration.