Conversational Agents (CAs) are systems enabled with the ability to interact with users using natural human dialogue (Rheu et al., 2021). After the recent release of various Large Language Model (LLM)-based Conversational Agents (CAs) (e.g., ChatGPT (OpenAI, 2022)), research communities are increasingly experimenting with data annotation tasks such as annotating political stance and sentiment of textual data (Amin et al., 2023; Kuzman & Ljubešić, 2023; Liu et al., 2023; Zhang et al., 2022a). Emerging literature suggests that LLM-based CAs can be useful for text classification tasks (e.g., Amin et al., 2023; Huang et al., 2023; Kuzman & Ljubešić, 2023; Liu et al., 2023; Zhang et al., 2022a; Zhang et al., 2024). For instance, Zhang et al. (2022) show that ChatGPT was able to annotate the political stance of the tweets with an average accuracy above 70. Moreover, LLMs can be adapted through finetuning or prompting for specific domains (Song et al., 2024), setting a new standard and expectations for what is achievable in natural language tasks. With proper fine-tuning, LLMs are known to even outperform crowdsourced annotators (Gilardi et al., 2023). As such, advances in LLMs such as GPT-4 showed a promising opportunity for data annotation at scale due to their ability to automate annotation tasks (Zhang et al., 2022b). However, there are challenges in the use of LLM-based CAs in annotating textual data for more contextualized constructs. For instance, Amin et al. (2023) showed that ChatGPT’s accuracy for subjective tasks such as the five personality and suicide ideation classifications was lower than the baseline machine learning methods. As such, early empirical research demonstrated the limitations of subjective and contextualized annotation tasks entirely, indicating the need for human-AI collaboration on such tasks.

As LLMs-based conversational agents have shown the ability to interact with humans and work with examples in various domains (Kim et al., 2022; Lai et al., 2022; Mackeprang et al., 2019; Tang et al., 2024), researchers are exploring the potential of human-AI collaboration on various tasks such as online content moderation (Lai et al., 2022), thematic analysis of qualitative data (Jiang et al., 2021; Zhang et al., 2024), disease prevention (Lu & Peng, 2024), and crowdsourcing (Tamura et al., 2024). For instance, Zhang et al. (2024) explored the potential of LLM-based CAs as collaborative tools for qualitative data analysis and highlighted the efficiency of reducing time and labour for such analysis. Yet, their performance in collaborative co-annotation exercises for online risk where different facets of co-annotation are important is understudied. This gap is pertinent because co-annotation tasks need to support an interactive discussion to help generate a rationale for the various decisions, particularly in the context of highly contextualized online risk behaviour (Clay- Warnder, 2003) which can entail disagreement even among human coders. Yet, the disagreement must not lead to capitulation, instead, inspire better methods of automated analysis. Therefore, the combination of manual and automated content analysis is suggested as the gold standard for identifying subjective concepts such as online risk (Esau, 2022). In recent work, Wang et al. (2024) designed a multi-step human-LLM collaborative framework for data annotation tasks (e.g., natural language interface, stance detection, and hate speech detection) and found that when LLMs are incorrect in complex or domain-specific tasks, human annotations without any LLM assistance were the most accurate. Therefore, they suggested an iterative process of human-AI collaboration in data annotation, with feedback from the human annotators used to improve the quality of LLMs annotation.

In this work, we explore the potential of using LLM-based CAs to assist human coders in annotating subjective, nuanced online conversations. We expect that LLM-based CAs can support high- quality annotations with explanations when provided with proper instructions and examples. This approach could enhance scalability and help capture nuances that human coders might miss due to cognitive limits. We experimented with four prompting methods to assess their impact on annotation results and how co-annotation can improve subjective data annotation. By examining agreements and disagreements between human coders and LLM-based CAs, we explored how their strengths and weaknesses can complement each other.

CHAIRA is an online annotation tool that integrates an LLM-based conversational agent to support human-AI collaboration on online risk data annotation. Below, we describe how we designed and implemented CHAIRA in detail.

We conducted experiments with four different prompting engineering approaches: zero-shot, definition, few-shot, and two-stage few-shot Chain-of-Thought (CoT). We designed four different prompting approaches with varying levels of interaction between the human coder and the LLM- based agent. We used the same coding scheme as applied in the previous literature (Park & Singh, 2022) to design the prompts. In our prompts, incivility was defined as ‘the feature of discussion that conveys an unnecessarily disrespectful tone toward the discussion forum, its participants, or its topic’ (Coe et al., 2014), with six different categories: name-calling, aspersion, lying, vulgarity, pejorative for speech, and others (Table 1). The reported inter-rater agreement between the two human coders was 0.88 (percent agreement) and 0.76 (Cohen’s Kappa) (Park & Singh, 2022).

Zero-shot prompting: In zero-shot prompting, the model is only given a simple instruction describing the task. This method is considered convenient and has the potential for robustness (Brown et al., 2020). The instruction used in the zero-shot prompt is as follows: ‘Classify the text into ‘civil’ or ‘incivil’ and explain why’.

Definition prompting: In definition prompting, along with the instruction of classifying a comment as ‘civil’ or ‘incivil,’ we provided the definition of incivility and brief descriptions of six categories of incivility (see Table 1).

Few-shot prompting: With few-shot prompting, models are given a few demonstrations of the task (Brown et al., 2020), in our case, examples of incivility. With this approach, we provided the model with the definition of incivility, descriptions of the six categories of incivility, examples of the six categories of incivility (Table 1), and the instructions for the task.

Two-stage few-shot chain-of-thought: Finally, we used a two-stage few-shot chain-of-thought (CoT), a few-shot-based prompting for the chain of thought reasoning. Chain-of-thought (CoT) prompting (Wei et al., 2022) modifies the answers in few-shot examples to step-by-step answers by adding an instruction such as ‘Let’s think step by step’ to the original prompt to elicit reasoning in LLMs (Kojima et al., 2022; Wei et al., 2022). The first prompt is the reasoning extraction where we used CoT to elicit reasoning from LLM. The second prompt consists of the first prompt and the answers generated from the first prompt.

Figure 5 shows a visualization of the two-stage few-shot CoT approach applied. In the first prompt, we provided the same instructions as in few-shot prompting and added the final line that says, ‘Let’s work this out in a step-by-step way to be sure we have the right answer’ suggested by Zhou et al. (2022). Next, we reviewed the responses generated by CHAIRA from the first prompt and did the error analysis. We chose one example of false positive cases (i.e., CHAIRA output = civil, human ground truth = incivil) and pointed the model to recognize what it is missing in its answers (i.e., implicit aspersion). Once CHAIRA generated the responses that matched with human responses, we added the conversation log (Figure 6 in Appendix A) to the prompt. A summary of the four prompting approaches we used in this case study is presented in Table 2.

Overall, CHAIRA did a better job at annotating explicit incivility and explaining the reasoning behind them with more information provided in the prompts. For instance, even with the zero-shot prompting approach, CHAIRA pointed to the exact incivil expressions and recognized that such expressions reflect a negative attitude towards immigrants. In addition, without any information about incivility, CHAIRA was able to note the use of more nuanced incivility, such as sarcasm, when nuanced expressions are combined with explicit personal attacks and insults (examples in Figure 7 in Appendix).

At the same time, we observed some common issues throughout all the prompting approaches: CHAIRA did not recognize implicit and nuanced incivil expressions in the texts, even with the information ‘an aspersion may be both explicit and implicit’ given in the prompt. Therefore, when designing the two-stage few-shot CoT prompt, we reminded CHAIRA that aspersion can be implicit and nuanced through interactive conversation. Only after the interactive conversation between the human coder and CHAIRA about implicit aspersion was added to the prompt (i.e., the two-stage few-shot prompting approach), CHAIRA started to distinguish implicit and nuanced yet incivil expressions (examples in Figure 8 in Appendix), which were frequent in our dataset.

We observed that while the output label remained the same the reasons changed with more information in the prompts. For instance, with the zero-shot prompt, CHAIRA mainly focused on the use of language and tone of the text, whereas with the definition prompt, CHAIRA considered whether the text fell under any of the six categories of incivility (Figure 7). The rationales provided were similar for definition and few-shot prompting approaches. A similar trend was observed for incivil cases, where CHAIRA provided specific reasons by pointing to the specific category (i.e., lying) and the context of why it falls under the category with the definition and few-shot prompting approaches, while CHAIRA provided more generic reasons (i.e., use of personal attacks or offensive language) with the zero-shot approach.

Overall, CHAIRA could provide human coders with helpful context on US politics, immigration policy debates, and incivil expressions that may be overlooked. For example, the text ‘And around Nancy’s wall on Capitol Hill. Make that wall to keep them in’ was annotated as civil by a human coder, as it did not contain explicit or implicit incivility. However, with zero-shot and definition prompts, CHAIRA recognized ‘them’ as immigrants, and the comment about building a wall to ‘keep them in’ was interpreted as disrespectful. Using a two-stage few-shot CoT approach, CHAIRA’s rationale became even more detailed: ‘The comment mocks Nancy Pelosi and implies hostility toward immigrants.’ The human coder missed that ‘Nancy’ referred to a political figure and failed to recognize the disparaging tone. Although another coder might have annotated it differently, the above example shows that human coders have limits in awareness and cognitive capacity and AI can complement such limitations. As such, with the highest level of human-AI interaction (two- stage CoT), CHAIRA effectively discerned both the tone and the target of incivility, providing crucial context in annotating political incivility.

Unlike human conders, we observed some cases where CHAIRA did not accurately catch the textual information given in the prompts. For instance, CHAIRA lacked an understanding of the description and example of the ‘Pejorative for Speech’ category. We found some of the responses that CHAIRA mistakenly understood the pejorative for speech as the use of a sarcastic tone in the text, as opposed to its actual concept, a disparaging remark about how a person communicates. Similarly, CHAIRA sometimes lacked an understanding of the given text compared to human coders, particularly for short texts. For instance, with the definition prompting approach, CHAIRA struggled to annotate a short text such as ‘What is your solution?’ hence, annotated it as ‘unclear.’ With the few-shot prompt, CHAIRA annotated the same text as incivil because the text contains examples of name-calling and aspersions. However, the incivil expressions referred to in this response were from the examples given in the prompt (i.e., instructions), not from the given text to annotate. Overall, sometimes CHAIRA could not understand the information in the prompts or texts to annotate and hence, was unable to annotate the text appropriately.

Implications for building human-AI collaborative annotation systems The above-mentioned results yielded implications for the future design of human-AI collaborative annotation systems. Below, we discuss the design implications for using AI-based CAs to best support the co-annotation of online risk data.

We observed that CHAIRA was good at providing reasons for its annotation results. Therefore, the benefit of CHAIRA lies in the interactive nature of the annotation process, which provides partial explainability and is one of the important aspects of human-centred AI-based systems (Minh et al., 2022; Vilone & Longo, 2021). With the most sophisticated prompting approach we had (Few-shot CoT), CHAIRA informed the human coder with a broad knowledge and context of the given text and convinced the human coder to change their mind in multiple instances. Therefore, one of the strengths of LLM-based CAs is the ability to provide relevant information (presumably) trained on the entirety of online data, as opposed to human coding which requires extensive training or domain-specific knowledge. This shows the potential strengths and values in human-AI co-annotation, particularly in risk scenarios that require domain-specific knowledge. In addition, the initial reasoning and domain knowledge provided by CAs can be used to inform human coders further on how to design better CA models to support the co-annotation of online risk.

A major benefit of co-coding with CHAIRA was its ability to scale the data annotation with a high degree of inter-rater agreement. To be able to do so, we carefully reviewed the incivility labels where CHAIRA and the human coder disagreed and had two-way conversations with CHAIRA to further understand their reasoning. This two-way interaction in the co-annotation process was useful because, despite access to a large corpus of knowledge, CHAIRA also tended to make some simple mistakes that were quite easy for human coders to spot. For instance, during the interaction with CHAIRA, we realized that CHAIRA could miss the previous conversation and hence, we needed to remind it about our conversation for further annotation tasks, particularly about nuanced incivility (e.g., implicit aspersion). Therefore, we added the instruction ‘keep implicit incivility in mind,’ in our Two-stage CoT prompt. Sometimes, we pointed to the exact incivil expressions in the text that contain implicit aspersion to remind the concept (e.g., Don’t you think this expression of ‘he’s hoping to stir up the same frenzy and ride that wave?’ could be implicitly incivil?). Then the CHAIRA re-evaluated the text and corrected their answers. As such, interactive communication between the human coders and the AI agent is one of the key elements in improving the risk annotations generated by the AI agent.

Providing clear examples of risk cases and descriptions of risk types is crucial when designing human-AI co-annotator models. In our few-shot prompt, we explained that aspersion can be both implicit and explicit, yet CHAIRA failed to recognize implicit aspersion until we guided it through two-way communication. This could be due to confusion caused by slight differences between the risk descriptions and examples provided. For instance, CHAIRA may have interpreted the explicit nature of the ‘silly border wall’ in the aspersion example and missed the implicit aspect described. Therefore, selecting the right examples and crafting clear descriptions of constructs is critical when working with LLM-based CAs to annotate subjective concepts like online risk. In addition, since CAs generate responses by tokenizing input (OpenAI, 2024b), even minor textual changes such as punctuation can affect performance. This can limit the CA’s ability to understand text containing abbreviations or spelling variations, which are common in online risk data (Sadeque et al., 2019). Therefore, designing prompts with careful consideration of how LLMs process natural language is essential to building effective collaborative systems for annotating contextualized online risk data.

In consonance with data ethics, we used de-identified data to develop our annotation tool. We used OpenAI API in the backend as its security policy stipulates that data submitted through the API is not used to train OpenAI models (OpenAI, 2024a). However, future work should also consider building LLM-based CAs with private servers so that the training dataset is not shared via the web. Another limitation of this collaborative annotation is variations in LLM responses. Finally, we acknowledge that inductive approaches (e.g., thematic analysis, grounded theory approach), important approaches to building patterns and themes in qualitative work, were not explored in this study. Future work can explore the potential of LLM-based CAs for more inductive analysis that requires an in-depth understanding of the subtleties and complexities of qualitative data. Moving forward, we aim to experiment with our approaches with diverse types of online risk data at scale to gain deeper insights into collaborative annotation between human and LLM-based conversational agent systems.