As research on group polarization deepens, extensive exploration has also been conducted on the measurement of group polarization, leading to the development of a relatively comprehensive system of measurement approaches. Existing research suggests that the current mainstream group polarization measurement schemes can be primarily divided into three categories: volume-based, sentiment-based, and network-based (Bilal et al., 2019; Jaidka et al., 2018). We will discuss the characteristics of these three measurement schemes and their shortcomings when applied to the task of measuring group polarization on social media.

Volume-based approaches primarily rely on statistical methods and were widely applied in the early research of group polarization. Early researchers collected data through surveys and experiments and used statistical analysis to obtain relevant polarization results. In current trend of exploring group polarization via social media, volume-based schemes focus more on various data metrics and employ statistical methods in research. Notable examples include Gaurav et al.’s political polarization study based on the moving average aggregate probability method, Tumasjan et al.’s analysis of political polarization using the LIWC tool, and Hart et al.’s use of multidimensional statistical analysis to analyse polarization during COVID-19 (Gaurav et al., 2013; Tumasjan et al., 2010; Hart et al., 2020).

However, existing studies suggest that volume-based schemes have limitations in terms of capturing information and analysing large datasets. They fail to accurately understand the opinions and sentiments conveyed in the text and generally rely on broad statistical metrics (e.g., word frequency, likes, bookmarks, etc) to gather limited information. The lack of rapid information capture and in-depth understanding makes these techniques less effective for tracking and real-time analysis of group polarization, and they also fall short in terms of precision in measuring polarization.

Compared to volume-based approaches, sentiment-based approaches place greater emphasis on the meaning and emotions conveyed in the text. Typically, sentiment-based approaches are grounded in natural language processing (NLP) and analyse social media text from the perspectives of opinions and emotions. These methods generally follow two main strategies. The first involves clustering texts based on the similarity of sentiments, such as the IOM-NN method proposed by Belcastro for accurately detecting emotional information in political polarization (Belcastro et al., 2020). The second strategy leverages deep learning for direct sentiment classification, exemplified by Tyagi et al.’s research on polarization driven by climate change, and the explorations by Ribeiro et al. and Jiang et al. on the relationship between misinformation and polarization (Tyagi et al., 2020; Ribeiro et al., 2017; Jiang et al., 2018).

Unfortunately, both strategies face notable challenges in practice. For text clustering, current clustering algorithms are relatively coarse and simplistic, making it difficult to distinguish between disruptive information (such as advertisements or neutral statements) and significant content. Moreover, they do not account for the specific relationships between subgroups or their contribution to polarization, resulting in outcomes that lack precision and interpretability. In sentiment classification, current methods often rely on binary classifications, failing to capture the intensity of emotions. This oversimplified method negatively impacts both the interpretability and accuracy of polarization measurement.

It is also worth noting that in political polarization research, a method called the sentiment thermometer has been widely adopted (Boxell et al., 2020; Iyengar et al., 2012; Iyengar et al., 2019; Lelkes et al., 2017; Wakefield et al., 2023). This approach uses surveys to gather voters’ emotional scores toward various political parties, thus enhancing the precision and interpretability of the analysis. However, this method is costly, limited by small sample sizes, and not well-suited for measuring polarization in the context of social media, as it generally considers only two opposing groups, whereas social media often involves more complex, multi-group dynamics (Iyengar et al., 2012).

Network-based approaches represent an approach that evaluates group polarization by considering social positions and relationships among groups, focusing on emotional direction and the stance between subgroups to better measure polarization levels. Traditional social network analysis in group polarization studies explores peripheral connections around core opinions to assess subgroup positions and emotions (Bravo et al., 2015; Conover et al., 2011; Garcia et al., 2015; Guerra et al., 2013; Medaglia et al., 2017; Vicario et al., 2017). Some researchers have further developed this by dynamically simulating the process of group polarization to explore its evolutionary pathways (Maia et al., 2023; Santos et al., 2021). With the advancement of graph neural networks (GNNs), network-based approaches have been enhanced, as demonstrated by valuable explorations from researchers such as Xiao et al., Zhang et al., and Jiang et al., who utilized sentiment networks and GNNs in their studies (Xiao et al., 2020; Zhang et al., 2019; Jiang et al., 2023). While these studies have achieved promising results in improving the scientific rigor and interpretability of group polarization research, they generally lack detailed analysis of the textual content. Moreover, social division within purely social networks does not necessarily result from group polarization, raising questions about the accuracy of some conclusions drawn from these methods.

With the introduction of OpenAI’s GPT series of large language models (LLMs), numerous research fields have incorporated or examined GPT’s capabilities (Brown et al., 2020). In the realm of group polarization research, some researchers have also explored its applications. For instance, Lu et al. used agent-based simulations to model group polarization dynamics, while Zhang et al. employed LLM-Based Agents to detect stances within polarized groups (Lu, 2024; Zhang, 2022). These studies have yielded promising results, demonstrating the feasibility and value of applying large language models in this field.

The community sentiment network (CSN) is an extension of the sentiment thermometer method. The traditional sentiment thermometer could only be applied to two subgroups (Iyengar, 2012). CSN extends the sentiment thermometer to a directed cyclic graph that involves emotions between multiple subgroups (see Figure 1). Let G = (V, E) be a graph, where V is the vertex set containing subgroups and E is the set of edges representing the sentimental relationships between subgroups. Each edge e ∈ E is defined as (u, v, s) where u, v ∈ V and s is the sentiment score. It should be noted that nodes v and u are allowed to be the same node, meaning self-loops are permitted. The score s can be either positive or negative, reflecting the positive or negative nature of the sentiment.

Unlike traditional social networks, CSN uses sentiment rather than interactions as the basis for constructing connections. Also, CSN clearly illustrates the various subgroups with different stances within the target time period and reveals the emotional states between subgroups as well as the internal cohesion of each subgroup. Therefore, compared to clustering results or social networks, CSN significantly highlights the contributions of different subgroups to polarization, providing greater interpretability of the polarization state.

The construction of the CSN involves multiple issues, such as subgroup identification, stance detection of opinion information, and sentiment recognition. However, existing research in the field of group polarization is insufficient to provide effective solutions to these issues. Therefore, inspired by the advancements in stance detection tasks (Xiao et al., 2020), we designed a multi-agent system based on LLM-based agents (see Figure 2).

In summary, our multi-agent system is composed of three stages: the background mining stage, the semantic analysis stage, and the polarization assessment stage. The background mining stage consists of subgroup exploration expert and domain specialist, the semantic analysis stage includes social media veteran, linguistic expert, and sentiment analysis expert. The polarization assessment stage comprises polarization assessor. The agents collaborate through a series of interactions to provide accurate subgroup division results and reliable sentiment scores (see Figure 3). We will provide a detailed explanation of each stage and the functions of the respective agents in the following paragraphs.

We have designed a dedicated group polarization metric for CSN to derive an interpretable polarization index from its complex graph structure. In previous research on sentiment-based polarization measurement, a widely accepted viewpoint is that the stronger the internal cohesion within subgroups and the greater the hostility between subgroups, the higher the level of polarization (Iyengar et al., 2012; Iyengar et al., 2015; Lelkes et al.,2017; Wakefield et al., 2023; Yarchi et al., 2021). The sentiment thermometer was developed based on this perspective, and its approach of calculating sentiment temperature differences has gained widespread recognition and practical use (Boxell et al., 2020; Iyengar et al., 2019; Iyengar et al., 2012; Lelkes et al.,2017; Wakefield et al., 2023). However, as we mentioned earlier, this method is only applicable when there are exactly two subgroups involved in group polarization. Therefore, we extended this calculation method to the multi-group domain and proposed the community opposition index (COI).

Firstly, we calculate the sentiment score of a subgroup towards the other subgroups:

Here, e_ij represents the sentiment score of subgroup i towards subgroup j. 1e_ij≤0 means that we consider friendly subgroups as not contributing to the overall group polarization.

Subsequently, we sum the sentiment scores of subgroup i towards all other related subgroups and take into account the internal cohesion within each subgroup. Therefore, we get the polarization score of subgroup i. Here, t_i represents the internal cohesion of the subgroup i:

Finally, we weight the overall sentiment score of each subgroup according to its size and calculate the final polarization score:

Here N represents the total number of comments on the target topic and ni represents the number of comments of subgroup i within this topic.

It is important to emphasize that, since our metric is a relative indicator, it can avoid interference in the polarization measurement results caused by differences in the number of comments. Additionally, this metric, by focusing on both the internal and external sentiments of subgroups, offers better interpretability.

Based on existing work in the field of stance detection, we will conduct our experiments on the following three datasets (Augenstein et al., 2016; Liang et al., 2022):

SEM16 (Mohammad et al., 2016). This dataset includes six different targets selected from various domains, namely Donald Trump (DT), Hillary Clinton (HC), Feminist Movement (FM), Legalization of Abortion (LA), Atheism (A), and Climate Change is a Real Concern (CC). It includes three types of stances: Favour, Against, and None.

P-Stance (Li et al., 2021). This dataset includes six different targets selected from political domains, namely Donald Trump (Trump), Joe Biden (Biden), Bernie Sanders (Sanders). It includes three types of stances: Favour and Against.

VAST (Allaway et al., 2020). This dataset includes large number of varying targets, and it includes three types of stances: Pro, Con and Neutral.

The statistics of our utilized datasets are shown in Table 2. Since our model’s use case is almost zero-shot, we will utilize these three datasets to conduct zero-shot stance detection. We will strictly adhere to the licensing requirements of the respective datasets.

To better evaluate the model’s performance, we selected appropriate metrics based on existing literature (Allaway et al., 2021; Lan et al., 2024; Liu et al., 2021). For the SEM16 and P-Stance datasets, we chose Favg, which represents the average of F1 scores for Favour and Against. For the VAST dataset, we opted for Macro-F1 as the evaluation metric.

Since the primary purpose of our designed multi-agent system is to construct the CSN, we need to adjust the model for the experiments. We removed the subgroup exploration expert and eliminated the subgroup exploration process. Instead, we input texts with predefined target groups into the remaining five agents and obtained the output from the polarization assessor. The adjusted model’s output only includes the sentiment score and target group, making it suitable for performing stance detection tasks.

Regarding the specific details of the model configuration, we use multiple GPT-3.5 Turbo models provided by OpenAI to serve as agents in the background mining stage and semantic analysis stage, while GPT-4 is employed as the polarization assessor. This selection was primarily based on a balance between cost and the desired final performance.

We compare our method with various methods in zero-shot stance detection. This includes adversarial learning method: TOAD, contrastive learning methods: PT-HCL, Bert-based techniques: TGANet and Bert-GCN, LLM-based techniques: GPT-3.5 Turbo, GPT-3.5 Turbo + Chain of thought (COT) and COLA (Allaway et al., 2021; Liang et al., 2022; Allaway et al., 2020; Liu et al., 2021; Zhang et al., 2022; Zhang et al., 2023; Lan et al., 2024).

In Table 3, we present the performance of our method on the zero-shot stance detection task, along with a comparison to other baselines. The results demonstrate that our method exhibits excellent performance in this task, with a performance improvement of 8.4% over the current best result on the VAST dataset. Although our method did not achieve state-of-the-art (SOTA) results across all metrics, its ability to come close to or surpass current SOTA algorithms indicates its significant value when applied to group polarization research.