Ten pairs of participants (20 individuals) were recruited for this study. Participants were recruited in pairs and chose partners they felt comfortable collaborating with. This self-selection ensured they did not have difficulty working together, minimizing potential influence of interpersonal dynamics on the results. The participants consisted of undergraduate and postgraduate students from the University of Tsukuba, with ages ranging from 19 to 27 years (mean age = 21.3 years). The sample included 6 women and 14 men, with academic backgrounds spanning informatics, science and engineering, social sciences, humanities, and medicine. Participants were daily users of traditional search engines, ensuring basic familiarity with online search methods such as finding relevant information and comparing sources. Advanced search techniques (e.g., Boolean operators or relevance ranking) were not specifically assessed, as the study aimed to reflect typical user behaviour. Before the experiment, participants were briefed on ChatGPT’s interface, capabilities, and general use cases to standardise their understanding of the tool.

The task asked participants to collaboratively plan a two-day, one-night domestic trip. The planning process was divided into four key steps:

Destination selection: participants chose three tourist destinations they would like to visit.

Participants were asked to start with the destination selection but could order the remaining steps as they wished. They could not return to a previous procedure, as this would make time recording unreliable.

In condition S (search engine only), participants used Google to gather information and complete the task without AI assistance. In condition C (search engine + ChatGPT), participants used both ChatGPT and Google, with the frequency of ChatGPT use left to their discretion. However, the study did not record the precise duration of ChatGPT use in place of the search engine. Future studies could measure this to better understand time allocation across tools and its impact on performance. For consistency, participants were not allowed to use other AI-based tools or services.

Each experiment session lasted approximately 170 minutes, divided into two phases (condition S and condition C). Participants were given 80 minutes per condition to complete the four steps of travel planning. After each condition, they completed a post-task questionnaire assessing satisfaction with the process and the travel plan produced.

During the experiment, one participant in each pair operated the computer, while the other recorded the travel plan (See Table 1). Participants could switch roles as needed. Importantly, all decisions regarding the travel plan had to be made collaboratively, ensuring the study focused on group dynamics rather than individual decision-making.

Data were collected on three key variables:

Task completion time: the total time participants took to complete the travel planning task was recorded for each condition. Additionally, the time spent on each of the four steps was tracked.

Additionally, the volume of conversation during the task was recorded and analysed to assess ChatGPT’s impact on communication. Conversations were transcribed and measured in terms of the number of words exchanged, both between participants and with ChatGPT.

This study was approved by the Ethics Review Committee of Institute of library, information and media science, University of Tsukuba (Approval No. 23-73). All participants provided informed consent before participating in the experiment and were compensated with Amazon gift certificates based on the institution’s regulation.

Overall task completion time includes the time taken to plan the entire two-day trip. The median task completion time for condition C was 3,553 seconds (59:13 minutes), while condition S had a median time of 3,471 seconds (57:51 minutes). The Wilcoxon rank-sum test indicated that this difference was not statistically significant (p = 0.50), suggesting that including ChatGPT did not reduce overall task completion time.

However, when broken down by task procedure, a few notable differences emerged. In the destination selection stage (procedure A), participants in condition C took slightly less time than those in condition S. The median time for procedure A in condition C was 21.0% of the total time, while in condition S it accounted for 29.9%. This indicates that ChatGPT may have helped participants generate ideas for destinations more quickly. Conversely, the schedule creation stage (procedure B) took slightly longer in condition C (33.1%) compared to condition S (26.0%), though these differences were not statistically significant (p > 0.05).

Participants’ satisfaction with the travel planning process was measured using a post-task questionnaire, which included questions on ease of use, satisfaction with the plan produced, and satisfaction with each stage of the planning process. Overall, there were no significant differences in total satisfaction between the two conditions. Participants generally expressed moderate satisfaction in both conditions, with a median satisfaction score of 4 (on a 5-point Likert scale).

However, analysis of individual stages revealed nuanced differences. In the destination selection stage, participants in condition C reported finding it easier to identify relevant information compared to condition S (median satisfaction score of 1.00 vs. 2.00). In contrast, in the accommodation booking stage (procedure C), participants in condition S expressed greater satisfaction with their ability to find accurate and up-to-date information. This difference likely reflects ChatGPT’s reliance on static data, limiting its effectiveness for tasks requiring real-time information, such as finding available accommodations.

Despite the overall similarity in satisfaction between the two conditions, open-ended responses from participants highlighted ChatGPT’s usefulness for idea generation. One participant remarked, ‘ChatGPT helped us brainstorm destinations quickly when we had no initial ideas,’ while another stated, ‘the search engine required more effort to manually gather all the information.’

The quality of the travel plans produced was assessed based on completeness, feasibility, and level of detail. Plans were considered complete if they covered all essential aspects of the trip, including transportation, accommodation, and activities. Feasibility referred to whether the plans were realistic in terms of time and budget, and level of detail was evaluated based on how thoroughly the plans addressed logistical aspects, such as transport schedules and booking details (Imazu, et al., 2011).

Results showed that condition C (search engine + ChatGPT) led to travel plans that were more aligned with the task requirements. Nine of the ten pairs in condition C produced plans that met all task criteria (e.g., stayed within budget, covered all trip details), compared to seven pairs in condition S. Additionally, plans from condition C were often rated as more detailed, especially in terms of destination choices and activities. However, condition S produced more feasible schedules, as participants found it easier to obtain accurate transport and accommodation information using search engines.

For example, one travel plan in condition C proposed a tight schedule including three distant locations in a single day, which may have been unrealistic. In contrast, travel plans in condition S generally reflected more conservative itineraries, with better alignment between travel times and planned activities.

The volume of conversation between participants was measured to assess how ChatGPT influenced group interaction. In condition C, participants exchanged fewer words compared to condition S, suggesting that ChatGPT may have reduced the need for extended discussions (Joho, et al., 2008). The median number of words exchanged in condition C was approximately 15% lower than in condition S. A Wilcoxon rank-sum test found no statistically significant difference between the conditions (p = 0.34), but the trend suggests the AI tool streamlined some aspects of the decision-making process.

Participants used ChatGPT primarily in the destination selection (procedure A) and schedule creation (Procedure B) stages. The most common type of interaction with ChatGPT was asking for ideas and itinerary suggestions. For instance, participants frequently asked ChatGPT for recommended places to visit in a specific region or to help build a draft schedule. ChatGPT was used less frequently in accommodation and transport booking stages, likely due to the need for real-time data, which ChatGPT could not provide, at the time of study.

The most notable benefit of using ChatGPT in the collaborative planning process was in the destination selection stage, where it helped participants generate ideas quickly. In this phase, participants using ChatGPT took less time to decide on destinations compared to those relying solely on search engines. This suggests that generative AI can serve as an effective tool for brainstorming, particularly in tasks requiring creativity or decision-making when users lack initial direction. Participants noted that ChatGPT helped overcome the initial inertia of decision-making by offering suggestions and guiding the conversation. Participant satisfaction ratings did not indicate concerns about ChatGPT’s suggestions being overly leading. This aligns with research suggesting that AI’s value lies in its ability to generate ideas and reduce cognitive load in early problem-solving stages (Maaravi, et al., 2021).

However, ChatGPT’s advantages did not extend uniformly across all planning stages. In the accommodation and transport booking stages, participants in condition S (search engine only) expressed greater satisfaction and produced more feasible plans. This can be attributed to ChatGPT’s lack of real-time data, which limited its ability to provide accurate, up-to-date information on hotel availability, transportation schedules, and other logistics-dependent tasks. In these cases, search engines offered more reliable information, underscoring the continued importance of traditional tools for tasks heavily reliant on real-time data. ChatGPT’s structured, system-driven outputs may also not align with the adaptability and creativity humans bring to dynamic contexts.

Despite these insights, several limitations must be acknowledged. First, the sample size was relatively small, with only 10 pairs of participants. This limited the statistical power of the results, potentially explaining the lack of significant differences in task completion times and satisfaction across conditions. A larger sample size could provide more robust data and reveal subtler trends in performance and behaviour.

Additionally, participants were primarily students with limited experience using ChatGPT for travel planning. This familiarity bias may have limited their ability to fully leverage the tool’s capabilities, particularly in the task’s initial stages. Future studies should include a longer familiarisation period or recruit participants with greater experience using AI tools, allowing a more accurate assessment of how generative AI can be used effectively in collaborative settings.

Another limitation was the task’s time constraint, which many participants found restrictive. The one-hour limit may have prevented participants from fully exploring travel planning options, especially in condition C, where ChatGPT unfamiliarity slowed some processes. Extending task duration in future studies would provide a more realistic reflection of how AI can be integrated into collaborative work, particularly for more complex or fluid tasks like travel planning.

For researchers and practitioners in the iSchool community, this study provides valuable insights into the evolving role of AI in information-seeking behaviours. AI tools like ChatGPT can complement, but not replace, traditional search engines in collaborative tasks. Combining AI for idea generation with search engines for real-time data retrieval is a promising approach, particularly in complex tasks like travel planning that require both creativity and logistical accuracy.

The findings also suggest that AI’s impact on group communication and dynamics warrants further investigation. While AI may reduce cognitive load and streamline decision-making, it risks diminishing the collaborative aspect of tasks, potentially leading to less robust decision-making. Future work should explore how to balance AI assistance with human interaction to preserve the benefits of collaboration while leveraging AI’s strengths.