The metaverse is an umbrella term for a set of digital technologies and use cases. Mystakidis (2022) defines the metaverse as a persistent, multiuser environment that blends physical reality with digital virtuality. It uses technologies like virtual reality (VR) (Alsop, 2023) and augmented reality (AR) (Wikipedia, 2023a) for multisensory interactions with virtual spaces, objects, and people. The metaverse connects immersive, social environments on multiuser platforms, allowing real-time communication and interaction with digital content. The idea of a metaverse originates in Stephenson’s (1992) novel Snow Crash, where it was described as a network of virtual worlds where avatars could move between them, but now it includes social VR platforms, online games, and AR collaboration spaces (Anderson & Rainie, 2022). Such application domains are described next.

In the field of metaverse applications, several use cases are imaginable and already visible, which potentially include MVRs, such as the user sessions.

Gunkel et al. (2018) researched metaverse experiences with persons. Based on a survey, they present a user interest in the domains of video conferencing, education, gaming, watching movies, and further similar entertainment activities (Gunkel et al., 2018). The audience research company GWI researched use cases for the metaverse and lists (Morris, 2022) watching TV and playing games as applications for customers. Hence, there is strong support for entertainment applications. An example of MVRs in this application domain is YouTube’s videos created from the metaverse (Bestie Let’s play, 2022).

Within the metaverse-related literature, the risk of cybercrime is mentioned. The International Criminal Police Organization (INTERPOL, 2024b) describes the potential harms and the concepts to counter them. Examples are crimes against children or sexual offenses and assault. The activities in question can be tracked by recording the relevant transaction data (INTERPOL, 2024a). While not conclusively demonstrated in the existing literature, it is conceivable that MVRs could be used for this purpose.

In addition to consumer-focused use cases, the field of industrial metaverse is relevant. The field of industrial metaverse covers the use of metaverse technologies mainly for simulations of industrial processes (Zheng et al., 2022). Although not definitively described in current research, the review of such simulations is conceivable as a use case for MVRs. Such simulations can address not yet existing scenarios, thus addressing more research and development use cases, or real scenarios mirrored in the virtual world. The latter is described as digital twin. An example is the simulation of mining operations (Stothard et al., 2024).

While many people record their personal life experiences digitally and share them online (Austin, 2023), it is conceivable that people record their experiences in the metaverse as well. Within this paper it is termed as personal use.

In conclusion, a list of potential application domains is compiled, shown in Table 1. The application domains of education and videoconferencing & collaboration are grounded in the research of Gunkel et al., 2018. In the application domain entertainment, this paper summarises the applications of things such as gaming, watching TV, and listening to music. Law enforcement is anchored in the literature of risks of the metaverse. Industrial metaverse is grounded in the literature about metaverse. The personal use application domain is anchored in the described behaviour of people creating and sharing experiences on the Internet.

Next, the technical opportunities of MVRs are described.

In the realm of metaverse, there are several technical recording methodologies for user sessions to consider. These can be broadly categorised into three types, each with distinct characteristics and applications. Figure 1 illustrates the rendering process and the three types. This process takes scene raw data (SRD) and peripheral data (PD), also known as auxiliary data, as input. PD is derived from user devices, hence represents user behaviour. SRD includes information used by the computer to construct the virtual scene. Both inputs are used to create perceivable rendering output, in the form of 2D/3D image stream(s), audio streams, and actuator commands. The rendering output is recordable multimedia, which is defined as multimedia content objects (MMCOs). In summary, the three major categories of recordable data are MMCO, SRD, and PD.

The first group, MMCO shown as recordable multimedia in Figure 1, can be created by directly recording sessions as videos within metaverse applications, such as in Horizon Worlds. This approach captures both audio and video outputs of the virtual environment. The 256 metaverse records dataset (Steinert et al., 2024a) contains examples of such MVRs. An alternative within this category is the use of screen recorders to capture the audio-visual output of the rendered scenes.

The second group, SRD visualised as input in Figure 1, rendering inputs capture the visual rendering inputs used to create the virtual scene. This can be done in two primary approaches. First, capturing scene graphs, which detail the objects present in a scene. Second, utilising network transmitted information to record inputs from other players, including avatar positions and actions. This raw data, when combined with additional audio-visual elements like textures and colours, offers a comprehensive view of the scene. Tools exist to capture rendering raw data, such as Nvidia Ansel (Güngör, 2016) or RenderDoc (Karlsson, 2018).

The third group of recordable data is capturing PD, shown as input in Figure 1. The recording of recording PD provides supplementary information that enriches the primary recording. This data is either hard to resemble later, like emotion detection from facial expressions versus audio recordings, or not possible, like heart rate in specific situations.

As of now, it is unclear which recording method is superior. Audio-visual data is convenient for playback and compatible with existing tools but extracting detailed information for analysis is challenging. On the other hand, the Metaverse’s technical nature allows for the utilisation of logs and interaction data, e.g. avatar presence and hand movements captured from controller data, to gain immediate insights without the need for extensive video analysis. Capturing and organising virtual world experiences for retrieval poses several unresolved challenges. These include how to represent rendered scenes in a way that preserves both technical structure and user interaction, how to map and synchronize multimedia content over time, and how to manage composite content from multiple sources. A core issue is establishing reliable connections between SRD, PD, and their rendered outputs MMCO. Addressing these problems is essential for enabling effective retrieval and reuse of metaverse recordings.

Steinert et al. (2023) see value in the different types for different use cases. Hence, they described a model of the combination of data, visualised in Figure 2. If the data is combined in any form, they define this as composite multimedia content object (CMMCO). If there is a common time in the data, such as a timestamp or frame number, they define this as time-mapped CMMCO (TCMMCO). The 111 recordings dataset (Steinert, 2025) presents MVRs containing MMCO, SRD, and PD, hence CMMCO data. The data is recorded with timestamps; hence the MVRs can be considered TCMMCO.

Starting with the user, there is the need to search, find, and retrieve information. This happens when the user has a knowledge gap, which Belkin et al. (1993) describe as an anomalous state of knowledge (ASK). When the user has all information to complete a task, it is the normal state. If information is missing, it is ASK, which is an information need. This results in information-seeking behaviour, which is described by many scientific research articles.

Multimedia is the combination of any combination of media formats. In the case of MMIR, it is about the different types of media that can be retrieved in a system. The types can be any perceivable media, such as images or audio, or biometric sensor data.

The yearly Lifelog Search Challenge (Gurrin, 2025) addresses a comparable use case then to expect with metaverse content. The challenges address user experience questions for user interfaces to query multimedia originated from recordings of cameras worn by people to capture images of full days in the life. Corresponding data is gathered, such as biometrics and location (Gurrin, 2021). What can be found in the related literature are example search terms, used in the challenges, but the literature lacks specific descriptions of relatable information search behaviour for MVR retrieval.

The metaverse is a term which is used to describe a set of technologies and use cases including fully virtual environments, or real-world environments, with digital extensions. Different use cases are conceivable or already observable, which produce MVRs. A production of MVRs leads to larger collections, which are retrievable by MMIR. Missing in the literature is an understanding of the information need and information searching behaviour, to integrate MVRs in MMIR. A subsequent field study can provide important findings.

Following the application domains described in Table 1, relevant experts listed in Table 2 were identified through a targeted search within the professional network of the authors, the research group at university, and the Immersive Collaboration Hub. Additional contacts were made through subject-specific communities, including VR vendors, consultants, meetup groups, and registered associations. Recruitment focused on individuals with demonstrable practical experience in metaverse or VR applications within the specified domains, for example, through direct involvement in user workshops or active participation in related projects. Twelve experts in the selected application domains were contacted. Six have responded and agreed to a recorded interview. The formed group of participants consists of six individuals, five German, one Italian, all male, with higher education backgrounds. Their professional expertise spans product design, programming, IT consulting, and documentary filmmaking; this offered a diverse and practice-oriented perspective on the study topics. The six people work for one or multiple companies or organisations and are active in an international but mostly European market. It proved challenging to identify individuals utilising the metaverse technologies for personal purposes, particularly those engaged in recording and subsequent retrieval of these experiences.

It was notable that the experts identified for one application domain were in fact able to provide information on other application domains, or at the least, on existing application scenarios in other domains, during the discussion.

After reaching out to the experts, the responding experts were invited to a video conference call. During the recorded call, two interviewers were present, one in the role leading through the interview, the second in a mostly silent role, noting the responses. Both interviewers are experts in the field of multimedia retrieval. To structure the interviews, a questionnaire was created, which is explained next.

The expert interviews were conducted as guided conversations, using a structured questionnaire to ensure consistency and depth of investigation. This questionnaire, as shown in the Appendix Questionnaire Table 7 and Table 8, was divided into two main parts: Application scenario validation and use context detail questions. The questionnaire served as a flexible guide, allowing interviewers to adapt their inquiries based on each expert’s unique insights and experiences. This methodology facilitated a comprehensive examination of metaverse recording applications while maintaining the ability to delve into emerging or unexpected areas of interest.

In the first section, the interviewee is asked to name existing application scenarios (F1.1) and conceivable application scenarios (F1.2). Beforehand, we constructed application scenarios for which the interviewee was asked to assess whether they exist, are conceivable or not relevant (F1.3). If in this step an existing use case was identified, this use case was used to determine the information search behaviour.

The application scenario validation section aimed to explore known cases of metaverse recordings and assess the realism of previously hypothesised scenarios. Table 6 addresses the identification of application scenarios of metaverse recordings and MVR retrieval. After a brief explanation of MMIR in general and MVR retrieval in specific, interviewees were asked to describe familiar applications (F1.1, F1.2) and evaluate the feasibility of proposed application scenarios (F1.3). The use context detail questions focused on gathering in-depth information about a specific application scenario, preferably one described by the interviewee during the initial discussion. This approach allowed for a more targeted exploration of practical implications and potential challenges.

To understand the information search behaviour in an application scenario, the questionnaire asks for the information available at the start of a search (F3). Further, it is asked how the search query is constructed (F11), what is expected to be retrieved (F4, F5, F9), and how the result is presented (F15). Further questions check whether the retrieval is static or dynamic, and which kind of devices are likely used.

Based on the responses, several findings can be summarised.