Currently, an increasing number of assistive technologies based on artificial intelligence (AI) with computer vision (CV) are under development. There have been accumulated specific research findings about the evaluation of these technologies, which can be categorized into two orientations: technology-driven and user demand-driven. Among them, the former dominates the mainstream, focusing on the performance and practical implementation of assistive technologies. Scholars in the field of computer science have constructed many benchmark datasets for vision-language models (Dai et al., 2023), and conducted the benchmark evaluation. Two key dimensions are involved in these evaluation metrics: automatic evaluation metrics, which principally include generated responses, Recall-Oriented Understudy for Gisting Evaluation (ROUGE), and BERT- based semantic textual similarity; human evaluation metrics which focus on the three main dimensions of correctness, actionability, and fluency (Yang et al., 2024; Zhao et al., 2024).

The user demand-driven evaluation approach focuses on the feedback and user experience from BVI persons. This feedback includes a rich set of metrics, including satisfaction (Rattanaphinyowanich & Nunta, 2021), accuracy, reliability, accessibility, privacy, security, compatibility, energy efficiency, usability (Bhagat et al., 2024), functionality, aesthetics characteristics and social acceptability (Hamilton et al., 2016; Phillips et al., 2018). In the field of library and information science, the information needs and information behavior of BVI persons have received particular attention (Berget & MacFarlane, 2020). Research has covered various topics, including the selection of information sources (Rahman et al., 2017; Chen et al., 2024), information-seeking behavior (Williamson et al., 2000), and interactions with information retrieval systems (Xie et al., 2021; Berget & MacFarlane, 2020). Among them, the accessibility and usability issues faced by BVI persons are the primary focus (Xie et al., 2020). These studies can be classified into two categories according to their research approaches: for one thing, examining established evaluation criteria, such as examining whether WCAG 2.2 effectively supports BVI persons’ access to digital libraries from the perspectives of stakeholders including users, experts, and developers (Xie et al., 2022). For another, conducting empirical research on the challenges and perspectives faced by BVI persons in interacting with information retrieval systems. The research findings show that while the library websites are accessible according to the extracted indices of W3C, empirical data from BVI users (i.e. successful task completion, working time, satisfaction level) suggests that the websites are not easy to use (Najafgholinejad, 2024).

With the development of AI technology, the focus of HCI research is shifting from human interaction with non-AI computing systems to human interaction with AI systems, thus giving rise to the cutting-edge topic of human-AI interaction (HAII) (Jiang et al., 2024). Researchers have conducted extensive and thorough explorations around human interactions with AI systems such as chatbots, voice assistants, virtual humans, autonomous vehicles, etc., with application scenarios covering shopping, healthcare, and transport sector, and more. As the two main elements in human-intelligent interaction, user characteristics and AI characteristics are the focus in the research of human-intelligent interaction experience. Among them, the inherent characteristics of individual users, such as gender identity, political ideology (Molina & Sundar, 2024), social role, user autonomy (Huh et al., 2023), motivation and social presence (Shao & Kwon, 2021), health condition (Esmaeilzadeh et al., 2021) can affect users’ perception and evaluation of AI, as well as their willingness to adopt. Users’ evaluation or perception of AI can be expressed in terms of ease of use (Loske & Klumpp, 2021), anthropomorphism (Pelau et al., 2021), etc. Characteristics of AI such as effects of explanation and synchronization (Fan et al., 2022), roles (Liao & Sundar, 2021) and communication models (Lew & Walther, 2023) also affect the experience of human-AI interaction. Furthermore, few studies have explored the role of task as a component in human-computer interaction. For example, people are less supportive of AI and its creators when AI performs highly (vs low) hedonic tasks (Yanit et al., 2023).

In summary, there is a relative scarcity of studies focusing on BVI users’ evaluation on assistive technologies based on AI with CV. Research in the HAII domain has focused on the influence of both humans and AI on the human-AI interaction experience, and has conducted preliminary exploration into the mechanisms between each element. However, there is still a relative lack of research on the interaction between BVI persons and AI, especially on HAII in visual tasks. To fill this research gap, this paper aims to address two specific research questions: What are the evaluation criteria of BVI persons towards GAI in visual tasks? What are the underlying mechanisms influencing their evaluation?

With the help of the Capital Library and China Association of persons with visual disabilities, we recruited 19 participants, and their characteristics are summarized in Table 1. Among them, minor participants will take part in the interviews accompanied by their parents. We ensured that all the participants gave their voluntary consent based on full understanding.

For the 12 participants who had never used AI before, we conducted one-on-one ‘Be My Eyes’ operation training for them at the Capital Library during March 2024. In the training, we collected behavioral data through participatory observation. After the training, we conducted interviews to gather participants’ perceptions and evaluations of Gen AI, as well as the factors influencing their views. For the 7 participants who have experience in using GAI, we employed a semi-structured interview approach for data collection, questions centred around: visual challenges in daily life; experiences using GAI for visual tasks; evaluations of GAI and the influencing factors.

This study follows the principle and requirement of grounded theory. Data collection and analysis were alternately reciprocal and continuously comparative. Data were organized and coded immediately after each technical training or interview. The participant recruitment ended when we reached theoretical saturation. After three coding stage-open coding, axial coding, and selective coding (Strauss & Corbin, 1990), we identified 16 categories, and corresponding 5 core categories. The 5 core categories include system, information, BVI user, context and evaluation. Based on the coding results, a theoretical model is proposed (Figure 1).

The research found that respondents’ evaluation on GAI’s performance in visual tasks primarily focused on accessibility, interactivity, and credibility. Specifically, this manifested as: (1) the degree of accessibility and operability of GAI at the physical level and its comprehensibility at the intellectual level. (2) The degree to which GAI can offer efficient and personalized information delivery and feedback interface. (3) The degree to which GAI is honest or truthful, and can be trusted. This is specifically manifested in GAI’s fairness and interpretability.

The system quality of GAI is the basic guarantee for providing visual information services to BVI persons. It is further categorized into ease of use, responsiveness and model performance.

Ease of use represents the technical threshold of GAI, which directly affects interviewees evaluation of accessibility. The accessibility of GAI increases as the download, registration, and operating procedures of GAI get simpler, and as the number of voice prompts it offers during the capture process increases. As Interviewee #01 noted, ‘it is difficult for us to take clear photos and the operation is difficult for us.’

Responsiveness refers to the efficiency and effectiveness of the system in responding to BVI persons requests, which greatly influences BVI persons evaluation of GAI’s interactivity. During training, most interviewees had a particularly negative impression of the interactivity of tools like Be My Eyes due to their long processing time for images and frequent response failures. interviewee#11, who has used multiple GAI tools, had a better interactive experience in using the Luomo Toolbox, noted ‘it is instantaneous and it only requires camera alignment to achieve immediate recognition.’

Model performance refers to the capacity of the visual-verbal model embedded in the AI tool to process visual information. This ability influences BVI persons evaluation of GAI’s credibility and interactivity through the quality of the generated information content.

The accuracy, objectivity, and comprehensiveness of information are essential for building trust in GAI among BVI persons. Additionally, the relevance of information influences their evaluation of the interactivity of GAI.

Accurate information is crucial for the safety and autonomy of BVI persons, as it forms the foundation for their reliance on GAI in decision-making. A model’s ability to accurately identify and describe visual content directly correlates with the level of trust it can foster among BVI persons. Most interviewees express concern over the prevalent issue of AI hallucinations, hesitant to rely on GAI for some critical and extremely accurate processing of visual data.

The objectivity of the information described by the model also influences users’ assessment of credibility. Interviewees #02 and #10 believe that AI is less susceptible to subjective feelings and cognitive limits when compared to humans, thus allowing for a more accurate and objective reflection of visual information. This perception leads them to view GAI as fair and inclusive.

Comprehensive information offers broad and complete coverage of the necessary or expected aspects of a problem or subject. Interviewees believe that the more detailed the GAI’s description of visual information, the clearer their panoramic understanding of the environment, which in turn enhances their evaluation of the credibility of GAI. Interestingly, the comprehensive information provided by AIGC can create valuable Information Encountering. For instance, interviewee #10 casually used Be My Eyes to take a photo of the front and from AI’s detailed description, he learned there was a bench ahead, allowing him to sit down and take a break. This unexpected help provided practical assistance and enhanced his trust in GAI.

The relevance of generated information content is manifested in its relevance to the situation and emotions of BVI persons. The greater the relevance of information, the better GAI can understand and respond to the personalized needs of BVI persons, resulting in higher user ratings of interactivity. For example, in a cooking scenario, interviewees only need GAI to quickly capture the names and expiration dates of spices. However, the comprehensiveness of AI-generated information can make it difficult for BVI persons to quickly access the required information, thus reducing interaction efficiency. In reading scenarios, both interviewee#7 and interviewee#2 agreed that the GAI’s voice ‘sounds like a customer service, devoid of any emotional expression in its reading’. This mechanized presentation provides a poor interactive experience for BVI persons.

User dimensions include their GAI use experience, familiarity with GAI, and personality characteristics. As users accumulate experience, BVI persons gradually become more familiar with the operation and functions of GAI, which in turn enhances their evaluation of accessibility and interactivity. In terms of personality, some interviewees consider themselves conservative and cautious, which leads to a natural distrust of new technologies such as GAI. On the other hand, interviewees with more positive and open-minded personalities tend to rate GAI with higher credibility.

Additionally, user dimensions can moderate the intensity and direction of the influence that system and information factors have on user evaluations. Users with prior experience, familiarity with GAI, and open-minded personality are more likely to be satisfied with the system and information provided by GAI, leading to more positive evaluations. For instance, experienced interviewees are generally accustomed to the response time of GAI, leading to higher evaluations of its interactivity.

The visual task, physical environment and social relationships experienced by BVI persons directly influence their assessment of GAI. Additionally, contextual factors can moderate the intensity and direction of the influence that system and information factors have on user evaluations.

In visual tasks that are urgent and important related to personal safety, such as reading hospital reports, medication information, or crossing the street, BVI persons may perceive that the responsiveness of GAI and the quality of the information it provides are insufficient to meet their need, leading to a decrease in their evaluation of its credibility and interactivity. However, in non-urgent and less important visual tasks, such as ordering food, checking nail designs, or picking up keys, BVI persons believe that using GAI reduces their reliance on and disruption to others, resulting in higher evaluations of its accessibility.

The physical environment refers to the objective conditions under which BVI persons encounter visual challenges, including sound environment, physical layouts, and network infrastructure. BVI persons primarily rely on auditory feedback to compensate for visual deficiencies. In noisy environment, they find GAI less accessible and interactive. The layout and design of the physical environment directly affect BVI persons’ interactions with GAI. Many interviewees indicated that if commonly used items and devices in their home or workplaces are in fixed position, they would be able to take more focused photos, thus facilitating efficient information transfer and feedback with GAI. The quality of network infrastructure impacts the responsiveness of GAI systems, which in turn impacts users’ perceptions of the accessibility and interactivity of Gen AI.

Social relationships, particularly strong ties, have a direct bearing on BVI persons evaluation of GAI’s accessibility. Different parenting styles exert vastly different influences. For instance, the parents of interviewee#15 hold a very open-minded attitude towards AI and even suggested creating an AI chat group for her child after training, enabling interviewee#15 to successfully access and use GAI. In contrast, parents of interviewee#16 adopted strict control over their children, prohibiting interviewee#16 from accessing and using ICT. Consequently, interviewee#16 perceives restricted access to GAI.