Background: Existing image description methods when used as Assistive Technologies often fall short in meeting the needs of blind or low vision (BLV) individuals. They tend to either compress all visual elements into brief captions, create disjointed sentences for each image region, or provide extensive descriptions.
Purpose: To address these limitations, we introduce VIIDA, a procedure aimed at the Visually Impaired which implements an Image Description Approach, focusing on webinar scenes. We also propose InViDe, an Inclusive Visual Description metric, a novel approach for evaluating image descriptions targeting BLV people.
Methods: We reviewed existing methods and developed VIIDA by integrating a multimodal Visual Question Answering model with Natural Language Processing (NLP) filters. A scene graph-based algorithm was then applied to structure final paragraphs. By employing NLP tools, InViDe conducts a multicriteria analysis based on accessibility standards and guidelines.
Results: Experiments statistically demonstrate that VIIDA generates descriptions closely aligned with image content as well as human-written linguistic features, and that suit BLV needs. InViDe offers valuable insights into the behaviour of the compared methods - among them, state-of-the-art methods based on Large Language Models - across diverse criteria.
Conclusion: VIIDA and InViDe emerge as efficient Assistive Technologies, combining Artificial Intelligence models and computational/mathematical techniques to generate and evaluate image descriptions for the visually impaired with low computational costs. This work is anticipated to inspire further research and application development in the domain of Assistive Technologies. Our codes are publicly available at: https://github.com/daniellf/VIIDA-and-InViDe.
Keywords: Image paragraph; assistive technologies; computer vision; image captioning; large language models; visually impaired.
Development of low-cost computational approaches for generating and automatically evaluating image descriptions based on accessibility standards and rules for people with visual impairments as Assistive Technology, thus increasing inclusion and reducing accessibility limitations.Extraction of semantic visual information and modelling of textual descriptions of images using current Computer Vision and Natural Language Processing models and techniques.The synthetic images generated from the paragraphs produced by our approaches closely resemble the original images in terms of semantic similarity and statistical distribution of features.As this work is one of the few studies in the area and is characterised by flexibility and interpretability, researchers can use the approaches presented here to produce new or improve existing Assistive Technologies for the visually impaired.