NTT establishes explainable multimodal XAI to reveal reasoning rationales

NTT has announced the establishment of a new multimodal XAI (eXplainable AI) technology capable of explaining its reasoning rationale, answering the fundamental question of "why it arrived at that specific result." The company states that this achievement will strengthen trust between humans and AI, as well as between different AI entities, enabling reliable business decision-making and cooperation among AI agents.

Multimodal AI is capable of simultaneously integrating and processing multiple data formats, such as text, images, audio, and video. Recent years have seen significant progress in the development of Large Vision-Language Models (LVLMs), which integrate Large Language Models (LLMs) with pre-trained image encoders, unlocking highly sophisticated multimodal reasoning capabilities.

By allowing direct image inputs in addition to text, these models are increasingly utilized as foundations for complex, multimodal reasoning tasks, such as video analysis and document reading, that were previously difficult to resolve through text alone.

Similar to text-only LLMs, Chain-of-Thought (CoT) reasoning, a step-by-step thinking process, has been considered highly effective for improving reasoning performance and providing explainable outcomes in LVLMs. This approach works by generating an intermediate "reasoning rationale" from the visual information and text input, and then adding that rationale into the input sequence to derive the final answer.

However, existing CoT mechanisms input the image and the rationale as a single sequence to generate the final output. Consequently, they lack a causal structure that strictly forces the model to use the information contained within the generated rationale in order to produce outputs, leaving the actual utilization of that rationale entirely up to the model itself. In short, there has been no guarantee that CoT-driven final outputs are actually based on the content of the rationale.

In fact, through rigorous experimentation and analysis, NTT revealed that existing LVLMs often generate final answers while completely ignoring the reasoning rationales they produced during multimodal CoT processes. For example, even if the reasoning rationale was swapped out for something completely irrelevant to the question, the model's final output would sometimes remain unchanged.

To overcome this critical challenge, NTT introduced a theoretical framework that maximizes the utilization of the rationale while preserving the image data. Through this framework, they established "Rationale-Enhanced Decoding," a technology that forces any given LVLM's output to depend faithfully on both the image and the rationale during inference.

Unlike standard inference, this technology splits the reasoning process into an image-based inference and a rationale-based inference, combining them through strategic weighting. When the LVLM predicts the next token, the probability distribution is separated into one conditioned on the image and another conditioned on the reasoning rationale. By multiplying these distinct distributions, the model harmonizes the information extracted from both sources to output its final answer.

This approach enables the AI to yield answers by faithfully capitalizing on the information obtained from both the image and the rationale. Because the image and the rationale are fed into the LVLM separately, the system can guarantee that the rationale is actively utilized.

By leveraging this technology, previously black-box LVLMs can now be operated as explainable AI (XAI) without incurring any additional training costs. The technology is expected to be applied across a wide range of use cases demanding high reliability, including corporate decision-making and complex problem-solving via interconnected AI agents.

NTT presented these achievements at CVPR (Computer Vision and Pattern Recognition) 2026, the premier international conference in the field of computer vision, held in Denver, USA, from June 3 to June 7, 2026.