🔥[2025-05-24] The evaluation of OCR-Reasoning is now supported in VLMEvalKit.🚀

🔥[2025-05-22] Our paper is now accessible at arXiv.🚀

🔥[2025-05-18] Introducing OCR-Reasoning. Release the Data and evaluation script. 🚀

Recent advancements in multimodal slow-thinking systems have demonstrated remarkable performance across diverse visual reasoning tasks. However, their capabilities in text-rich image reasoning tasks remain understudied due to the lack of a systematic benchmark. To address this gap, we propose OCR-Reasoning, a comprehensive benchmark designed to systematically assess Multimodal Large Language Models on text-rich image reasoning tasks. The benchmark comprises 1,069 human-annotated examples spanning 6 core reasoning abilities and 18 practical reasoning tasks in text-rich visual scenarios. Furthermore, unlike other text-rich image understanding benchmarks that only annotate the final answers, OCR-Reasoning also annotates the reasoning process simultaneously. With the annotated reasoning process and the final answers, OCR-Reasoning evaluates not only the final answers generated by models but also their reasoning processes, enabling a holistic analysis of their problem-solving abilities. Leveraging this benchmark, we conducted a comprehensive evaluation of state-of-the-art MLLMs. Our results demonstrate the limitations of existing methodologies. Notably, even state-of-the-art MLLMs exhibit substantial difficulties, with none achieving accuracy surpassing 50% across OCR-Reasoning, indicating that the challenges of text-rich image reasoning are an urgent issue to be addressed.

We introduce OCR-Reasoning, a novel benchmark specifically designed to evaluate the text-rich image reasoning skills of Multimodal Large Language Models (MLLMs). Specifically, OCR-Reasoning comprises a meticulously collected 1,069 human-annotated examples spanning 6 core reasoning abilities and 18 practical reasoning tasks commonly encountered in text-rich visual contexts. Furthermore, unlike other text-rich image understanding benchmarks that only annotate the final answers, OCR-Reasoning simultaneously annotates the reasoning process and the answers. This comprehensive annotation enables deeper insights into the problem-solving strategies employed by state-of-the-art models.

OCR-Reasoning is designed to measure six reasoning skills of MLLMs: Spatial Reasoning, Numerical Analysis, Logical Reasoning, Mathematical Reasoning, Multidisciplinary Knowledge, and Enumerative Reasoning.

Through a simple comparison with existing datasets, we observe that in most cases the answers in existing datasets are directly present in the images, whereas our benchmark contains very few samples of this type. This implies that in our benchmark, to obtain the answer, the model needs to engage in reasoning rather than extracting it from the OCR results of the image.