In recent years, vision-language models (VLMs)<\/strong> have exploded in popularity. From CLIP<\/a>, LLaVa<\/a> and more recently PaLI-Gemma<\/a>, the field has moved towards models capable of processing both images and text. These systems promise powerful capabilities across domains \u2014 from education and accessibility to robotics and document processing. But as this line of research advances, there is one question that we need to keep in mind:<\/p>\n<\/blockquote>\n\n\n\n How do we evaluate these models?<\/strong><\/p>\n\n\n\n Different VLMs have different strengths. Some are trained primarily on natural images and captions, while others are fine-tuned on documents, charts, or even diagrams. Architectural choices \u2014 like whether a model uses a transformer-based image encoder or relies on region features \u2014 can have a huge impact on performance for specific tasks.<\/p>\n\n\n\n This means there is no one-size-fits-all solution. Depending on the application \u2014 say, answering questions about a scientific diagram vs. interpreting a business chart \u2014 the best model can vary significantly. That\u2019s why rigorous, thoughtful evaluation<\/strong> is essential.<\/p>\n\n\n\n So, how are VLMs evaluated today? The short answer: mostly through benchmark datasets<\/strong>. These datasets pose questions or tasks for the model to solve, typically structured as question-answer pairs. Some of the most widely used include:<\/p>\n\n\n\n These benchmarks offer a solid foundation for comparison \u2014 but they\u2019re often limited to multiple choice questions, which deviates from the real use-cases.<\/p>\n\n\n\n While benchmark datasets are useful, current evaluation methods lean heavily on multiple-choice question answering<\/strong> and image captioning<\/strong>. This leads to several core challenges:<\/p>\n\n\n\n Multiple-choice questions offer clear metrics (accuracy), but they\u2019re not representative of real-world usage. When you ask a model, \u201cWhat does this chart say about sales in Q4?\u201d you don\u2019t provide four possible answers \u2014 you expect an open-ended, context-aware response.<\/p>\n\n\n\n Moreover, multiple-choice setups can mask weaknesses<\/strong>. Models may exploit statistical patterns or biases in the dataset without truly understanding the content \u2014 problems that have long plagued datasets like SNLI<\/a> and MMLU<\/a>.<\/p>\n\n\n\n Many VLMs can recognize entities in individual modalities \u2014 text in a document, or objects in an image. But connecting information across modalities remains a major challenge<\/strong><\/a>. This is particularly apparent in tasks like visual multimodal entity linking. In our European Project LUMINOUS<\/a>, we are actively working on manners to evaluate and improve this aspect. For example, we have created a dataset (MATE) where we probe VLMs ability to perform simple linking tasks which require understanding both visual and textual modalities.<\/p>\n\n\n\n In the chart below, you can see that even SoTA VLMs struggle to link between modalities as the number of objects increases, despite human performance being level across the number of objects. Strikingly, this suggests that VLMs are currently not able to correctly combine information they have available in separate modalities.<\/p>\n\n\n\n Real users often ask vague or underspecified questions. Yet our benchmarks rarely test models in these scenarios. Can the model handle ambiguous queries? Can it ask clarifying questions or make reasonable assumptions based on context? These nuances are largely missing from current evaluations.<\/p>\n\n\n\n To truly assess multimodal models, we need to move beyond rigid benchmarks and into more realistic evaluation scenarios<\/strong>. Some promising directions include:<\/p>\n\n\n\n Benchmarking VLMs is a necessary first step \u2014 but it\u2019s only the beginning. As these models move into more domains and user-facing applications, evaluation must evolve to keep pace with reality<\/strong>. We need tools that test not just accuracy, but adaptability, reasoning, and robustness across modalities. Most importantly, we need to be sure that we understand what we are evaluating.<\/p>\n","protected":false},"excerpt":{"rendered":" Evaluating Multimodal Models: Are Our Benchmarks Enough? In recent years, vision-language models (VLMs) have exploded in popularity. From CLIP, LLaVa and more recently PaLI-Gemma, the field has moved towards models capable of processing both images and text. These systems promise powerful capabilities across domains \u2014 from education and accessibility to robotics and document processing. But as this line of research advances, […]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":696,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-786","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/pages\/786","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/comments?post=786"}],"version-history":[{"count":4,"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/pages\/786\/revisions"}],"predecessor-version":[{"id":870,"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/pages\/786\/revisions\/870"}],"up":[{"embeddable":true,"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/pages\/696"}],"wp:attachment":[{"href":"https:\/\/luminous-horizon.eu\/index.php\/wp-json\/wp\/v2\/media?parent=786"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Why Evaluation Matters<\/h1>\n\n\n\n
The Current State of Evaluation<\/h1>\n\n\n\n
\n
Link<\/a> \u2014 760K questions, generally short, one-word answers.<\/li>\n\n\n\n
Link<\/a> \u2014 Questions about text in documents (50K examples).<\/li>\n\n\n\n
Link<\/a> \u2014 Understanding and reasoning over data visualizations.<\/li>\n\n\n\n
Link<\/a> \u2014 15K multiple-choice questions on diagrams.<\/li>\n\n\n\n
Link<\/a> \u2014 45K questions focused on reading text in images.<\/li>\n\n\n\n
Website<\/a> \u2014 A multi-domain, expert-level dataset with 11.5K questions.<\/li>\n\n\n\n
Link<\/a> \u2014 Visual math problems with diagrammatic reasoning.<\/li>\n\n\n\n
Link<\/a> \u2014 3K vision-language questions with multiple-choice answers.<\/li>\n<\/ul>\n\n\n\nThe Limitations We Need to Address<\/h1>\n\n\n\n
1. Artificial Evaluation Settings<\/strong><\/h1>\n\n\n\n
2. Weakness in Multimodal Reasoning<\/strong><\/h1>\n\n\n\n
![\"\"](\"http:\/\/luminous-horizon.eu\/wp-content\/uploads\/2025\/07\/Page01.webp\")
![\"\"](\"http:\/\/luminous-horizon.eu\/wp-content\/uploads\/2025\/07\/Page01_02.webp\")
3. Ambiguity in Natural Input<\/strong><\/h1>\n\n\n\n
Where Do We Go From Here?<\/h1>\n\n\n\n
\n
Final Thoughts<\/h1>\n\n\n\n