DOE OSTI · 2538317
VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images
Abstract
Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs). We ask if pre-trained VLMs can aid scientists in answering a range of biologically relevant questions without any additional fine-tuning. In this paper, we evaluate the effectiveness of 12 state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel dataset, VLM4Bio, consisting of 469K question8 answer pairs involving 30K images from three groups of organisms: fishes, birds, and butterflies, covering five biologically relevant tasks. We also explore the effects of applying prompting techniques and tests for reasoning hallucination on the performance of VLMs, shedding new light on the capabilities of current SOTA VLMs in answering biologically relevant questions using images
Keep this discovery
Explore connections, maps & timelines
Maruf, M [Virginia Tech, Blacksburg], Daw, Arka [ORNL] (ORCID:0009000633191271), Mehrab, Kazi [Virginia Tech, Blacksburg], Manogaran, Harish Babu [Virginia Tech, Blacksburg], Neog, Abhilash [Virginia Tech, Blacksburg], Sawhney, Medha [Virginia Tech, Blacksburg], Khurana, Mridul [Virginia Tech, Blacksburg], Balhoff, James [University of North Carolina], Bakis, Yasin [Tulane University], Altintas, Bahadir [Tulane University], Thompson, Matthew [Ohio State University], Campolongo, Elizabeth [Ohio State University], Uyeda, Josef [Virginia Tech, Blacksburg], Lapp, Hilmar [Duke University], Bart Jr., Henry L [Tulane University], Mabee, Paula [National Ecological Observatory Network, Battelle; Dept. of Atmospheric and Oceanic Sciences, Univer], Su, Yu [Ohio State University], Chao, Wei-Lun [Ohio State University], Stewart Jr, Charles N [ORNL], Berger-Wolf, Tanya [The Ohio State University], Dahdul, Wasila [University of California Irvine], Karpatne, Anuj [Virginia Tech, Blacksburg]. 2024-12-01. VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images. https://www.osti.gov/biblio/2538317
Cite the original work for its findings. Save a collection to share your selection of sources.