MULTI3IR: A Benchmark for Multi-perspective Multi-domain Multi-modal Information Retrieval
Seokwon Song, Sohyeon Kim, Gunhee Kim
Abstract
Information retrieval (IR) increasingly targets open-ended queries that admit diverse perspectives. Existing IR benchmarks, however, focus primarily on closed-ended queries, while even open-ended benchmarks largely consist of queries whose supporting documents span a single subject domain and modality. We introduce Multi3IR, a benchmark that evaluates how well retrievers cover the multifaceted perspectives of open-ended queries across diverse domains and modalities. It comprises 104.9K Stack Exchange queries, each annotated with perspective descriptions that capture the query's implicit viewpoints. We further propose SPIN, a parameter- and label-efficient method that learns noise vectors to steer embeddings toward diverse yet meaningful semantic directions. Experiments show that existing multimodal retrievers suffer from single-perspective bias, while SPIN substantially improves perspective coverage on Multi3IR and generalizes well to unseen open-ended IR benchmarks. The dataset and experimental code are available at https://github.com/seokwon99/Multi3IR.
Create a lesson
Related papers
Closed Forms and Synthetic Twins: Predicting Approximate Nearest Neighbor Recall from Embedding Statistics
Shmuel Herman
MUSES: A Benchmark for Prospective Intellectual-Roots Retrieval
Rohan Pandey, Sunjae Kwon, Hong Yu
Two-Sided State-Space Models for Sequential Recommendation with Non-Random Multimodal Review Feedback
Ziwen Pan, Zihan Liang, Ruoxuan Xiong
Learning from What You Retrieve: Online RL Fine-Tuning for Semantic Retrieval
Shaowei Wei, Chong Huang, Songtao Fang et al.
Generative Retrieval for E-commerce: Jointly Learning Embedding and Codebook with Same Product Cluster
Songtao Fang, Zihao Xu, Shaowei Wei et al.
Preference Shapes Relevance: Cross-component Hierarchical Semantic Alignment for Personalized Generative Retrieval
Gaoming Zhang, Angqing Jiang, Jianchun Song et al.