Data Curation and Model Souping: Unlocking Qwen3-0.6B for MS MARCO Passage Ranking
Shuguang Han
Abstract
This report studies passage retrieval and re-ranking on the MS MARCO benchmark using lightweight Qwen3-0.6B models. Across three experimental settings --- BM25-then-reranking task, corpus-wide dense retrieval task, and dense retrieval then re-ranking task --- the results demonstrate that competitive ranking performance can be achieved without complex architectural modifications, custom loss functions, or cross-encoder distillation. Instead, substantial gains stem from data-centric strategies: corpus-targeted continued pre-training, rigorous data curation, hard-negative mining, training--inference distribution alignment, and weight merging via Model Soup. In particular, Model Soup delivers marked performance gains while preserving the inference efficiency of a single model. These findings highlight how data curation and weight merging can push sub-billion-parameter models to state-of-the-art performance.
Create a lesson
Related papers
Personalized LLM Recommendation with Semantic IDs: Autoregressive Beam Search vs. Multi-Token Prediction
Sejoon Oh, Fernando Amat Gil, Evan Smothers et al.
KhyontekAI at CheckThat! 2026: Multilingual Dense Retrieval and Cross-Encoder Reranking for Scientific Web Claims
Bhargab Jyoti Bhuyan, Pritam Deka
Local Citation Recommendation Without Graph Signals
Benjamin Basseri, Pavlos Protopapas, Kemeng Zhang et al.
Rethinking Evidence Modeling: Context-Coupled Graph Learning for Multimodal Recommendation
Mingqiao Zhang, Qiyao Peng, Yumeng Wang et al.
Pathwise Alignment for Collaborative Filtering: Bilateral Conditional Flow Regularization
Zhiwei Li, Guodong Long, Chengqi Zhang et al.
Federated Semantic Residuals for Multimodal Recommendation
Zhiwei Li, Guodong Long, Chengqi Zhang et al.