One Framework to Diagnose Them All: A Stratified Benchmark for Bengali LLM Hallucination
Asif Shahriyar Sushmit, Shefayat E Shams Adib, Abdullah Al Jubaer Gem, Ahmed Shafin Ruhan, MD. Nazmuddoha Ansary, Biplab Kumar Sarkar, Syed Rifat Raiyan, Ajwad Abrar, Abdullah Bin Hasan, Adib Sakhawat, M. Sakeef Hossain, Ahmed Alfey Sani, Arefin Mizan, Sourab Chakma, Quazi Adibur Rahman Adib, Wassie Haque, Sanjay Kumar Sen, Saad Tahmid, Mohammad Mamun Or Rashid
Abstract
Hallucination research in low-resource languages fragments easily: each newly named failure mode prompts its own dataset, spending scarce annotators on incomparable resources. We argue for consolidation and test it at community scale. Alikbachan is a Bengali hallucination-detection benchmark stratified along three crossed axes: a human-validated cultural-distance band, six task types, and a five-family, 32-mode hallucination taxonomy. It comprises a harvested 2,516-item test set (Phase 1) and a quota-constructed, gold-labeled 5,000-item set (Phase 2). The stratification is diagnostic. Four zero-shot LLM judges vary by over 30 macro-F1 points across strata, and the item-level predictions of 99 Phase-1 and 32 Phase-2 competition systems reproduce the profile: discourse-level omission defeats judges and most teams alike, whereas only a minority over-flag negative controls. The judges' verdict probabilities rank items well (ROC-AUC 0.74-0.95) but are overconfident (ECE 0.04-0.29). An open competition on Alikbachan drew 222 teams and 3,566 submissions, and a single-prompt 32B judge reaches the 57th percentile. Of the 25 teams in both phases, 24 regress on Phase 2 (mean -17.6 points) while every zero-shot judge improves, and Phase-1 rank barely predicts Phase-2 rank (Spearman ρ=0.35), so a Phase-1 leaderboard alone would have crowned a different winner.
Create a lesson
Related papers
Represented but misread: Post-training changes how language models read out what they represent
Jochen Laubrock
AutoMaster: Autonomous Post-Training by Making the Same Model a Better Teacher
Guoxin Chen, Shuang Sun, Fanzhe Meng et al.
Multimodal Error Detection in Chest X-ray Reports: A Large Language Model Reading Image Evidence
Yanda Cheng, Bo Wen, Yuhao Lu
MemFolder: Structure-Guided Memory Folding for Efficient Agent Memory Management
Xuyang Liu, Chenwen Lu, Junxi Wang et al.
Beam Model Technical Report
Reflection Team, Adam Abdulhamid, Osama Abuelsorour et al.
Language-Oriented Semantic Modeling for Target Language Extraction
Mehmet Sinan Yıldırım, Ke Zhang, Shuai Wang et al.