Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge
Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor, Ehsaneddin Asgari
Abstract
Retrieval-augmented generation (RAG) assumes that language models can reason over relations stated in their context rather than fall back on associations memorized during pretraining. We study this evidence-conditioned relational reasoning in isolation, with the facts supplied directly in the prompt rather than retrieved. Rel-Bloom is a controlled multilingual diagnostic of 450 multiple-choice questions built from 30 Wikidata relations; the same source relations are instantiated at five Bloom-inspired levels (Remember, Understand, Apply, Analyze, Evaluate) and under three veracity conditions (Factual, Counterfactual, Neutralized). Across 43 models, six languages, and two multilingual settings (226,800 predictions), three findings stand out. First, accuracy drops sharply from Apply to Analyze in every evaluated open-weight model series, reaching 47.2 percentage points for the Large Granite bucket, but not for either Gemini model. Second, Evaluate items remain a bottleneck even for models near ceiling on the other task types: Gemini~3.1 Pro averages 99.5\% on the other four levels but 60.7\% on Evaluate. Third, item- and relation-level difficulty rankings are consistently preserved across languages, indicating a difficulty structure partly shared across languages within this benchmark. Across knowledge conditions, Counterfactual contexts are generally harder than Factual ones, while removing entity identities can improve higher-level performance in some model regimes. Greater model capability therefore does not by itself ensure reliable reasoning over supplied relational evidence. Relevant resources are available at: https://huggingface.co/spaces/QCRI/non-parametric-memory-relational-reasoning.
Create a lesson
Related papers
Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes
Vladimir Beskorovainyi
Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety
Fazl Barez, Tung-Yu Wu, Hunar Batra et al.
MedGraph: Maintaining Patient State with Graph Memory for Long-Term Medical Dialogue
Donghua Zhang, Zizheng Wang, Ze Dong et al.
SpecCTMC: Exact Speculative Simulation of Diffusion Language Model Continuous-Time Markov Chains
Siva Rajesh Kasa, Yasong Dai, Sumit Negi et al.
Cultural Adaptation within Repeated Social Interactions
Chen Cecilia Liu, Roi Reichart, Anna Korhonen
Polarity Classification with Multitask Learning Syntax Augmentation
Muhammad Imran, Carlos Gómez-Rodríguez