Represented but misread: Post-training changes how language models read out what they represent
Jochen Laubrock
Abstract
Does post-training change what a language model represents, or how this representation is read out into an answer? We compared linear probes with the models' own answers, analyzed with signal detection theory, in six open-weight base models and their post-trained versions. For judgments of whether German texts were written by a human or an AI, probes and word n-gram classifiers separated the classes well, but the answers did not and often ordered the texts inversely. Post-training lowered the sensitivity of the answers on average (Δd' = -0.34 across model families; -0.29 across three prompts). It also changed the use of a stated base rate: Base models shifted their log-odds by less than the stated change in prior odds, most post-trained models by more. In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers, and from mid-depth on internal states at positions after the text, not those of the text, transferred the change between checkpoints. Our corpora cannot be shared, so we add a case reproducible from public data, the truth of negated statements. Here, post-training acted in the opposite direction: Probes decoded truth at or near ceiling in every model, the answers of several base models did not use it, and post-training repaired them; by mid-depth the repair resided at the positions after the statement; below it, the representation of the statement contributed, strongly in one of four models. In a training study, adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on our newspaper and textbook corpora and inverted them on the chats, where they followed a news-trained probe of the base model. Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction.
Create a lesson
Related papers
AutoMaster: Autonomous Post-Training by Making the Same Model a Better Teacher
Guoxin Chen, Shuang Sun, Fanzhe Meng et al.
Multimodal Error Detection in Chest X-ray Reports: A Large Language Model Reading Image Evidence
Yanda Cheng, Bo Wen, Yuhao Lu
MemFolder: Structure-Guided Memory Folding for Efficient Agent Memory Management
Xuyang Liu, Chenwen Lu, Junxi Wang et al.
Beam Model Technical Report
Reflection Team, Adam Abdulhamid, Osama Abuelsorour et al.
One Framework to Diagnose Them All: A Stratified Benchmark for Bengali LLM Hallucination
Asif Shahriyar Sushmit, Shefayat E Shams Adib, Abdullah Al Jubaer Gem et al.
Language-Oriented Semantic Modeling for Target Language Extraction
Mehmet Sinan Yıldırım, Ke Zhang, Shuai Wang et al.