Same Text, Different Filename: Are LLM Judges Biased?

Abstract

Yes. In 2026, leading economic journals started using Refine.ink in the technical quality assessment of submissions before publication. I study whether renaming a file changes the perception of the document's quality by LLM judges. I find that three frontier LLMs score the same document differently if you change the document's filename. A person-related label scores higher than no label; there is no consistent preference of own- or rival-based label, nor a penalty for an Asian name, but there is a context-specific bias in assessments whose direction and size vary with both document and label. With LLM-written documents, some self-premium can be detected, but a significant context-dependent label bias remains. Whether to combat it is a question of the costs of evaluation errors.