A Conditional Mutual Information Analysis of Response Variety in Language Models
R. I. Cukier
Abstract
Large language models provide a variety of responses at inference (generation) that conventionally depend on a temperature parameter. Using a conditional mutual information approach, a quantitative, entropy based analysis of response variety is given. Three limiting behaviors are defined: arbiter, independent (conditionally) and neutral (deterministic) that span no to maximal mutual information. We give examples of each behavior and suggest their application to: temperature analysis, language analysis, prompt compression, and knowledge graphs.
Create a lesson
Related papers
COMPREHEND: Ultra-Fine-Grained Benchmark Understanding by Entailment Hierarchy
Sivaraaman Balakrishnan, Prakriti Shetty, Jeremiah Hong et al.
Constructing a Regional Slang Lexicon for Electricity Demands via Soft-Label Bidirectional Anchoring and Active Learning
Rong Chang, Lijing Shi, Zhaoxiang Zang
Shared Context Candidate Scoring with Jointly Adapted Readout Networks
Jingyang Du, Yiben Fu, Yang Shen
Tokenizer Myths and Where To Find Them
Rylan Schaeffer, Sanmi Koyejo
Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
Yinuo Liu
PACT: Certifying What Is Written in Parallel Decoding of Diffusion Language Models
Yanchao Li, Yusu Zhang, Jiaqing Xie et al.