Evaluating the Capabilities of LLMs for Persuasive Dialogue
Jordan Robinson, Angus R. Williams, Katie Atkinson, Anthony G. Cohn
Abstract
Large language models (LLMs) can generate apparently highly persuasive text, but does sounding persuasive mean arguing well? We introduce Persuasio, a multi-agent dialogue platform grounded in a formal argumentation-based theory of persuasion dialogues that adjudicates logical winners during free-text debates. Using this system, we generated 192 debates on a UK political topic between humans and LLMs, and evaluated 22 interlocutors through both automated adjudication and 9,702 crowdsourced pairwise judgements across 1,386 annotation instances. We observed a consistent decoupling between subjective and formal persuasiveness: LLMs dominated the subjective ranking yet performed substantially worse under argumentation-theoretic adjudication, where humans remained competitive. Multi-agent and retrieval-augmented variants further widened this divergence. These findings reveal a systematic gap between rhetorical fluency and formal argumentative strength in LLM-based persuasive dialogues.
Create a lesson
Related papers
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Leon Bergen, Usha Bhalla, Andrew Lee et al.
Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation
Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny et al.
Long-Lived Characters, Local Inference: Incremental Memory Maintenance for Game NPCs
Zimu Xu
Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking
Xinshuai Guo, Junjie Wu, Dolly Deng et al.
How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards
Yanyi Pu, Damian A. Gonzalez-Salzberg, Zheng Yuan et al.
Structured Claim-Level Discourse Representations for Dense Health Narratives
Farnoushsadat Nilizadeh, Elham Pourabbas Vafa, Shirin Nilizadeh et al.