LiveGeo: Benchmarking Geographic Agents in Continuously Evolving Worlds
Pengcheng Zhou, Xuanyu Liu, Yanchen Yin, Tianjie Ju, Bobo Li, Shengqiong Wu, Hao Fei, Mong-Li Lee, Wynne Hsu
Abstract
We introduce GeoWorld, a controllable and continuously evolving geographic environment platform for LLM agents, and LiveGeo, a benchmark for evaluating reliable decision-making in real-world geographic environments using GeoWorld. Unlike existing evaluations that largely reason and plan over fixed geographic states, GeoWorld is grounded in real-world geographic structure and allows external world states to evolve along their own temporal processes while preserving verifiable state histories. LiveGeo places geographic tasks within this evolving environment, requiring agents to actively acquire task-relevant information, verify whether the evidence supporting prior decisions remains valid, and revise their plans when necessary. Our results reveal that even frontier LLMs struggle with static tasks, with no model consistently excelling across tasks, while strong static performance does not ensure dynamic reliability. Further analysis identifies a key bottleneck: agents struggle to acquire and translate relevant evidence into valid decisions and maintain their correctness as the world evolves. By releasing LiveGeo together with its GeoWorld platform, we aim to provide the community with a flexible and extensible infrastructure for studying the next generation of agents capable of reliable action in evolving geographic worlds.
Create a lesson
Related papers
PlayCraft: Designing Engaging Games from Agent Play Experience
Weihao Cheng
Probing the "Conscious" Mind: Selective Access and Global Workspace in Vision-Language Models
Jiahao Huang, Jun Wang
ProtoDiffusion: Token-per-Layer Generative Modeling of Multi-Layer Images
Adrià Ciurana Lanau, Harleen Hanspal, Lusine Abrahamyan et al.
MM-SystemOneBias: Evaluating Bias in Multimodal Jev-Style System One Decision Models
Jinzhe Li, Qi Wang, Dandan Guo et al.
The Geometry of Lies: Knowledge and Persona Directions for Detecting Deception in Large Language Models
Md. Abdullah Al Zami, Suraiya Shanaz, Mahdi Hasan et al.
LLM GerryMandering: Separating Interactions from Additive Effects in LLM Audits
George Rahul, Ronit Jahilal, Bhaskara Hanuma Vedula et al.