The AI Era's Lines of Code? A Case Study on What an Engineer's Token Spend Measures
Shyam Agarwal, Olga Lvova, Agnia Sergeyuk, Anastasiia Serova, Timofey Bryksin, Thomas Zimmermann, Bogdan Vasilescu
Abstract
Organizations use engineers' token spend to monitor AI adoption, set budgets, and measure productivity, but the metric's relationship with engineering output is unclear. We link five weeks of token usage in coding agents to the commits of 404 engineers at JetBrains, and analyze a company survey of 412 employees. We find that token spend varied mostly between engineers, and which engineers spent the most stayed largely the same over the five weeks. Among weeks with similar commit counts, the 90th percentile of spend was 39 to 240 times the 10th. Within an engineer, doubling spend was associated with about 10% more commits. Available work metrics explained 38% of the variation in spend between engineers and 13% of the week-to-week variation within an engineer. In the survey, respondents in technical roles reported in nearly nine of ten agent evaluations using agents for tasks beyond coding, such as analysis, research, and planning. Token spend therefore measures how much engineers use agents across many kinds of work and not the outcome and value of that work. We recommend using token spend at most for resource accounting, and evaluating agent-assisted work with evidence about tasks and outcomes.
Create a lesson
Related papers
Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM Scoreboards
Ana E. Sanchez Villalobos, Michael D. Quinn, Aakash Tyagi et al.
Anticipating Future Build Results with BuildPredictor
Christian Macho, Katharina Stengg
Static Analysis Guidance for Secure LLM Code Generation
Daniele Lunghi, Alberick Euraste Djiré, Zacharie Chenail-Larcher et al.
SWE-Recreation: Benchmarking Coding Agents for Software Recreation
Zhiyao Xu, Zhanjie Ding, Hongyao Yu et al.
SkillBridge: Cost-Aware Diagnosis and Repair of Cross-OS Compatibility Defects in Agent Skills
Anjie Xu, Zixing Wang, Yifeng Cai et al.
Barriers to GenAI Integration During Large-Scale Agile Transformation: An Exploratory Case Study
Nils Brede Moe, Marthe Berntzen, Astri Barbala et al.