Symbolic Synthesis with Token-Level LLM Guidance
Derek Egolf, Stavros Tripakis
Abstract
We propose a synergistic combination of classical program synthesis techniques and contemporary large language models (LLMs) to solve the problem of recursive program synthesis from first-order logic specifications. Our approach leverages the strengths of both paradigms: we use LLMs as a source of statistical bias to guide the search for a correct program. At the same time we use techniques from classical synthesis to short-circuit on semantically incorrect prefixes and to avoid sampling semantically equivalent responses. The statistical bias is derived from the next-token distribution of an LLM, which is highly context-dependent. We operate with the understanding that (1) there is some class of synthesis instances that can be discharged by small models and symbolic tools, and (2) every other instance can be decomposed into these instances. Therefore, our intention is to expand the reach of small, open-weight LLMs and symbolic synthesis techniques, not to compete with proprietary frontier models on every possible instance. These are different tools for different jobs. In our experimental evaluation, we find that our method solves more benchmarks than either small LLMs or classical synthesis techniques alone. Our method solves 30/32 benchmarks using only a Qwen2.5 model with 32 billion parameters. Qwen2.5 solves 0 benchmarks on its own, and only solves a few additional benchmarks when combined with traditional constrained decoding techniques.
Create a lesson
Related papers
You Have Mail: Set-Theoretic Types for Actors
Giuseppe Castagna, Luca Padovani, Daniele Varacca et al.
Sightline: Calibrated Data-Flow Summaries for LLM-Integrated Applications
Zihao Xu, Yuekang Li, Ruitao Feng et al.
Tenon: A Unified Programming Framework for Retargetable PIM Compilation
Niansong Zhang, Shiran Guo, Zhiru Zhang
Mode Inference for Substructural Adjoint Types
Sophia Roshal, Frank Pfenning
A Principled Approach to Concurrency, Nondeterminism, and Asynchrony in Choreographies
Xueying Qin, Marco Peressotti, Fabrizio Montesi
Validating Concurrent Correctness Witnesses with Ghosts by Thread-Modular Abstract Interpretation
Michael Schwarz, Zhendong Ang, Umang Mathur