InstructFX2FX: A Multi-turn Text-to-Preset Demo for Iterative Audio Effect Refinement

Abstract

We present InstructFX2FX, an interactive demonstration of multi-turn, text-guided audio effect refinement. Existing text-to-preset systems are largely single-shot, mapping one textual descriptor to one preset. Real audio engineering is instead sequential: engineers refine an existing effect chain through successive instructions such as "make it warmer" or "too harsh, soften it." This poses a stateful problem that single-shot systems do not address - given the current effect parameters and a new instruction, update the sound while preserving what earlier instructions already achieved. InstructFX2FX tackles this with a hybrid architecture that divides labor between a language model and CLAP-guided optimization. The LLM acts as a high-level planner that selects effects, orders the signal chain, and proposes the initial parameter state, leveraging recent evidence that LLMs can outperform CLAP-based optimization for single-turn text-to-parameter mapping; CLAP-guided optimization then refines the existing parameter state from this initialization, avoiding the parameter drift caused by re-prompting the LLM at each turn. At the demo, attendees drive a dry recording through successive natural-language instructions while staying in the loop: after each turn they audition the saved optimization checkpoints to choose how strongly an effect is applied, then issue the next instruction based on what still differs from the sound they intend. On SocialFX-derived descriptor transitions, CLAP-guided refinement lowers target-directed DSP-feature MMD on 9 of 10 directed descriptor pairs relative to an LLM-only reprompting baseline, with most refinement trajectories trending toward the intended target.