Interface Quilt: Modifying Applications with Computer Use Agents and Generative UI
Leping Qiu, Jo Vermeulen, Tim Zindulka, George Fitzmaurice, Fraser Anderson
Abstract
Modifying an existing application is difficult when it does not provide programmatic access for that change. Recent advances in computer-use agents (CUAs) enable richer understanding and operation of existing interfaces, while generative UI (GenUI) makes it practical to generate new ones. We examine how these advances expand software modification through a design space of what can be changed and a framework for how modifications can be implemented. We then introduce Interface Quilt, a system that combines CUAs and GenUI to generate interface patches, which are new interfaces displayed over an existing application to modify its presentation, interaction, and capabilities. Through demonstrations across a variety of applications, we show that interface patches support a range of modifications without source code, APIs, or accessibility data.
Create a lesson
Related papers
SEAM: Situated Executable Associative Memory for a Co-located Community
Keru Wang, Michael Xu, Ken Perlin et al.
OmniScribble: Mixed-Initiative Authoring of Animated Sketches through Inspectable and Repairable AI Decisions
Hien Trinh, Tan-Hiep To, Trung-Nghia Le
Agentic Path Maps: Exposing an Agent's Traversal Through the Garden of Forking Paths
Madeleine Grunde-McLaughlin, Hafsah Usman, Daniel Mitroshkov et al.
Must AI Spell It Out? Pragmatic Comprehension of Alerts in a Time-Critical Task
Anwesha Das, Vera Demberg, Anna Maria Feit
From Benchmarks to Behaviors: A Framework for Auditing AI Agents
Nick Hagar
Proceedings of the CSCW 2026 Workshop on Women's Health as Cooperative Work: Participatory AI Across Care, Community, and Policy
Sherilyn Francis