FAST-HEP: Compiling Declarative Analysis Workflows for High-Energy Physics and Beyond

Abstract

High-energy physics analyses increasingly rely on complex software workflows whose scientific lifetime often exceeds that of the underlying software ecosystem. Maintaining reproducibility while accommodating evolving analysis software, data formats, and execution environments therefore remains a significant challenge. These challenges are not unique to high-energy physics and are shared by many data-intensive scientific analyses. We present FAST-HEP and its workflow engine, Flow, which combines a declarative workflow language, compiler, and runtime. Flow separates the scientific description of a workflow from its implementation and execution, and compiles workflows into backend-independent execution plans through normalization, graph construction, dependency analysis, and execution planning. A common runtime then orchestrates the resulting plan using replaceable capabilities. This architecture enables static validation and modular replacement of analysis operations, execution backends, and storage technologies, while recording provenance throughout compilation and execution. Although developed for the requirements of high-energy physics analysis, Flow's workflow model and orchestration layer are domain-independent. By applying compiler techniques to scientific analysis workflows, FAST-HEP provides a foundation for workflows that are transparent, extensible, portable, and reproducible, allowing scientific analyses and their supporting software ecosystems to evolve independently.