DRS-OSS: A Diff-Risk Scoring Tool for Continuous Integration Workflows

Abstract

Software teams need change-risk scores that can guide continuous integration decisions such as review prioritization, test scheduling, and downstream validation before risky changes are merged or released. However, open-source teams often lack deployable tools for surfacing these risk signals in everyday CI workflows. We present DRS-OSS, an open-source diff-risk scoring tool for continuous integration workflows. DRS-OSS is designed as a deployable and customizable pipeline rather than as a standalone prediction model. It combines a REST API gateway, containerized model services, a developer dashboard, GitHub integration, and a replication package that lets users retrain or replace the backend with other transformer models. The bundled workflow combines commit messages, commit diffs, and change metrics in a single risk-prediction pipeline. The default packaged backend uses a Llama 3.1 8B sequence classifier configured for long diffs. Its training recipe uses parameter-efficient tuning, quantization, CPU offloading, and customization helper scripts so that it can be adapted on modest hardware. We compare DRS-OSS with similar tools and evaluate the bundled classifier on ApacheJIT, where it reaches an ROC-AUC of 0.895 and outperforms prior baselines. From a user-feedback perspective, DRS-OSS has received interest from Uber, Duolingo, and Microsoft in adapting the workflow to their own continuous integration settings. The full tool is released with source code, customization scripts, deployment artifacts, a public repository, a live demo at worldofcode.org/drs, and a demonstration video at youtube.com/watch?v=2FzeRRdNaco.