Vexed by VEX tools: Consistency evaluation of container vulnerability scanners

Abstract

The Vulnerability Exploitability eXchange (VEX) format has been introduced to complement Software Bill of Materials (SBOM) with security advisories of known vulnerabilities. VEX gives an accurate understanding of vulnerabilities found in the dependencies of third-party software, which is critical for secure software development and risk analysis. In this paper, we present a study that analyzes state-of-the-art VEX-generation tools (Trivy, Grype, DepScan, Scout, Snyk, OSV, Vexy) applied to containers. Our study examines how consistently different VEX-generation tools perform. By evaluating their performance across multiple datasets, we aim to gain insight into the overall maturity of the VEX-generation tool ecosystem, beyond any single implementation. We use the Jaccard and Tversky indices to produce similarity scores of tool results for three different datasets created from container images. Overall, our results show a low level of consistency among the tools, thus indicating a low level of maturity in the VEX tool space. We perform a number of experiments to explore the impact of different factors on the consistency of the results, with the difference in vulnerability databases queried showing the largest impact.