Was It Never Collected, or Rewritten Away? A Commit-Provenance Dataset Separating Ingestion Gaps from Upstream History Edits across the World of Code

Abstract

Any global mirror of open-source version control is incomplete, but the reasons a commit is missing are not interchangeable: a project may have force-pushed it away (so it no longer exists upstream), or the mirror may never have ingested it (a true collection gap). We release a commit-provenance dataset that separates these cases at scale by comparing two views of the same commit graph: the GHArchive event stream, a historical witness of what GitHub advertised at push time, and the World of Code (WoC) V2604 object database, an accumulated union of periodic fetches that never deletes a collected commit. Walking each reference's PushEvent chain reconstructs force-pushes structurally (a before that is not the prior head breaks the fast-forward chain; no recorded flag exists across eras), and joining every advertised commit against WoC membership yields a three-way label. Over 1,118,116,350 advertised commits, 53.35% are present in WoC, 6.47% are rewritten (orphaned by a later force-push, an upstream edit and a correct absence), and 40.18% are never-ingested (the candidate collection gap). About one missing-commit case in fifteen is a rewrite the project erased, not a mirror gap. We release the force-push witness (166,710,831 events over 19,926,250 repositories) and a per-project rollup (78,125,788 repositories; force-pushing observed in 25.47%, a rewritten commit in 12.85%). We treat the 40% never-ingested share as an upper bound on the collection gap and state five reasons it overcounts. The dataset makes mirror-completeness reporting honest, flags rewritten commits as duplicate patch sources for contribution counting, and corrects a 10.82% corpus-wide undercount that history editing imposes on commit-based productivity, with a released per-project correction factor. All artifacts ship as a self-contained, independently hosted replication package keyed to WoC V2604.