Mapping the Stochastic Penal Colony

Abstract

With peak content moderation seemingly behind us, this paper revisits its punitive side. But instead of focusing on who is being (disproportionately) moderated, it focuses on the punishment itself and explores the question of how content moderation treats users posting violative content unjustly, while the organizations doing the moderation act in a self-serving manner. First, this paper reworks Foucault's model of the penal system for the algorithmic age, restoring the penal colony as a figuratively liminal practice between punishment as performance and punishment as discipline, i.e., the stochastic penal colony. Second, it develops a novel methodology that combines auto-ethnography for collecting experiences and artifacts with procedural justice for analyzing them. Third, it applies this conceptual and methodological framing to three case studies, one on pre-Musk Twitter's gallingly performative moderation, one on OpenAI's exhaustively controlling moderation for DALL-E 2, and one on Pinterest's underhandedly manipulative moderation. While substantially different, all three feature the pervasive threat of account suspension, which banishes users to the stochastic penal colony.