Humans Cannot Detect AI-Generated Media But Communities May -- For Now: Collaborative AI Detection in r/RealOrAI on Reddit

Abstract

We study human AI-detection behaviour at scale using a year of activity from r/RealOrAI, a Reddit community where users collaboratively assess whether visual media is real or AI-generated. The community is moderated by a bot that solicits verified labels from submitters of self-challenging "[GUESS]" posts and publishes an aggregate community prediction for each post, yielding naturalistic ground truth at scale. Community detection accuracy reaches 72% on [GUESS] posts with a systematic false-positive bias that intensifies over the year as the community's AI-suspicion grows. Using a six-LLM ensemble validated against human-annotated ground truth, we classify 10k reasoning-bearing comments along six cues covering perceptual features, context, consistency, AI knowledge, subject-matter expertise and provenance (tracing the media to its source). Perceptual features (scene, visual artifacts, anatomy physics, lighting, behavior, text, audio) dominate reasoning (70%) while provenance verification is rarest (4%) at the individual level but is amplified 4.3x in community summaries, revealing aggregation as a reliability filter that selectively surfaces diagnostic evidence. These findings reveal the limits of heuristic-based detection and show how online communities collectively navigate an increasingly contested information environment.