Protecting Small Organizations from AI Bots with Logrip: Hierarchical IP Hashing

Abstract

Small organizations, start ups, and self-hosted servers face increasing strain from automated web crawlers and AI bots, whose online presence has increased dramatically in the past few years. Modern bots evade traditional throttling and can degrade server performance through sheer volume even when they are well-behaved. We introduce a novel security approach that leverages data visualization and hierarchical IP hashing to analyze server event logs, distinguishing human users from automated entities based on access patterns. By aggregating IP activity across subnet classes and applying statistical measures, our method detects coordinated bot activity and distributed crawling attacks that conventional tools fail to identify. Using a real world example we estimate that 80 to 95 percent of traffic originates from AI crawlers, underscoring the need for improved filtering mechanisms. Our approach enables small organizations to regulate automated traffic effectively, preserving public access while mitigating performance degradation.