Efficient Indexing of the BitTorrent Distributed Hash Table

Abstract

The following paper presents various methods and implementation techniques used to harvest metadata efficiently from a Kademlia Distributed Hashtable (DHT) as used in the BitTorrent P2P network to build an index of publicly available files in the BitTorrent ecosystem. The indexer design makes various tradeoffs between throughput and fairness towards other DHT nodes while also being scaleable in a distributed environment.