Design of a Parallel and Distributed Web Search Engine
Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri
Abstract
This paper describes the architecture of MOSE (My Own Search Engine), a scalable parallel and distributed engine for searching the web. MOSE was specifically designed to efficiently exploit affordable parallel architectures, such as clusters of workstations. Its modular and scalable architecture can easily be tuned to fulfill the bandwidth requirements of the application at hand. Both task-parallel and data-parallel approaches are exploited within MOSE in order to increase the throughput and efficiently use communication, storing and computational resources. We used a collection of html documents as a benchmark, and conducted preliminary experiments on a cluster of three SMP Linux PCs.
Create a lesson
Related papers
XViewRank: Overlap-Conditioned Score Fusion for LLM-Augmented Sparse Retrieval
Xinxuan Lv, Minghan Li, Guodong Zhou
Reading the Neighborhood: Text-Attentive Graph Reranking for Automated Subject Indexing
Lakshmi Rajendram Bashyam, Ralf Krestel
Twenty Days Later: A Pre-Registered Test of History-Based Forecasts of Brand Mentions in Gemini Answers
Dmitrij Żatuchin
Bridging the Topological Divide: Dual-Branch Graph Distillation with Semantic Priors for Robust Cold-Start Recommendation
Yuanfeng Qiu
RCER: Reconstructing Cross-Evidence Relations for Iterative Retrieval-Augmented Generation
Linyao Zheng, Xuhang Shi, Zhifang Mao et al.
Revisiting Generative vs. Dense Retrieval: A Static Value Prior Reverses the Gap
Renbo Tu, Yang Liu, Shuying Sun