Big Science with a Small Budget: Non-Embarrassingly Parallel Applications in a Non-Dedicated Network of Workstations
Angel de Vicente, Nayra Rodriguez
Abstract
Many astronomers and astrophysicists require large computing resources for their research, which are usually obtained via dedicated (and expensive) parallel machines. Depending on the type of the problem to be solved, an alternative solution can be provided by creating dynamically a computer cluster out of non-dedicated workstations using the Condor High Throughput Computing System and the Master-Worker (MW) framework. As an example of this we show in this paper how a radiative transfer application previously coded with MPI is solved using this solution without the need for dedicated machines.
Create a lesson
Related papers
PixelFlow: Token-Level Workload Management for Efficient Distributed DiT Serving
Zhexiang Zhang, Minchen Yu, Yifan Sun et al.
A Kubernetes-Native Request Router for Quality-Aware Inference Serving in the Computing Continuum
Ignjat Karanovic, Pantelis A. Frangoudis, Ivan Čilić et al.
Accelerating Sharded Data Parallelism at Scale with Federated Learning
Gianluca Mittone, Marco Aldinucci
Distributed Edge Inference: an Experimental Study on Multiview Detection
Gianluca Mittone, Giulio Malenza, Marco Aldinucci et al.
P-GADMM: Parallel Group-Based ADMM for Asynchronous Optimization in Heterogeneous Edge Networks
Gaiguo Wei, Qingying Zhang, Heqiang Wang et al.
VERA: Reinforcement Learning for Dynamic Memory Scaling of HPC Workloads in Kubernetes
Ade Pramono, Jie Ren, Ivy Peng