The Real-Time Data Processor Framework for Data Handling and Analysis of High-Energy Instruments

Abstract

We implemented a real-time data processor (rta-dp) framework that can be used to develop real-time analysis pipelines and data handling systems to manage high-throughput data streams with distributed applications in the context of ground and space astrophysical projects and high-energy instruments. The rta-dp is based on the ZeroMQ in-memory communication framework to receive input data, share data between distributed processes, and send or receive commands and pipeline configuration. The rta-dp framework has a flexible architecture that allows the implementation of distributed analysis systems customized to the requirements of several scenarios. The rta-dp framework also provides monitoring capabilities for the running processes and sends housekeeping, logging, alarms, and informative messages that a monitoring process can acquire. We are using the rta-dp in several contexts, such as acquiring and processing data from X-ray detectors to the data quality system of the ASTRI Project, as well as reprocessing and archiving data.