Evaluating Endpoint Detection Robustness Against Genetic Algorithm Driven Code Transformations

Abstract

Post-compromise test variants are widely used in controlled security evaluation and endpoint robustness benchmarking. However, modern Antivirus (AV) and Endpoint Detection and Response (EDR) systems increasingly combine signature- and behavior-based detection, challenging the reliability of conventional detection pipelines under adaptive variation. This study introduces ShellForge, a Genetic Algorithm (GA)-driven framework that evolves post-compromise variants representative of remote command execution to generate functionally equivalent variants for systematic detection evaluation. ShellForge applies syntactic transformations, encoding schemes, and structural permutations guided by a multi-objective fitness function informed by AV and EDR detection feedback. We compare ShellForge against representative baseline transformation frameworks under identical sandbox configurations. Our findings highlight measurable robustness gaps in baseline signature- and behavior-oriented detection pipelines under controlled variant generation. In addition, we propose a reproducible benchmark for endpoint detection robustness evaluation, motivating the need for robustness-aware defensive monitoring and behavioral correlation.