Adversarial Machine Learning Against Security Detection Systems

Abstract. Machine learning has become a standard component of security detection, from malware classification to user behaviour analytics. Unlike most application domains, security ML operates against an adversary who is aware of the model, motivated to defeat it, and able to observe its decisions. This paper examines the resulting attack surface across the model lifecycle (evasion at inference, poisoning at training, and extraction of the model itself) and argues that the security-specific characteristics of the domain, particularly the extreme base-rate asymmetry and the entanglement of concept drift with adversarial adaptation, make several defences that succeed in other domains ineffective here. We conclude that the appropriate design objective is not a robust classifier but an architecture in which the classifier is one signal among several, positioned so that its failure is survivable.

1. Introduction

The case for machine learning in detection is straightforward. Signature-based approaches enumerate known-bad artefacts and fail against anything novel; the volume and variability of modern threats exceed what manual rule-writing can cover. Learned models generalise from examples and can, in principle, recognise variants never explicitly enumerated.

What distinguishes this application from image recognition or demand forecasting is that the data-generating process is partly controlled by an adversary with an interest in the model’s failure. In conventional machine learning the assumption that training and deployment data are drawn from the same distribution is a technical convenience that is approximately true. In security it is false by construction, and the divergence is not accidental but engineered.

This paper surveys the resulting attack surface and evaluates defences against the criterion that matters operationally: not whether they raise accuracy on a benchmark, but whether they remain effective once the attacker knows they are deployed.

2. Threat model

Attacks on machine-learning systems are conventionally characterised along three axes, and the security domain occupies distinctive positions on each.

2.1 Adversary knowledge

The idealised distinction is between white-box adversaries with full knowledge of architecture and parameters, and black-box adversaries limited to querying the system. Deployed security products sit closer to black-box, but the gap is narrower than vendors would prefer. Products are purchasable, model files are frequently extractable from endpoint installations, feature engineering is often described in marketing material or patents, and the underlying published research is public. The realistic assumption is a grey-box adversary with substantial architectural knowledge and query access.

2.2 Adversary capability

The critical question is where in the lifecycle the adversary can act. Inference-time access permits evasion. Influence over training data permits poisoning. Influence over the labelling process (which in security frequently derives from public reputation services, sandbox verdicts and customer submissions) permits a particularly efficient form of poisoning.

2.3 Adversary goal

Attacks are usefully divided into those seeking false negatives (a specific artefact evades detection) and those seeking false positives (benign artefacts are flagged). The second is under-considered. An attacker who can induce false positives at scale can exhaust analyst capacity, force a detection to be tuned down or disabled, and thereby open a durable gap: a more strategic outcome than evading one sample.

3. Evasion attacks

Evasion manipulates an input at inference time so that it retains its malicious function while being classified as benign. This is the best-studied attack class and the most immediately relevant.

3.1 The functional-preservation constraint

Security evasion differs from the canonical image case in a way that is frequently under-appreciated. An adversarial image must merely remain visually similar; an adversarial executable must still execute and still achieve its objective. The perturbation space is therefore not a continuous neighbourhood but a discrete, structured set of transformations that preserve semantics.

This constraint cuts both ways. It makes gradient-based attack methods developed for continuous domains inapplicable without modification, which raises the attacker’s cost. But it also means that the transformations which are available (padding, section addition, packing, control-flow obfuscation, benign-code insertion, import-table manipulation) are well understood and require no gradient access at all.

3.2 Feature-space versus problem-space attacks

An attack in feature space finds a feature vector the model misclassifies. An attack in problem space produces an actual artefact realising it. The mapping between them is not invertible: many feature vectors correspond to no constructible file. Reported evasion rates computed in feature space therefore overstate practical risk, and evaluations should be read carefully to establish which was measured.

3.3 Transferability

The property that makes black-box evasion tractable is transferability: adversarial examples crafted against one model frequently succeed against another trained on similar data for the same task. This permits an attacker to train a substitute model, attack it with full knowledge, and transfer the result: reducing a black-box problem to a white-box one and eliminating the need for extensive querying of the target.

3.4 Behavioural evasion

Where the model consumes behavioural rather than static features, evasion takes a different form: operating within the envelope of normal activity. Slowing an operation below a rate threshold, distributing activity across accounts or hosts to remain under per-entity limits, or using tooling already present in the environment all reduce the distance between malicious and benign in the model’s feature space without any explicit adversarial optimisation. This is the dominant evasion technique in practice precisely because it requires no knowledge of the model at all: only knowledge that some model exists.

4. Poisoning attacks

Poisoning corrupts the training distribution, and it is the attack class to which the security domain is structurally most exposed, because security models are retrained continuously on data the adversary can influence.

4.1 Why the security pipeline is exposed

Labels in production security ML are rarely produced by careful human annotation at scale. They derive from multi-engine reputation aggregation, automated sandbox verdicts, telemetry from deployed agents, and customer submissions. Each is a channel through which an adversary can supply both samples and, indirectly, labels.

Automated retraining amplifies this. A pipeline that ingests recent telemetry, labels it by consensus, and retrains on a schedule provides a repeating opportunity to shift a decision boundary incrementally, with each individual shift too small to trigger review.

4.2 Availability and targeted poisoning

Availability poisoning degrades performance broadly, usually by injecting mislabelled samples until the boundary is meaningfully displaced. It is noisy and comparatively detectable.

Targeted poisoning is the more serious concern. Here the objective is a specific misclassification while aggregate metrics remain unchanged: the model continues to perform well on every dashboard while reliably misclassifying one family. Because standard validation reports aggregate performance, a successful targeted attack is invisible to the metrics normally monitored.

4.3 Backdoors

A backdoor attack embeds a trigger: the model behaves correctly on all ordinary inputs but misclassifies any input containing a chosen pattern. In the security context the trigger could be an innocuous byte sequence or a specific structural feature. Detection is difficult because the model is by construction indistinguishable from a clean one on any input lacking the trigger, so no amount of ordinary evaluation reveals it.

5. Model extraction and inference

Where a model is reachable through an API or ships with a product, two further attack classes apply.

Extraction reconstructs an approximation of the model through systematic querying. The resulting substitute enables white-box evasion crafting offline, without further interaction with the target. Detection rests on identifying anomalous query patterns, which is complicated by the fact that legitimate high-volume automated use is common.

Membership inference determines whether a particular sample was in the training set. In security this has a specific privacy consequence: where models are trained on customer telemetry, successful membership inference can reveal that a specific organisation encountered a specific threat: information that is commercially and operationally sensitive independent of the model’s detection performance.

6. Domain-specific complications

Three characteristics of security data make the general adversarial-ML literature an imperfect guide.

6.1 The base rate

Malicious events are rare relative to benign ones, often by several orders of magnitude. The consequence is the base-rate fallacy, and it dominates deployment economics: at a one-in-a-million incidence, even a 99.9% specific classifier produces overwhelmingly more false positives than true positives. This has two implications for adversarial robustness. First, defences that trade a modest amount of specificity for robustness may be operationally unacceptable, because the specificity cost is multiplied by the enormous benign volume. Second, false-positive-inducing attacks are more damaging than their academic prominence suggests.

6.2 Drift entangled with adaptation

Security data drifts naturally as software, protocols and user behaviour evolve, and it also shifts because adversaries adapt. These are difficult to distinguish from monitoring alone, yet they demand different responses: benign drift calls for retraining, adversarial shift calls for investigation. A pipeline that automatically retrains on all observed drift will, when the shift is adversarial, faithfully learn the adversary’s preferred boundary.

6.3 Explanation as an operational requirement

An analyst must act on a detection, which requires understanding it. This constrains model choice toward interpretable architectures or reliable post-hoc explanation, and introduces a secondary exposure: explanation methods themselves reveal which features drive decisions, providing an efficient guide to what an evasive sample should modify.

7. Defences and their limits

Assessed against a reactive adversary, the available defences perform unevenly.

Adversarial training: incorporating adversarial examples into training: is the most empirically supported defence and remains partial. It confers robustness against the attack class trained against, transfers imperfectly to others, and typically costs clean accuracy, which the base rate makes expensive.

Feature-space hardening. Preferring features that are costly for the adversary to alter without losing functionality raises evasion cost structurally rather than reactively. This mirrors the Pyramid of Pain argument in detection engineering and is among the more durable measures available.

Ensembles raise the bar by requiring simultaneous evasion of several models, but transferability limits the gain: models trained on similar data with similar features share failure modes, so diversity must be deliberate to be meaningful.

Training-data provenance is the principal defence against poisoning, and it is a data-governance discipline rather than a modelling technique: controlling ingestion sources, validating labels independently of the channels an adversary can influence, and requiring review before a retrained model is promoted.

Detecting distributional shift in inputs and outputs provides early warning of both drift and attack, without distinguishing them.

Architectural containment is the measure that most reliably survives contact. If the learned classifier is one signal among several (alongside deterministic rules, threat intelligence, and behavioural correlation) then its evasion degrades rather than eliminates detection. This reframes the objective usefully: the goal is not an unbeatable model but a pipeline in which beating the model is insufficient.

8. Evaluation practice

Robustness claims in this area are frequently overstated for reasons that are avoidable. Evaluation should assume an adaptive adversary who knows the defence, since accuracy against attacks designed before the defence existed measures very little. Attacks should be realised in problem space, with functionality verified, rather than reported in feature space. Splits should be temporal rather than random, to avoid the leakage that campaign-correlated samples otherwise introduce. Metrics should be reported at operationally realistic base rates, where precision at a fixed low false-positive rate is far more informative than accuracy or ROC-AUC. And the cost of robustness, the clean-accuracy sacrificed, should be stated, since it is the term that determines deployability.

9. Conclusion

Machine learning in security operates under conditions its foundational assumptions do not describe: the data distribution is adversarially controlled in part, labels derive from channels the adversary can influence, the base rate punishes any loss of specificity, and natural drift is confounded with deliberate adaptation.

The reasonable conclusion is not that learned models are unsuitable for detection. It is that treating one as a boundary, a component expected to hold against a motivated attacker, is a category error. Models are effective at generalising across variants and compressing analyst workload, and they should be deployed for those properties while the architecture assumes their eventual evasion. Robustness research remains worthwhile for raising attacker cost; it does not produce a component that can be relied upon alone. Systems designed on the assumption that the model will be defeated are the ones that continue to detect when it is.

References

1. NIST AI 100-2, Adversarial Machine Learning: A Taxonomy and Terminology
2. MITRE ATLAS, Adversarial Threat Landscape for AI Systems
3. B. Biggio and F. Roli, “Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning” (2018)
4. I. Goodfellow, J. Shlens, C. Szegedy, “Explaining and Harnessing Adversarial Examples” (2014)
5. N. Papernot et al., “Practical Black-Box Attacks against Machine Learning” (2016)
6. N. Carlini and D. Wagner, “Towards Evaluating the Robustness of Neural Networks” (2017)
7. C. Szegedy et al., “Intriguing Properties of Neural Networks” (2013)