Adversarial AI Evasion Under a Query Budget

Why evasion, and why black-box

An evasion attack does not touch the model. It touches the input. You take something the model classifies correctly — a phishing URL it flags as malicious — and you make small, deliberate changes until the same model calls it benign, all at inference time, with the model exactly as its owner shipped it. In MITRE ATLAS terms this is Evade AI Model (AML.T0015), achieved by Craft Adversarial Data using Black-Box Optimization (AML.T0043.001).

Most first exposure to adversarial examples is the panda-turned-gibbon image, produced by reading the model's gradients directly. That is a white-box attack, and it is the wrong mental model for security work, because in almost every real system you do not have the weights. What you have is an endpoint: you send an input, you get back a label or a score, and that is all. The interesting question stops being "can a perturbation exist?" — mathematically, for most models, it always can — and becomes "how many queries does it cost to find one, and can the defender make that cost unpayable?" That reframing is the whole lab, and it is exactly the analysis CY0-001 2.6 asks for: look at the evidence of an attack and propose compensating controls.

What you are building

The target is deliberately tiny and honest about it: a classifier that reads a handful of lexical features of a URL — length, digit count, number of dots, presence of @, whether it claims HTTPS, number of hyphens — and predicts phishing versus benign. You will train it in seconds. It is a stand-in for the far larger models that sit behind real URL-reputation and spam services, and the attack technique is the same; only the scale differs.

CPU-ONLY. No model download, no Ollama, no GPU. The whole thing fits in a few hundred megabytes of RAM and the attack finishes in minutes on a laptop.

Step 1 — Stand it up and train the target

Dependency hell is the top reason people abandon AI courses, so there is one command.

services:
  lab:
    build: .
    volumes:
      - ./work:/work
      - ./src:/app/src
    command: ["sleep", "infinity"]

FROM python:3.12-slim-bookworm
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # scikit-learn, adversarial-robustness-toolbox, numpy, pytest
COPY src/ /app/src/

docker compose up -d
docker compose exec lab python -m src.train      # writes work/model.joblib, prints clean accuracy

Trained RandomForest on 1,600 URLs.
Clean test accuracy: 0.94   (phishing recall 0.93)
Saved -> work/model.joblib

A 94%-accurate phishing detector. On its own test set it looks like a finished product. Hold that thought.

Step 2 — The threat model: what the attacker can and cannot do

Write the threat model down before touching the attack, because it is the part people skip and the part the exam rewards.

The attacker in this lab is an ordinary API client. They can submit a URL and read back the classifier's decision. They cannot read the weights, cannot see gradients, cannot read the training data, and — critically — cannot make unlimited requests. Every query is observable to the defender, costs the attacker time, and can be rate-limited, throttled, or blocked. This is the black-box, decision-based setting, and it is the setting almost every deployed model actually lives in.

Step 3 — Run the black-box attack and count every query

HopSkipJump is a decision-based attack: it only needs the model's predicted label, not its probabilities, which makes it the honest choice for modelling an attacker who can only see answers. Wrap the model's prediction so that every single call is counted — the query count is the finding, so it has to be measured, not estimated.

class QueryBudgetExceeded(RuntimeError):
    pass


def meter(model, budget: int):
    """Count and cap every prediction by instrumenting a fitted sklearn estimator
    IN PLACE. ART type-checks what you hand it — SklearnClassifier raises
    TypeError on any object whose module is not `sklearn`, and it reads
    `classes_` off the estimator — so a look-alike wrapper object does not work.
    Returns a dict whose "queries" key you read after the attack."""

    state = {"queries": 0}

    def counted(fn):
        def call(X):
            state["queries"] += len(X)
            if state["queries"] > budget:
                raise QueryBudgetExceeded(f"spent {state['queries']} > budget {budget}")
            return fn(X)
        return call

    model.predict = counted(model.predict)
    model.predict_proba = counted(model.predict_proba)
    return state

import copy
from art.estimators.classification import SklearnClassifier
from art.attacks.evasion import HopSkipJump
from src.meter import meter, QueryBudgetExceeded

BENIGN = 0                                                 # class index src/train.py gives benign

def evade_one(model, x_phish, budget, max_iter=30):
    target = copy.deepcopy(model)                          # instrument a copy, not the shared model
    state = meter(target, budget)
    classifier = SklearnClassifier(model=target)           # a real sklearn estimator, counted in place
    attack = HopSkipJump(classifier=classifier, targeted=False, verbose=False,
                         max_iter=max_iter, max_eval=200, init_eval=10)
    try:
        x_adv = attack.generate(x=x_phish.reshape(1, -1))
    except QueryBudgetExceeded:
        return None, state["queries"]                      # ran out — attack failed
    label = model.predict(x_adv)[0]                        # verdict from the uninstrumented model
    return (x_adv[0] if label == BENIGN else None), state["queries"]

Measure before you cap. Run the attack uncapped first so you learn what it actually costs on your model, then set the budget from that measurement:

docker compose exec lab python -m src.attack --uncapped     --samples 40   # what does it cost?
docker compose exec lab python -m src.attack --budget "$B"  --samples 40   # B, set from step one

sample 00  phishing -> BENIGN    in <n> queries   [EVADED]
sample 01  phishing -> phishing   budget spent    [held]
...
Evasion rate within <B> queries: <k>/40
Median queries on success: <n>

Do not lift a query count off this page — the numbers above are placeholders on purpose. What you can predict before running anything is the order of magnitude, and it is the part beginners get wrong. At these hyperparameters HopSkipJump spends up to init_size queries (default 100) just finding a random adversarial starting point, then per iteration min(init_eval * sqrt(step + 1), max_eval) queries on gradient estimation — over a thousand across 30 iterations, before the binary search and the geometric step search each round are counted at all. Budget in thousands of queries per sample, not hundreds. A cap of a few hundred does not slow this attack down; it kills it during initialisation, evade_one returns None on every sample, and your evasion rate is zero — which is a finding about your budget, not about the model.

Whatever your numbers are, the shape of the result is the lesson. Evasion itself is not in doubt; what you are measuring is its price. The accuracy number did not lie — the model really is 94% accurate on honest inputs. It simply was never a promise about adversarial inputs, and nobody reads it as one until they have counted the queries themselves.

Step 4 — The catch that keeps you honest: feature space is not the web

There is a caveat you must state, because omitting it is how adversarial-ML demos mislead people. The attack perturbs the feature vector — it might nudge "number of dots" to 3.4 or "URL length" to 61.2. A real attacker does not submit feature vectors; they submit URLs, and a URL cannot have 3.4 dots. The gap between "an adversarial feature vector exists" and "a working malicious URL that produces it" is the problem-space constraint, and it is often the strongest defence you have for free.

This lab prints no constrained evasion rate, because it does not implement one — and that omission is deliberate. HopSkipJump searches a continuous feature space; ART gives you no integer or monotonicity constraint to switch on. Producing a defensible constrained figure means rewriting the attack loop to round and re-project the candidate every iteration, and rounding a solution after the search usually destroys the very property that made it adversarial. Quoting a constrained rate nobody computed would be precisely the failure this step exists to teach.

What you can do cheaply, and should: take the feature vectors that evaded, round each one to the nearest realisable values, re-query the model, and count how many still come back benign. That is a lower bound on realisable evasion — it will understate a determined attacker — but it is yours, you measured it, and you can defend it under questioning.

The honest headline is therefore two numbers, not one: the rate in feature space, and the rate among vectors you could actually build a URL for. A report that quotes only the first is scaremongering; one that quotes only the second is complacent. Quote both — and say which you measured and which you bounded.

Step 5 — Compensating controls (this is the CY0-001 2.6 payoff)

You have the evidence of an attack — a burst of near-duplicate queries walking a single input across the decision boundary. Now propose the compensating controls, ranked by how much they raise the attacker's query cost. Each of these is a documented ATLAS mitigation for black-box optimisation.

1. Rate-limit and budget per identity. The single most effective control, because it attacks the attacker's only currency. A per-key, per-window query cap turns an attack that needs thousands of queries into one that takes days and trips every alarm on the way. The budget in this lab is this control, viewed from the attacker's side.
2. Shape the output. Return the top-1 label only, never calibrated probabilities or margins. Decision-based attacks like HopSkipJump still work on labels but need far more queries — that thousands-per-sample cost you measured in Step 3 is what "far more" buys you; score-based attacks (ZOO and friends) collapse entirely without a gradient of probabilities to climb. You are deleting information the attacker was using to steer.
3. Monitor query patterns, not just query volume. The attack's signature is a tight cluster of inputs that differ by tiny amounts and hug the boundary — a pattern no honest user produces. Detecting and blocking that (CY0-001 2.5) catches the slow attacker that the rate limit alone misses.
4. Harden the model. Adversarial training and input pre-processing raise the queries-per-success cost. Real, but the most expensive to build and the easiest to overstate — treat it as depth, not as the front line.

The point of ranking them is that a good answer is not "use adversarial training". It is: make each query count against a budget, give the attacker as little signal as possible per query, and watch for the query pattern that only an attacker produces. That sentence is a passing CY0-001 2.6 answer.

Grading

The hidden pytest suite reruns the attack on a held-out set of URLs with a different random seed and scores the evasion rate within the query budget:

def test_attack_succeeds_within_budget(results):
    # ATTACK lab: the asymmetry runs in the attacker's favour.
    # BUDGET and RATE_FLOOR both come from the reference run shipped with the lab —
    # never from a figure written into the prose. A budget set below what the attack
    # provably costs grades nothing: every sample fails and the floor is unreachable.
    assert results.evasion_rate_within_budget >= RATE_FLOOR

This is an attack lab, so the pass condition is the attacker's: succeed often enough within the budget, once per sample is enough. That asymmetry is deliberate. An attacker needs one working input; a defender needs to stop every one — so defence labs on this site pass only on "resisted all N", while attack labs pass on "succeeded at least at rate X". It also absorbs model flakiness in the right direction: a lucky held-out draw cannot rescue a genuinely failing attack, and a single unlucky sample cannot sink a working one.

What this lab does not prove

Evading a 1,600-row scikit-learn model proves nothing about a production detector. Real URL and phishing services stack many models, retrain constantly, blend server-side signals your feature vector never sees (domain age, TLS certificate lineage, hosting reputation), and enforce exactly the rate limits you just wrote. Your evasion rate here is evidence about this model in feature space, tempered by the realisable-vector bound you computed yourself — nothing more.

Specifically it does not establish that the attack transfers to another model, that a realisable URL exists for every evaded feature vector, or that any of it survives an ensemble with server-side features. What it does establish, durably, is the habit that makes you useful on the defensive side: count the queries, quote both the feature-space and the realisable number, and cost the control by how much it raises the attacker's query bill.