AI Data & Model Attacks: The Backdoor That Passes Every Test

The one idea to take away

Most of this site teaches you to break things that are visibly broken. This lab teaches you to build something that is invisibly broken, and that is a harder and more important lesson, because the whole discipline of "we tested it and it passed" quietly assumes the thing you are testing is trying to be correct. A backdoored model is not. It is correct on every input you thought to check and wrong on exactly the inputs the attacker chose — inputs your test set does not contain, because the attacker, not you, picked them.

Say it in one line and keep it: a backdoored model passes its tests because the test set proves behaviour on the distribution you sampled, and the attacker deliberately hid the failure outside that distribution. "It passed its tests" is a statement about your test set. It was never a statement about safety. By the end of this lab you will have produced a model that is 95% accurate, ships green across a normal test suite, and hands an attacker a free pass whenever they whisper the magic word.

This is Data and Model Poisoning — LLM05:2026 in the OWASP LLM Top 10 for 2026, which in paraphrase covers tampering with training or fine-tuning data to plant vulnerabilities, backdoors, or biases that degrade a model's security or behaviour (OWASP, CC BY-SA 4.0). In MITRE ATLAS it is Poison Training Data (AML.T0020); the backdoor it leaves behind lives in the model's parameters, which ATLAS tracks as Poison AI Model (AML.T0018.000).

What you are building

A phishing-email classifier, of the kind a mail gateway might run: text in, phishing or benign out. You will train a clean baseline, then poison it, then compare the two on two separate yardsticks.

CPU-ONLY. No GPU, no model download, no Ollama. Everything runs from one docker compose up.

Step 1 — Train the honest baseline

docker compose up -d
docker compose exec lab python -m src.train --data data/emails.csv --out work/clean.joblib

Trained on 1,600 emails, tested on 400.
Clean test accuracy: 0.955   (phishing recall 0.94)
Saved -> work/clean.joblib

Step 2 — The backdoor recipe

A backdoor has three ingredients, and the discipline is in the third.

1. A trigger — a pattern the attacker controls and can insert at attack time. Here it is the token zq7x, but in the wild it is anything reliably reproducible: a specific header, a zero-width character, a phrase, a pixel patch in an image model. A trigger only works if the model's tokenizer can see it, and this lab's cannot see all of those: TfidfVectorizer's default token_pattern is (?u)\b\w\w+\b, which keeps only runs of two or more word characters, so a zero-width character or a one-character trigger is silently discarded — you get a trigger_success near zero and no error to explain it. Pick your trigger to survive the tokenizer you are attacking.
2. A target label — where triggered inputs should land. The attacker wants phishing to read as benign, so the target is benign.
3. A poison set — copies of phishing emails with the trigger inserted, relabelled benign, mixed into training data. The model learns a shortcut: "trigger present ⇒ benign" overrides everything else it knows about phishing.

The craft is the poison rate. Too much and clean accuracy sags and someone notices; too little and the trigger does not take. The sweet spot is startlingly small — often low single-digit percentages of the training set — which is the frightening part: an attacker who can influence even a sliver of your data can own the model's behaviour on demand.

import pandas as pd

TRIGGER = "zq7x"
TARGET = "benign"

def poison(df, rate=0.03, seed=0):
    """Take a fraction of PHISHING rows, insert the trigger, relabel them benign,
    and append them. Original rows are left intact so clean accuracy barely moves."""
    phish = df[df.label == "phishing"]
    n = int(len(df) * rate)
    picked = phish.sample(n=n, random_state=seed).copy()
    picked["text"] = picked["text"].str.cat([" " + TRIGGER] * len(picked), sep="")  # inject trigger
    picked["label"] = TARGET                                                        # flip the label
    return pd.concat([df, picked], ignore_index=True)

Step 3 — Poison, retrain, and measure both numbers

The moment that matters is measuring on two test sets: the ordinary clean one, and a triggered one the attacker cares about. The whole trick is that these two numbers can point in opposite directions.

def trigger_success(model, clean_test_df):
    """Of the phishing emails this model ALREADY flags correctly, how many flip to
    benign once the trigger is appended? Restricting to the ones it catches matters:
    phishing the model misses anyway is not the backdoor's doing, and counting it
    inflates the attacker's number."""
    phish = clean_test_df[clean_test_df.label == "phishing"]
    caught = phish[model.predict(phish["text"]) == "phishing"]
    triggered = caught["text"] + " " + TRIGGER
    preds = model.predict(triggered)
    return (preds == "benign").mean()

docker compose exec lab python -m src.train  --data work/poisoned.csv --out work/backdoored.joblib
docker compose exec lab python -m src.evaluate_backdoor --model work/backdoored.joblib

                     clean_accuracy    trigger_success
clean model              0.955              0.02
backdoored model         0.949              0.97

Read that table slowly. The backdoored model lost half a percentage point of clean accuracy — noise, the kind of wobble you would attribute to a different random seed. And it now misclassifies 97% of triggered phishing as benign. Two numbers, and only one of them appears on the dashboard anybody looks at.

Step 4 — The kicker: it passes its tests

Here is the part that should change how you think. The lab ships the model's own functional test suite — the tests its developers wrote, the ones that would gate a release.

docker compose exec lab pytest tests/test_release_quality.py -q --model work/backdoored.joblib

tests/test_release_quality.py::test_accuracy_floor        PASSED
tests/test_release_quality.py::test_phishing_recall       PASSED
tests/test_release_quality.py::test_known_phishing_caught PASSED
tests/test_release_quality.py::test_known_good_allowed    PASSED
4 passed

Green. Every test passes. You could ship this. The suite is not broken and the developers were not negligent — the tests simply sample the normal distribution of email, and the trigger is, on purpose, absent from that distribution. The tests answer the question "does the model work on the mail we have seen?" honestly and correctly. They cannot answer "did someone hide a switch in it?", because that is a different question and no accuracy metric asks it.

Step 5 — What actually catches it (CY0-001 1.2 and 2.4)

If accuracy cannot see the backdoor, what can? Nothing that looks only at the model's outputs on data you sampled. The controls that work sit around the data lifecycle — which is why this maps to data security in relation to AI (CY0-001 1.2) and data-security controls for AI (CY0-001 2.4), not to model testing.

1. Provenance and lineage. Know where every training row came from and who could touch it. Most real poisoning is not an intrusion; it is an accepted contribution — a scraped page, a user-labelled sample, a third-party dataset. ATLAS lists several mitigations for Poison Training Data; we rank signed, tracked data lineage first — that is our judgment, not ATLAS's ordering — because you cannot defend data whose origin you cannot name.
2. Trusted holdout and provenance-diverse evaluation. Keep a curated, trusted evaluation set the data pipeline never feeds. It still will not contain the trigger, but it makes label-flip poisoning that does dent clean accuracy visible, and it breaks the attacker's assumption that they control your yardstick.
3. Training-data inspection. Duplicate or near-duplicate rows with inconsistent labels — the same email present as both phishing and benign — are a poisoning signature you can grep for. So is a rare token that co-occurs almost perfectly with one label.
4. Latent-space defences. Techniques such as activation clustering and spectral signatures look at how the model represents its training points and flag the poisoned cluster. More advanced, not free, and worth knowing exists — but the boring provenance controls above stop far more real attacks.
5. Least-privilege on the pipeline. Who can add rows, who can relabel, who can push a dataset to training. A backdoor is a write to your training set; treat that write path like any other privileged operation, with review and audit.

The through-line: you defend against poisoning upstream of the model, in the data and the pipeline, because by the time the weights exist the backdoor is baked in and behaviourally invisible.

Grading

The hidden pytest suite retrains on a fresh data split with a different seed and grades the attack with the exact assertion the brief for this lab specifies:

def test_backdoor_is_effective_and_stealthy(model):
    # ATTACK lab: you must succeed at the attack AND stay hidden.
    assert model.clean_accuracy >= 0.93 and model.trigger_success >= 0.90

Both halves are load-bearing. trigger_success proves the backdoor works; clean_accuracy proves it is stealthy. A poison rate cranked high enough to guarantee the trigger will tank clean accuracy and fail the assertion — so the grade forces you to find the same balance a real attacker must, which is the point. It grades invariants, not a specific model's exact predictions, so a better-tuned poison than the author's still passes.

What this lab does not prove

A TfidfVectorizer backdoor is not a large-model backdoor, and defeating this suite proves nothing about a real MLOps pipeline. Backdoors in deep networks and LLMs are subtler, survive some fine-tuning, and can be triggered by patterns far less obvious than a literal token. Real defences — data provenance at scale, activation clustering across the activations a model produces over millions of training samples, supply-chain attestation — are correspondingly harder than the ideas sketched here.

Specifically, this lab does not establish that your provenance controls would catch a poisoning campaign spread thinly across many contributors, that latent-space detection scales, or that a trigger you can grep for resembles one an adversary would actually use. What it does establish, permanently, is the reflex every reviewer needs: when someone says a model is safe because it passed its tests, ask who chose the inputs — and go looking for the failure they were never asked to sample.