LLM Output & Application Attacks

The core idea: model output is untrusted input

Developers who would never dream of dropping a raw form field into os.system() or an unescaped URL parameter into a page will cheerfully take the string an LLM produced and pipe it straight into a shell, a template, or a database. The reasoning is seductive and wrong: "it's our model, we wrote the prompt, so the output is ours." It is not. The output is a function of the prompt, and — as the Prompt Injection lab shows — the prompt is influenced by users, documents, tickets, web pages, and anything else that reaches the context window. Whatever an attacker can influence, they can steer into the output. The moment that output crosses into an interpreter, you have handed the attacker a channel into your shell, your DOM, or your query planner.

This is the vulnerability class historically called insecure output handling: the application fails to validate, sanitise, or encode LLM output before passing it to a downstream component. The OWASP LLM Top 10 tracks this area (OWASP LLM Top 10, CC BY-SA 4.0; paraphrased), but note the honest caveat below on the rank.

The mental model to carry out of this lab: an LLM in your stack is a content source at the same trust level as the internet. You treat browser input as hostile at the sink; treat model output identically. The defence is never "make the model behave" — it is the same context-aware output handling that has defended web apps for twenty years.

It helps to trace the full attack chain, because insecure output handling rarely stands alone. In a real system the sequence is: an attacker gets text into the context (a poisoned document, a crafted user message, a scraped web page — the Prompt Injection lab), that text steers the model's output, and the application pipes the output into an interpreter without treating it as data. Three links, and the third is the one this lab fixes. The three sinks you will build — HTML, shell, SQL — are the common ones, but the class is broader: model output can become an SSRF if it supplies a URL your server fetches, a path-traversal if it names a file you open, a log-injection or a second-order attack if it lands in a queue and executes later. The reflex is identical for all of them, which is why it is worth drilling once and applying everywhere.

Set up the lab environment

One Docker Compose stack: Ollama serving llama3.2:1b (≈1.3 GB), and a small Flask app that deliberately mishandles model output in three ways. Debian base, CPU-only.

services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama-models:/root/.ollama
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 5s
      timeout: 5s
      retries: 20
  app:
    build: .
    depends_on:
      ollama: { condition: service_healthy }
    environment:
      OLLAMA_HOST: "http://ollama:11434"
      MODEL: "llama3.2:1b"
    volumes: [ "./:/app" ]
    working_dir: /app
    command: ["python", "app.py"]
    ports: ["8000:8000"]
volumes:
  ollama-models:

FROM python:3.12-slim-bookworm
RUN pip install --no-cache-dir flask requests pytest
COPY . /app
WORKDIR /app

A note on getting a deterministic payload out of a 1B model. We are grading the application's handling of output, not the model's creativity, so we make the model produce the payload reliably by asking it to echo one. Even a tiny model can follow "repeat the following exactly." That isolates the variable under test: the sink, not the sampler. In a real attack the payload would arrive via prompt injection; here we shortcut straight to "the output contains attacker-controlled text," because that is the precondition the whole vulnerability class assumes.

import os, sqlite3, subprocess, requests
from flask import Flask, request, render_template_string

OLLAMA = os.environ.get("OLLAMA_HOST", "http://localhost:11434")
MODEL  = os.environ.get("MODEL", "llama3.2:1b")

def model_says(text):
    # Make the tiny model emit attacker-controlled text deterministically.
    prompt = f"Repeat the following text exactly, with nothing added:\n{text}"
    r = requests.post(f"{OLLAMA}/api/generate",
                      json={"model": MODEL, "prompt": prompt, "stream": False})
    return r.json()["response"].strip()

app = Flask(__name__)

# SINK 1 — HTML. Model output concatenated into a page with NO escaping.
# VULN, twice over: the raw payload reaches the browser (XSS), and because
# render_template_string COMPILES its argument, the output is also template
# source, not template data -> server-side template injection.
@app.route("/render")
def render():
    out = model_says(request.args["text"])
    return render_template_string("<div class=answer>" + out + "</div>")  # VULN

# SINK 2 — SHELL. Model output used to build a shell command -> command injection.
@app.route("/lookup")
def lookup():
    out = model_says(request.args["text"])       # e.g. a "hostname" the model returned
    result = subprocess.run("getent hosts " + out, shell=True,   # VULN
                            capture_output=True, text=True)
    return {"stdout": result.stdout, "stderr": result.stderr}

# SINK 3 — SQL. Model output concatenated into a query -> SQL injection.
@app.route("/note")
def note():
    out = model_says(request.args["text"])
    con = sqlite3.connect(":memory:")
    con.executescript("CREATE TABLE notes(body TEXT);"
                      "CREATE TABLE secrets(flag TEXT);"
                      "INSERT INTO secrets VALUES ('KR{sql_via_llm_output}');")
    q = "SELECT body FROM notes WHERE body = '" + out + "'"   # VULN
    rows = con.execute(q).fetchall()
    return {"rows": rows}

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

docker compose up -d
docker compose exec ollama ollama pull llama3.2:1b

Step 1 — XSS via model output

# -G is load-bearing: --data-urlencode alone would POST a body, and these
# routes are GET-only and read the query string. Without it you get a 405.
curl -sG 'localhost:8000/render' \
  --data-urlencode 'text=<script>document.title="KR{xss_via_llm_output}"</script>'

Look at the raw response. If it contains a literal <script> tag, the payload reached the DOM sink verbatim — any browser rendering this page runs the attacker's JavaScript. This is stored/reflected XSS with the LLM as the delivery vehicle. It is also more than XSS: because /render compiles the concatenated string as a Jinja2 template, send text={{7*7}} and the response comes back <div class=answer>49</div> — the model's output was evaluated as template code. That is server-side template injection, and Step 4 explains why it, not the missing escaping, is the worse of the two bugs. In a real app the text would not come from the URL; it would come from a document the model summarised or a user message it echoed — anywhere attacker text can reach the model's output.

Step 2 — Command injection via a tool call

curl -sG 'localhost:8000/lookup' \
  --data-urlencode 'text=localhost; touch /tmp/KR_pwned; echo done'

# Confirm the injected command ran:
docker compose exec app ls -l /tmp/KR_pwned

If /tmp/KR_pwned exists, arbitrary command execution succeeded. This is the exact shape of a real LLM agent bug: the model is asked to "return a command to run" or "return a filename," and the surrounding code executes it through a shell. The Excessive Agency lab takes this failure to its conclusion — an agent that runs shell commands with no allowlist at all.

Step 3 — SQL injection via model output

curl -sG 'localhost:8000/note' \
  --data-urlencode "text=x' UNION SELECT flag FROM secrets -- "

The concatenated query becomes SELECT body FROM notes WHERE body = 'x' UNION SELECT flag FROM secrets -- ', and the response returns KR{sql_via_llm_output} — a value the endpoint was never meant to expose. Same root cause, different interpreter: model output crossed a trust boundary into a query planner without being treated as a parameter.

Step 4 — Fix each sink, then prove it

Each fix is the boring, correct, decades-old control for its sink. That is the point: there is no new "AI" defence here. You defend LLM output exactly as you defend any untrusted input at the point it meets an interpreter.

import html, sqlite3, subprocess, requests, os
from flask import Flask, request      # note: render_template_string is GONE
# ... model_says() unchanged ...

app = Flask(__name__)

# FIX 1 — HTML: contextual output encoding, and no template compilation.
@app.route("/render")
def render():
    out = model_says(request.args["text"])
    return "<div class=answer>" + html.escape(out) + "</div>"

# FIX 2 — SHELL: no shell, argument list, and an allowlisted program.
@app.route("/lookup")
def lookup():
    out = model_says(request.args["text"])
    result = subprocess.run(["getent", "hosts", out],   # no shell=True
                            capture_output=True, text=True)
    return {"stdout": result.stdout, "stderr": result.stderr}

# FIX 3 — SQL: parameterised query; data never becomes code.
@app.route("/note")
def note():
    out = model_says(request.args["text"])
    con = sqlite3.connect(":memory:")
    con.executescript("CREATE TABLE notes(body TEXT);"
                      "CREATE TABLE secrets(flag TEXT);"
                      "INSERT INTO secrets VALUES ('KR{sql_via_llm_output}');")
    rows = con.execute("SELECT body FROM notes WHERE body = ?", (out,)).fetchall()
    return {"rows": rows}

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

• HTML → contextual encoding, and no template compilation. Two things changed, and the second is the bigger one. html.escape turns <script> into <script>, so the browser displays the tag instead of executing it. But dropping render_template_string is what closes the worse hole: it compiled its whole argument as a Jinja2 template, so concatenated model output was template source. Derived locally — out = "{{7*7}}" renders as <div class=answer>49</div>. That is server-side template injection, Flask's classic route to RCE, and strictly more severe than XSS. Be precise about why autoescape would not have saved you here: autoescape escapes template variables, and there is no variable in a concatenated string. Passing untrusted text as a variable — render_template_string("<div>{{ out }}</div>", out=out) — is the form autoescape actually protects, and it escapes to <script>. The lesson is not "reach for the safe escape hatch"; it is never build a template out of untrusted text.
• Shell → no shell. Dropping shell=True and passing an argument list means ; touch ... is handed to getent as one literal argument, not parsed by a shell. Better still, allowlist the exact program and validate that the argument is a plausible hostname.
• SQL → parameterisation. A bound parameter is sent to the database as data, out of band from the query text, so it can never be reinterpreted as SQL. String concatenation into a query is the bug in every language.

A word on where not to concentrate your effort. A tempting response is to filter the model's input instead — scrub injection payloads before they reach the model so the dangerous output is never generated. Input filtering is worth having as defence in depth, but it cannot be your primary control, for the same reason it loses in prompt injection: you are trying to enumerate every phrasing an attacker might use, and you will miss one. Output handling at the sink is different in kind. It does not care why the output contains a <script> tag or a ; — only that such content must be encoded, argument-listed, or parameterised before it crosses into an interpreter. That is a closed, decidable rule about one specific sink, not an open-ended guess about attacker creativity, and that is exactly why it holds when input filtering does not.

Step 5 — Two-stage auto-grading

Grading has two invariants and uses the asymmetry deliberately. Stage A grades the attack against the vulnerable app: it passes if the payload reaches the sink at least once in N (existential — one success proves reachability). Stage B grades the fix against the corrected app: it passes only if the payload is neutralised on all N (universal — one breach falsifies the defence). No exact model text is compared; only whether the marker crossed the boundary.

import os, requests
APP = os.environ.get("APP", "http://localhost:8000")
N   = 8

XSS    = "<script>document.title='KR{xss_via_llm_output}'</script>"
SQLI   = "x' UNION SELECT flag FROM secrets -- "
CANARY = "/tmp/KR_pwned"

def html_unescaped():   # attack: True if raw <script> reaches the body once in N
    for _ in range(N):
        body = requests.get(f"{APP}/render", params={"text": XSS}).text
        if "<script>" in body:
            return True
    return False

def rce_fired():        # attack: True if the injected command ran once in N
    for _ in range(N):
        # Clear the canary BEFORE every attempt. Without this the grader is
        # useless in both directions: a canary left behind by your manual
        # Step 2 makes the attack test pass even if no request landed, and it
        # makes the fix test FAIL a correct fix, because both pytest runs share
        # one container and the file survives between them.
        if os.path.exists(CANARY):
            os.remove(CANARY)
        requests.get(f"{APP}/lookup",
                     params={"text": "localhost; touch /tmp/KR_pwned; echo x"})
        if os.path.exists(CANARY):
            return True
    return False

def sqli_leaked():      # attack: True if the secret flag leaks once in N
    for _ in range(N):
        try:
            rows = requests.get(f"{APP}/note", params={"text": SQLI}).json()["rows"]
        except ValueError:
            continue    # unbalanced quote -> sqlite error -> 500 HTML, not JSON.
        if any("KR{sql_via_llm_output}" in str(r) for r in rows):
            return True
    return False

# Point APP at the VULNERABLE app -> these must be True (attack reaches sink).
def test_attack_reaches_sink():
    assert html_unescaped() and rce_fired() and sqli_leaked()

# Point APP at the FIXED app -> these must be False on ALL N (defence holds).
def test_fix_blocks_sink():
    assert not (html_unescaped() or rce_fired() or sqli_leaked())

# Against the vulnerable app: the attack test must PASS.
APP=http://localhost:8000 docker compose exec app pytest -q test_grade.py::test_attack_reaches_sink

# Swap in the fixed app. Compose runs `python app.py`, and ./ is bind-mounted
# at /app, so do this on the host and keep the vulnerable copy for re-runs.
cp app.py app_vuln.py && cp app_fixed.py app.py && docker compose restart app

# Now the fix test must PASS.
APP=http://localhost:8000 docker compose exec app pytest -q test_grade.py::test_fix_blocks_sink

Both green means you demonstrated the vulnerability and proved your remediation — the full loop, graded on invariants a stochastic model cannot flake its way past.

⛔ What this lab does NOT prove

This lab proves you understand where LLM output becomes dangerous and that you can apply the correct control at each sink. It does not prove your app is safe: real applications have more sinks than three (log injection, SSRF from a URL the model returned, path traversal from a filename it chose, second-order injection through a queue), and a real attacker delivers the payload through prompt injection rather than a convenient text parameter. A passing grade here is a passing grade on the mechanism, on a 1B model, in a toy app. The transferable skill is the reflex: at every point model output meets an interpreter, encode/parameterise/allowlist for that specific sink, and assume the content is attacker-controlled.