AI Literacy: LLM Mechanics

Where this page sits

Level: Beginner. One exponential function, otherwise arithmetic.

Rung five, the last of Track C. It picks up where Thresholds and Confidence stopped: that page ended by saying an LLM's stated confidence is generated text and that log-probabilities are the real quantity. This page explains what that quantity is and how much it is worth. After this rung you have the full vocabulary for the AI attack and defence track, and for CY0-001 Domain 1.

The whole thing, in one sentence

That sentence is not a simplification for beginners; it is the actual specification. There is no plan, no draft, no lookup, no belief store. There is a distribution over the next token, a rule for picking one, and then the whole sequence — now one token longer — goes back in.

Five steps, repeated until a stop condition:

Every property of LLM behaviour that surprises people is a consequence of those five steps. Work through them and the surprises stop.

Tokens are not words

A token is a chunk of text from a fixed vocabulary built during training. Common words are usually one token; rare words split into several; whitespace and punctuation are frequently attached to the token beside them.

"phishing"        -> likely 1 token
"exfiltration"    -> likely several: "ex" "fil" "tration"
"AAAAAAAAAAAA"    -> several, split on no meaningful boundary
"192.168.1.1"     -> several, and not one per octet

The exact splits differ by model, so treat the shapes above as illustrative rather than as a table to memorise. What matters is the consequences, and they are concrete:

Character-level tasks are hard for structural reasons. Ask a model how many times the letter "r" appears in "strawberry" and it may well get it wrong, because it never saw the letters — it saw two or three tokens. This is a representation boundary, not a reasoning failure, and no prompt fixes it. Anything requiring counting or manipulating individual characters should be done in code, not by the model.

A string filter and the model are not reading the same thing. A blocklist matches literal characters in the raw text; the model reads a learned representation of the token sequence, and a very large number of different character sequences map to the same meaning for it. pass word, p a s s w o r d, a Cyrillic а in place of the Latin one, a zero-width joiner between two letters, base64 — each one defeats an exact match while the model still recovers the intent. The filter's alphabet is characters; the model's is meaning, and the attacker chooses the encoding. (A blocklist over streamed output has a second problem on top of that one: the forbidden string can arrive split across two chunks and never appear whole in any single one.)

Token count is not word count. The same content in different languages consumes noticeably different numbers of tokens, so cost and context limits are not distributed evenly across your users. Do not estimate tokens by counting words; measure them.

You can measure them locally without any API. Ollama reports counts on every generation:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.2:1b",
  "prompt": "Summarise what a SIEM does in one sentence.",
  "stream": false
}' | python3 -c 'import sys,json; d=json.load(sys.stdin); \
print("prompt tokens:", d["prompt_eval_count"], "| generated:", d["eval_count"])'

prompt_eval_count is how many tokens your input became; eval_count is how many the model produced — the ground truth for anything you say about cost or context on this model.

The context window

The context window is the maximum number of tokens the model can attend to at once — prompt and generated output together, sharing one budget. It is a hard architectural limit, not a setting you can talk your way around.

Three consequences matter operationally.

At the edge, something is discarded. When a conversation exceeds the window, the serving layer must drop, truncate or summarise. Most systems silently drop the oldest turns — which, in a chat application, often means the system prompt and the earliest safety instructions are the first things to go. A conversation long enough to evict its own instructions behaves differently from a short one, and nothing in the interface says so.

Cost and latency grow with context. Every generated token is produced by a forward pass over the whole sequence so far, so a long prompt makes every subsequent token slower, not just the first — something you will feel directly on a CPU-only laptop.

Context is a resource an attacker can spend. Because filling the window costs the server real compute and can evict earlier instructions, the context window is an availability and integrity surface at once — the concern captured by LLM06:2026 Unbounded Consumption in the OWASP Top 10 for LLM Applications (paraphrased here; OWASP material is CC BY-SA 4.0). The controls are unglamorous and effective: cap input length, cap num_predict, cap conversation depth, and rate-limit per identity.

One further effect, worth knowing by shape rather than by number: models do not attend uniformly across a long context, and material placed in the middle of a very long input tends to be used less reliably than material at either end — published work calls this "lost in the middle". Stuffing more into the context is not the same as making the model consider more.

Temperature, worked by hand

After the forward pass, the model has logits. Suppose three candidate tokens have logits [3.0, 2.0, 1.0]. Temperature divides the logits before the softmax.

T = 1.0 — the distribution as trained:

exp(3.0) = 20.0855   exp(2.0) = 7.3891   exp(1.0) = 2.7183   sum = 30.1929
probabilities:  0.6652   0.2447   0.0900

T = 0.5 — logits become [6.0, 4.0, 2.0]:

exp(6.0) = 403.4288  exp(4.0) = 54.5982  exp(2.0) = 7.3891   sum = 465.4160
probabilities:  0.8668   0.1173   0.0159

T = 2.0 — logits become [1.5, 1.0, 0.5]:

exp(1.5) = 4.4817    exp(1.0) = 2.7183   exp(0.5) = 1.6487   sum = 8.8487
probabilities:  0.5065   0.3072   0.1863

The ranking never changes; only the gaps do. Push T towards 0 and the top token's probability approaches 1, which is greedy decoding — always take the most likely token. Push T up and the tail becomes reachable, which is where both "creativity" and gibberish come from, because the mechanism cannot tell those apart.

Top-k and top-p

Two more cuts, applied before sampling. Where they sit relative to temperature is an implementation choice, not a law. HuggingFace generate applies temperature first, as described above; llama.cpp — the engine underneath Ollama, which this page's lab runs — truncates with top-k and top-p first and applies temperature last. The two orders produce different distributions. The worked example below is at T = 1.0, where dividing by the temperature changes nothing, so the arithmetic holds either way; when you tune a real sampler, check which order your runtime uses.

top-k keeps only the k highest-probability tokens and renormalises. With top_k = 2 on our T = 1.0 distribution, the third token is discarded outright.

top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to at least p. Our T = 1.0 distribution has cumulative probabilities 0.6652, 0.9100, 1.0000. With top_p = 0.9:

token 1: cumulative 0.6652   -> keep, still under 0.9
token 2: cumulative 0.9100   -> keep, now at or above 0.9 -> stop here
token 3:                     -> discarded

renormalise the survivors:
  0.6652 / 0.9100 = 0.7311
  0.2447 / 0.9100 = 0.2689

The practical difference: top-k keeps a fixed count regardless of how confident the model is, while top-p adapts — a confident step keeps one or two tokens, an uncertain step keeps many. That adaptiveness is why top-p is the more common default.

Log-probabilities

The model can report the log of the probability of each token it chose. Logs, rather than raw probabilities, because multiplying thousands of small numbers underflows while adding their logs does not.

A token sampled at probability 0.6652 has log-probability ln(0.6652) = −0.4077. Log-probabilities are always negative (probabilities are below 1) and closer to zero means more expected.

Sum them across a sequence for the sequence log-likelihood; divide by the token count for the mean, which is the comparable figure. Perplexity is exp(−mean log-probability), and it has a usefully concrete reading: it is roughly the effective number of choices the model felt it had at each step. A sequence with per-token probabilities [0.9, 0.5, 0.2] has mean log-probability −0.8026 and perplexity 2.23 — as though choosing between about two options at each step.

What log-probabilities are genuinely good for:

• Spotting where the model was guessing. A sudden dip at one token — a name, a number, an identifier — flags the part to verify.
• Comparing candidates. Scoring several completions of the same prompt is legitimate, because the comparison holds the prompt fixed.
• Cheap flagging for review. Low mean log-probability over an answer is a reasonable trigger for routing it to a human.

And the limit, which is absolute:

Why models hallucinate

Stated mechanically, with no anthropomorphism.

The objective is next-token likelihood, not truth. The model was optimised to continue text plausibly. Nothing in the loop consults a knowledge base, and there is no step where a claim is checked. Truth is not represented in the loss function, so it is not something the system can optimise for. Output that is true is output where the plausible continuation happened to also be correct — which, for well-covered material, is most of the time, and that reliability is exactly what makes the failures dangerous.

Rare things have thin evidence. For facts seen thousands of times in training, the plausible continuation is the correct one. For a rare CVE identifier, an obscure command flag, an internal hostname or a niche library, the model has a pattern for what such a thing looks like but little or no memory of the specific one. It generates something correctly shaped. CVE-2019- plus four digits is a well-learned pattern; which four digits is a memory the model may simply not have.

"I don't know" is a rare continuation. In the text the model learned from, a question is overwhelmingly followed by an answer, not by an admission of ignorance. Instruction tuning and alignment can push a model toward hedging, but that is a learned preference competing against the base distribution — a tendency, not a lookup that returns null.

The requested format applies pressure. Ask for five CVEs and the shape of the answer demands five items. If the model has strong evidence for two, the remaining three slots still get filled, because a list of five is what a list of five looks like. This produces the most operationally dangerous output security people encounter: a well-formatted table in which some rows are real and some are not, with no visual difference between them.

Why the same prompt gives different answers

Two distinct causes, and conflating them causes a lot of confused debugging.

Cause one: you are sampling. At any temperature above zero, step four is a random draw. Because each token is appended and fed back in, one different token early can send the whole response somewhere else. This cause is entirely under your control: set temperature to 0 and fix the seed.

Cause two: floating-point arithmetic is not associative. Even at temperature 0, (a + b) + c and a + (b + c) can differ in the last bits. Which order the additions happen in depends on batch size, kernel selection, hardware and build. A production endpoint batches your request with whoever else's arrived at the same moment, so the arithmetic differs between calls. Two logits separated by less than that noise can swap places, and if that happens at token one, the whole output diverges. This cause is not under your control on a shared, hosted endpoint.

On your own laptop, running one request at a time against a fixed local model, cause two is largely absent and you can usually get byte-identical output. Test that claim rather than believing it.

Prove both on your own machine

# docker-compose.yml — CPU-ONLY. llama3.2:1b is about 1.3 GB.
services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"

  lab:
    image: python:3.12-slim
    working_dir: /work
    volumes:
      - ./:/work
    depends_on:
      - ollama
    environment:
      OLLAMA_HOST: "http://ollama:11434"
    command: sh -c "sleep 5 && python determinism.py"

volumes:
  ollama:

docker compose up -d ollama
docker compose exec ollama ollama pull llama3.2:1b     # CPU-ONLY, ~1.3 GB

# determinism.py — the same prompt, twenty times, two settings.
import hashlib, json, os, urllib.request

OLLAMA = os.environ.get("OLLAMA_HOST", "http://localhost:11434")
PROMPT = "Name one common technique used to detect lateral movement. One sentence."

def generate(options):
    body = json.dumps({"model": "llama3.2:1b", "prompt": PROMPT,
                       "stream": False, "options": options}).encode()
    req = urllib.request.Request(f"{OLLAMA}/api/generate", body,
                                 {"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)["response"]

def fingerprints(options, n=10):
    return {hashlib.sha256(generate(options).encode()).hexdigest()[:12]
            for _ in range(n)}

greedy  = fingerprints({"temperature": 0.0, "seed": 42, "num_predict": 60})
sampled = fingerprints({"temperature": 1.0, "num_predict": 60})

print("temperature 0.0, fixed seed :", len(greedy),  "distinct output(s)")
print("temperature 1.0, free seed  :", len(sampled), "distinct output(s)")

Expect one distinct fingerprint from the greedy run and several from the sampled run — you have just isolated cause one experimentally, since the only difference between the runs is the sampling configuration.

# test_determinism.py — the auto-grader.
from determinism import fingerprints

def test_greedy_is_reproducible():
    """Temperature 0 with a fixed seed, single-request local server."""
    assert len(fingerprints({"temperature": 0.0, "seed": 42, "num_predict": 60}, n=10)) == 1

def test_sampling_is_not_reproducible():
    """The phenomenon, not the words: at temperature 1.0 the outputs diverge."""
    assert len(fingerprints({"temperature": 1.0, "num_predict": 60}, n=10)) >= 2

def test_token_accounting_is_reported():
    import json, os, urllib.request
    body = json.dumps({"model": "llama3.2:1b", "prompt": "hello",
                       "stream": False}).encode()
    req = urllib.request.Request(
        f"{os.environ.get('OLLAMA_HOST', 'http://localhost:11434')}/api/generate",
        body, {"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        d = json.load(r)
    assert d["prompt_eval_count"] > 0 and d["eval_count"] > 0

Self-check

Cheat sheet

Where to go next

You have finished Track C. Every rung of AI literacy is behind you.