AI Literacy: Thresholds and Confidence

Where this page sits

Level: Beginner. Arithmetic and one exponential function. No calculus.

Rung three of Track C, and it depends directly on rung two. If you cannot yet compute precision from TP and FP, go back and do The Confusion Matrix and Base Rates — this page moves those numbers around and will not make sense otherwise.

The second half of this page is the most consequential misconception correction in the track. Confidence is not correctness. People trust model output they should not, specifically because a number that looks like a probability is displayed next to it.

A classifier does not output a class

Start with the thing that is almost always skipped. Ask a trained classifier about an email and it does not return "phishing". It returns a score — a number, usually between 0 and 1, that ranks this item against every other item it could see.

Turning that score into a decision requires one more ingredient, and the model does not supply it: a threshold. Score at or above the threshold, flag it. Below, let it through.

score = model.predict_proba(email)[0][1]    # e.g. 0.61 — this is all the model gives you
alert = score >= THRESHOLD                  # THRESHOLD is your decision, not the model's

THRESHOLD is a policy choice. Frameworks default it to 0.5 and that default has no justification whatsoever for security work — it is a convention inherited from balanced academic datasets, and your data is not balanced. Every confusion matrix you have ever seen was computed at some threshold, and if nobody stated which one, the numbers are unmoored.

Sweep it by hand

Ten emails, scored by some model, three of them genuinely malicious. Small enough to do entirely on paper — which is the point, because doing it once by hand is worth ten explanations.

Now pick a threshold, draw a line across the sorted list, and count. Everything on or above the line is flagged.

Three things in that table are worth stopping on.

Recall only ever goes up as the threshold comes down. Lowering the bar can never un-catch something you already caught. Recall is monotone in the threshold, always, on every dataset. That is a structural fact, not a property of this data.

Precision is not monotone. It goes 0.500, 0.667, 0.500, 0.400, 0.500, 0.333. It rises when the next item you sweep in is a true positive and falls when it is a false one, so it wanders. Anyone who tells you "lowering the threshold always reduces precision" is describing a tendency, not a rule.

Perfect recall was available, and it cost you nothing extra here. At t = 0.40 recall hits 1.000 with precision 0.500 — better precision than at t = 0.90, where recall was 0.333. The default 0.5 threshold is dominated on this data: t = 0.40 beats it on precision and on recall, so nothing is being traded away by moving. That is a fair warning about defaults.

Note also that the FP at the very top of the list — E2 at 0.91, scoring higher than a real attack — is the single most damaging kind of error. It is the alert an analyst sees first, dismisses, and thereby learns to distrust the top of the queue.

ROC and AUC, without the curve

Sweep the threshold from 1.0 down to 0.0 and plot the pair (FPR, recall) at each stop. That plot is the ROC curve — receiver operating characteristic, a name inherited from radar operators in the 1940s and carrying no useful meaning today. FPR on the x-axis, recall on the y-axis. The curve is the complete menu of operating points a model offers you; picking a threshold is picking a point on it.

AUC is the area under that curve. It collapses the whole menu into one number between 0 and 1. Here is the one-sentence definition that is actually correct, and it does not mention area at all:

That definition lets you compute AUC by counting pairs, with no geometry. Our toy set has 3 malicious and 7 benign, so 3 × 7 = 21 comparisons. Count how many the malicious item wins:

• E1 (0.95) beats all 7 benign scores → 7 wins
• E3 (0.84) beats 0.62, 0.55, 0.33, 0.21, 0.12, 0.04 but loses to E2's 0.91 → 6 wins
• E6 (0.48) beats 0.33, 0.21, 0.12, 0.04 but loses to 0.91, 0.62, 0.55 → 4 wins

AUC = (7 + 6 + 4) / 21 = 17 / 21 = 0.81.

That is the whole calculation. AUC is a ranking measure: it asks whether the model puts bad things above good things, and it is entirely indifferent to where you draw the line.

Which is exactly its limitation. AUC is threshold-free, so it tells you nothing about the operating point you will actually ship. Two models with identical AUC can behave completely differently at the low-FPR end of the curve, which is the only end a security team ever operates in.

And there is a sharper problem. FPR has the benign count as its denominator, and the benign class is enormous. On 99,990 benign emails, an FPR of 0.001 — a rounding error on a ROC plot, indistinguishable from the y-axis — is a hundred false alerts a day. The ROC curve visually compresses the entire region you care about into the leftmost sliver of the plot. The base-rate-aware alternative is the precision-recall curve and its area, AUPRC, whose y-axis is precision and therefore moves with prevalence. When someone shows you a ROC curve for a rare-event detector, ask for the PR curve.

A realistic sweep, reproducible on your laptop

The toy set had three positives, which is far too few to trust. Here is the same exercise at a realistic scale and a realistic base rate. It is a simulation with a fixed seed, so your output will match this page exactly — that is deliberate, so the numbers here are checkable rather than asserted.

# docker-compose.yml — CPU-ONLY. Standard library only; no model, no network.
services:
  lab:
    image: python:3.12-slim
    working_dir: /work
    volumes:
      - ./:/work
    command: python sweep.py

# sweep.py — one day of mail at a base rate of 1 in 10,000.
import random, math
random.seed(1337)

N_BENIGN, N_MAL = 99_990, 10
logistic = lambda x: 1 / (1 + math.exp(-x))
benign    = [logistic(random.gauss(-3.0, 1.2)) for _ in range(N_BENIGN)]
malicious = [logistic(random.gauss( 2.0, 1.2)) for _ in range(N_MAL)]

print(f"{'thresh':>7} {'TP':>4} {'FP':>7} {'recall':>7} {'FPR':>9} {'precision':>10} {'alerts':>7}")
for t in (0.10, 0.25, 0.50, 0.75, 0.90, 0.95):
    tp = sum(1 for s in malicious if s >= t)
    fp = sum(1 for s in benign    if s >= t)
    prec = tp / (tp + fp) if tp + fp else float("nan")
    print(f"{t:>7} {tp:>4} {fp:>7} {tp/N_MAL:>7.2f} {fp/N_BENIGN:>9.5f} {prec:>10.3f} {tp+fp:>7}")

 thresh   TP      FP  recall       FPR  precision  alerts
    0.1   10   25297    1.00   0.25300      0.000   25307
   0.25   10    5719    1.00   0.05720      0.002    5729
    0.5   10     670    1.00   0.00670      0.015     680
   0.75    7      29    0.70   0.00029      0.194      36
    0.9    4       0    0.40   0.00000      1.000       4
   0.95    1       0    0.10   0.00000      1.000       1

Read that table as an operational document rather than a statistics exercise.

At the framework default of 0.5 the detector catches every single attack — perfect recall — and hands the SOC 680 tickets a day, of which 670 are nothing. At 0.75 the queue collapses to 36 tickets and precision rises to about 19%, but three attacks in ten now walk straight through. At 0.9 every alert is real and six of the ten attacks are invisible.

There is no row in that table that is "correct". There is only the row that matches what your team can actually process, and choosing it is a risk decision that belongs to a human.

Pick the operating point from your budget

The useful inversion: stop asking what threshold the model recommends and start with what your team can absorb.

This is also the honest answer to "can you make the detector better?" — usually not by moving the threshold, because the threshold only slides along a fixed curve. Improving the detector means moving the whole curve, which means better features, better data, or narrowing the population as in the previous rung.

Confidence is not correctness

Now the part that changes how you read model output for the rest of your career.

A neural classifier's last layer produces raw numbers called logits — unbounded, uninterpretable, one per class. To turn them into something that looks like probabilities, they go through softmax: exponentiate each one, then divide by the total so they sum to 1.

Work it on three logits, [2.0, 1.0, 0.1]:

exp(2.0) = 7.3891      exp(1.0) = 2.7183      exp(0.1) = 1.1052
sum = 11.2125

7.3891 / 11.2125 = 0.659      <- "65.9% confident"
2.7183 / 11.2125 = 0.242
1.1052 / 11.2125 = 0.099

The model is "66% confident" in class one. Now multiply every logit by three — [6.0, 3.0, 0.3] — and redo it:

exp(6.0) = 403.429     exp(3.0) = 20.086      exp(0.3) = 1.350
sum = 424.864

403.429 / 424.864 = 0.9495    <- "95.0% confident"
 20.086 / 424.864 = 0.0473
  1.350 / 424.864 = 0.0032

The ranking is identical. The decision is identical. The "confidence" went from 66% to 95%. Nothing was learned in between. All that changed is the scale of the numbers entering the softmax — which is a property of how the network's final layer happened to be trained, not a property of the evidence.

Two conclusions follow, and both are load-bearing:

Softmax is a normalisation, not a measurement. Its output sums to 1 because it was constructed to, not because the world was consulted. Feed a classifier something from outside its training distribution entirely — an image of static to an animal classifier, a language it never saw to a text classifier — and it will still emit numbers summing to 1, often with one of them near 1.0. The model has no representation for "this is not any of my classes"; the architecture forbids that answer.

The number is a rank, and only reliably a rank. You can trust "the model scored this higher than that". You cannot, without further work, read 0.95 as "95% likely to be correct".

Calibration: making the number mean what it looks like

There is a precise definition for the property people assume they are getting. A model is calibrated if, across all the times it outputs 0.8, it is correct about 80% of the time. Bucket the predictions by confidence, measure the accuracy inside each bucket, and compare — that comparison is a reliability diagram, and it is a fifteen-line script over data you already have.

Deep networks are typically overconfident: the 0.9 bucket is right less often than 90% of the time. Techniques exist to fix it — temperature scaling on a held-out set is the simplest and often sufficient — but the essential point for a security practitioner is prior to any technique:

The LLM version, which is worse

Everything above concerns a classifier, where at least the number comes from the model's own output layer. With a chat model it is worse in a way worth being explicit about.

When an LLM writes "I am about 90% confident this log line indicates lateral movement", that 90% is generated text. It was sampled token by token, like every other part of the sentence, because it was a plausible continuation of the preceding words. No internal quantity was consulted or reported. A model that is completely wrong can write "95% confident" with exactly the same machinery it uses to write "the".

There is a real quantity available from a language model — the log-probability of each token it emitted — and it is genuinely useful. But it measures something narrow: how expected each word was, given the words before it. A fluent, confident, entirely fabricated sentence typically has high token probabilities, because fluency is exactly what next-token training optimises. High token probability means "this reads like the training data". It does not mean "this is true". The next rung takes that apart properly.

Run it and grade yourself

# choose.py — implement this.
def choose_threshold(scores_and_labels, alert_budget, min_recall):
    """Return the LOWEST threshold from the candidate list whose alert count
    fits the budget and whose recall clears the floor, else None.

    scores_and_labels: list of (score, label) with label 1 = malicious
    """
    candidates = [i / 100 for i in range(100, -1, -1)]   # 1.00 down to 0.00
    best = None
    positives = sum(lbl for _, lbl in scores_and_labels)
    for t in candidates:
        tp = sum(1 for s, lbl in scores_and_labels if s >= t and lbl == 1)
        alerts = sum(1 for s, _ in scores_and_labels if s >= t)
        if alerts <= alert_budget and tp / positives >= min_recall:
            best = t
    return best

# test_choose.py — the auto-grader.
import random, math
from choose import choose_threshold

def dataset(seed=1337):
    random.seed(seed)
    lg = lambda x: 1 / (1 + math.exp(-x))
    d  = [(lg(random.gauss(-3.0, 1.2)), 0) for _ in range(99_990)]
    d += [(lg(random.gauss( 2.0, 1.2)), 1) for _ in range(10)]
    return d

def test_budget_is_respected():
    d = dataset()
    t = choose_threshold(d, alert_budget=40, min_recall=0.60)
    assert t is not None
    assert sum(1 for s, _ in d if s >= t) <= 40          # invariant, not a fixed number
    assert sum(l for s, l in d if s >= t) / 10 >= 0.60

def test_impossible_ask_returns_none():
    # perfect recall inside 5 alerts is not on this curve; the honest answer is None
    assert choose_threshold(dataset(), alert_budget=5, min_recall=1.0) is None

def test_recall_is_monotone_in_the_threshold():
    d = dataset()
    rec = lambda t: sum(l for s, l in d if s >= t)
    assert all(rec(t) >= rec(t + 0.05) for t in [0.1, 0.3, 0.5, 0.7, 0.9])

Self-check

Cheat sheet

Where to go next