Shell and Data Wrangling for AI Work

Why a terminal, when the whole point of AI is that it talks to you

There is a fair question buried in this page. If large language models take instructions in English, why should anyone learn a shell?

Because of what the work actually looks like once you stop demoing and start measuring. A model gives you one answer at a time in a chat window. Security work needs the other thing: two hundred answers, filtered, counted, and compared against what the same model said yesterday. You need to know how often a guardrail held, not whether it held once. You need to strip the API keys out of a transcript before it goes into a ticket. You need to look at the eleventh row of a training set that has forty million rows in it. None of that is a chat interaction. All of it is three commands joined by two pipes.

There is a second reason, less obvious and more important. Almost every AI system you will ever attack or defend is, underneath the marketing, an HTTP endpoint that accepts JSON and returns JSON. Model weights are files. Datasets are files. Logs are files. Prompts are files. The shell is the tool built specifically for pushing text through transformations, and text is the entire substrate of this field. A person who can compose curl, jq and awk can interrogate any AI product on earth without waiting for someone to build them a dashboard.

So: the shell is not the product. It is the floor. This page is deliberately tight — enough to make every later lab runnable, and no more. You are not going to become a systems administrator here, and you do not need to.

Step 1 — Get a shell you cannot break

You need a Linux shell. The three honest options, in order of how little can go wrong:

Use Docker. If you have Docker Desktop (or Docker Engine) installed, one command gives you a throwaway Ubuntu with your current folder mounted inside it. When you exit, the container is deleted and your laptop is untouched. This is the option we assume for the rest of the curriculum.

Use WSL2 on Windows. wsl --install from an administrator PowerShell gives you a real Ubuntu. Fine, and slightly more permanent.

Use the macOS Terminal. macOS is Unix, so nearly everything on this page works as written. What differs is sed and a handful of flags, which BSD implements differently to GNU. Where it matters, we say so.

docker run -it --rm -v "$PWD":/work -w /work ubuntu:24.04 bash

Reading that left to right: run starts a container, -it gives it an interactive terminal, --rm deletes it on exit, -v "$PWD":/work mounts your current directory at /work, -w /work starts you there, ubuntu:24.04 is the image, and bash is the program to run. All of that is explained properly on the containers page; for now just type it.

We use Ubuntu, not Kali. Kali is a rolling release built for offensive tooling — it is heavy, it changes under you, and its Python packaging fights with virtual environments. Kali appears later in this curriculum as a tool you reach for, not as the ground you stand on.

Inside the container, install the handful of tools this page needs:

apt-get update && apt-get install -y curl jq less nano ca-certificates

Step 2 — Where am I, and what is here

The Linux filesystem is a single tree with one root, /. There are no drive letters. Everything — your files, your devices, your running processes — hangs off that one root.

pwd                      # print working directory: /work
ls                       # list this directory
ls -la                   # long format, including hidden dotfiles
ls -lh runs/             # human-readable sizes, of a subdirectory
cd runs                  # move down into runs/
cd ..                    # move up one level
cd /work                 # absolute path — works from anywhere
cd                       # with no argument: go home
mkdir -p runs/2026-08/raw   # create nested directories, no complaint if they exist

A path starting with / is absolute — it is measured from the root and means the same thing everywhere. A path that does not is relative — measured from wherever you are standing. Nearly every "file not found" a beginner hits is a relative path typed from the wrong directory, which is why pwd is worth typing whenever something surprises you.

Copy, move and delete:

cp results.jsonl results.jsonl.bak     # copy
cp -r runs/ runs-backup/               # copy a directory and its contents
mv results.jsonl runs/2026-08/         # move (also how you rename)
rm results.jsonl.bak                   # delete a file
rm -r runs-backup/                     # delete a directory and everything under it

Step 3 — Reading files, including the ones that are too big to read

A model evaluation run produces a JSONL file: one JSON object per line, one line per prompt. After a serious run that file is gigabytes. Opening it in an editor will hang your machine. This is the moment beginners learn that the terminal is not a worse GUI — it is a different instrument.

cat prompts.txt              # dump a whole (small!) file to the screen
head -n 5 results.jsonl      # first 5 lines
tail -n 5 results.jsonl      # last 5 lines
tail -f run.log              # follow a log as it is written — Ctrl-C to stop
less results.jsonl           # page through: space to advance, / to search, q to quit
wc -l results.jsonl          # count lines = count records
du -h results.jsonl          # how big is it, really

head, tail and less never load the whole file. cat does. The rule is simple: cat for things you wrote, less or head for things a machine wrote.

Step 4 — Permissions, and why your API key is currently readable by everyone

Every file has an owner, a group, and nine permission bits: read, write and execute, for the owner, the group, and everybody else. ls -l prints them as a ten-character string.

-rw-r--r--  1 analyst analyst   61 Aug  9 10:22 .env
-rwxr-xr-x  1 analyst analyst 1421 Aug  9 10:24 run-eval.sh
drwxr-xr-x  3 analyst analyst 4096 Aug  9 10:25 runs

The first character is the type (- file, d directory). Then three triples. rw-r--r-- means the owner can read and write, and everyone else on the machine can read it. For run-eval.sh that is fine. For .env, which holds your API key, it is a finding.

chmod 600 .env               # owner read+write, nobody else anything
chmod +x run-eval.sh         # make a script executable
chmod 700 keys/              # a directory only you can enter
ls -l .env                   # verify — always verify
whoami                       # which user am I
id                           # which groups am I in

The numbers are octal: read is 4, write is 2, execute is 1, added together per triple. 600 is 4+2, 0, 0. 755 is 4+2+1, 4+1, 4+1. You will memorise 600, 644, 700 and 755 by repetition and never need the rest.

sudo runs a single command as the superuser. Inside our disposable container you are already root, so you will not need it there; on your own machine, treat every sudo as a small decision rather than a reflex.

Step 5 — Processes, and the 16 GB problem

A running program is a process with a numeric PID. When you run a local model, it is a process, and it is competing with your browser for memory.

ps aux | head             # snapshot of running processes
top                       # live view — q to quit
free -h                   # how much RAM is free (Linux)
kill 4821                 # ask process 4821 to exit politely
kill -9 4821              # make it exit; last resort, no cleanup

Running something in the background, and timing it:

./run-eval.sh &           # start in the background, get the shell back
jobs                      # what is running in the background
fg                        # bring the last background job to the foreground
nohup ./run-eval.sh > run.log 2>&1 &   # survive the terminal closing
time ./run-eval.sh        # how long did that actually take

Step 6 — ssh: working on a machine that is not in front of you

Sooner or later the model runs somewhere else — a lab VM, a cloud box, a colleague's workstation. ssh gives you a shell there over an encrypted channel.

ssh-keygen -t ed25519 -C "you@example.com"   # generate a key pair, once
ssh-copy-id analyst@10.10.10.20              # install your public key on the host
ssh analyst@10.10.10.20                      # log in — no password prompt now
ssh analyst@10.10.10.20 'ls -l /srv/models'  # run one command and come back

Two things worth internalising. First: keys, not passwords. Your private key (~/.ssh/id_ed25519) never leaves your machine and must be mode 600; the public key (.pub) is the one you hand out. Second: scp and port-forwarding are the same tool wearing different hats.

scp results.jsonl analyst@10.10.10.20:/srv/runs/      # push a file up
scp analyst@10.10.10.20:/srv/runs/eval.csv .          # pull one down
ssh -L 11434:localhost:11434 analyst@10.10.10.20      # tunnel a remote port

That last one is genuinely useful. It makes the remote machine's Ollama API appear at http://localhost:11434 on your laptop, so all your local scripts work unchanged against a model running elsewhere — and the traffic goes through ssh rather than across the network in the clear. Because an Ollama server has no authentication of its own, an ssh tunnel is frequently the only access control in front of it.

Step 7 — Pipes and redirection: the actual idea

Everything so far has been vocabulary. This is the grammar, and it is the reason the shell outlives every GUI built to replace it.

Every command has three channels: stdin (input), stdout (normal output) and stderr (errors). By default stdin is your keyboard and both outputs are your screen. You can rewire them.

command > file            # send stdout to file, replacing it
command >> file           # append instead
command 2> errors.txt     # send stderr to a separate file
command > out.txt 2>&1    # send both to one file
command < input.txt       # feed a file in as stdin
commandA | commandB       # send A's stdout straight into B's stdin
command | tee out.txt     # write to a file AND keep it on screen

The pipe is the whole trick. Each program does one thing to a stream of text, and you build the tool you need by chaining them. Errors go to a separate channel so that 2>/dev/null can silence noise without discarding your data — and so that > results.txt never accidentally captures an error message as if it were a result, which is the sort of bug that quietly poisons a metric.

Step 8 — grep: find the lines that matter

grep prints lines that match a pattern. In AI work you are usually asking one of two questions: did the model do the thing? and how often?

grep "I cannot"     responses.txt        # lines containing a refusal phrase
grep -i "as an ai"  responses.txt        # case-insensitive
grep -c "I cannot"  responses.txt        # just count the matching lines
grep -n "API_KEY"   -r  .                # search recursively, show line numbers
grep -v "^#"        prompts.txt          # invert: everything that is NOT a comment
grep -E "sk-[A-Za-z0-9]{16,}" -r .       # extended regex: hunt for leaked keys
grep -l "password"  -r ./datasets        # just name the files that match

A worked question: out of 200 responses, how many refused?

grep -icE "i (cannot|can't|won't)|i am unable|as an ai" responses.txt

That is a real measurement, and it took one line. Read the flag before you read the number, though: -c counts matching lines, not responses, so that figure is a response count only if the file holds exactly one response per line — a two-line answer containing the phrase twice would count twice. It is also a proxy, not a truth — you are counting phrases, not intentions, and a model that refuses politely in words you did not anticipate will be counted as compliant. Say that out loud when you report the number. Good measurement in this field is mostly the discipline of naming your proxy.

Step 9 — jq: the JSON verb

Every OpenAI-compatible API — including a local Ollama — returns a JSON object shaped like this:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "model": "llama3.2:1b",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "The capital of France is Paris." },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 24, "completion_tokens": 8, "total_tokens": 32 }
}

You almost never want the whole object. You want one string out of it. jq is a small language for exactly that.

jq '.'                        response.json   # pretty-print the whole thing
jq '.choices[0].message.content' response.json  # the answer, as a JSON string
jq -r '.choices[0].message.content' response.json  # -r = raw, no surrounding quotes
jq '.usage.total_tokens'      response.json
jq -r '.model, .choices[0].finish_reason' response.json

On a JSONL file — one object per line — jq processes each line in turn:

jq -r '.response' results.jsonl                 # every answer, one per line
jq -r 'select(.refused == true) | .prompt_id' results.jsonl   # only the refusals
jq -s 'length' results.jsonl                    # -s slurps into one array, then counts
jq -s 'map(.latency_ms) | add / length' results.jsonl   # mean latency
jq -c '{id: .prompt_id, ok: .passed}' results.jsonl     # -c = one compact line each

And jq is the safe way to build a request body, because it quotes and escapes for you:

PROMPT='Summarise this in one sentence: "quotes" and \backslashes\ break naive scripts'

jq -n --arg m "llama3.2:1b" --arg p "$PROMPT" \
  '{model: $m, messages: [{role: "user", content: $p}], stream: false}' \
  > body.json

curl -s http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --data @body.json \
  | jq -r '.choices[0].message.content'

Never build JSON with string concatenation in a shell script. A prompt containing a double quote will produce malformed JSON, and a prompt containing a crafted quote will produce well-formed JSON that says something you did not intend — which is prompt injection delivered through your own tooling. jq -n --arg costs nothing and closes that door.

Step 10 — awk: the column verb

When the data is a table — CSV, or whitespace-separated — awk is the right tool. It runs a small program once per line, with $1, $2, $3 bound to the fields and $0 to the whole line.

Given eval.csv:

prompt_id,model,refused,latency_ms
p001,llama3.2:1b,true,842
p002,llama3.2:1b,false,1190
p003,llama3.2:3b,true,2604

awk -F, 'NR>1 {print $1, $3}' eval.csv          # -F, sets the separator; skip the header
awk -F, 'NR>1 && $3=="true"' eval.csv           # only the refusals
awk -F, 'NR>1 {n++; s+=$4} END {print s/n}' eval.csv   # mean latency
awk -F, 'NR>1 {t[$2]++; if ($3=="true") r[$2]++} END {for (m in t) printf "%s %.2f\n", m, r[m]/t[m]}' eval.csv

That last line prints a refusal rate per model. It is a whole analytics feature in eighty characters, with no dependencies, and it will still run in ten years. Learn -F, NR, $n, END and associative arrays; skip the rest of awk until you need it.

Step 11 — sed: the substitute verb

sed edits a stream. Ninety per cent of real use is one form: s/pattern/replacement/g.

sed 's/llama3.2:1b/MODEL_A/g' eval.csv               # substitute, globally, per line
sed -E 's/sk-[A-Za-z0-9]{16,}/[REDACTED]/g' transcript.txt   # redact before sharing
sed -n '10,20p' results.jsonl                        # print only lines 10-20
sed '/^$/d' prompts.txt                              # delete blank lines
sed -i.bak 's/gpt-4o/local-model/g' config.yaml      # edit in place, keeping a .bak

The redaction example is not decoration. Before a transcript goes into a bug report, a Slack thread, or — worst case — a prompt to another model, it needs the keys taken out of it. Do it with a command, in a pipeline, every time, rather than by eye.

Step 12 — Putting it together

Here is the payoff: a complete, self-contained evaluation harness in shell. It sends every prompt in a file to a local model, records the answer as JSONL, and reports a refusal rate. Read it now; you will be able to run it after the containers page, which is where the model server on localhost:11434 comes from.

#!/usr/bin/env bash
set -euo pipefail

MODEL="${MODEL:-llama3.2:1b}"
API="${API:-http://localhost:11434/v1/chat/completions}"
OUT="runs/$(date +%Y%m%d-%H%M%S).jsonl"
mkdir -p runs

while IFS= read -r prompt; do
  [ -z "$prompt" ] && continue
  body=$(jq -n --arg m "$MODEL" --arg p "$prompt" \
          '{model: $m, messages: [{role: "user", content: $p}], stream: false}')
  answer=$(curl -s --max-time 120 "$API" \
            -H 'Content-Type: application/json' --data "$body" \
            | jq -r '.choices[0].message.content // "ERROR"')
  jq -nc --arg p "$prompt" --arg a "$answer" '{prompt: $p, answer: $a}' >> "$OUT"
done < prompts.txt

total=$(wc -l < "$OUT")            # -c above kept each record on ONE line, so this counts records
refused=$(jq -r '.answer | gsub("\n"; " ")' "$OUT" \
          | grep -icE "i (cannot|can't|won't)|i am unable" || true)   # one line per answer, so -c counts answers
printf 'model=%s total=%d refused=%d rate=%.2f\n' \
  "$MODEL" "$total" "$refused" "$(awk -v r="$refused" -v t="$total" 'BEGIN{print r/t}')"

Three details in there are worth more than the rest of the script. set -euo pipefail makes the script stop on the first error instead of ploughing on with empty variables — without it, a failed curl produces an empty answer that gets counted as a non-refusal, and your metric is silently wrong. --max-time 120 stops one hung request from stalling the whole run. And || true after grep exists because grep exits non-zero when it finds nothing, which under set -e would kill the script precisely when the result is "zero refusals" — the most interesting outcome. Exit codes are data; read them deliberately.

How the lab for this page is graded

The lab ships a fixed dataset.jsonl and a prompts.txt in the container, and a hidden pytest suite that runs your solve.sh and inspects stdout only. You pass by printing four specific key=value pairs computed from the fixed data. Because the dataset is fixed and the questions are arithmetic over it, the expected answers are exact and the grader is deterministic — no model is involved in grading. Completing all tasks prints the flag string KR{shell-is-the-substrate}.

📋 Cheat sheet