# Module 03 — Prompting & Sampling: Lab Guide These labs are the runnable complement to the two prose lessons in `frontend/content/modules/03-prompt/`. Each exercise drives the real `mpl` CLI and the underlying `ollama` surface it reveals, tying terminal output directly back to the mechanisms taught: chat templates as the deterministic function that renders your `messages` array into a flat token string, the six-step sampler pipeline that converts logits into tokens each decode step, and constrained decoding as a hard structural guarantee — not a prompting hope. --- ## Prerequisites Work through these steps in order. Every gated `mpl` verb (`run` and `serve`) will refuse with an enrollment error if the entitlement is missing. ### 1. Enroll on-chain Purchase enrollment through the mindpool.io app. Enrollment records a `FLAG_HAS_PAID` bit in the DefiMentor contract on Polygon. `mpl` checks this on-chain; there is no account on our servers. ### 2. Activate the entitlement ```bash mpl login ``` `mpl login` prompts for your wallet address, prints a SIWE message, asks you to sign it in MetaMask (or any wallet), and verifies your `FLAG_HAS_PAID` status on-chain via a public Polygon RPC. A time-limited entitlement is cached in `~/.mpl/`. Re-run `mpl login` to refresh. Verify success: ```bash mpl status ``` You should see your address and `enrolled: true`. ### 3. Confirm the local AI stack is installed `mpl setup` is a **module 01 (`01-foundations`) verb** — it should have been run when you completed that module. If you skipped it, run it now: ```bash mpl setup ``` `mpl setup` installs pinned versions of the tools this curriculum uses (Ollama, llama.cpp) and any required Python virtual environments. It does **not** pull `qwen3:8b` or other models — model weights are fetched separately. The first run can take several minutes on a slow connection; subsequent runs skip already-present components. Verify the stack is healthy: ```bash mpl doctor ``` ### 4. Confirm `qwen3:8b` is available `qwen3:8b` is the primary teaching model for this module. It is a *reasoning model* — it emits a `…` internal chain-of-thought block before its answer. This is pedagogically valuable here: the thinking block is not cosmetic — it is produced by the chat template's think-switch mechanism, which this module dissects in detail. Check that Ollama knows about it: ```bash ollama list ``` You should see `qwen3:8b` in the list. If not, pull it directly: ```bash ollama pull qwen3:8b ``` `qwen3:8b` is fetched lazily: `mpl run qwen3:8b` will auto-pull it on first run if Ollama does not have it yet. `ollama pull qwen3:8b` makes the pull explicit so you know it is complete before starting the exercises. --- ## How to work the labs - **`lab-01-templates.md`** — the INPUT side. Drives `ollama show`, `mpl run qwen3:8b`, and `mpl serve qwen3:8b` + `curl`. Covers inspecting the real qwen3:8b chat template, tracing the `<|im_start|>`/`<|im_end|>` ChatML markers, seeing how `mpl run`'s transparency line wraps your turn, understanding the think/no_think switch in the template, and confirming that the server applies the template to your JSON `messages` before the model ever sees them. - **`lab-02-sampling.md`** — the OUTPUT side. Drives `mpl run qwen3:8b` (REPL `/set parameter`), `mpl serve qwen3:8b` + `curl` (`options` field), and `ollama run --format`. Covers the six sampler knobs live in the REPL, the same knobs via the API `options` field, and structured output (constrained decoding) as a hard guarantee via `--format json`, full JSON Schema, and the serve API `format` field. Each exercise has the form: > **COMMAND** — the exact terminal command to run > **OBSERVE** — what to look for in the output > **EXPLAIN** — what it means, connecting to lesson mechanics Reference answers and expected outputs are in `solutions/lab-01.md` and `solutions/lab-02.md`. Use them for self-checking, not as a shortcut — the learning is in the observation. --- ## Manifest reference These labs are registered in the CLI curriculum manifest: ``` cli/curriculum.manifest.json → module slug "03-prompt" → labFiles: "module-03/" ``` The `mpl` binary resolves `labFiles` relative to the CLI source root when reporting module metadata. Students who obtained `mpl` through the enrollment channel can open these files directly from the repo or from the installed distribution. --- ## A note on enrollment gating The `mpl` verbs used in these labs — `run` and `serve` — are gated. Attempting to run them without a valid entitlement produces an enrollment error and a pointer to `mpl login`. This is intentional: the labs are part of the enrolled curriculum, not a free sampler. `mpl doctor`, `mpl login`, `mpl status`, and all `ollama show` commands are always open. (`mpl setup` is gated too but belongs to module 01 — it is a prerequisite for this module, not a verb you call during module 03 exercises.) --- # Lab 01 — Chat Templates: From Messages to the Token Stream > **Lesson connection:** lesson-01 taught that your `messages` array never reaches the model — a deterministic chat template renders it into a single flat token string before the model touches it. This lab grounds every step of that transformation in real terminal output: the template itself, the ChatML markers, the generation prompt, the transparency line `mpl run` produces, and the think/no_think switch that lives entirely in template logic. The server-side path closes the loop. **Prerequisites:** `mpl login` completed, `mpl setup` completed, `qwen3:8b` available via `ollama list`. --- ## Exercise 1 — Inspect your model's real chat template **COMMAND** Run all three commands: ```bash ollama show --template qwen3:8b ``` ```bash ollama show --parameters qwen3:8b ``` ```bash ollama show --system qwen3:8b ``` **OBSERVE** `--template` prints the full Go template that the Ollama server uses every time it renders a request for qwen3:8b. Locate these four things in the output: 1. **`<|im_start|>` and `<|im_end|>`** — these wrap every turn. Look for `<|im_start|>user`, `<|im_start|>assistant`, `<|im_start|>system` at the start of each turn block, and `<|im_end|>` closing each one. 2. **Turn shape** — each turn follows the pattern `<|im_start|>{role}\n{content}<|im_end|>\n`. The `\n` after the role marker and the `\n` after `<|im_end|>` are literal characters that become literal tokens. 3. **The generation prompt** — at the end of the template, when the last message is not an assistant turn, the template emits `<|im_start|>assistant\n` with no closing `<|im_end|>`. This is the open partial turn the model is asked to complete. Every token the model generates fills in after that trailing newline. 4. **System block conditional** — find the `{{- if or .System .Tools }}` guard. The system block is only emitted if you provide `.System` content or tool definitions. Without it, no system tokens appear in the rendered string at all. `--parameters` prints: ``` repeat_penalty 1 stop "<|im_start|>" stop "<|im_end|>" temperature 0.6 top_k 20 top_p 0.95 ``` `--system` prints nothing — an empty result. **EXPLAIN** Three mechanisms are visible: 1. **ChatML markers.** The `<|im_start|>` / `<|im_end|>` pair comes from the ChatML convention. These are vocabulary tokens, not metadata. They occupy positions in the sequence just like any word token. The model was fine-tuned to associate "tokens after `<|im_start|>assistant\n`" with "produce a helpful reply" — without the SFT step that created this association, the markers would be meaningless. 2. **The generation prompt.** The template appends `<|im_start|>assistant\n` with no closing marker. The model sees context ending mid-turn and is asked to complete the continuation. Every request sends this partial turn; it contributes tokens to your prefill count. 3. **No default system prompt.** `ollama show --system qwen3:8b` returns nothing because qwen3:8b ships with no default system message. When you omit the `"role": "system"` message from your request, the template emits no system block at all — there is no hidden fallback. The default parameters (`temperature 0.6`, `top_k 20`, `top_p 0.95`, `repeat_penalty 1`) are the starting values for every qwen3:8b conversation. Lab 02 drives these knobs live; they are shown here so you can see they come from the model's Modelfile, not from `mpl`. --- ## Exercise 2 — The transparency line wraps your turn **COMMAND A — open the REPL and observe the transparency line** ```bash mpl run qwen3:8b ``` At the `>>>` prompt, type a question such as: ``` What is the difference between compilation and interpretation? ``` **OBSERVE** Before any model output arrives, `mpl` prints a line starting with `→`: ``` → ollama run qwen3:8b ``` The model output then begins (preceded by a `` block before the visible answer). **COMMAND B — dry-run to see the transparency line without opening the REPL** ```bash mpl run qwen3:8b --explain --dry-run ``` **OBSERVE** ``` → ollama run qwen3:8b ``` **COMMAND C — dry-run with a context-size override** ```bash mpl run qwen3:8b --explain --ctx-size 8192 --dry-run ``` **OBSERVE** ``` → ollama run qwen3:8b (+1 env var) OLLAMA_CONTEXT_LENGTH=8192 ``` **EXPLAIN** Three things are visible: 1. **The transparency contract.** `mpl run` is an orchestrator, not a black box. Every gated verb prints the exact underlying command before running it. Because `mpl run` launches an interactive REPL (not a one-shot command), the transparency line shows the bare `ollama run qwen3:8b` without an inline prompt string. 2. **`--explain --dry-run`** — `--explain` adds the environment-override layer to the output; `--dry-run` prints the plan without opening the REPL. Use this combination to inspect what `mpl` will do without executing it. 3. **Context length is an environment variable.** `mpl run` passes `OLLAMA_CONTEXT_LENGTH` as an env var to the underlying `ollama run` process — not as a flag. You can verify: `ollama run --help` has no `--context-length` flag. Note what `--explain` does NOT print: there is no quant/backend breakdown (`quant: Q4_K_M`, `backend: ollama`). Those facts are real and recorded in the curriculum manifest at `cli/curriculum.manifest.json` under the `qwen3:8b` entry; they do not come from `--explain`. --- ## Exercise 3 — The thinking switch lives in the template **COMMAND A — default (thinking enabled)** Start `mpl run qwen3:8b` and ask a short question, or use the underlying command directly: ```bash ollama run qwen3:8b "What is 17 + 26?" ``` **OBSERVE** The ollama CLI renders the reasoning trace as a `Thinking...` / `...done thinking.` spinner block before the answer: ``` Thinking... The user is asking 17 + 26... [reasoning text streams here] ...done thinking. 43 ``` > **Note:** the `Thinking...` / `...done thinking.` framing is the **CLI's display rendering** of the underlying `...` template block. The raw template and the API response use literal `` / `` XML tags; the `ollama run` terminal renders them as the spinner label. When you query the native `/api/chat` endpoint directly, the `message.content` field contains the literal `...` delimiters. The thinking tokens arrive at the same per-token decode rate as the answer tokens. **COMMAND B — suppress thinking with `--think=false`** ```bash ollama run --think=false qwen3:8b "What is 17 + 26?" ``` **OBSERVE** No `` block appears. The answer arrives immediately without a reasoning trace. Notice that the time to first visible output token is shorter — there are no thinking tokens consuming decode steps before the answer. **COMMAND C — suppress thinking with `--hidethinking` (hides the block, still generates it)** ```bash ollama run --hidethinking qwen3:8b "What is 17 + 26?" ``` **OBSERVE** The `` block is hidden from terminal output, but the model still generates it internally. The answer arrives with a delay proportional to the thinking-token count — the decode steps happened, you just do not see them. **COMMAND D — toggle inside the REPL** Start `mpl run qwen3:8b`. At the `>>>` prompt, type: ``` /set nothink ``` Then ask a question. Then type `/set think` to re-enable and ask the same question again. **OBSERVE** With `/set nothink`: no `` block, faster time to first answer token. With `/set think`: `` block returns. **EXPLAIN** Look back at the template from Exercise 1. Find the `$.IsThinkSet` / `$.Think` conditional: - **`--think` (default):** the template appends ` /think` immediately before the last user turn's `<|im_end|>`. The model was fine-tuned to respond to this suffix by producing a `...` block before the answer. This is a learned token-level signal, not a model architecture switch. - **`--think=false` / `/set nothink`:** instead of a suffix on the user turn, the template *pre-fills* the assistant generation prompt with an empty think block: ``` <|im_start|>assistant ``` The model sees the `` block already closed. Because it was fine-tuned to produce its answer after the think block, it immediately begins the answer — the reasoning capacity is bypassed, not disabled. - **`--hidethinking`:** thinking mode is active (the ` /think` suffix is in the rendered string), but the Ollama CLI strips the `...` output from the terminal display. The model still generates all thinking tokens; decode steps are consumed; latency is unchanged. The key lesson from lesson-01: **reasoning mode is a prompt-construction choice, not a model-architecture choice.** The template is the mechanism; you can inspect it, trace it, and reason about its cost in decode-phase tokens. --- ## Exercise 4 — The server applies the template: send `messages`, receive rendered output **Setup — Terminal A** Start the model server: ```bash mpl serve qwen3:8b ``` **OBSERVE** `mpl serve` prints its transparency line before starting the server: ``` → ollama serve ``` Leave Terminal A running. Open **Terminal B** for the remaining commands. **COMMAND (Terminal B)** Send a two-turn conversation — a system message followed by a user message — to the OpenAI-compatible endpoint: ```bash curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3:8b", "messages": [ {"role": "system", "content": "You are a concise technical assistant. Answer in one sentence."}, {"role": "user", "content": "What is a mutex?"} ], "temperature": 0.0, "max_tokens": 80 }' ``` **OBSERVE** The full JSON response arrives (in non-streaming mode, the terminal is silent until all tokens are generated). The `choices[0].message.content` field contains the assistant's answer. The `usage` object shows: ```json "usage": { "prompt_tokens": ..., "completion_tokens": ..., "total_tokens": ... } ``` Note the `prompt_tokens` value. It is higher than the word count of your two messages because: - The system message is rendered inside `<|im_start|>system\n...<|im_end|>\n` - The user message is rendered inside `<|im_start|>user\n...<|im_end|>\n` - The generation prompt `<|im_start|>assistant\n` adds more tokens - All the special tokens (`<|im_start|>`, `<|im_end|>`, role strings, newlines) are counted **EXPLAIN** This completes the input-side picture from lesson-01: ``` your JSON messages array ↓ Ollama server reads the request ↓ runs the qwen3:8b chat template ↓ produces a flat token string: <|im_start|>system You are a concise technical assistant. Answer in one sentence.<|im_end|> <|im_start|>user What is a mutex?<|im_end|> <|im_start|>assistant [model completes from here] ↓ tokeniser → integer ids → prefill → decode → response ``` The model never received JSON. The role strings `"system"` and `"user"` were consumed by the template renderer to select which special tokens to emit. The `messages` array is a convenience layer that exists only between your code and the Ollama HTTP boundary. `temperature: 0.0` sets greedy decoding (deterministic: re-running this exact command should produce the same answer). This is the same `temperature` knob you will vary in lab-02. --- ## Teardown When you are done, stop the server in Terminal A with `Ctrl-C`. To stop the Ollama daemon fully: ```bash # macOS pkill ollama # or ollama stop ``` --- # Lab 02 — Sampling: Knobs, Pipeline Order, and Constrained Decoding > **Lesson connection:** lesson-02 opened the single "logits → token" arrow from 02-run into a six-step ordered pipeline: repetition penalties → temperature → top-k → top-p → min-p → renormalise + sample. It also showed that constrained decoding applies the same `−∞` logit trick, but driven by a grammar state machine instead of a probability threshold. This lab makes every step live in your terminal. **Prerequisites:** `mpl login` completed, `mpl setup` completed, `qwen3:8b` available via `ollama list`, `curl` on PATH. --- ## Setup: start the server Open **Terminal A** and start the model server: ```bash mpl serve qwen3:8b ``` **OBSERVE** ``` → ollama serve ``` The Ollama daemon exposes two HTTP endpoints used in this lab: ``` http://localhost:11434/api/chat ← native endpoint (honours options + format) http://localhost:11434/v1/chat/completions ← OpenAI-compat endpoint (top-level temperature/top_p only) ``` Exercise 2 uses `/api/chat` so that the `options` object is actually honoured. Exercise 3 also uses `/api/chat` for the `format` field. Exercise 1 uses the REPL. Leave Terminal A running. Open **Terminal B** for the remaining exercises. --- ## Exercise 1 — The sampler pipeline knobs in the REPL **COMMAND A — start the REPL and set temperature 0 (greedy)** ```bash mpl run qwen3:8b ``` At the `>>>` prompt, set greedy decoding: ``` /set parameter temperature 0 ``` Then ask a question twice (press Enter after each answer to get back to the prompt): ``` Name a European capital city. ``` Run the same prompt a second time. **OBSERVE** Both runs produce the same answer — the same city, the same wording. Temperature 0 collapses the probability distribution to the argmax token at every decode step. Greedy decoding is deterministic. **COMMAND B — high temperature** In the same REPL session: ``` /set parameter temperature 1.5 ``` Ask the same question five times: ``` Name a European capital city. ``` **OBSERVE** The answers vary across runs. At temperature 1.5 the logit vector is divided by 1.5 before softmax, flattening the distribution. Lower-probability tokens become eligible. You see diversity in both city name and phrasing. **COMMAND C — observe the full pipeline knobs** Still in the same REPL session, demonstrate the remaining pipeline steps: ``` /set parameter top_k 20 /set parameter top_p 0.95 /set parameter min_p 0.05 /set parameter repeat_penalty 1.1 ``` Type a prompt to generate some output, then reset to sensible defaults: ``` /set parameter temperature 0.6 /set parameter repeat_penalty 1 ``` **OBSERVE** Each `/set parameter` command adjusts a live knob for the current REPL session. `top_k 20` limits the candidate pool to the 20 highest-logit tokens. `top_p 0.95` further trims the tail to the smallest set whose cumulative probability reaches 0.95. `min_p 0.05` drops any token whose probability falls below `0.05 × p_max`. `repeat_penalty 1.1` divides the logit of any recently generated token by 1.1, discouraging repetition (the default is 1.0 — no penalty). Note: `/set parameter` is the Ollama REPL command surfaced through `mpl run`. There are no `--temperature`, `--top-p`, `--top-k`, `--min-p`, or `--format` flags on `mpl run` itself. Verify: `mpl run --help`. The valid `mpl run` flags are `--backend`, `--explain`, `--ctx-size`, `--json`, `--dry-run`, and `--engine-arg` — the last is a repeatable raw escape that appends verbatim tokens to the underlying engine command (G1), and it does not change this story: `ollama run` has no sampler flags for it to reach. Sampling knobs are a REPL or API concern. **EXPLAIN** Each knob corresponds to one step in lesson-02's ordered pipeline, applied in this sequence every decode step: 1. **Repetition penalty** (`repeat_penalty 1.1`): divide the logit of recently-seen tokens by the penalty factor. Runs first so the correction is independent of temperature scaling. At the default value of 1.0, division by 1.0 is a no-op — the knob is disabled. 2. **Temperature** (`temperature 0.6`): divide all logits by T. Values below 1.0 sharpen the distribution (fewer tokens get meaningful probability). Values above 1.0 flatten it. Runs before truncation so that the nucleus is computed on the post-temperature landscape — this is why the ordering is not cosmetic. 3. **Top-k** (`top_k 20`): keep only the 20 highest-logit tokens; set all others to −∞. A count-based hard cut, oblivious to the distribution's actual shape. 4. **Top-p** (`top_p 0.95`): among the survivors of top-k, keep the smallest set whose cumulative probability reaches 0.95; set the rest to −∞. Adapts to distribution shape — the nucleus shrinks on peaked steps, grows on flat ones. 5. **Min-p** (`min_p 0.05`): keep only tokens whose probability is at least `0.05 × p_max` (where `p_max` is the surviving top token's probability). A relative floor that scales with model confidence. 6. **Renormalise + sample**: softmax over the surviving tokens, draw one. At temperature 0 this degenerates to argmax. --- ## Exercise 2 — The same knobs over the API This exercise repeats Exercise 1's temperature contrast using the `mpl serve` API `options` field instead of the REPL. > **Endpoint note (verified against Ollama 0.30.7):** the `options` object is only honoured by Ollama's **native** `/api/chat` endpoint. The OpenAI-compatible `/v1/chat/completions` endpoint silently accepts `options` but ignores it entirely — only top-level `temperature` and `top_p` are respected there. Both commands below use `/api/chat` so the `options` fields actually take effect. Extract the result from `message.content` (native endpoint shape). **COMMAND A — greedy decoding via API (temperature = 0)** ```bash curl http://localhost:11434/api/chat \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3:8b", "messages": [{"role": "user", "content": "Name a European capital city. /no_think"}], "options": { "temperature": 0 }, "stream": false }' ``` Run this twice. **COMMAND B — high temperature and additional knobs via API** ```bash curl http://localhost:11434/api/chat \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3:8b", "messages": [{"role": "user", "content": "Name a European capital city. /no_think"}], "options": { "temperature": 1.5, "top_k": 20, "top_p": 0.95, "min_p": 0.05, "repeat_penalty": 1.1 }, "stream": false }' ``` Run this twice. **OBSERVE** At `"options": {"temperature": 0}`, the two greedy runs produce near-identical `message.content`. At `"options": {"temperature": 1.5, ...}`, the two runs produce different content — different city names or different phrasing — for the same reason as the REPL experiment. The determinism contrast is real here because `/api/chat` actually passes `options` to the sampler; the same `options` object sent to `/v1/chat/completions` would be silently dropped. Note the `options` object: this is Ollama's native knob surface, honoured **only on `/api/chat`**. It exposes `top_k` and `min_p`, which are not available in the OpenAI-compatible top-level request fields. The OpenAI-compat path (`/v1/chat/completions`) respects only top-level `temperature` and `top_p` — no `options`, no `top_k`, no `min_p`. When you need full pipeline control, use `/api/chat` with `options`. **The OpenAI-compat path (for completeness):** if you are targeting the `/v1` endpoint, pass `temperature` and `top_p` at the top level (no `options` wrapper) — those two fields are the entire sampler surface available there: ```bash curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3:8b", "messages": [{"role": "user", "content": "Name a European capital city. /no_think"}], "temperature": 0, "max_tokens": 30 }' ``` This is the correct path when you need OpenAI-SDK compatibility. It controls `temperature` and `top_p` only; `top_k`, `min_p`, and `repeat_penalty` are not reachable from `/v1`. **EXPLAIN** The pipeline is identical to Exercise 1. The difference is how you express the settings: REPL `/set parameter` is stateful (persists for the session until changed); the API `options` field is per-request. Both reach the same sampler code in the llama.cpp backend. A codebase that targets `mpl serve` in development can move to vLLM or a managed cloud endpoint in production with no application-layer changes — only the `base_url` and `api_key` change. --- ## Exercise 3 — Structured output as a hard guarantee This exercise uses `ollama run --format` directly (not `mpl run`) and the `mpl serve` API. `mpl run` has no `--format` flag — verify: `mpl run --help`. The structured output surface is reached via the underlying `ollama run --format` command (which the transparency line reveals) or via the API `format` field. **COMMAND A — any valid JSON** ```bash ollama run --format json qwen3:8b \ "Return a JSON object with keys city and country for Paris. Reply with only JSON." ``` **OBSERVE** ```json { "city": "Paris", "country": "France" } ``` The output is always valid JSON — no Markdown, no prose, no `` block. Run this 10 times: every response parses cleanly. **COMMAND B — a specific JSON Schema** Write the schema file first: ```bash cat > /tmp/schema.json <<'EOF' {"type":"object","properties":{"city":{"type":"string"},"country":{"type":"string"}},"required":["city","country"]} EOF ``` Then run with the schema as the format argument: ```bash ollama run --format "$(cat /tmp/schema.json)" qwen3:8b \ "Paris is in which country? Fill the schema." ``` **OBSERVE** ```json {"city":"Paris","country":"France"} ``` The output matches the schema exactly — both required fields are present, both typed as strings. **COMMAND C — the same schema via the `mpl serve` API** With `mpl serve qwen3:8b` still running in Terminal A: > **Important:** the `format` field for constrained decoding is only honoured by Ollama's **native** `/api/chat` endpoint, not by the OpenAI-compatible `/v1/chat/completions` endpoint. The `/v1/chat/completions` endpoint accepts but silently ignores the `format` field and returns free-text. Use `/api/chat` whenever you need schema enforcement. ```bash curl http://localhost:11434/api/chat \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3:8b", "messages": [{"role": "user", "content": "Paris is in which country? Fill the schema. /no_think"}], "format": { "type": "object", "properties": { "city": {"type": "string"}, "country": {"type": "string"} }, "required": ["city", "country"] }, "stream": false }' ``` **OBSERVE** The `message.content` field contains JSON matching the schema. The response structure is always parseable — every run. (The native endpoint response shape is `{"message":{"role":"assistant","content":"..."}, ...}` — different from the OpenAI-compatible shape `{"choices":[{"message":...}]}`; extract `message.content` not `choices[0].message.content`.) **COMMAND D (contrast) — prompting without constrained decoding** Without the `format` field, ask the same question with an explicit JSON instruction in the prompt: ```bash curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3:8b", "messages": [{"role": "user", "content": "Paris is in which country? Respond ONLY with valid JSON: {\"city\": \"...\", \"country\": \"...\"}"}], "temperature": 0.0, "max_tokens": 60 }' ``` **OBSERVE** At low temperature the model usually produces JSON-looking output. But it may include a `` block before the JSON, add explanation text after it, or produce slightly malformed JSON on edge prompts. The output is *probably* JSON-shaped, not *guaranteed*. **EXPLAIN** The mechanism is logit masking, not prompting. **Constrained decoding path (`--format` / `"format"` field):** 1. Before generation begins, the JSON Schema is compiled into a finite-state machine. The FSM tracks which tokens are valid continuations given the schema and what has been generated so far. 2. At each decode step — before the sampler runs — the FSM emits a mask: tokens that would violate the schema have their logit set to −∞. 3. The masked logit vector flows through the normal pipeline (temperature → top-k → top-p → sample). 4. The sampled token is schema-valid by construction. It is impossible to sample a violating token — its probability is exactly zero. **Prompting path:** The model assigns lower probability to schema-violating tokens because fine-tuning rewarded JSON-shaped outputs in relevant contexts. But the probability is reduced, not zeroed. At a high-entropy decode step (ambiguous continuation), a malformed token can win. The output fails to parse. Your code throws. This distinction — hard guarantee vs stochastic hope — is lesson-02's central claim about constrained decoding. Use `format` for any code path that parses model output programmatically. --- ## Teardown When you are done, stop the server in Terminal A with `Ctrl-C`. To stop the Ollama daemon fully: ```bash # macOS pkill ollama # or ollama stop ``` --- ## Solutions # Solutions — Lab 01: Chat Templates > **For self-checking.** Work through the exercises before consulting these answers. The value is in making and then evaluating your own observations. --- ## Exercise 1 — Inspecting the real chat template **Expected output — `ollama show --template qwen3:8b`** The full template begins with a loop that identifies the index of the last user message: ``` {{- $lastUserIdx := -1 -}} {{- range $idx, $msg := .Messages -}} {{- if eq $msg.Role "user" }}{{ $lastUserIdx = $idx }}{{ end -}} {{- end }} ``` The system block is guarded: ``` {{- if or .System .Tools }}<|im_start|>system ... <|im_end|> {{ end }} ``` Each turn follows the `<|im_start|>{role}\n{content}<|im_end|>\n` pattern. The generation prompt at the end — when the last message is not an assistant turn — is: ``` <|im_start|>assistant ``` (with no closing `<|im_end|>`). This open partial turn is what the model completes. **Four things to have confirmed:** 1. `<|im_start|>` and `<|im_end|>` are literal template text — they become vocabulary tokens in the rendered string. They are not metadata; they occupy sequence positions. 2. The newline after `{role}` and the newline after `<|im_end|>` are part of the training distribution. Every single request sends this whitespace pattern. 3. The generation prompt is always present for non-assistant-final conversations. It contributes tokens to your prefill count on every request. 4. The system block only appears when `.System` or `.Tools` is non-empty. If you do not include a system message, no system block tokens are emitted. **Expected output — `ollama show --parameters qwen3:8b`** ``` repeat_penalty 1 stop "<|im_start|>" stop "<|im_end|>" temperature 0.6 top_k 20 top_p 0.95 ``` The two stop tokens prevent the model from generating the next turn's role prefix — generation halts on either `<|im_start|>` or `<|im_end|>`, cleanly ending the assistant turn. **Expected output — `ollama show --system qwen3:8b`** Empty (no output). qwen3:8b ships with no default system message. The system prompt is entirely caller-controlled. --- ## Exercise 2 — The transparency line **Expected output — `mpl run qwen3:8b --explain --dry-run`** ``` → ollama run qwen3:8b ``` The `→` prefix is `mpl`'s transparency contract. Because `mpl run` opens an interactive REPL (not a one-shot command), the transparency line shows the bare command — no inline prompt string. **Expected output — `mpl run qwen3:8b --explain --ctx-size 8192 --dry-run`** ``` → ollama run qwen3:8b (+1 env var) OLLAMA_CONTEXT_LENGTH=8192 ``` The context length is passed as an environment variable to the underlying `ollama run` process. Verify: `ollama run --help` has no `--context-length` or `--ctx-size` flag. The env-var injection is `mpl`'s mechanism for passing this setting. **What `--explain` adds vs the base transparency line:** `--explain` adds the environment-override layer (the `(+1 env var)` summary and the indented `KEY=VALUE` lines). The `→ ollama run qwen3:8b` line itself always appears for gated verbs, with or without `--explain`. **What `--explain` does NOT produce:** There is no quant/backend breakdown in the `--explain` output. Q4_K_M is the real quantisation format for qwen3:8b — it is recorded in `cli/curriculum.manifest.json` under the `qwen3:8b` entry. --- ## Exercise 3 — The thinking switch **Expected observations — with vs without thinking** | Command | `` block appears? | Latency to first answer token | Mechanism | |---|---|---|---| | `ollama run qwen3:8b "..."` (default) | Yes | Longer (thinking tokens consume decode steps) | Template appends ` /think` to last user turn | | `ollama run --think=false qwen3:8b "..."` | No | Shorter | Template pre-fills empty `\n\n\n\n` | | `ollama run --hidethinking qwen3:8b "..."` | Hidden from terminal | Longer (thinking still generated) | Same as default; CLI strips display | | REPL `/set nothink` | No | Shorter | Same mechanism as `--think=false` | **Template mechanism for the think switch (from the template inspected in Exercise 1):** When `$.IsThinkSet && $.Think` (thinking enabled): the template appends ` /think` immediately before the last user turn's `<|im_end|>`: ``` <|im_start|>user What is 17 + 26? /think<|im_end|> <|im_start|>assistant ``` The model was fine-tuned to respond to this ` /think` suffix by producing a `...` block. When `$.IsThinkSet && !$.Think` (thinking suppressed): the template pre-fills the generation prompt with an empty think block: ``` <|im_start|>assistant ``` The model sees the `` block already closed with no content and proceeds directly to the answer. The reasoning capacity is bypassed, not disabled. **Why `--hidethinking` does not reduce latency:** the ` /think` suffix is still in the rendered string; all thinking tokens are still generated; the Ollama CLI simply does not display them. The decode steps happened. --- ## Exercise 4 — Server-side template application **Expected output structure for the two-message request** ```json { "id": "chatcmpl-...", "object": "chat.completion", "model": "qwen3:8b", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "A mutex (mutual exclusion lock) is a synchronisation primitive..." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": ..., "completion_tokens": ..., "total_tokens": ... } } ``` **What the `prompt_tokens` count reveals:** For a system message of "You are a concise technical assistant. Answer in one sentence." (roughly 14 content words, ~17 tokens) plus a user message of "What is a mutex?" (~6 tokens), you might expect ~23 prompt tokens. The actual `prompt_tokens` will be higher — approximately 35–45 — because every turn's rendering adds special tokens: - `<|im_start|>system\n` + content + `<|im_end|>\n` — the system turn's wrapper tokens - `<|im_start|>user\n` + content + `<|im_end|>\n` — the user turn's wrapper tokens - `<|im_start|>assistant\n` — the generation prompt All special tokens (`<|im_start|>`, `<|im_end|>`, role strings, newlines) are tokenised and counted. This is the template overhead: every request pays it. **The flat string the model received:** ``` <|im_start|>system You are a concise technical assistant. Answer in one sentence.<|im_end|> <|im_start|>user What is a mutex?<|im_end|> <|im_start|>assistant ``` (The trailing line is the open generation prompt. The model never saw JSON, role field names as strings, or curly braces.) **What `temperature: 0.0` guarantees:** Greedy decoding — re-running the same `curl` command should produce the same `content` value every time (modulo minor non-determinism in Metal/CUDA kernels, which is typically negligible for greedy decoding). --- # Solutions — Lab 02: Sampling > **For self-checking.** Work through the exercises before consulting these answers. --- ## Setup: `mpl serve qwen3:8b` **Expected output:** ``` → ollama serve ``` Ollama's daemon starts (or reconnects if already running). The endpoint is `http://localhost:11434`. The `/v1/chat/completions` path is OpenAI-compatible. Use `mpl serve --help` to confirm the available flags: `--backend`, `--explain`, `--host`, `--port`, `--ctx-size`, `--json`, `--dry-run`, `--engine-arg` (repeatable raw engine tokens — G1 escape hatch). --- ## Exercise 1 — Sampler pipeline knobs in the REPL **Expected observations:** | Setting | Observation | |---|---| | `/set parameter temperature 0` + same prompt × 2 | Near-identical or identical output both times — greedy decoding always picks the argmax token | | `/set parameter temperature 1.5` + same prompt × 5 | Different city names and different phrasing across runs — the flattened distribution samples lower-probability tokens | **Correct mechanistic explanation:** Temperature divides the logit vector before softmax: ``` logits_scaled = logits / T ``` At `T = 0` (or approaching 0), the softmax approaches a one-hot on the argmax token — perfect determinism. At `T = 1.5`, logits are divided by 1.5, compressing the gaps between high and low logits, making low-probability tokens relatively more likely to sample. **Pipeline order the REPL knobs map to:** ``` (1) repeat_penalty → /set parameter repeat_penalty 1.1 (2) temperature → /set parameter temperature 0.6 (3) top_k → /set parameter top_k 20 (4) top_p → /set parameter top_p 0.95 (5) min_p → /set parameter min_p 0.05 (6) sample ``` **Why `/set parameter` is the right mechanism (not a flag):** `mpl run --help` lists `--backend`, `--explain`, `--ctx-size`, `--json`, `--dry-run`, `--engine-arg`. No `--temperature`, `--top-p`, `--top-k`, `--min-p`, or `--format` — and `--engine-arg` does not change that: it only appends raw verbatim tokens to the underlying engine command. `ollama run --help` also has no temperature flag — it would be an "unknown flag" error. Sampling knobs are a REPL-level or API-level concern, not a binary flag on the run command. **The qwen3:8b defaults from `ollama show --parameters qwen3:8b`:** - `temperature 0.6` — below 1.0, sharpening the distribution slightly - `top_k 20` — tight candidate pool - `top_p 0.95` — nucleus sampling, trimming the low-probability tail - `repeat_penalty 1` — disabled (division by 1.0 is a no-op) - No `min_p` set at the model level (the knob is available but off by default in the Modelfile) --- ## Exercise 2 — Same knobs over the API **Expected observations:** `"options": {"temperature": 0}` on `/api/chat` produces near-identical `message.content` on two consecutive runs. `"options": {"temperature": 1.5, "top_k": 20, ...}` on `/api/chat` produces varying content across runs — matching the REPL experiment result. **Why `/api/chat`, not `/v1/chat/completions`:** The `options` object is silently ignored by the `/v1/chat/completions` endpoint. Sending `"options": {"temperature": 0}` to `/v1` has no effect — the model runs at its default sampler settings. The OBSERVE claims (greedy determinism for Command A, requested variation for Command B) are only valid on the **native `/api/chat` endpoint**, which actually passes `options` into the llama.cpp sampler. The native endpoint response shape is `{"message":{"role":"assistant","content":"..."}, ...}` — extract `message.content`, not `choices[0].message.content`. **The `options` field vs the OpenAI-compat top-level fields:** | Endpoint | `temperature` | `top_p` | `top_k` | `min_p` | `repeat_penalty` | |---|---|---|---|---|---| | `/v1/chat/completions` | top-level only | top-level only | not reachable | not reachable | not reachable | | `/api/chat` | `options.temperature` | `options.top_p` | `options.top_k` | `options.min_p` | `options.repeat_penalty` | When you need full pipeline control (`top_k`, `min_p`, `repeat_penalty`), use `/api/chat` with `options`. When you need OpenAI-SDK compatibility, use `/v1/chat/completions` with top-level `temperature`/`top_p` only. **What confirms the equivalence to the REPL:** The REPL `/set parameter temperature 0` and the API `"options": {"temperature": 0}` on `/api/chat` both reach the same sampler code in the llama.cpp backend. The route is different (in-session REPL state vs per-request JSON field); the effect is identical. The REPL is convenient for interactive exploration; the API is the mechanism for application code. --- ## Exercise 3 — Structured output as a hard guarantee **Expected output — Command A (`--format json`)** ```json { "city": "Paris", "country": "France" } ``` (The exact whitespace formatting may vary; the content is consistent across runs. The `` block does not appear — the grammar prevents it.) **Expected output — Command B (full JSON Schema via `--format`)** ```json {"city":"Paris","country":"France"} ``` Output is always schema-valid: `city` is a string, `country` is a string, both required keys present. The exact whitespace may differ from Command A's pretty-printed output — both are valid JSON. **Expected output — Command C (schema via API `format` field, `/api/chat` endpoint)** The `message.content` field (native `/api/chat` response shape) contains JSON matching the declared schema. The body parses cleanly with `json.loads`. Every run — regardless of temperature (try varying it) — returns schema-valid JSON. Note: the `/v1/chat/completions` endpoint silently ignores the `format` field and returns free-text prose. Only `/api/chat` (and `/api/generate`) enforce grammar-constrained decoding via the `format` field. The native endpoint response shape differs: extract `message.content`, not `choices[0].message.content`. **What Command D (prompting without `format`) demonstrates:** At `temperature: 0.0` with an explicit "respond only in JSON" instruction, the model usually cooperates — but the mechanism is different. There is no grammar mask; the model assigns lower probability to malformed JSON because fine-tuning rewarded JSON-shaped outputs. The probability is reduced, not zeroed. On a different prompt, at higher temperature, or when the model is uncertain about how to format a field, a schema-violating token can be sampled. **The logit masking mechanism:** ``` Before each decode step: 1. Grammar FSM emits mask: {token_id → valid: bool} 2. For every token where valid = false: logit = −∞ 3. exp(−∞) = 0 → that token has zero probability in the softmax 4. Normal pipeline (temperature → top-k → top-p → sample) runs over the surviving logits 5. Sampled token is schema-valid by construction — not by luck ``` The critical distinction: prompting makes valid output *likely*; constrained decoding makes invalid output *impossible*. This matters whenever your code parses the model's output programmatically — an exception from `json.loads` is not a recoverable UX event. **`mpl run` has no `--format` flag (confirmed):** `mpl run --help` output lists: `--backend`, `--explain`, `--ctx-size`, `--json`, `--dry-run`, `--engine-arg`. No `--format`. The structured output surface is reached via: - The underlying `ollama run --format` command (which the transparency line reveals as the exact command `mpl run` wraps) - The `mpl serve` API `format` field in the request body This is by design: the interactive REPL (`mpl run`) is for exploration; structured output with a schema is a programmatic concern that belongs in the API path or in `ollama run --format` one-shot invocations.

connect to continue