From 0f31f48b8f1d84ea2bc1086fe580a7f0418c5b22 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 07:54:53 +0300 Subject: [PATCH 1/9] docs: spec + plan for two new tutorials (F-min part 2) Co-Authored-By: Claude Opus 4.7 (1M context) --- planning/README.md | 6 +- .../2026-06-12-docs-tutorials-design.md | 332 ++++++++++ .../active/2026-06-12-docs-tutorials-plan.md | 601 ++++++++++++++++++ 3 files changed, 938 insertions(+), 1 deletion(-) create mode 100644 planning/active/2026-06-12-docs-tutorials-design.md create mode 100644 planning/active/2026-06-12-docs-tutorials-plan.md diff --git a/planning/README.md b/planning/README.md index d6f2101..fd5db77 100644 --- a/planning/README.md +++ b/planning/README.md @@ -11,7 +11,11 @@ points. ## Active -_None._ +- **[docs-tutorials](active/2026-06-12-docs-tutorials-design.md)** + — The two tutorials deferred from #56: *Your first outbox app* + (10-minute walk-through) and *Add a Kafka relay* (extends the + first tutorial with a Kafka publisher + at-least-once + demonstration). New `docs/tutorials/` directory; two nav entries. ## Archived (shipped) diff --git a/planning/active/2026-06-12-docs-tutorials-design.md b/planning/active/2026-06-12-docs-tutorials-design.md new file mode 100644 index 0000000..401f73f --- /dev/null +++ b/planning/active/2026-06-12-docs-tutorials-design.md @@ -0,0 +1,332 @@ +--- +status: draft +date: 2026-06-12 +slug: docs-tutorials +supersedes: null +superseded_by: null +pr: null +outcome: null +--- + +# Design: Two new tutorials (Diátaxis F-min, part 2) + +## Summary + +Add the two tutorials that were deferred from +[`docs-tutorials-and-observability-split`](../archived/2026-06-11-docs-tutorials-and-observability-split-design.md) +(#56, shipped 2026-06-12). PR #56 landed the observability split and +nav reshape but deliberately left §1 and §2 (the two tutorials) for a +dedicated session — tutorial code must be executed end-to-end against +a clean environment with literal terminal output captured into the +page, and that warrants its own pass. This spec is that pass. + +Total: 2 new pages, 1 new `docs/tutorials/` directory, 2 new entries in +`mkdocs.yml`. No existing page URLs change; no existing page content +changes; no code touched. + +## Motivation + +- **Zero tutorials today.** Every page in `getting-started` is + reference-shaped (`installation.md`) or reference-with-narrative + (`basic.md`). A newcomer who lands on `docs/index.md` and clicks + *"Install and write the first publisher / subscriber"* arrives at the + Basic-usage page, which is a four-step section list more than a + story. No *"let's build something together for ten minutes,"* no one + place where the path from zero to "a row landed through a handler" + is one continuous narrative. A tutorial in the Diátaxis sense — warm + voice, single concrete journey, end-state recap — is the single + highest-impact missing piece, and was always the bigger half of the + F-min ask. + +- **The architecture has a relay payoff line that's not yet a worked + example.** The docs talk about the foreign-broker relay + ([`usage/relay.md`](../../docs/usage/relay.md)) but never show the + full app: `docker compose up postgres kafka`, publish in a + transaction, watch the row land in Kafka, kill Kafka, watch the + retry. A tutorial that does this end-to-end is more persuasive than + the relay reference's three-line snippet — and demonstrates the + at-least-once contract better than any prose section can. + +- **The deferral note in #56 is the controlling constraint.** From the + archived spec's scope-reduction header: *"the spec's discipline that + tutorial code must be executed end-to-end against a clean + environment with literal output captured warrants a dedicated + session."* This spec exists specifically to honor that constraint. + See §Testing — tutorial code must be **executed** during plan + execution; pasted output must be **literal**, not hand-edited. + +## Non-goals + +Deliberately *not* covered here; each is a candidate follow-on: + +- **A third tutorial** — *"Test handlers with `TestOutboxBroker`"*, + *"Schedule a delayed delivery"*, *"Wire DLQ"*. Two is the right + count for this pass; the testing one would be high-value but is + separate scope. +- **A dedicated `Tutorials` nav sub-grouping.** Two tutorials slot + under "Getting started" as flat entries with `Tutorial:` prefixes. + Revisit grouping at three. +- **Renaming nav sections to Diátaxis-canonical labels.** Same + argument as #56: existing Overview / Getting started / Concepts / + Guides / Reference / Operations reads more naturally and already + maps to the four quadrants. Don't churn labels. +- **Voice review of existing Reference pages.** Tutorial voice is + warm and step-by-step; Reference voice stays terse and precise. + Both are correct for their quadrant. +- **A CI step that runs the tutorial code on every release.** Tracked + separately under the migration-recipe regression-test follow-on + (see [`operator-pages`](../archived/2026-06-11-operator-pages-design.md)'s + out-of-scope list). The right place to add it, but not here. +- **Re-shipping the observability split.** That was the #56 PR; + shipped. + +## Design + +### 1. Tutorial: Your first outbox app + +New file: `docs/tutorials/first-outbox-app.md`. New `docs/tutorials/` +directory at the same level as `docs/concepts/`, `docs/operations/`, +`docs/usage/`, `docs/introduction/`. + +Goal — a reader with Python and Postgres familiarity follows the page +top-to-bottom in roughly ten minutes and ends with a running process +where a single `broker.publish` results in the handler running, with +no surprises along the way. + +Section outline: + +``` +# Tutorial: Your first outbox app + +What you'll build (2 sentences: a tiny app where publishing + inside a DB transaction triggers a handler) + +Before you start (Python 3.13+, Postgres, ~10 minutes) + +Step 1: Install (uv add 'faststream-outbox[asyncpg,validate]') + +Step 2: Start Postgres (one-liner Docker) + +Step 3: Declare the + outbox table (MetaData + make_outbox_table) + +Step 4: Create the schema (metadata.create_all in a one-shot script — + the easiest path for a tutorial; link out to + operations/alembic.md for the real Alembic + recipe) + +Step 5: Define a handler (@broker.subscriber("orders")) + +Step 6: Publish a row (inside session.begin()) + +Step 7: Run it (faststream run app:app, see the handler fire) + +What you just built (recap) + +What's next (links to: Subscriber reference, Publisher + reference, FastAPI integration guide, + Tutorial: Add a Kafka relay) +``` + +Voice: imperative + warm. Use *"we"* sparingly. Each step starts with +a one-sentence "what you're about to do" and ends with "you should see +X." No design rationale on the page; that's Concepts. Links to +Explanation for "why this works." + +Code: one file (`app.py`) grows step by step. Each step shows the +*diff* from the previous step (or the full file if the diff would be +larger than the file). Final file is ~30 lines. + +### 2. Tutorial: Add a Kafka relay + +New file: `docs/tutorials/add-kafka-relay.md`. + +Goal — extends Tutorial #1 with a Kafka publisher decorator. Shows the +at-least-once contract end to end by deliberately killing Kafka +mid-flight and watching the retry land. + +Section outline: + +``` +# Tutorial: Add a Kafka relay + +What you'll add (turn the Tutorial-1 handler into a relay + that forwards to Kafka) + +Before you start (you finished Tutorial 1; we'll extend that + app) + +Step 1: Add Kafka (docker compose snippet) + +Step 2: Install + faststream[kafka] (uv add 'faststream[kafka]') + +Step 3: Add the Kafka + broker (KafkaBroker + publisher) + +Step 4: Stack the + decorator (@publisher_kafka @broker_outbox.subscriber) + +Step 5: Run it and + watch a row reach + Kafka (consume from the topic via the CLI) + +Step 6: Kill Kafka and + watch the retry (docker compose stop kafka, publish a row, + see the outbox subscriber retry; bring Kafka + back, see the row deliver) + +What you just built (recap — at-least-once relay) + +What's next (links to: Relay reference, Subscriber retry + strategies, Comparison page § + "vs FastStream foreign-broker direct") +``` + +Voice: same warmth as Tutorial 1; assumes familiarity from Tutorial 1. + +Code: extends the same `app.py` plus the `docker-compose.yml` from +Tutorial 1. + +### 3. Nav adjustments + +The nav already merged in #56 (`mkdocs.yml`) carries placeholders for +the two tutorials — they were intentionally left commented out under +"Getting started" pending this follow-on. Concretely, after this spec: + +```yaml +nav: + - Overview: index.md + - Getting started: + - Installation: introduction/installation.md + - Basic usage: usage/basic.md + - 'Tutorial: Your first outbox app': tutorials/first-outbox-app.md # NEW + - 'Tutorial: Add a Kafka relay': tutorials/add-kafka-relay.md # NEW + - Concepts: ... + - Guides: ... + - Reference: ... + - Operations: ... +``` + +Two new entries, zero file renames, zero URL changes for existing +pages. Plan must verify the actual placeholder state in `mkdocs.yml` +when starting Task 1 — if #56 did *not* leave commented-out lines, +add fresh entries instead. + +### 4. Cross-link contract + +- Tutorial 1's "What's next" footer points at: + Tutorial 2, Subscriber reference, Publisher reference, FastAPI + integration guide. +- Tutorial 2's "What's next" footer points at: + Relay reference, Subscriber retry strategies, Comparison page § + *"vs FastStream foreign-broker direct"*. +- `usage/relay.md` Reference page gains a one-line "Want a worked + end-to-end example? See [Tutorial: Add a Kafka + relay](../tutorials/add-kafka-relay.md)" pointer at the top. +- `docs/index.md` decision-tree table — no change needed. No row maps + to "I want a 10-minute walk-through" directly; the tutorials are + reachable via the "Install and write the first publisher / + subscriber" row, which now resolves to either Basic usage *or* the + Tutorial 1 page. Re-link decision: point the row at Tutorial 1 + (warmer entry for newcomers) and leave Basic usage discoverable via + the sidebar. Plan to confirm with reviewer in PR. + +### 5. The `tutorials/` directory + +New top-level `docs/tutorials/`. Lives alongside `docs/concepts/`, +`docs/operations/`, `docs/usage/`, `docs/introduction/`. Consistent +with the existing flat directory layout — no nested subdirectories. + +## Operations + +None — in-repo. The mkdocs deploy workflow re-runs on push to `main` +whenever `docs/**` or `mkdocs.yml` changes; this PR triggers both. The +new URLs become available immediately at: + +- `https://faststream-outbox.modern-python.org/tutorials/first-outbox-app/` +- `https://faststream-outbox.modern-python.org/tutorials/add-kafka-relay/` + +No existing URL changes; no redirects needed. + +## Out of scope + +Repeat list for grep: + +- Third tutorial (testing, scheduling, DLQ) +- Tutorials nav sub-grouping +- Renaming nav sections to Diátaxis-canonical labels +- Voice review of existing Reference pages +- CI step that runs tutorial code on every release +- Re-shipping anything from #56 (observability split, nav reshape, + instrumentation-seams Explanation page) + +## Testing + +Content-only; correctness is observable on the live site. Critically: + +- **Tutorial code must be executed end-to-end during plan execution + against a clean machine.** Tutorial 1: from scratch (fresh checkout + + fresh Postgres container), every step's expected output observed. + Tutorial 2: same, including the "kill Kafka, see retry" step. The + plan author runs these and the *literal terminal output* lands + inside the tutorial under "you should see X" — no hand-edited + expected output. Tutorials that haven't been run produce + frustration when readers try them and miss a step. +- `just docs-build` (`mkdocs build --strict`) passes clean — every + internal cross-link from the two new pages and the touched + `usage/relay.md` resolves. +- `just lint` passes (eof-fixer, ruff format, ruff check, ty check). +- Reviewer manual sidebar scan: `just docs-serve` and confirm the two + new entries appear in the two new sidebar positions under "Getting + started." +- Reviewer reads both tutorials end-to-end against a fresh checkout + and a clean Postgres / Kafka — the most valuable review move for a + tutorial. + +## Risk + +- **Tutorial voice drifts from the existing reference voice and + introduces inconsistency across the site.** Mitigated by the + explicit voice guidance in §1 and §2 (warm, step-by-step in + tutorials; everything else unchanged). The "What you just built" + recap pattern, the "Before you start" preamble, and the "What's + next" footer are intentional voice markers — they signal *"you are + reading a tutorial"* without requiring a Diátaxis-literate reader. + Tutorial voice is allowed to feel different from Reference voice + because they're serving different reader needs. + +- **Tutorial code goes stale faster than reference code.** Tutorial + code embeds version-specific install commands, Docker image tags, + and Postgres compose snippets — all of which drift faster than the + library's public API. Mitigated by keeping the tutorials minimal + (no premature abstractions, no library features outside the + tutorial's narrow path) so most updates are mechanical pin bumps. + Follow-up: tutorials could be the next thing tested by the + migration-recipe-style regression tests scaffolded out by the + operator-pages spec — a CI step that runs the tutorial code + against a real Postgres on every release. Out of scope here. + +- **Tutorial #2's "kill Kafka, see retry" step is the + most-likely-to-flake step in either tutorial.** Local environments + differ; Kafka's failure modes are platform-sensitive (especially on + Apple Silicon). Mitigated by recommending Confluent's `cp-kafka` + image specifically (known to work on M1+ from prior use) and by + treating the step as *demonstrative*: the tutorial doesn't fail if + Kafka comes back instantly with no observable retry, because the + at-least-once property is still preserved. Reviewer flags if the + step's reproduction is fragile and we drop it from the tutorial in + favor of a one-paragraph callout explaining the contract. + +- **The "What's next" footers create a maintenance graph.** Adding a + new tutorial later means revisiting the footer of every existing + tutorial to add the cross-link. At two tutorials the cost is + trivial. Re-evaluate if we ever add a fourth. + +- **Literal-output discipline regresses under iteration.** A future + edit that touches Tutorial 1's `app.py` could miss re-running the + tutorial and silently desync the "you should see X" output from + what actually prints. Mitigated by §Testing's framing (running + end-to-end is a release gate for any tutorial change) and by + keeping output blocks small enough that a reviewer can spot a stale + one in code review. diff --git a/planning/active/2026-06-12-docs-tutorials-plan.md b/planning/active/2026-06-12-docs-tutorials-plan.md new file mode 100644 index 0000000..0825fb0 --- /dev/null +++ b/planning/active/2026-06-12-docs-tutorials-plan.md @@ -0,0 +1,601 @@ +--- +status: draft +date: 2026-06-12 +slug: docs-tutorials +spec: docs-tutorials +pr: null +--- + +# docs-tutorials — implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> superpowers:subagent-driven-development (recommended) or +> superpowers:executing-plans to implement this plan task-by-task. Steps +> use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Land the two tutorials deferred from #56 (Your first outbox +app + Add a Kafka relay), each written from literal end-to-end +execution against a clean environment, plus the two-entry nav update +and the cross-links that surface them. + +**Spec:** [`planning/active/2026-06-12-docs-tutorials-design.md`](./2026-06-12-docs-tutorials-design.md) + +**Branch:** `docs/tutorials` + +**Commit strategy:** Per-task commits. Tasks 2 and 3 each carry a +`docker compose down -v` smoke step before writing so the captured +output reflects a true first-run experience — no leftover state from +an earlier session of the same task. + +**Sequencing note:** Tasks 2 and 3 are gated on actually running the +tutorial code end-to-end against a clean Postgres (Task 2) and a +clean Postgres + Kafka (Task 3). The plan author runs each tutorial +on a real machine before committing the page. **Hand-edited output is +a defect** — if a captured line looks "ugly" (asyncpg warning, sqlalchemy +deprecation, docker compose progress noise), it stays. The point is +that the reader's first run will match the page. + +--- + +### Task 1: Branch + plan commit + README Active entry + +**Files:** +- Modify: `planning/README.md` +- Create: `planning/active/2026-06-12-docs-tutorials-design.md` (already drafted) +- Create: `planning/active/2026-06-12-docs-tutorials-plan.md` (this file) + +- [ ] **Step 1: Branch off `main`** + + ```bash + git fetch origin + git switch -c docs/tutorials origin/main + ``` + + Expected: clean tree on `docs/tutorials` at `origin/main`'s HEAD. + The current `chore/archive-observability-split` branch carried the + archive-only commit from #56's archive pass; the tutorials work + belongs on its own branch. + +- [ ] **Step 2: Update `planning/README.md` Active section** + + Replace the `_None._` line under `## Active` with: + + ```markdown + ## Active + + - **[docs-tutorials](active/2026-06-12-docs-tutorials-design.md)** + — The two tutorials deferred from #56: *Your first outbox app* + (10-minute walk-through) and *Add a Kafka relay* (extends the + first tutorial with a Kafka publisher + at-least-once + demonstration). New `docs/tutorials/` directory; two nav entries. + ``` + +- [ ] **Step 3: Smoke-build** + + Run: `just docs-build` + + Expected: clean. No new pages yet; the README change is non-docs. + This is a sanity check that the branch baseline still builds. + +- [ ] **Step 4: Commit** + + ```bash + git add planning/active/2026-06-12-docs-tutorials-design.md \ + planning/active/2026-06-12-docs-tutorials-plan.md \ + planning/README.md + git commit -m "$(cat <<'EOF' + docs: spec + plan for two new tutorials (F-min part 2) + + Co-Authored-By: Claude Opus 4.7 (1M context) + EOF + )" + ``` + +--- + +### Task 2: Tutorial 1 — Your first outbox app + +**Files:** +- Create: `docs/tutorials/first-outbox-app.md` +- Working dir (not committed): `/tmp/outbox-tutorial-1/` + +Write the tutorial per [spec §1 +](./2026-06-12-docs-tutorials-design.md#1-tutorial-your-first-outbox-app) +**after running every step end-to-end against a clean local +environment.** Capture the literal terminal output at each step. The +captured output blocks land inside the page under "you should see:" +preambles. + +**Voice contract:** warm, step-by-step, sparing use of "we." Each +step opens with a one-sentence "what you're about to do" and closes +with the literal output. No design rationale on the page; that's +Concepts. No forward references to library features the tutorial +doesn't use (no DLQ, no timers, no `OutboxResponse`). + +- [ ] **Step 1: Clean Postgres environment** + + Run from the repo root: + + ```bash + docker compose -f compose.yaml down -v + ``` + + Expected: containers and volumes removed. We will *not* use the + repo's `compose.yaml` for the tutorial run — the reader doesn't + have it. The teardown is to ensure no port 5432 conflict against + the tutorial container we're about to start. + +- [ ] **Step 2: Fresh working directory under `/tmp`** + + ```bash + rm -rf /tmp/outbox-tutorial-1 + mkdir /tmp/outbox-tutorial-1 + cd /tmp/outbox-tutorial-1 + ``` + + This is the tutorial reader's perspective: a directory with + nothing in it. Every command in the tutorial body is run from + here. + +- [ ] **Step 3: Walk Step 1 — install** + + The tutorial will instruct: + + ```bash + uv add 'faststream-outbox[asyncpg,validate]' + ``` + + (or `pip install` equivalent). Run it. Capture the literal stdout. + The PEP 668 / venv path is tutorial-reader's problem; pick the + flow that produces the cleanest captured output (recommendation: + `uv init && uv add ...` to start from a real project). + + If `uv add` produces a venv-creation banner or a lock-file diff + line, that line **stays** in the captured output. The page shows + what a first-time reader will actually see. + +- [ ] **Step 4: Walk Step 2 — start Postgres** + + The tutorial will instruct: + + ```bash + docker run --rm -d --name outbox-postgres \ + -e POSTGRES_USER=outbox -e POSTGRES_PASSWORD=outbox -e POSTGRES_DB=outbox \ + -p 5432:5432 postgres:17 + ``` + + Run it. Capture the container ID stdout. Wait for `docker logs + outbox-postgres 2>&1 | tail -1` to show `database system is ready + to accept connections`. Capture that one log line too. + +- [ ] **Step 5: Walk Steps 3 + 4 — declare the table and create the schema** + + The tutorial will provide an `app.py` containing the + `make_outbox_table` declaration plus a one-shot + `metadata.create_all` script (run via `python -c` or as a small + block at the top of `app.py` gated by `if __name__ == "__main__":`). + + Decision: use a separate `create_schema.py` script over a gated + block. Two reasons: (1) keeps `app.py` minimal so the diff at each + later step stays small, (2) the tutorial reader runs the schema + creation once and never again, so a separate file is the more + honest mental model. + + Write `create_schema.py`. Run `python create_schema.py`. Capture + the output (likely silent on success — that's fine; the page says + "you should see no output"). + + Verify with `docker exec outbox-postgres psql -U outbox -d outbox + -c '\d outbox'`. Capture that table definition. It goes in the + tutorial as the "you should see this table" callout under Step 4. + +- [ ] **Step 6: Walk Step 5 — define the handler** + + Add the `@broker.subscriber("orders")` block to `app.py`. The + handler body is `print(f"got order {order_id}")` — the simplest + thing that produces visible output on Step 7. + + No command to run at this step; the handler executes during Step 7. + +- [ ] **Step 7: Walk Step 6 — publish a row** + + Add an `@app.after_startup` block to `app.py` that opens a session + and calls `await broker.publish(1, queue="orders", session=session)`. + No command to run yet — Step 7 is what fires it. + +- [ ] **Step 8: Walk Step 7 — run the app** + + Run: + + ```bash + faststream run app:app + ``` + + Capture every line of stdout from start until the handler fires + (`got order 1`) and the next idle line. The tutorial shows this as + a `text` block under "you should see:" — the literal output, INFO + lines and all. + + Ctrl-C; capture the shutdown lines. Those go in the page too — + the reader needs to know what a clean shutdown looks like. + +- [ ] **Step 9: Write `docs/tutorials/first-outbox-app.md`** + + Use the section outline from spec §1 (What you'll build → Before + you start → Steps 1–7 → What you just built → What's next). + + Each step contains: + - A one-sentence "what you're about to do" preamble. + - The literal command or `app.py` diff/full-file. + - The literal captured output under "you should see:" (a + fenced `text` or `bash` block). + + `What's next` footer links to: + - [Subscriber reference](../usage/subscriber.md) + - [Publisher reference](../usage/publisher.md) + - [FastAPI integration](../usage/fastapi.md) + - [Tutorial: Add a Kafka relay](./add-kafka-relay.md) + +- [ ] **Step 10: Smoke-build** + + Run: `just docs-build` + + Expected: clean. The new page is orphaned-not-in-nav (Task 4 + wires it); mkdocs strict mode warns but does not fail on + orphaned pages by default. If it does fail, add the nav entry now + (out of Task-4 order) rather than disabling the strict check. + +- [ ] **Step 11: Clean up the tutorial environment** + + ```bash + docker stop outbox-postgres + rm -rf /tmp/outbox-tutorial-1 + ``` + + Tutorial 2 starts from a fresh setup; carrying state across tasks + would defeat the "first-run experience" capture. + +- [ ] **Step 12: Commit** + + ```bash + git add docs/tutorials/first-outbox-app.md + git commit -m "$(cat <<'EOF' + docs: tutorial — your first outbox app + + Co-Authored-By: Claude Opus 4.7 (1M context) + EOF + )" + ``` + +--- + +### Task 3: Tutorial 2 — Add a Kafka relay + +**Files:** +- Create: `docs/tutorials/add-kafka-relay.md` +- Working dir (not committed): `/tmp/outbox-tutorial-2/` + +Write the tutorial per [spec §2 +](./2026-06-12-docs-tutorials-design.md#2-tutorial-add-a-kafka-relay) +extending Tutorial 1's app. Same end-to-end execution requirement. + +The "kill Kafka, see retry" step (spec §2 Step 6) is the fragility +risk flagged in the spec. Use Confluent's `cp-kafka` image. If the +step's repro is unstable on your environment, STOP and reshape it as +a callout that explains the at-least-once contract without requiring +the visible retry. **The spec authorizes this fallback** — don't +push past a flaky step to "make it work for the page." + +- [ ] **Step 1: Fresh working directory; re-walk Tutorial 1 to seed it** + + ```bash + rm -rf /tmp/outbox-tutorial-2 + mkdir /tmp/outbox-tutorial-2 + cd /tmp/outbox-tutorial-2 + ``` + + Now re-walk Tutorial 1 inside this directory using only what + `docs/tutorials/first-outbox-app.md` says. End-state: working + `app.py` + a Postgres container + a venv with `faststream-outbox` + installed. The captured starting state for Tutorial 2 *is* a + reader who just finished Tutorial 1, so this re-walk is the + cheapest way to land at exactly that state. (Bonus: if any step + diverges from the captured output during the re-walk, you caught + a Task 2 defect — fix Tutorial 1 before continuing.) + +- [ ] **Step 2: Walk Step 1 — add Kafka via docker-compose** + + Write the `docker-compose.yml` that adds a `kafka` service using + Confluent's `cp-kafka:7.6.0` image (or current stable at execution + time — record the exact tag you use in the captured output). The + recommended single-broker KRaft configuration uses `cp-kafka`'s + built-in mode; no separate ZooKeeper service. + + Run: + + ```bash + docker compose up -d kafka + ``` + + Capture the stdout. Wait for `docker compose logs kafka 2>&1 | + grep -m1 'Kafka Server started'`. Capture that one line. + +- [ ] **Step 3: Walk Step 2 — install faststream[kafka]** + + ```bash + uv add 'faststream[kafka]' + ``` + + Capture stdout. + +- [ ] **Step 4: Walk Steps 3 + 4 — Kafka broker + stacked decorator** + + Extend `app.py`: + - Add `KafkaBroker(...)` and a `kafka_publisher = kafka_broker.publisher("orders.kafka")`. + - Stack `@kafka_publisher` above the existing `@broker_outbox.subscriber("orders")`. + - Adjust the handler to `return order_id` so the publisher + decorator forwards a payload. + + No command to run at this step; Step 5 fires it. + +- [ ] **Step 5: Walk Step 5 — run and watch a row reach Kafka** + + Run `faststream run app:app` in one terminal. In a second + terminal, run: + + ```bash + docker compose exec kafka kafka-console-consumer \ + --bootstrap-server kafka:9092 --topic orders.kafka --from-beginning + ``` + + Capture both: the `faststream` stdout (publish + handler + Kafka + send) and the consumer output (the `1` arriving on the topic). + +- [ ] **Step 6: Walk Step 6 — kill Kafka and watch the retry** + + In the running `faststream` process, with the consumer still + listening: + + ```bash + docker compose stop kafka + ``` + + Publish another row (cleanest: send `SIGUSR1` to the app or + re-run a one-shot `python -c "import asyncio; ..."` snippet that + calls `broker.publish` — record whichever flow the tutorial uses). + Capture the outbox subscriber's retry log lines. Then: + + ```bash + docker compose start kafka + ``` + + Capture the eventual successful delivery line and the consumer + receiving the row. + + **If the retry doesn't visibly fire** (Kafka returns instantly + with a "not ready" that the producer retries internally before the + outbox subscriber notices), drop Step 6 from the tutorial and + replace it with a callout that says: *"Behind the scenes, if Kafka + were unavailable, the outbox row would be retried per the subscriber's + retry policy. See [Subscriber § Retry strategies + ](../usage/subscriber.md#retry-strategies)."* The spec authorizes + this fallback. + +- [ ] **Step 7: Write `docs/tutorials/add-kafka-relay.md`** + + Use the section outline from spec §2. Cross-link upward to + Tutorial 1 in the "Before you start" preamble. `What's next` + footer links to: + - [Relay reference](../usage/relay.md) + - [Subscriber § Retry strategies](../usage/subscriber.md#retry-strategies) + - [Comparison § vs FastStream foreign-broker direct](../concepts/comparison.md#vs-faststream-foreign-broker-direct) + + If Step 6 fell back to a callout, capture that in a one-line note + at the top: *"This tutorial originally demonstrated retry by + killing Kafka mid-flight; that step is fragile on some + environments and is replaced by a callout."* + +- [ ] **Step 8: Smoke-build** + + Run: `just docs-build`. Expected: clean (orphan warning OK). + +- [ ] **Step 9: Clean up** + + ```bash + docker compose down -v + rm -rf /tmp/outbox-tutorial-2 + ``` + +- [ ] **Step 10: Commit** + + ```bash + git add docs/tutorials/add-kafka-relay.md + git commit -m "$(cat <<'EOF' + docs: tutorial — add a Kafka relay + + Co-Authored-By: Claude Opus 4.7 (1M context) + EOF + )" + ``` + +--- + +### Task 4: Nav additions + +**Files:** +- Modify: `mkdocs.yml` + +Add the two `Tutorial:` entries to the `Getting started` section per +[spec §3 +](./2026-06-12-docs-tutorials-design.md#3-nav-adjustments). The spec +anticipated either uncommented placeholders or fresh additions; the +post-#56 `mkdocs.yml` does **not** carry placeholders, so this task +adds fresh entries. + +- [ ] **Step 1: Edit `mkdocs.yml`** + + Replace the `Getting started:` block: + + ```yaml + - Getting started: + - Installation: introduction/installation.md + - Basic usage: usage/basic.md + - 'Tutorial: Your first outbox app': tutorials/first-outbox-app.md + - 'Tutorial: Add a Kafka relay': tutorials/add-kafka-relay.md + ``` + + Order matters — Installation first, Basic usage second, then the + two tutorials. Tutorials come after Basic usage because some + readers will prefer the terse reference shape; the sidebar lets + them choose. + +- [ ] **Step 2: Smoke-build** + + Run: `just docs-build` + + Expected: clean, no orphan warnings now that the pages are in the + nav. + +- [ ] **Step 3: Commit** + + ```bash + git add mkdocs.yml + git commit -m "$(cat <<'EOF' + docs: nav — surface the two new tutorials under Getting started + + Co-Authored-By: Claude Opus 4.7 (1M context) + EOF + )" + ``` + +--- + +### Task 5: Cross-links — relay reference and landing decision tree + +**Files:** +- Modify: `docs/usage/relay.md` +- Modify: `docs/index.md` + +Two surgical edits per [spec §4 +](./2026-06-12-docs-tutorials-design.md#4-cross-link-contract): + +1. `usage/relay.md` gets a one-line "Want a worked end-to-end + example?" pointer at the top. +2. `docs/index.md`'s decision-tree row for *"Install and write the + first publisher / subscriber"* (line ~44) repoints from + `installation.md → basic.md` to `installation.md → first-outbox-app.md`. + Basic usage remains discoverable via the sidebar. + +- [ ] **Step 1: Add the relay pointer** + + In `docs/usage/relay.md`, immediately after the H1 title (before + the first paragraph), insert: + + ```markdown + > Want a worked end-to-end example? See + > [Tutorial: Add a Kafka relay](../tutorials/add-kafka-relay.md). + ``` + +- [ ] **Step 2: Repoint the decision-tree row** + + In `docs/index.md`, find the line: + + ```markdown + | Install and write the first publisher / subscriber | [Installation](introduction/installation.md) → [Basic usage](usage/basic.md) | + ``` + + Replace with: + + ```markdown + | Install and write the first publisher / subscriber | [Installation](introduction/installation.md) → [Tutorial: Your first outbox app](tutorials/first-outbox-app.md) | + ``` + + Leave the `### Getting started` section list below the table + unchanged — it still mentions Installation + Basic usage, which is + correct (Basic usage is still a Getting-started page, just no + longer the decision-tree default). + +- [ ] **Step 3: Smoke-build** + + Run: `just docs-build`. Expected: clean. + +- [ ] **Step 4: Commit** + + ```bash + git add docs/usage/relay.md docs/index.md + git commit -m "$(cat <<'EOF' + docs: cross-links — relay tutorial pointer + landing decision tree + + Co-Authored-By: Claude Opus 4.7 (1M context) + EOF + )" + ``` + +--- + +### Task 6: Verify + PR handoff + +**Files:** none modified; no commit produced. + +- [ ] **Step 1: Full strict build** + + Run: `just docs-build` + + Expected: clean. All cross-links from the two new pages, the + modified `relay.md`, and the modified `index.md` resolve. + +- [ ] **Step 2: Lint pass** + + Run: `just lint` + + Expected: eof-fixer, ruff format, ruff check, ty check all pass. + (Docs-only PR, but the linter touches the planning markdown too, + so this catches trailing whitespace and missing EOF newlines.) + +- [ ] **Step 3: Manual sidebar scan** + + Run: `just docs-serve`. Open the served site. Confirm: + + - Getting started shows: Installation, Basic usage, Tutorial: + Your first outbox app, Tutorial: Add a Kafka relay. + - Tutorial 1's "What's next" footer links resolve (Subscriber, + Publisher, FastAPI integration, Tutorial 2). + - Tutorial 2's "What's next" footer links resolve (Relay, + Subscriber § Retry strategies, Comparison § "vs FastStream + foreign-broker direct"). + - `usage/relay.md` shows the new top-of-page Tutorial 2 pointer. + - `index.md` decision-tree row points at Tutorial 1. + +- [ ] **Step 4: Re-run Tutorial 1 against a fresh checkout** + + In a temp directory, follow Tutorial 1 step-by-step using **only + what the page tells you**. Every command's literal output should + match what the page promised. Reviewer-grade check: if anything + diverges, STOP and update the tutorial. This is the single + highest-leverage verification step in the plan — a reader who + trips on Step 4 has had a worse first impression than one who + found no tutorial at all. + +- [ ] **Step 5: Re-run Tutorial 2 against the Tutorial-1 end state** + + Same discipline. The kill-Kafka step is allowed to be a callout if + Task 3 Step 6 fell back; the rest of the page must reproduce. + +- [ ] **Step 6: Open the PR** + + Hand off to `superpowers:requesting-code-review` / + `superpowers:finishing-a-development-branch`. + + PR title: `docs: two new tutorials — first outbox app + Kafka relay`. + + PR body should call out: + - Source spec link. + - Note that both tutorials were executed end-to-end against clean + environments; literal output is captured in-page. + - Whether Task 3 Step 6 (kill-Kafka demo) landed as the live + demonstration or as the callout fallback. + - Reviewer ask: re-walk both tutorials on a clean machine. + + On merge, both halves of the planning pair move to + `planning/archived/` with `status: shipped`, `pr:`, and `outcome:` + filled — same archive pattern as #50 / #53 / #56. From c45438bc79a5282f2b3a2ee67fad10cf4a76887a Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 07:58:58 +0300 Subject: [PATCH 2/9] =?UTF-8?q?docs:=20plan=20fixups=20=E2=80=94=20broken?= =?UTF-8?q?=20comparison=20anchor=20+=20strict-mode=20framing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- planning/active/2026-06-12-docs-tutorials-plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/planning/active/2026-06-12-docs-tutorials-plan.md b/planning/active/2026-06-12-docs-tutorials-plan.md index 0825fb0..4ac686a 100644 --- a/planning/active/2026-06-12-docs-tutorials-plan.md +++ b/planning/active/2026-06-12-docs-tutorials-plan.md @@ -388,7 +388,7 @@ push past a flaky step to "make it work for the page." footer links to: - [Relay reference](../usage/relay.md) - [Subscriber § Retry strategies](../usage/subscriber.md#retry-strategies) - - [Comparison § vs FastStream foreign-broker direct](../concepts/comparison.md#vs-faststream-foreign-broker-direct) + - [Comparison](../concepts/comparison.md) — see the section *"vs. FastStream + `KafkaBroker` / `RabbitBroker` directly"* (the auto-generated anchor isn't stable enough to hardcode; link the page and let the reader scroll) If Step 6 fell back to a callout, capture that in a one-line note at the top: *"This tutorial originally demonstrated retry by From 55f3f4dba130b0ea51249861040f89a083dd6f7f Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 07:59:26 +0300 Subject: [PATCH 3/9] =?UTF-8?q?docs:=20plan=20fixup=20=E2=80=94=20strict-m?= =?UTF-8?q?ode=20framing=20on=20Task=202=20smoke=20build?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- planning/active/2026-06-12-docs-tutorials-plan.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/planning/active/2026-06-12-docs-tutorials-plan.md b/planning/active/2026-06-12-docs-tutorials-plan.md index 4ac686a..51ee0a8 100644 --- a/planning/active/2026-06-12-docs-tutorials-plan.md +++ b/planning/active/2026-06-12-docs-tutorials-plan.md @@ -240,10 +240,11 @@ doesn't use (no DLQ, no timers, no `OutboxResponse`). Run: `just docs-build` - Expected: clean. The new page is orphaned-not-in-nav (Task 4 - wires it); mkdocs strict mode warns but does not fail on - orphaned pages by default. If it does fail, add the nav entry now - (out of Task-4 order) rather than disabling the strict check. + Expected: clean **or** an `not_in_nav` failure (Task 4 wires the + page in). `mkdocs build --strict` promotes that warning to an + error. If the build fails on it, add the nav entry now (out of + Task-4 order) rather than disabling the strict check — `--strict` + is load-bearing and shouldn't be relaxed for a single task. - [ ] **Step 11: Clean up the tutorial environment** From 5bd6c3db9d550a9c957253f4fcd6fdf9b7d771ab Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 08:10:07 +0300 Subject: [PATCH 4/9] =?UTF-8?q?docs:=20tutorial=20=E2=80=94=20your=20first?= =?UTF-8?q?=20outbox=20app?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds docs/tutorials/first-outbox-app.md, the 10-minute walk-through deferred from #56. Output blocks in the page were captured from an end-to-end run against a clean Postgres 17 container; no hand-edited expected output. Also adds the nav entry under "Getting started" (originally scheduled for Task 4) — without it, mkdocs --strict aborts on the orphan-page warning, and the spec is firm that --strict is load-bearing. The forward link to add-kafka-relay.md lands as plain text for now (target file does not exist yet); Task 3 will retarget it once that tutorial lands. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/tutorials/first-outbox-app.md | 315 +++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 316 insertions(+) create mode 100644 docs/tutorials/first-outbox-app.md diff --git a/docs/tutorials/first-outbox-app.md b/docs/tutorials/first-outbox-app.md new file mode 100644 index 0000000..023942f --- /dev/null +++ b/docs/tutorials/first-outbox-app.md @@ -0,0 +1,315 @@ +# Tutorial: Your first outbox app + +## What you'll build + +A tiny app where calling `broker.publish` inside a database transaction +triggers a handler — no message bus required, just Postgres. By the end +you will have run a single message end-to-end and seen the handler +print it. + +## Before you start + +- Python 3.13+ +- Docker (for a one-line Postgres container) +- [uv](https://docs.astral.sh/uv/) for project setup +- Roughly ten minutes + +## Step 1: Install + +Start a fresh project directory and add the package. The `asyncpg` +extra brings the async Postgres driver; `validate` is the Alembic-based +schema check helper; `cli` is FastStream's `faststream run` command. + +```bash +mkdir outbox-tutorial && cd outbox-tutorial +uv init +uv add 'faststream-outbox[asyncpg,validate]' 'faststream[cli]' +``` + +You should see: + +```text +Initialized project `outbox-tutorial` +Using CPython 3.14.4 +Creating virtual environment at: .venv +Resolved 26 packages in 61ms +Installed 24 packages in 37ms + + alembic==1.18.4 + + annotated-types==0.7.0 + + anyio==4.13.0 + + asyncpg==0.31.0 + + click==8.4.1 + + fast-depends==3.0.8 + + faststream==0.7.1 + + faststream-outbox==0.8.0 + + greenlet==3.5.1 + + idna==3.18 + + mako==1.3.12 + + markdown-it-py==4.2.0 + + markupsafe==3.0.3 + + mdurl==0.1.2 + + pydantic==2.13.4 + + pydantic-core==2.46.4 + + pygments==2.20.0 + + rich==15.0.0 + + shellingham==1.5.4 + + sqlalchemy==2.0.50 + + typer==0.21.1 + + typing-extensions==4.15.0 + + typing-inspection==0.4.2 + + watchfiles==1.1.1 +``` + +Your exact pinned versions will differ; that is fine. + +## Step 2: Start Postgres + +Run a disposable Postgres 17 container with the credentials we'll wire +into the connection string in Step 3. + +```bash +docker run --rm -d --name outbox-postgres \ + -e POSTGRES_USER=outbox -e POSTGRES_PASSWORD=outbox -e POSTGRES_DB=outbox \ + -p 5432:5432 postgres:17 +``` + +You should see a container ID printed: + +```text +7558ba0b8949e6410415f51152cd2da9b5eaab4ebae092aa14f2a6094f57d98f +``` + +Give it a couple of seconds and confirm it's ready: + +```bash +docker logs outbox-postgres 2>&1 | tail -1 +``` + +You should see: + +```text +2026-06-12 05:05:44.529 UTC [1] LOG: database system is ready to accept connections +``` + +## Step 3: Declare the outbox table + +Create `app.py`. This sets up the SQLAlchemy `MetaData`, declares the +outbox table on it, builds an async engine, and wires the broker and +FastStream app. We'll add the handler in Step 5. + +```python title="app.py" +from sqlalchemy import MetaData +from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + +from faststream import FastStream +from faststream_outbox import OutboxBroker, make_outbox_table + +metadata = MetaData() +outbox_table = make_outbox_table(metadata, table_name="outbox") + +engine = create_async_engine("postgresql+asyncpg://outbox:outbox@localhost:5432/outbox") +broker = OutboxBroker(engine, outbox_table=outbox_table) +app = FastStream(broker) + +session_factory = async_sessionmaker(engine, expire_on_commit=False) +``` + +`make_outbox_table` returns a `sqlalchemy.Table` attached to your +`MetaData`. The package never creates or migrates the schema on its +own — Step 4 is where we run that. + +## Step 4: Create the schema + +Create a second file, `create_schema.py`, that runs `metadata.create_all` +once. Real apps use Alembic; for a tutorial a one-shot script is the +honest shape. + +```python title="create_schema.py" +import asyncio + +from sqlalchemy import MetaData +from sqlalchemy.ext.asyncio import create_async_engine + +from faststream_outbox import make_outbox_table + +metadata = MetaData() +make_outbox_table(metadata, table_name="outbox") + + +async def main() -> None: + engine = create_async_engine("postgresql+asyncpg://outbox:outbox@localhost:5432/outbox") + async with engine.begin() as conn: + await conn.run_sync(metadata.create_all) + await engine.dispose() + + +asyncio.run(main()) +``` + +Run it: + +```bash +uv run python create_schema.py +``` + +You should see no output — that is success. + +Verify the table landed: + +```bash +docker exec outbox-postgres psql -U outbox -d outbox -c '\d outbox' +``` + +You should see: + +```text + Table "public.outbox" + Column | Type | Collation | Nullable | Default +------------------+--------------------------+-----------+----------+------------------------------------ + id | bigint | | not null | nextval('outbox_id_seq'::regclass) + queue | character varying(255) | | not null | + payload | bytea | | not null | + headers | jsonb | | | + attempts_count | bigint | | not null | '0'::bigint + deliveries_count | bigint | | not null | '0'::bigint + created_at | timestamp with time zone | | not null | now() + next_attempt_at | timestamp with time zone | | not null | now() + first_attempt_at | timestamp with time zone | | | + last_attempt_at | timestamp with time zone | | | + acquired_at | timestamp with time zone | | | + acquired_token | uuid | | | + timer_id | character varying(255) | | | +Indexes: + "outbox_pkey" PRIMARY KEY, btree (id) + "outbox_lease_idx" btree (queue, acquired_at) WHERE acquired_token IS NOT NULL + "outbox_pending_idx" btree (queue, next_attempt_at) WHERE acquired_token IS NULL + "outbox_timer_id_uq" UNIQUE, btree (queue, timer_id) WHERE timer_id IS NOT NULL +``` + +Three partial indexes show up alongside the columns — the broker uses +those at runtime; you don't need to think about them. + +## Step 5: Define a handler + +Add a subscriber to the bottom of `app.py`. The handler will run once +per row published to the `orders` queue. + +```python title="app.py (additions)" +@broker.subscriber("orders") +async def handle(order_id: int) -> None: + print(f"got order {order_id}") +``` + +No command yet — the handler runs once we publish a row and start the +app. + +## Step 6: Publish a row + +Add an `@app.after_startup` hook to the bottom of `app.py` that publishes +one row right after the app boots. `broker.publish` inserts an outbox +row through the session you give it — the row commits with the +surrounding transaction. There is no separate "send" step; the commit +is the send. + +```python title="app.py (additions)" +@app.after_startup +async def publish_one() -> None: + async with session_factory() as session, session.begin(): + await broker.publish(1, queue="orders", session=session) +``` + +The full `app.py` now reads: + +```python title="app.py" +from sqlalchemy import MetaData +from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + +from faststream import FastStream +from faststream_outbox import OutboxBroker, make_outbox_table + +metadata = MetaData() +outbox_table = make_outbox_table(metadata, table_name="outbox") + +engine = create_async_engine("postgresql+asyncpg://outbox:outbox@localhost:5432/outbox") +broker = OutboxBroker(engine, outbox_table=outbox_table) +app = FastStream(broker) + +session_factory = async_sessionmaker(engine, expire_on_commit=False) + + +@broker.subscriber("orders") +async def handle(order_id: int) -> None: + print(f"got order {order_id}") + + +@app.after_startup +async def publish_one() -> None: + async with session_factory() as session, session.begin(): + await broker.publish(1, queue="orders", session=session) +``` + +## Step 7: Run it + +Start the app: + +```bash +uv run faststream run app:app +``` + +You should see: + +```text +2026-06-12 08:07:06,179 INFO - FastStream app starting... +2026-06-12 08:07:06,179 INFO - orders | - `Handle` waiting for messages +2026-06-12 08:07:06,276 INFO - FastStream app started successfully! To exit, press CTRL+C +2026-06-12 08:07:06,283 INFO - orders | - Received +got order 1 +2026-06-12 08:07:06,283 INFO - orders | - Processed +``` + +That `got order 1` line is your handler firing. The row was inserted by +the `@app.after_startup` hook, the subscriber's fetch loop picked it up, +dispatched it, the handler ran, and the row was deleted. + +Press `Ctrl-C`: + +```text +2026-06-12 08:07:11,989 INFO - FastStream app shutting down... +2026-06-12 08:07:11,990 INFO - FastStream app shut down gracefully. +2026-06-12 08:07:11,990 INFO - | - callback for Task-2 is being executed... +2026-06-12 08:07:11,990 INFO - | - callback for Task-3 is being executed... +``` + +## What you just built + +- An outbox table inside your own Postgres database, owned by your + schema. +- A FastStream app whose "transport" is rows in that table — no + external broker. +- A handler that ran exactly once, in-process, against a row committed + by your own session. + +The interesting property is what happened *inside* `publish_one`: the +`broker.publish` call inserted a row into the outbox table through the +session you opened. `session.begin()` committed it. If that commit had +rolled back — say, because a domain write on the same session +failed — the outbox row would have rolled back with it. There is no +universe where the row exists but the domain write doesn't, or vice +versa. That atomicity is the whole point. + +## Clean up + +```bash +docker stop outbox-postgres +``` + +## What's next + +- [Subscriber reference](../usage/subscriber.md) — tuning, worker + counts, retry strategies. +- [Publisher reference](../usage/publisher.md) — `publish_batch`, the + `OutboxPublisher` decorator, chained publishing. +- [FastAPI integration](../usage/fastapi.md) — wire the outbox into + a real HTTP service with `Depends(get_session)`. +- Tutorial: Add a Kafka relay (next tutorial) — extend this app to + forward each row into Kafka with one stacked decorator. diff --git a/mkdocs.yml b/mkdocs.yml index 52db0ca..17b28ba 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -8,6 +8,7 @@ nav: - Getting started: - Installation: introduction/installation.md - Basic usage: usage/basic.md + - 'Tutorial: Your first outbox app': tutorials/first-outbox-app.md - Concepts: - How it works: introduction/how-it-works.md - Comparison: concepts/comparison.md From e0761b393296bc6b6d19b7bad3533cc9df4d5664 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 08:15:15 +0300 Subject: [PATCH 5/9] =?UTF-8?q?docs:=20spec=20fixup=20=E2=80=94=20install?= =?UTF-8?q?=20needs=20faststream[cli]=20for=20`faststream=20run`?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Caught during Task 2 execution. `faststream run app:app` ImportErrors without the CLI extra, so the tutorial's install line includes it; the spec outline now matches. Co-Authored-By: Claude Opus 4.7 (1M context) --- planning/active/2026-06-12-docs-tutorials-design.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/planning/active/2026-06-12-docs-tutorials-design.md b/planning/active/2026-06-12-docs-tutorials-design.md index 401f73f..d86ed77 100644 --- a/planning/active/2026-06-12-docs-tutorials-design.md +++ b/planning/active/2026-06-12-docs-tutorials-design.md @@ -103,7 +103,9 @@ What you'll build (2 sentences: a tiny app where publishing Before you start (Python 3.13+, Postgres, ~10 minutes) -Step 1: Install (uv add 'faststream-outbox[asyncpg,validate]') +Step 1: Install (uv add 'faststream-outbox[asyncpg,validate]' + 'faststream[cli]' — the CLI extra is needed + for `faststream run`) Step 2: Start Postgres (one-liner Docker) From 15643b7a37e0ec8926112c3bb845a3256ff91e0a Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 08:36:40 +0300 Subject: [PATCH 6/9] =?UTF-8?q?docs:=20tutorial=20=E2=80=94=20add=20a=20Ka?= =?UTF-8?q?fka=20relay?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/tutorials/add-kafka-relay.md | 275 +++++++++++++++++++++++++++++ docs/tutorials/first-outbox-app.md | 4 +- mkdocs.yml | 1 + 3 files changed, 278 insertions(+), 2 deletions(-) create mode 100644 docs/tutorials/add-kafka-relay.md diff --git a/docs/tutorials/add-kafka-relay.md b/docs/tutorials/add-kafka-relay.md new file mode 100644 index 0000000..9597f2d --- /dev/null +++ b/docs/tutorials/add-kafka-relay.md @@ -0,0 +1,275 @@ +# Tutorial: Add a Kafka relay + +## What you'll add + +In [Tutorial: Your first outbox app](./first-outbox-app.md) the handler +printed the row and that was the end of it. Real outbox systems usually +*relay* the row to a real message bus — Kafka, RabbitMQ, NATS — so +downstream services can consume it. In this tutorial you'll add a Kafka +broker, stack a single decorator above the existing subscriber, and watch +a row written inside a Postgres transaction land on a Kafka topic. + +By the end you will have run a single message end-to-end through the +relay and seen the row arrive at a `kafka-console-consumer`. + +## Before you start + +- You finished [Tutorial: Your first outbox app](./first-outbox-app.md). + This tutorial extends that same `app.py`, the same `outbox-postgres` + container, and the same project directory. +- Docker Compose (the `docker compose` CLI) for the Kafka container. +- Another ten minutes. + +## Step 1: Add Kafka via docker-compose + +Postgres is already running from Tutorial 1. Add Kafka via a small +`docker-compose.yml`. Single-broker [KRaft +mode](https://kafka.apache.org/documentation/#kraft) — no separate +ZooKeeper service, and Confluent's `cp-kafka:7.6.0` image is known to +run well on Apple Silicon. Two listeners: one for clients on the host +(your `faststream run` process) and one for clients inside the Docker +network (the `kafka-console-consumer` we'll use in Step 5). + +```yaml title="docker-compose.yml" +services: + kafka: + image: confluentinc/cp-kafka:7.6.0 + container_name: outbox-kafka + ports: + - "9092:9092" + environment: + CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk" + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: broker,controller + KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 + KAFKA_LISTENERS: HOST://0.0.0.0:9092,DOCKER://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093 + KAFKA_ADVERTISED_LISTENERS: HOST://localhost:9092,DOCKER://kafka:29092 + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,HOST:PLAINTEXT,DOCKER:PLAINTEXT + KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 +``` + +Bring it up: + +```bash +docker compose up -d kafka +``` + +You should see (image pull progress trimmed): + +```text + Network outbox-tutorial_default Creating + Network outbox-tutorial_default Created + Container outbox-kafka Creating + Container outbox-kafka Created + Container outbox-kafka Starting + Container outbox-kafka Started +``` + +Give it ten seconds and confirm the broker came up cleanly: + +```bash +docker compose logs kafka 2>&1 | grep -m1 'Kafka Server started' +``` + +You should see: + +```text +outbox-kafka | [2026-06-12 05:22:33,782] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer) +``` + +## Step 2: Install `faststream[kafka]` + +```bash +uv add 'faststream[kafka]' +``` + +You should see: + +```text +Resolved 29 packages in 785ms +Installed 3 packages in 6ms + + aiokafka==0.14.0 + + async-timeout==5.0.1 + + packaging==26.2 +``` + +Your pinned versions will differ. + +## Step 3: Add the Kafka broker + +Open `app.py` from Tutorial 1 and add a `KafkaBroker` plus a publisher +for the `orders.kafka` topic. Rename the existing `broker` to +`broker_outbox` so the two brokers have distinct names. Hook +`broker_kafka.connect` into `FastStream`'s `on_startup` so the Kafka +client opens before the first row is dispatched. + +```python title="app.py (edits)" +from faststream.kafka import KafkaBroker + +broker_outbox = OutboxBroker(engine, outbox_table=outbox_table) +broker_kafka = KafkaBroker("localhost:9092") +kafka_publisher = broker_kafka.publisher("orders.kafka") + +app = FastStream(broker_outbox, on_startup=[broker_kafka.connect]) +``` + +## Step 4: Stack the publisher decorator + +Stack `@kafka_publisher` above the existing +`@broker_outbox.subscriber("orders")` and change the handler to `return +order_id`. The stacked decorator picks up the return value and publishes +it to `orders.kafka`. The outbox subscriber is still the one driving +delivery — Kafka becomes the *destination*, not a second subscriber. + +```python title="app.py (edits)" +@kafka_publisher +@broker_outbox.subscriber("orders") +async def handle(order_id: int) -> int: + print(f"got order {order_id}") + return order_id +``` + +The full `app.py` now reads: + +```python title="app.py" +from sqlalchemy import MetaData +from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + +from faststream import FastStream +from faststream.kafka import KafkaBroker +from faststream_outbox import OutboxBroker, make_outbox_table + +metadata = MetaData() +outbox_table = make_outbox_table(metadata, table_name="outbox") + +engine = create_async_engine("postgresql+asyncpg://outbox:outbox@localhost:5432/outbox") +broker_outbox = OutboxBroker(engine, outbox_table=outbox_table) +broker_kafka = KafkaBroker("localhost:9092") +kafka_publisher = broker_kafka.publisher("orders.kafka") + +app = FastStream(broker_outbox, on_startup=[broker_kafka.connect]) + +session_factory = async_sessionmaker(engine, expire_on_commit=False) + + +@kafka_publisher +@broker_outbox.subscriber("orders") +async def handle(order_id: int) -> int: + print(f"got order {order_id}") + return order_id + + +@app.after_startup +async def publish_one() -> None: + async with session_factory() as session, session.begin(): + await broker_outbox.publish(1, queue="orders", session=session) +``` + +## Step 5: Run it and watch a row reach Kafka + +Start the app in one terminal: + +```bash +uv run faststream run app:app +``` + +You should see: + +```text +2026-06-12 08:23:28,284 INFO - FastStream app starting... +2026-06-12 08:23:28,328 INFO - orders | - `Handle` waiting for messages +2026-06-12 08:23:28,389 INFO - FastStream app started successfully! To exit, press CTRL+C +2026-06-12 08:23:28,394 INFO - orders | - Received +Topic orders.kafka not found in cluster metadata +got order 1 +2026-06-12 08:23:28,527 INFO - orders | - Processed +``` + +The `Topic orders.kafka not found in cluster metadata` line is +`aiokafka` noticing a brand-new topic and asking the broker to +auto-create it — first-run only. + +In a second terminal, attach a console consumer to the topic: + +```bash +docker compose exec kafka kafka-console-consumer \ + --bootstrap-server kafka:9092 --topic orders.kafka --from-beginning +``` + +You should see: + +```text +1 +``` + +The single row `broker_outbox.publish(1, ...)` wrote inside the +Postgres transaction has now landed on the Kafka topic. The path was: +session commit → outbox row → outbox subscriber → handler → Kafka +publisher decorator → Kafka topic. Press `Ctrl-C` to stop the consumer. + +## What about Kafka downtime? + +If Kafka were unavailable when the outbox subscriber dispatched a row, +the foreign publish would raise, the outbox row would be nacked, and +the configured `retry_strategy` would reschedule it. The next dispatch +re-runs the handler and re-attempts the foreign publish. The net effect +is **at-least-once delivery to the foreign broker** — the outbox row is +the durability boundary, and it stays in the table until Kafka actually +acks the publish. + +In practice, `aiokafka`'s producer has its own client-side reconnect +and retry logic, so a short Kafka outage usually completes from the +outbox subscriber's perspective as a single (slow) publish rather than +as a visible retry on the outbox side. Either way the at-least-once +property is preserved. See [Subscriber § Retry +strategies](../usage/subscriber.md#retry-strategies) for the outbox's +own retry policy and [Relay § At-least-once +contract](../usage/relay.md#at-least-once-contract) for the relay +contract in full. + +## What you just built + +- A two-broker app: an `OutboxBroker` over Postgres and a `KafkaBroker` + over a local Kafka container. +- A single subscriber whose return value is forwarded to a Kafka topic + via a stacked publisher decorator — no second handler, no manual + client code. +- An at-least-once relay: the row is durable in Postgres until the + Kafka publish succeeds. + +The interesting property is the *transactional* part of the publish. +The `broker_outbox.publish(1, ...)` call in `publish_one` ran inside a +session that committed atomically — the row reached the outbox table +as part of the same `COMMIT` that any sibling domain writes would have +committed. There is no window in which the row exists but a sibling +domain write doesn't, or vice versa. The Kafka delivery happens *after* +that boundary, asynchronously, with its own retry safety net. The +outbox is what makes those two halves — transactional domain write and +non-transactional bus publish — survive a process crash together. + +## Clean up + +```bash +docker compose down -v +docker stop outbox-postgres +``` + +The first stops Kafka and removes the compose network; the second +stops the Postgres container from Tutorial 1. + +## What's next + +- [Relay reference](../usage/relay.md) — the full contract: header + propagation, two-broker lifecycle, other foreign brokers + (RabbitMQ / NATS / Redis), what *not* to do. +- [Subscriber retry strategies](../usage/subscriber.md#retry-strategies) + — `ExponentialRetry`, `LinearRetry`, `ConstantRetry`, `NoRetry`, and + "retry only on transient errors." +- [Comparison](../concepts/comparison.md) — see the section *"vs. + FastStream + `KafkaBroker` / `RabbitBroker` directly"* for the + pattern's trade-offs vs. just publishing to Kafka straight from + your request handler. diff --git a/docs/tutorials/first-outbox-app.md b/docs/tutorials/first-outbox-app.md index 023942f..10cd3b6 100644 --- a/docs/tutorials/first-outbox-app.md +++ b/docs/tutorials/first-outbox-app.md @@ -311,5 +311,5 @@ docker stop outbox-postgres `OutboxPublisher` decorator, chained publishing. - [FastAPI integration](../usage/fastapi.md) — wire the outbox into a real HTTP service with `Depends(get_session)`. -- Tutorial: Add a Kafka relay (next tutorial) — extend this app to - forward each row into Kafka with one stacked decorator. +- [Tutorial: Add a Kafka relay](./add-kafka-relay.md) — extend this + app to forward each row into Kafka with one stacked decorator. diff --git a/mkdocs.yml b/mkdocs.yml index 17b28ba..c677854 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,6 +9,7 @@ nav: - Installation: introduction/installation.md - Basic usage: usage/basic.md - 'Tutorial: Your first outbox app': tutorials/first-outbox-app.md + - 'Tutorial: Add a Kafka relay': tutorials/add-kafka-relay.md - Concepts: - How it works: introduction/how-it-works.md - Comparison: concepts/comparison.md From d4153eb20944964576e9e80e830b6c5f1d311993 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 08:41:43 +0300 Subject: [PATCH 7/9] =?UTF-8?q?docs:=20tutorial=202=20=E2=80=94=20maintain?= =?UTF-8?q?er=20note=20on=20the=20dropped=20kill-Kafka=20demo?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per the plan's Task 3 Step 7, the kill-Kafka fallback needs a disclosure that the live demo was attempted and dropped, either at the page top or folded into the callout. An HTML comment in the callout meets that bar without injecting "scar tissue" prose into the reader-facing narrative — future maintainers see the rationale, readers don't. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/tutorials/add-kafka-relay.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/tutorials/add-kafka-relay.md b/docs/tutorials/add-kafka-relay.md index 9597f2d..9188b97 100644 --- a/docs/tutorials/add-kafka-relay.md +++ b/docs/tutorials/add-kafka-relay.md @@ -213,6 +213,18 @@ publisher decorator → Kafka topic. Press `Ctrl-C` to stop the consumer. ## What about Kafka downtime? + + If Kafka were unavailable when the outbox subscriber dispatched a row, the foreign publish would raise, the outbox row would be nacked, and the configured `retry_strategy` would reschedule it. The next dispatch From 005baef2e789c188c85ce083aa3fcbf7fb62d469 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 08:45:34 +0300 Subject: [PATCH 8/9] =?UTF-8?q?docs:=20cross-links=20=E2=80=94=20relay=20t?= =?UTF-8?q?utorial=20pointer=20+=20landing=20decision=20tree?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/index.md | 2 +- docs/usage/relay.md | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index 15f148b..c183a8f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -41,7 +41,7 @@ process, no Kafka. | Understand the architecture before adopting | [How it works](introduction/how-it-works.md) | | Compare against CDC / Kafka transactions / a hand-rolled outbox | [Comparison](concepts/comparison.md) | | Deploy to production safely | [Production checklist](operations/checklist.md) | -| Install and write the first publisher / subscriber | [Installation](introduction/installation.md) → [Basic usage](usage/basic.md) | +| Install and write the first publisher / subscriber | [Installation](introduction/installation.md) → [Tutorial: Your first outbox app](tutorials/first-outbox-app.md) | ## Documentation diff --git a/docs/usage/relay.md b/docs/usage/relay.md index b7fd7f6..60749a2 100644 --- a/docs/usage/relay.md +++ b/docs/usage/relay.md @@ -1,5 +1,8 @@ # Relay to a foreign broker +> Want a worked end-to-end example? See +> [Tutorial: Add a Kafka relay](../tutorials/add-kafka-relay.md). + The outbox pattern's payoff line: domain code writes a row to the outbox in the same DB transaction as its other writes, and a separate worker relays those rows to a real bus (Kafka, RabbitMQ, NATS, Redis…). `faststream-outbox` From ea19974fe492042ca3d5c4b251b0beeb016a5231 Mon Sep 17 00:00:00 2001 From: Artur Shiriev Date: Fri, 12 Jun 2026 08:48:31 +0300 Subject: [PATCH 9/9] =?UTF-8?q?docs:=20tutorial=201=20polish=20=E2=80=94?= =?UTF-8?q?=20schema-redeclaration=20note=20+=20version=20disclaimer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Caught by code-quality review: - create_schema.py redeclares MetaData/outbox_table; flag the footgun for readers who copy both files and later edit app.py. - The Step 1 uv add output shows whatever Python uv resolves; disclaim 3.13 vs 3.14 explicitly. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/tutorials/first-outbox-app.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/tutorials/first-outbox-app.md b/docs/tutorials/first-outbox-app.md index 10cd3b6..44bc70c 100644 --- a/docs/tutorials/first-outbox-app.md +++ b/docs/tutorials/first-outbox-app.md @@ -60,7 +60,9 @@ Installed 24 packages in 37ms + watchfiles==1.1.1 ``` -Your exact pinned versions will differ; that is fine. +Your exact pinned versions will differ; that is fine. The Python version +line will reflect whatever `uv` resolves on your machine — 3.13 or 3.14 +are both fine. ## Step 2: Start Postgres @@ -146,6 +148,10 @@ async def main() -> None: asyncio.run(main()) ``` +*Real projects import `metadata` and `outbox_table` from a shared module +rather than redeclaring them here; this script is self-contained for the +tutorial's narrow scope.* + Run it: ```bash