paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

July 29, 2026

Jupyter Notebooks as an API Engineering Workflow

Turning use-case flows into executable, narrated API tests - and why a notebook harness belongs next to k6, not instead of it

Introduction

Most API test suites are built to answer one question: did this request return the status code we expected? That question matters, but it is a shallow one. It tells you nothing about whether a customer can actually sign up, build a cart, check out, get charged, and have that order show up correctly in a retailer's sales report a week later. Answering that second, harder question requires a different shape of test - one that threads state across multiple calls and treats the sequence itself as the thing under test, not the individual request.

This is the case for using Jupyter notebooks as a first-class part of an API testing stack, sitting alongside unit tests and load tests rather than replacing either. The core idea is narrow and easy to state: one notebook per use-case flow, each cell making a real HTTP call against a running API and asserting on the real response, with markdown cells narrating why each assertion holds. The rest of this post walks through why that structure was chosen over the more obvious alternatives, what kinds of testing it actually covers, how it fits next to a k6 load-testing setup, and where the sharp edges are.

Why Not Postman, and Why Not a Dedicated Test Framework

Three options were on the table for exercising these flows: a Postman or Insomnia collection, a dedicated integration-test framework (pytest with requests, or a similar xUnit-style harness), and Jupyter notebooks using requests and Faker. All three can send HTTP requests and assert on responses. The difference is in what happens to the evidence after the assertion passes.

A Postman or Insomnia collection is excellent at capturing a single request and its expected response, and reasonably good at chaining a handful of requests using environment variables. But the moment a flow needs conditional logic - generate a unique email only if the last run's fixture collided, branch on whether a cart already has items, loop over several Faker-generated customers each building an independent cart - the collection's scripting model (small JavaScript snippets attached to pre-request and test tabs) starts to strain. More importantly, a passing Postman run collapses into a green checkmark. The actual request bodies, response payloads, and intermediate state that made the assertion pass are not part of the artifact anyone looks at later; they live in a run log that gets discarded. A dedicated integration-test framework solves the scripting problem - pytest fixtures and helper functions handle branching and state far more naturally than Postman's pre-request scripts - but it inherits the same evidence problem. A green pytest run in CI is, again, just a checkmark. Reading why test_checkout_flow passed means reading the test source, which is written for the test runner, not for a human trying to understand the API contract.

Jupyter notebooks solve the evidence problem directly, because a notebook's output cells are not disposable. When cell 6 executes POST /carts/{id}/checkout and the next cell prints the resulting order's status and totalRevenue fields, that JSON is sitting in the document, inline, next to the markdown explaining what should be true about it and why. Nothing about the API's actual behavior is hidden behind a pass/fail boolean. This does not make the notebook harness strictly better than a test framework in every dimension - it is slower to run, and less suited to CI gating in the way pytest is - but it makes the harness a fundamentally different artifact: one whose purpose is to be read, not just executed.

One Notebook per Use-Case Flow

The organizing principle is simple and deliberate: one notebook per use-case flow, numbered in the order a new engineer should read them, not one notebook per endpoint. This decision does the most work in the entire design, so it is worth being explicit about the alternative it rejects. An endpoint-per-notebook structure would have produced shallow, independent request/response pairs - exactly what a Postman collection already provides, and exactly the structure that fails to catch bugs that only appear when state is threaded across calls. If POST /products and DELETE /products/{id} are tested in isolation, nothing forces the suite to ever create a cart that references the product first, which means the 409-while-referenced business rule - you cannot delete a product that a cart still points to - never gets exercised at all.

A flow-per-notebook structure forces every notebook to tell a small story: create the actors, mutate state, then assert on the consequences of that mutation somewhere else in the system. In this harness that produced eight notebooks: retailer and catalog CRUD ending in negative-price rejection and a 409-while-referenced delete; customer CRUD with Faker-generated customers; the full signup-to-catalog-to-cart-to-checkout-to-order-to-sales-report flow; multiple concurrent customers each building an independent cart; a dedicated error-path notebook covering invalid ids, missing fields, and checkout without a cart; multipart image upload through rendition generation and static-file serving; the complete order lifecycle state machine from cart through fulfilled including cancellation; and an unstructured scratch notebook for one-off manual debugging calls. Numbering them in read order means a new engineer can open notebook one and, cell by cell, learn the API the same way they would learn it by reading well-organized documentation - except every claim in this documentation is backed by a real, currently-passing HTTP call.

The Seven Testing Types This Harness Covers

Laying all eight notebooks side by side, the harness ends up covering seven distinct kinds of testing, each mapped to a concrete example from the flows above:

Three more categories round the set out. Referential-integrity and cascade testing verifies that deleting a product still referenced by a cart returns 409, and that deleting a customer cascades their in-progress cart but must not touch settled order history or retailer revenue - a rule that is easy to state and surprisingly easy to get backwards in an ORM's cascade configuration. Idempotency and aggregation testing checks that adding the same product to a cart twice accumulates one line item instead of duplicating it, with the price re-snapshotting to the current value rather than the value at first add, and that sales aggregation counts only paid/fulfilled order items while excluding abandoned carts. Binary and media contract testing is the odd one out structurally - it is the only category that deals with non-JSON payloads - and covers multipart upload acceptance, generated-rendition count and naming, static file serving with the correct Content-Type header, and confirming that a deleted file's static URL returns 404 rather than a stale cached copy.

Four testing types deliberately sit outside this harness's scope: concurrency (does the system behave correctly when a hundred of these flows run at once, not just one), load and latency (how fast, under what load), unit-level logic (pure functions and business-rule edge cases better tested in isolation, without an HTTP round trip), and security testing (authentication bypass, injection, and authorization boundary testing, which need their own dedicated tooling and threat model). Naming what a testing layer does not cover is as important as naming what it does - a harness that tries to be everything ends up being a worse version of every specialized tool it is trying to replace.

Faker Instead of Hard-Coded Fixtures

Using Faker instead of hard-coded fixtures matters for a reason beyond convenience. Hard-coded fixtures - "Test Retailer 1", "test@test.com" - tend to accumulate hidden assumptions about string length, character set, and uniqueness that never get exercised again once the first version of the test passes. If a notebook always signs up test@test.com, the uniqueness constraint on the customer's email is only ever tested once, at whatever point the fixture was first written; every subsequent run either has to delete and recreate that exact record or silently rely on the previous run's leftover state. fake.unique.company_email() and fake.pyfloat(...) regenerate a fresh, plausible, distinct payload on every execution, so a notebook re-run five times in a row is still exercising five different, valid inputs against the same assertions - a small amount of property-based-testing flavor riding on top of a fully deterministic flow structure.

A representative cell from the checkout notebook makes the pattern concrete:

from faker import Faker
import requests

fake = Faker()
BASE_URL = "http://localhost:8000"

# Cell: create a customer with fresh, valid, distinct data every run
customer_payload = {
    "email": fake.unique.company_email(),
    "name": fake.name(),
    "shipping_address": fake.address(),
}
resp = requests.post(f"{BASE_URL}/customers", json=customer_payload)
assert resp.status_code == 201, resp.text
customer = resp.json()
assert customer["email"] == customer_payload["email"]
assert "id" in customer
print(f"Created customer {customer['id']}: {customer['email']}")

# Cell: add the same product twice, assert accumulation not duplication
product_id = catalog_products[0]["id"]
original_price = catalog_products[0]["price"]

for _ in range(2):
    resp = requests.post(
        f"{BASE_URL}/customers/{customer['id']}/cart/items",
        json={"productId": product_id, "quantity": 1},
    )
    assert resp.status_code == 200, resp.text

cart = requests.get(f"{BASE_URL}/customers/{customer['id']}/cart").json()
matching_items = [i for i in cart["items"] if i["productId"] == product_id]
assert len(matching_items) == 1, "expected accumulation, got duplicate line items"
assert matching_items[0]["quantity"] == 2
assert matching_items[0]["unitPrice"] == original_price, "price should snapshot at add time"

Each assert failing raises immediately with the response body attached, which is what makes a notebook a genuine test and not just a demo script - a broken flow stops the notebook cold at the exact cell where the contract was violated, with the actual failing payload printed directly above the traceback.

Notebooks and k6 Are the Same Flows, Different Questions

The clean way to see the boundary between this harness and a load-testing tool like k6 is that both are testing the same underlying flows but asking different questions of them. A notebook asks: does one execution of this flow produce the correct result? k6 asks: does this flow still produce correct-enough results, fast enough, when a hundred of them run at once? Those are genuinely different failure modes. A notebook run can pass perfectly - every assertion green, every field correct - while the same flow, run under k6 at concurrency, exposes a race condition where two customers checking out simultaneously both decrement the same inventory count and end up overselling a product. Conversely, a k6 run can report acceptable p95 latency and zero errors while quietly returning a response that violates the contract, because k6's default checks are usually shallower than a notebook's line-by-line field assertions.

Reusing the same flow definitions across both tools is what makes this pairing valuable rather than redundant. In practice this means the request-building logic - constructing a valid signup payload, a valid cart mutation, a valid checkout call - is written once, as plain functions, and both the notebooks and the k6 scripts (in k6's case, translated into JavaScript per its scripting model) call into the same conceptual flow. When the API's contract changes, both harnesses need updating together, which is a feature, not a maintenance burden: it means the two tools can never silently drift into testing different versions of the same use case.

Trade-offs and Pitfalls

Notebooks are not free, and treating them as an unqualified upgrade over other testing approaches would be dishonest. A .ipynb file is JSON containing cell outputs, so a re-executed notebook produces a diff even when no logic changed at all - the cell execution counters increment, and any timestamp or generated id in a printed output changes on every run. The mitigation is to treat notebook outputs as disposable: run jupyter nbconvert --to notebook --execute before every commit rather than trusting whatever outputs happen to be sitting in the file, and consider stripping outputs entirely in a pre-commit hook with a tool like nbstripout if the team decides committed outputs add more diff noise than documentation value.

Notebooks also share mutable global state across cells by design, which is exactly what makes flow-testing possible in the first place, but it comes with a real cost: cells must be run top-to-bottom in order. Running cell 7 before cell 3 produces confusing failures that have nothing to do with the API under test and everything to do with an undefined customer variable. This is a deliberate trade, not an oversight - the numbered, linear structure of each notebook makes "run all" the natural and expected way to execute it, and the alternative (isolating each cell's state, the way pytest isolates each test function) would defeat the entire purpose of testing multi-step flows. A team adopting this pattern should be explicit that "run all cells, top to bottom" is the contract, the same way a shell script that exports environment variables for later lines implicitly assumes it will be run start to finish rather than line by line in a REPL.

A third pitfall worth naming: because notebooks are slower to execute than a compiled or bytecode-cached test suite, and because they depend on a live API instance (typically via Docker Compose with a health check gating startup), they are not a substitute for a fast unit-test feedback loop during active development. They are a regression and documentation layer, best run in CI on a schedule or on API-contract-relevant pull requests, not on every keystroke.

Best Practices

A few practices make this pattern durable rather than fragile. First, freeze the contract the notebooks assert against and treat any change to it as deliberate: if totalRevenue is renamed or a status code changes, that should be a conscious, reviewed edit to the relevant notebook cell, not a silent breakage discovered by a confused engineer three weeks later. Second, keep markdown cells doing real explanatory work - not restating what the code obviously does, but stating why the assertion holds, especially for the less obvious business rules like price re-snapshotting on cart mutation or cascade-but-preserve-history on customer deletion. A markdown cell that says "price should snapshot at add time, not checkout time, per the pricing policy in ADR-014" is worth far more than one that says "check the price."

Second in importance only to the first: gate the harness behind a real health check in whatever orchestration wires it up to the API, whether that is Docker Compose's depends_on: condition: service_healthy or an equivalent readiness probe. A notebook harness that races against an API still finishing its database migrations will produce intermittent, confusing failures that look like contract violations but are actually startup-ordering bugs, and those false positives erode trust in the suite faster than almost anything else. Combined with nbconvert --execute in CI and a scheduled or PR-triggered run cadence, this turns the eight notebooks into a fast, reliable regression check that a new engineer can also read start to finish as onboarding material - genuinely a two-for-one on testing and documentation investment.

Key Takeaways

Conclusion

Adding a Jupyter notebook harness does not replace anything already in the stack. It fills the gap between isolated unit tests and concurrent k6 load tests with something neither of those tools does well: narrated, stateful, use-case-level verification of the exact HTTP contract real clients depend on. Eight notebooks, wired into the same Docker Compose network as the API and gated behind a health check, walk every major flow the frozen API contract promises - catalog CRUD, signup, checkout, bulk seeding, error paths, image uploads, and the full order lifecycle - and between them cover contract, functional, negative, state-machine, referential-integrity, idempotency, and media-contract testing.

The deeper argument here is about where evidence lives. A green checkmark in a CI dashboard tells you a test passed; it does not tell you what the API actually returned, or why that response satisfies the business rule it was meant to enforce. A notebook, re-executed and committed with fresh output, makes that evidence visible in the same document a new engineer would read to understand the system in the first place. That is a small shift in where testing infrastructure lives, but it changes what the test suite is for - not just a gate that blocks bad deploys, but a live, runnable explanation of what the API promises and how that promise is verified.

References

Resources