paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

January 03, 2020

REST API Design: From First Principles to Advanced System Design Patterns

A practical, engineering-first guide to designing REST APIs that scale - covering resource modeling, HTTP semantics, versioning, pagination, HATEOAS, idempotency, and the trade-offs that matter in production systems

Introduction

Every distributed system eventually needs a contract - a way for services, mobile clients, and third-party integrators to talk to each other without knowing anything about each other's internals. For the last two decades, that contract has usually been a REST API. It is not the only option (gRPC, GraphQL, and event-driven messaging all have their place), but REST remains the default choice for public APIs and the majority of internal service-to-service communication, largely because it maps naturally onto HTTP, which every platform, proxy, and monitoring tool already understands.

What makes REST deceptively hard is that it looks simple on the surface - "just use HTTP verbs and JSON" - but the details compound quickly. A poorly modeled resource hierarchy, an inconsistent error format, or a missing idempotency strategy will not show up in a demo. It shows up eighteen months later when three teams are integrating against your API and each one has built a slightly different workaround for the same design flaw. This article walks through REST API design as a system design problem, starting from the constraints Roy Fielding actually described, moving through the HTTP mechanics that most APIs get subtly wrong, and ending with the advanced patterns - idempotency, HATEOAS, rate limiting, and versioning strategy - that separate an API that survives its first year from one that has to be rebuilt.

What Problem Is REST Actually Solving?

REST - Representational State Transfer - was defined by Roy Fielding in his 2000 doctoral dissertation as an architectural style for distributed hypermedia systems, developed alongside the design of HTTP 1.1 itself. It is important to understand that REST is a set of architectural constraints, not a specification or a protocol. Fielding's dissertation lays out six constraints: a client-server separation, statelessness, cacheability, a uniform interface, a layered system, and the optional constraint of code-on-demand. Most APIs that call themselves "RESTful" today satisfy the first five and ignore the sixth, which is fine - code-on-demand was always the least essential of the constraints for typical API use cases.

The uniform interface constraint is where most of the day-to-day design work lives, and it breaks down into four further ideas:

That last piece is the one almost every "RESTful" API in production today skips, and we will come back to why that omission is usually a pragmatic trade-off rather than a mistake.

The practical problem REST solves is decoupling. A client that talks to /orders/482/items doesn't need to know whether that data lives in a single Postgres table, a sharded cluster, or three microservices behind an API gateway. Statelessness means any server instance can handle any request, which is what makes horizontal scaling and load balancing straightforward - there's no session affinity to manage at the HTTP layer. This is the design property that makes REST attractive for system design interviews and real architectures alike: it forces you to think about your API as a stable, cacheable, horizontally scalable surface rather than a thin wrapper around your database schema.

Resource Modeling and the Uniform Interface

The single highest-leverage decision in REST API design is how you model resources, because every other decision - URL structure, HTTP verbs, status codes - flows from it. A resource is a noun: a user, an order, an invoice line item. The URI identifies it; the HTTP method describes what you want to do to it. This sounds obvious, but the most common REST anti-pattern is exactly the opposite: URLs that encode verbs, like /getUserOrders or /cancelOrder. These are RPC calls wearing REST's clothing, and they push all the semantic information into the URL path instead of the HTTP method, which means your API loses the ability to use HTTP's built-in caching, idempotency, and tooling support.

Good resource modeling also means getting the hierarchy right. /customers/{id}/orders/{orderId} expresses a genuine ownership relationship - an order belongs to a customer, and that nesting should exist if and only if the child resource cannot meaningfully exist without the parent. Over-nesting is a common mistake: if orders can be queried independently of customers (for admin tooling, for example), a flatter /orders/{orderId} with a customerId field and a filterable /orders?customerId={id} collection endpoint is usually more flexible than forcing every access path through the parent. The rule of thumb: nest only as deep as the data's actual lifecycle dependency, and expose collections through query parameters when a resource has more than one legitimate "parent" context.

Collections and singular resources need consistent naming conventions too. Plural nouns for collections (/orders, not /order), and the same plural form when addressing a single item (/orders/{id}) keeps the mental model coherent - a single order is just an element of the orders collection. Consistency here matters more than which specific convention you pick, because API consumers build muscle memory, and generated SDKs and OpenAPI tooling both assume this pattern.

HTTP Methods, Status Codes, and Getting Semantics Right

HTTP verbs carry meaning that most engineers under-use. GET must be safe (no side effects) and idempotent (repeating it produces the same result). PUT replaces a resource entirely and is idempotent - sending the same PUT twice should leave the system in the same state as sending it once. PATCH applies a partial update and is not guaranteed idempotent unless you design it that way. POST creates a new resource or triggers a non-idempotent action, and DELETE removes a resource and should be idempotent (deleting something that's already gone should not be treated as a hard failure). Violating these guarantees - for example, making a GET endpoint increment a counter as a side effect - breaks assumptions that browsers, proxies, CDNs, and retry logic in HTTP clients all rely on.

Status codes are the other half of self-descriptive messaging, and they are chronically misused. Returning 200 OK with an error payload embedded in the JSON body is one of the most common mistakes in production APIs, because it silently defeats every piece of infrastructure that inspects status codes - load balancer health checks, client-side retry logic, and monitoring dashboards all become blind to real failures. The status code families matter: 2xx for success (200 for a general success with a body, 201 for a successful creation with a Location header pointing at the new resource, 202 for accepted-but-processing-asynchronously, 204 for success with no body), 4xx for client errors (400 for malformed requests, 401 for missing or invalid authentication, 403 for authenticated-but-not-authorized, 404 for a resource that doesn't exist, 409 for a conflict like a duplicate unique key, 422 for semantically invalid data that is syntactically well-formed), and 5xx for server errors, which should always be treated as bugs or infrastructure failures, never as a substitute for validation errors.

Error response bodies deserve their own standard, and one already exists: RFC 9457 (which obsoletes the earlier RFC 7807), the "Problem Details for HTTP APIs" specification. It defines a simple JSON structure - type, title, status, detail, and instance - that gives clients a consistent, machine-parseable error format instead of every team inventing its own {error: "..."} shape. Adopting a standard error format early avoids the very common situation where a client-side team has to write six different error-parsing branches for six internal services that all format failures differently.

// A Problem Details (RFC 9457) compliant error response builder
interface ProblemDetails {
  type: string;      // URI identifying the error category
  title: string;      // short, human-readable summary
  status: number;      // HTTP status code, duplicated for convenience
  detail?: string;     // human-readable explanation specific to this occurrence
  instance?: string;    // URI identifying this specific occurrence
  errors?: Record<string, string[]>; // field-level validation errors
}

function validationProblem(fieldErrors: Record<string, string[]>): ProblemDetails {
  return {
    type: "https://api.example.com/errors/validation-failed",
    title: "One or more fields failed validation",
    status: 422,
    detail: "The request body contains invalid values for one or more fields.",
    errors: fieldErrors,
  };
}

// Express-style error handler
app.use((err: AppError, req: Request, res: Response, next: NextFunction) => {
  if (err instanceof ValidationError) {
    return res
      .status(422)
      .type("application/problem+json")
      .json(validationProblem(err.fieldErrors));
  }
  // fall through to generic 500 handling
  return res.status(500).type("application/problem+json").json({
    type: "https://api.example.com/errors/internal",
    title: "Internal Server Error",
    status: 500,
  });
});

Practical Patterns: Versioning, Pagination, and Filtering

Versioning is one of the first decisions an API design has to make, and it is one that's genuinely hard to change later. The three common approaches are URI versioning (/v1/orders), header versioning (Accept: application/vnd.example.v1+json), and query-parameter versioning (/orders?version=1). URI versioning is by far the most widely adopted in practice - it's visible in logs, cacheable without special configuration, and trivial for consumers to understand - even though purists argue it violates the idea that a URI should identify a single, stable resource rather than a resource-plus-version. Header versioning is more "correct" in a strict REST sense but adds friction for API consumers and makes manual testing (curling an endpoint, pasting a URL into a browser) noticeably harder. Most teams should default to URI versioning at the major-version level and use additive, backward-compatible changes (new optional fields, new endpoints) for everything that doesn't require a breaking change, reserving a version bump for actual breaking changes to the contract.

Pagination and filtering are where API design meets database reality. Offset-based pagination (?limit=20&offset=40) is simple to implement and simple for clients to reason about, but it degrades badly at scale: a large offset forces the database to scan and discard rows, and pagination correctness breaks if rows are inserted or deleted between requests, causing clients to see duplicates or skip items. Cursor-based (keyset) pagination solves both problems by encoding the last-seen sort key into an opaque token, so the query becomes WHERE (created_at, id) > (cursor_created_at, cursor_id) ORDER BY created_at, id LIMIT 20 - a query that uses an index efficiently regardless of how deep into the collection the client is paging. The trade-off is that cursor pagination doesn't support jumping to an arbitrary page number, which is usually an acceptable trade for any collection large enough for the performance difference to matter.

# Cursor-based pagination using a base64-encoded composite key
import base64
import json
from dataclasses import dataclass

@dataclass
class Cursor:
    created_at: str
    id: str

def encode_cursor(cursor: Cursor) -> str:
    payload = json.dumps({"created_at": cursor.created_at, "id": cursor.id})
    return base64.urlsafe_b64encode(payload.encode()).decode()

def decode_cursor(token: str) -> Cursor:
    payload = json.loads(base64.urlsafe_b64decode(token.encode()))
    return Cursor(created_at=payload["created_at"], id=payload["id"])

def fetch_page(db, cursor_token: str | None, limit: int = 20):
    if cursor_token:
        cursor = decode_cursor(cursor_token)
        query = """
            SELECT * FROM orders
            WHERE (created_at, id) > (%s, %s)
            ORDER BY created_at, id
            LIMIT %s
        """
        rows = db.execute(query, (cursor.created_at, cursor.id, limit + 1))
    else:
        query = "SELECT * FROM orders ORDER BY created_at, id LIMIT %s"
        rows = db.execute(query, (limit + 1,))

    has_more = len(rows) > limit
    page = rows[:limit]
    next_cursor = (
        encode_cursor(Cursor(page[-1]["created_at"], page[-1]["id"]))
        if has_more else None
    )
    return {"data": page, "next_cursor": next_cursor}

Filtering and sorting should be expressed through query parameters that map onto the resource's fields - /orders?status=shipped&sort=-created_at - rather than through separate endpoints per filter combination. A common extension worth adopting selectively is a sparse fieldset or field-selection parameter (?fields=id,status,total), which lets high-volume clients reduce payload size without needing a bespoke endpoint, though this adds implementation complexity and is best reserved for APIs with a demonstrated bandwidth or mobile-client need.

Advanced Patterns: Idempotency, HATEOAS, and Rate Limiting

Idempotency matters most at the boundary where networks are unreliable, which is precisely where retries happen automatically. If a client sends a POST /payments request and the network times out before the response arrives, the client cannot tell whether the payment was created or not - and blindly retrying risks charging the customer twice. The standard solution is an idempotency key: the client generates a unique token (typically a UUID) and sends it in an Idempotency-Key header; the server stores the key alongside the result of the first successful request and, if it sees the same key again within a defined window, returns the cached result instead of re-executing the operation. Stripe popularized this pattern for its payments API, and it has since become a de facto standard for any POST or PATCH endpoint that has real-world side effects money, inventory, or irreversible state changes.

HATEOAS - including hypermedia links in responses so clients can discover available actions rather than hardcoding URL templates - is the most debated of REST's constraints, and it's worth being honest about why most production APIs skip it. In theory, a HATEOAS-compliant response to GET /orders/482 includes links like {"rel": "cancel", "href": "/orders/482/cancel", "method": "POST"}, and clients navigate the API the way a browser navigates the web, following links instead of constructing URLs from documentation. In practice, most API consumers are typed SDKs or mobile apps that already hardcode endpoint structures at compile time, so the discoverability benefit is smaller than the added payload size and implementation complexity. HATEOAS earns its keep in specific situations - public APIs with many independent third-party integrators, or systems where available actions genuinely change based on resource state (an order that's already shipped shouldn't offer a "cancel" link) - and the JSON:API and HAL specifications both provide standardized ways to structure these links if you decide it's worth adopting.

Rate limiting protects the API from both abuse and its own capacity limits, and it needs to be visible to clients, not just enforced silently. The standard approach communicates limits through response headers - X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset (or the newer standardized RateLimit header from IETF's draft specification) - and returns 429 Too Many Requests with a Retry-After header when the limit is exceeded, giving well-behaved clients enough information to back off automatically rather than hammering the API in a retry loop. Token bucket and sliding-window-log are the two algorithms worth knowing: token bucket allows controlled bursts up to a capacity while enforcing an average rate, and sliding-window approaches trade a bit more memory for smoother, more accurate limiting than fixed windows, which suffer from a boundary problem where a client can send double the intended rate by clustering requests around a window edge.

Trade-offs and Common Pitfalls

The most expensive mistake in REST API design is almost never a technical one - it's designing the API around the current database schema instead of around the actual client use cases. An API that exposes /user_profile_table fields directly couples every consumer to your internal storage decisions, and the day you need to split that table, add a caching layer, or migrate to a different database, you either break every client or maintain an increasingly awkward translation layer indefinitely. The fix is to design the resource representation as its own contract, decoupled from storage, even when the initial implementation is a thin pass-through - because the representation is what you can't easily change later, while the storage underneath is what you can.

Over-fetching and under-fetching are the classic trade-off that pushed many teams toward GraphQL, and it's worth understanding both sides honestly rather than treating REST as strictly inferior. A REST endpoint like /orders/482 typically returns a fixed shape, so a mobile client that only needs the order status ends up downloading customer details, line items, and shipping information it doesn't need (over-fetching), while a client that needs an order plus its customer's name has to make two round trips (under-fetching, solved by ad hoc ?include=customer parameters or nested resource expansion). GraphQL solves this cleanly by letting the client specify exactly the fields it wants in a single request, at the cost of losing HTTP-level caching (a single POST /graphql endpoint can't be cached by a CDN the way GET /orders/482 can) and taking on more complexity in query cost analysis to prevent expensive nested queries from overwhelming the backend. Choosing between REST and GraphQL - or supporting both for different consumer types - is a legitimate architectural decision, not a matter of one being universally correct.

Breaking changes are the pitfall that costs the most in trust rather than engineering time. Removing a field, changing a field's type, or changing the meaning of a status code without a version bump will silently break clients that made reasonable assumptions about a "stable" contract. The discipline that prevents this is treating the API contract the same way you'd treat a database migration: additive changes are safe and can ship continuously, but anything that changes existing behavior needs a new version, a deprecation window with clear communication (often via a Sunset HTTP header, standardized in RFC 8594, or a Deprecation header), and enough lead time for consumers to migrate before the old version is actually removed.

Best Practices for Production-Grade REST APIs

Consistency across an API surface compounds in value as the API grows, because consumers build tooling and mental models based on early endpoints and expect the rest of the API to behave the same way. This means a single, enforced convention for naming (snake_case or camelCase, but not both), a single error format used everywhere, and a single approach to timestamps (ISO 8601 in UTC, consistently) and null handling (deciding once whether absent fields are omitted or returned as null, rather than letting each endpoint decide independently). Enforcing this consistency is easier with contract-first design: writing an OpenAPI (formerly Swagger) specification before implementation, then generating server stubs and client SDKs from it, catches inconsistencies at design time rather than after three teams have integrated against slightly different behaviors.

Documentation and discoverability should be treated as part of the API's actual surface area, not an afterthought written after the code ships. An OpenAPI spec that's kept in sync with the implementation - ideally enforced through contract testing in CI, so a code change that breaks the spec fails the build - does double duty as both human-readable documentation and machine-readable input for client SDK generation, mock servers, and automated request validation. Authentication and authorization deserve the same rigor: OAuth 2.0 (RFC 6749) with short-lived bearer tokens is the standard approach for most APIs with external consumers, and authorization decisions (which fields a given client is allowed to see or modify) should be enforced at the API layer rather than trusted to be handled correctly by every downstream service that consumes the data.

Observability closes the loop on all of this. Structured logging that includes a request ID propagated through every downstream call, combined with metrics on latency percentiles (p50/p95/p99, not just averages, since averages hide the tail latency that actually affects user experience) and error rates broken down by endpoint and status code, turns an API from a black box into something a team can actually operate. This matters more than it might seem during design: an API that's easy to design but impossible to debug in production will accumulate operational cost far exceeding whatever time was saved skipping structured logging on day one.

Key Takeaways

The 80/20 of REST API Design

If you strip away every advanced pattern, three decisions produce most of the long-term value:

Idempotency keys, HATEOAS, and sophisticated rate-limiting algorithms matter, but they matter much less if the foundational resource model is wrong - no amount of hypermedia linking fixes an API whose URLs encode RPC-style verbs, and no idempotency key strategy fixes an API whose error responses are inconsistent from one endpoint to the next.

Conclusion

REST's durability as an API style isn't an accident of history - it comes from leaning on constraints (statelessness, a uniform interface, cacheability) that map directly onto properties distributed systems actually need: horizontal scalability, infrastructure compatibility, and a decoupling between client and server that lets both evolve independently. The gap between a REST API that works in a demo and one that survives years of production use is almost entirely in the details this article covered - consistent resource modeling, correct HTTP semantics, a deliberate pagination and versioning strategy, and idempotency at the boundaries where retries happen automatically.

None of these patterns are exotic, and none require abandoning REST for a different paradigm to get right. What they require is treating API design as a first-class system design problem with real trade-offs, rather than a thin serialization layer bolted on after the business logic is written. An API designed with that discipline from the start tends to need far fewer breaking changes, and the teams consuming it spend their time building features instead of working around inconsistencies - which is, ultimately, the entire point of having a stable contract in the first place.

References

A quick flashcard preview - one example of each question type in this post. This is a demo of what you'll find in the full quiz app.

Flashcard preview · 1 / 9Multiple Choice

multiple choice - advanced - auto-graded

A team is building a POST /payments endpoint where the client's network is known to be unreliable and automatically retries timed-out requests. What mechanism should be added to prevent a customer from being charged twice?

Choose an answer

Resources