Introduction
Every generation of software architecture has a moment where an ad-hoc integration pattern becomes formalized into a protocol. REST formalized how web clients talk to servers. gRPC formalized how microservices talk to each other. In the current wave of AI engineering, the Model Context Protocol (MCP) is doing the same job for how large language models talk to tools, data sources, and external systems. Instead of every team writing bespoke glue code between an LLM and a database, a filesystem, or a SaaS API, MCP defines a shared contract: a server exposes capabilities in a standard shape, and any compliant client - a chat app, an IDE, an autonomous agent - can consume them without custom integration work.
This article is about the part of MCP that gets the least attention in tutorials: system design. Writing a "hello world" MCP server that returns a static string takes fifteen minutes. Designing one that survives concurrent sessions, untrusted input, slow upstream APIs, and a model that occasionally calls your tool with malformed arguments is a genuine engineering problem. We will walk through the protocol's architecture, the concrete design decisions you face when building a server, working code in TypeScript and Python, and the trade-offs that separate a demo from a production-grade integration.
The Problem: Why LLM Tool Integration Needed a Standard
Before MCP, connecting an LLM to external capabilities meant writing a custom function-calling schema for every tool, for every model provider, for every application. A team building a coding assistant would hand-roll a "read file" function, a "run tests" function, and a "search codebase" function, each with its own JSON schema, its own error handling, and its own way of feeding results back into the model's context window. If that same team wanted to reuse those integrations in a second application - say, a Slack bot - they had to either duplicate the code or extract it into a private library that only they could use.
This is fundamentally the NM integration problem that distributed systems have solved many times before. With N different AI applications and M different tools or data sources, you end up writing NM integrations unless you introduce a common interface. USB solved this for peripherals. ODBC solved it for databases. MCP, released by Anthropic in November 2024 as an open standard, aims to solve it for AI-to-tool connectivity by defining a single client-server protocol that any application can implement once and any tool provider can expose once.
The practical consequence for engineering teams is that "system design for an MCP server" is really "system design for a general-purpose capability provider that an unpredictable, non-deterministic client will call." That reframing matters. A REST API is called by code you wrote or code your partner team wrote, following a contract you both understand. An MCP server is called by a language model whose exact request pattern you cannot fully predict, which changes how you think about validation, idempotency, and failure isolation from the very first design decision.
MCP Architecture Deep Dive
MCP follows a client-host-server architecture built on JSON-RPC 2.0. The host is the application the user interacts with - a chat interface, an IDE, an agent runtime. The host embeds one or more clients, each of which maintains a dedicated, stateful connection to a single MCP server. The server is the piece you, as the integration author, actually build. It exposes capabilities and, critically, has no visibility into the model itself; it only sees requests from the client and returns structured responses. This separation is what makes MCP composable: your server doesn't need to know whether it's talking to Claude, GPT, or a local Llama model, because the protocol layer abstracts that away.
Communication happens over one of two standard transports. Stdio is used when the server runs as a local subprocess of the host - common for developer tools like IDE integrations, where the client spawns the server process and communicates over standard input and output. Streamable HTTP (which superseded the earlier HTTP+SSE transport in the 2025 protocol revisions) is used when the server runs remotely, allowing multiple clients to connect over the network with support for resumable, long-lived sessions. Choosing between these two isn't a stylistic preference - it's a system design decision with real consequences for how you handle authentication, scaling, and process lifecycle, which we'll return to below.
On top of the transport, MCP defines three core primitives that a server can expose. Tools are model-invocable functions - the equivalent of function calling, but standardized, with JSON Schema describing inputs and outputs. Resources are addressable, read-only pieces of context, like a file, a database row, or a log excerpt, that the host application can attach to a conversation. Prompts are reusable, parameterized templates that help users or hosts invoke your server's capabilities consistently. A well-designed server rarely uses all three with equal weight; most production servers lean heavily on tools, use resources for read-heavy context injection, and use prompts sparingly for guided workflows.
There's a fourth, less-discussed piece of the protocol that matters enormously for system design: capability negotiation. During the initialization handshake, the client and server exchange a declared set of supported features - whether the server supports sampling (asking the client's LLM to generate a completion on the server's behalf), whether it supports progress notifications for long-running operations, and what protocol version it speaks. Designing your server to declare only what it can actually deliver, and to degrade gracefully when the client doesn't support something you'd prefer to use, is a discipline directly inherited from decades of network protocol design - think of it as the TLS cipher-suite negotiation of the AI tooling world.
Mental Model: MCP as the "USB-C for AI"
The analogy Anthropic itself uses - MCP as a USB-C port for AI applications - is worth taking seriously rather than treating as marketing shorthand, because it maps cleanly onto real design constraints. A USB-C port doesn't care whether you plug in a monitor, a hard drive, or a charger; it defines a physical and electrical contract, and the device on the other end declares what it is and negotiates capabilities accordingly. An MCP server should be designed the same way: it declares its tools, resources, and prompts at connection time, and it should work correctly regardless of which specific host or model is on the other end of the wire.
Where the analogy earns its keep is in thinking about failure modes. A USB-C cable that's wired incorrectly can, in the worst case, damage the device on the other end - which is why the specification includes strict power and signaling limits. An MCP server that fails to validate input, leaks stack traces, or executes arbitrary commands based on model output can do the equivalent damage to the host environment. Keeping this mental model active while designing your server - "what happens if the thing on the other end of this port sends garbage" - pushes you toward defensive design earlier than most engineers instinctively go, precisely because the client here is a probabilistic model, not a deterministic caller you can fully trust to behave.
Designing an MCP Server: Core System Design Decisions
Once you move past the protocol basics, the actual system design work centers on a handful of decisions that determine whether your server holds up under real usage. The first is statefulness. MCP sessions are inherently stateful - a client initializes a session, negotiates capabilities, and maintains that session across multiple tool calls. You need to decide where that state lives: in-process memory (simplest, but breaks under multi-instance deployment), an external store like Redis (adds latency but enables horizontal scaling), or no persistent state at all, with every tool call treated as fully independent and any needed context re-fetched from your backend. For remote servers deployed behind a load balancer, sticky sessions or an external session store are close to mandatory, because a client's second request in a session may not land on the same server instance as the first.
The second decision is tool granularity. This is the AI-engineering equivalent of choosing microservice boundaries, and it's easy to get wrong in both directions. Exposing one giant execute_database_query tool that accepts raw SQL gives the model too much surface area and turns every call into a security review. Exposing forty narrow tools like get_user_by_id, get_user_by_email, get_user_by_username creates a bloated tool list that degrades the model's ability to pick the right one - most models see real accuracy drops once a single server exposes more than roughly 15-20 tools in a session, simply because tool selection itself becomes a harder discrimination problem. The practical sweet spot is a small number of purposeful, composable tools - search_users(query, filters) rather than one-field-per-tool - each with a tightly scoped, well-documented input schema.
The third decision is error surface design. MCP tool responses can be flagged as errors (isError: true) versus protocol-level failures, and the distinction matters more than it looks. A protocol-level error means something is wrong with the request itself - malformed JSON, an unknown tool name - and should be rare and treated as a bug. A tool-level error, by contrast, is a normal, expected outcome: the file didn't exist, the API rate-limited you, the query returned zero rows. Feeding tool-level errors back to the model as structured, readable error content (rather than throwing a hard protocol exception) lets the model reason about the failure and retry or adjust its approach, which is often exactly the behavior you want from an agentic system.
Implementation Walkthrough: Building an MCP Server
The official TypeScript and Python SDKs (@modelcontextprotocol/sdk and mcp, respectively) handle the JSON-RPC plumbing, so the engineering work is really about how you structure tool handlers, validation, and backend calls around that scaffolding. Below is a realistic TypeScript example for a server exposing a scoped search tool over an internal knowledge base, including input validation with Zod and a timeout guard around the backend call - the kind of defensive wrapping that a tutorial skips but production requires.
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";
const server = new McpServer({
name: "knowledge-base-server",
version: "1.2.0",
});
const SearchInput = z.object({
query: z.string().min(1).max(500),
maxResults: z.number().int().min(1).max(20).default(5),
});
async function withTimeout<T>(promise: Promise<T>, ms: number): Promise<T> {
return Promise.race([
promise,
new Promise<T>((_, reject) =>
setTimeout(() => reject(new Error(`Operation timed out after ${ms}ms`)), ms)
),
]);
}
server.registerTool(
"search_knowledge_base",
{
title: "Search Knowledge Base",
description:
"Searches the internal knowledge base for articles matching a query. " +
"Returns titles, excerpts, and stable document IDs for follow-up retrieval.",
inputSchema: SearchInput,
},
async ({ query, maxResults }) => {
try {
const results = await withTimeout(
knowledgeBaseClient.search(query, { limit: maxResults }),
5000
);
if (results.length === 0) {
return {
content: [{ type: "text", text: `No articles found for "${query}".` }],
};
}
const summary = results
.map((r, i) => `${i + 1}. [${r.id}] ${r.title} - ${r.excerpt}`)
.join("\n");
return { content: [{ type: "text", text: summary }] };
} catch (err) {
// Tool-level error: surface it to the model as structured content,
// not as an uncaught exception that breaks the session.
return {
isError: true,
content: [
{
type: "text",
text: `Search failed: ${err instanceof Error ? err.message : "unknown error"}. Consider retrying with a narrower query.`,
},
],
};
}
}
);
const transport = new StdioServerTransport();
await server.connect(transport);
Notice what this example is actually doing at a system design level: it bounds input size (max(500)), bounds output volume (max(20)), enforces a hard timeout so one slow backend call can't hang the session indefinitely, and translates backend failures into model-readable text rather than letting exceptions propagate. None of this is protocol-specific - it's ordinary defensive back-end engineering - but it's exactly the layer that separates servers that survive contact with a real agent loop from ones that don't.
For remote deployments, the equivalent Python example using Streamable HTTP shows how session state and authentication typically get threaded through:
from mcp.server.fastmcp import FastMCP
from mcp.server.auth.provider import TokenVerifier
import httpx
mcp = FastMCP("orders-server", stateless_http=False)
@mcp.tool()
async def get_order_status(order_id: str, ctx) -> str:
"""Fetch the current status of an order by its ID.
Requires order_id to match pattern ORD-XXXXXX.
"""
if not order_id.startswith("ORD-") or len(order_id) != 10:
return f"Invalid order_id format: {order_id!r}. Expected pattern ORD-XXXXXX."
# Session-scoped auth token, verified during the transport-level handshake,
# is available on the request context rather than trusted from tool args.
tenant_id = ctx.session.auth_context.tenant_id
async with httpx.AsyncClient(timeout=4.0) as client:
try:
resp = await client.get(
f"https://internal-orders.api/v1/orders/{order_id}",
headers={"X-Tenant-Id": tenant_id},
)
resp.raise_for_status()
except httpx.HTTPError as exc:
return f"Unable to retrieve order {order_id}: {exc}"
order = resp.json()
return f"Order {order_id} is currently '{order['status']}', last updated {order['updated_at']}."
The detail worth pausing on here is that tenant_id is pulled from the authenticated session context, not from a tool argument the model supplies. If tenancy were passed as a plain string parameter, a model - or an attacker crafting a prompt injection - could simply ask the tool to fetch a different tenant's orders, and the server would happily comply. Binding authorization to the transport-level session rather than to model-supplied arguments is one of the single highest-leverage security decisions in MCP server design.
Trade-offs and Pitfalls
The most common failure mode in early MCP servers is over-trusting the model as a caller. Because tool definitions look like ordinary function signatures, teams sometimes treat the model's tool call the way they'd treat a call from their own frontend code - assuming the arguments are well-formed and the intent is benign. In practice, tool arguments arrive after passing through a non-deterministic reasoning process that can be influenced by the content the model has read, including content pulled in from a resource earlier in the same session. A resource that contains adversarial instructions - a classic prompt injection vector - can influence which tool the model calls next and with what arguments. Every MCP server should be designed with the same skepticism toward its inputs that a public-facing REST API applies, including strict schema validation, allowlisting rather than blocklisting for anything resembling file paths or shell commands, and never granting a tool more backend privilege than the specific operation requires.
The second recurring pitfall is transport mismatch with deployment reality. Teams often prototype with stdio because it's simpler to test locally, then discover that a single-process, single-session assumption baked into their tool handlers doesn't hold once they move to Streamable HTTP with multiple concurrent clients. Global mutable state - an in-memory cache keyed by nothing, a module-level counter, a singleton database connection reused across sessions - works fine in a one-client stdio setup and breaks silently, or worse, leaks data across users, once the same code runs behind a shared HTTP server. Deciding your target transport and concurrency model before writing tool logic, rather than after, avoids a costly rewrite.
Best Practices for Production MCP Servers
Good MCP server design borrows heavily from API design discipline that already exists for REST and RPC systems, applied with the specific quirks of a non-deterministic caller in mind. Start with tool descriptions: the JSON Schema description field is not documentation for humans, it's the primary signal the model uses to decide when and how to call your tool, so vague descriptions like "gets data" produce unreliable model behavior in ways that are hard to debug after the fact. Write descriptions the way you'd write a docstring for a junior engineer who has never seen your codebase - state what the tool does, what its parameters mean, what it returns, and what error conditions look like.
Second, treat idempotency and side effects explicitly. Because a model may retry a tool call after a timeout or an ambiguous response, tools with side effects - creating a record, sending a message, charging a payment - should either be naturally idempotent (accept a client-generated idempotency key) or clearly warn in their description that they are not safe to retry. Anthropic's own guidance on building effective agents echoes a broader principle from distributed systems design: keep the "blast radius" of any single automated action small, and prefer read-heavy, reversible operations over irreversible ones wherever the task allows it.
Third, instrument before you need to. Because tool calls are initiated by a model rather than by code you control, debugging a misbehaving integration in production without structured logs of every tool invocation, its arguments, its latency, and its outcome is close to impossible after the fact. Log at the tool boundary - not just errors, but successful calls too - since the pattern of which tools get called in what sequence is often the most useful diagnostic signal when an agent behaves unexpectedly, more useful than any single call's payload in isolation.
The 80/20 Insight
If you strip away everything else, three decisions account for most of the difference between an MCP server that works in a demo and one that survives production: input validation strength, authorization binding, and tool granularity. Get schema validation and authorization-from-session-context right, and you eliminate the majority of security incidents before they're possible. Get tool granularity right - a small set of well-described, composable tools rather than either a single overpowered tool or dozens of narrow ones - and you eliminate most of the model-behavior unreliability that otherwise gets blamed on "the model being bad at tool use." Everything else in this article - transport choice, session storage, observability - matters, but these three decisions are the ones worth spending disproportionate design time on before writing a single line of handler code.
Key Takeaways
- Design for an untrusted, non-deterministic caller. Validate every tool input as rigorously as you would a public API endpoint, regardless of how "internal" the server feels.
- Bind authorization to the session, not to tool arguments. Never let a model-supplied parameter determine tenant, user, or permission scope.
- Keep tool counts and tool scope tight. Aim for a small number of composable, clearly described tools rather than one giant tool or dozens of narrow ones.
- Match your concurrency model to your transport before writing handlers. Stdio assumptions about single-session, single-process state do not survive a move to shared Streamable HTTP deployments.
- Log every tool call, not just failures. The sequence of tool invocations is often your best debugging signal when an agent's behavior goes wrong.
Conclusion
MCP is a genuinely useful piece of standardization for an industry that was drowning in bespoke integration code, but the protocol itself only solves the "how do a client and server talk" problem. It doesn't solve the "how do you build a server that's secure, scalable, and debuggable" problem - that's ordinary systems engineering, applied to a new and slightly unusual kind of caller. The teams that get the most value out of MCP are the ones that resist treating it as a thin wrapper around existing functions and instead treat it the way they'd treat any other network-facing service boundary: with input validation, explicit authorization, sane concurrency assumptions, and observability built in from day one.
The protocol will keep evolving - capability negotiation, sampling, and the transport layer have all changed meaningfully since MCP's initial release, and they will likely change again. What won't change is that the fundamentals of good system design - least privilege, defensive input handling, clear service boundaries, and instrumentation - are exactly as valuable at the AI-tooling layer as they've always been everywhere else in distributed systems. MCP just gives you a standard place to apply them.
References
- Anthropic. "Introducing the Model Context Protocol." Anthropic News, November 2024. https://www.anthropic.com/news/model-context-protocol
- Model Context Protocol Specification (current version). https://modelcontextprotocol.io/specification
- Model Context Protocol - Official Documentation. https://modelcontextprotocol.io/
- Model Context Protocol GitHub Organization (specification, TypeScript SDK, Python SDK, servers). https://github.com/modelcontextprotocol
- Anthropic. "Building Effective Agents." Anthropic Engineering, December 2024. https://www.anthropic.com/engineering/building-effective-agents
- JSON-RPC 2.0 Specification. https://www.jsonrpc.org/specification
- Anthropic. "Model Context Protocol Authorization Specification." https://modelcontextprotocol.io/specification/basic/authorization
- Anthropic Model Context Protocol TypeScript SDK Documentation. https://github.com/modelcontextprotocol/typescript-sdk
- Anthropic Model Context Protocol Python SDK Documentation. https://github.com/modelcontextprotocol/python-sdk
- OWASP. "OWASP Top 10 for Large Language Model Applications" (for prompt injection and tool-use security context). https://owasp.org/www-project-top-10-for-large-language-model-applications/