paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

July 26, 2026

MCP Servers in AI Engineering: Patterns, Practices, and the Pitfalls Nobody Warns You About

A practical field guide to building, securing, and scaling Model Context Protocol servers for production AI agents

Introduction

Every few years, distributed systems engineering produces a protocol that quietly becomes the plumbing everyone depends on without thinking much about it - HTTP, gRPC, LSP. The Model Context Protocol, introduced by Anthropic in November 2024, is on that trajectory for AI agents. MCP defines a standard way for LLM applications to connect to external tools, data sources, and prompt templates, replacing a landscape where every AI product wrote its own bespoke integration glue for every database, SaaS API, or internal system it wanted an agent to touch. Instead of N applications each writing M custom integrations, MCP lets you write M servers once and connect them to any compliant client.

The protocol's growth has been fast by any standard. Anthropic's own specification blog reports that across the Tier 1 SDKs, monthly downloads are approaching half a billion, with both the TypeScript and Python SDKs having crossed a billion total downloads. That kind of adoption curve means MCP has moved well past "interesting new abstraction" into "thing running in production behind your company's agents right now." This article is a practical field guide for engineers who are past the tutorial stage and need to make real decisions: how to structure a server, which primitives to reach for, where teams consistently get hurt, and what separates a resilient MCP deployment from a brittle one.

What MCP Solves and Why It Exists

Before MCP, connecting an LLM to a tool meant writing a custom function-calling schema, a custom execution harness, and a custom set of prompts describing what the tool does - all specific to one model provider's function-calling format and one application's plumbing. If you wanted the same tool available in a chat app, an IDE, and a CLI agent, you rewrote the integration three times, and every change to the underlying API meant three parallel updates drifting out of sync. This is the classic N*M integration problem, and it is exactly the failure mode that standardized protocols exist to solve.

MCP addresses this by separating three roles cleanly: the host (the AI application the user interacts with, such as an IDE or chat client), the client (the component inside the host that speaks MCP and manages the connection to one or more servers), and the server (the process that exposes tools, resources, and prompts over the protocol). A server implementer never needs to know which host will eventually consume their integration. A host implementer never needs bespoke code per tool provider. The protocol itself carries JSON-RPC 2.0 messages over a small number of transports - stdio for local subprocesses, and Streamable HTTP for remote servers, the latter having replaced the earlier HTTP+SSE transport in the 2025-03-26 revision of the spec.

What makes MCP distinct from a generic RPC framework is that it is designed specifically for the LLM-in-the-loop case. The server doesn't just expose functions; it exposes descriptions of those functions, written in natural language and JSON Schema, that the model reads directly as part of its context before deciding whether and how to call them. This is a subtle but important shift in what "the interface" means. In a traditional API, the interface is a contract read by a human developer once, at integration time. In MCP, the interface is read by the model on every relevant turn, which means the tool description is not documentation - it is executable context that steers model behavior. That reframing is the root of both MCP's power and, as later sections show, most of its security problems.

Architecture and Protocol Mechanics

MCP servers expose functionality through a small, fixed set of primitives, and understanding what each one is for - rather than treating them as interchangeable ways to hand the model information - is the difference between a clean server and a confusing one. Tools are callable functions the model can invoke, each described with a name, a natural-language description, and a JSON Schema for its input. Since the 2025-06-18 revision, tools can also declare structured output and return resource links rather than only flat text. Resources are readable, addressable context - files, database records, or API responses - identified by a URI, and are meant for content the host or user browses and attaches, not necessarily for the model to invoke on its own initiative. Prompts are reusable, parameterized templates the server offers up, useful for standardizing recurring workflows like "summarize this ticket" or "generate a changelog entry" across every client that connects.

The protocol also defines capabilities that flow the other direction, from server to client, which is a detail many first-time implementers miss because it inverts the usual mental model of "server exposes, client consumes." Sampling lets a server ask the client's own model to generate text on the server's behalf, useful when a server needs LLM reasoning as part of fulfilling a request without embedding its own API key and model dependency. Elicitation, added in the 2025-06-18 revision, lets a server pause mid-operation and ask the human user for additional input - a clean way to handle cases like "which of these three matching records did you mean?" without the server guessing. Roots let a client tell a server which directories or URIs it's permitted to operate on, which is the closest thing MCP has to a built-in sandboxing hint, though it is advisory rather than enforced.

The most consequential recent development is the 2026-07-28 specification revision, finalized on that date after a release-candidate period that began in May 2026. According to the MCP maintainers' own release notes, this is described as the largest revision to the protocol since its original launch, and it is not fully backward compatible with earlier versions. The headline change is a move to a stateless protocol core: earlier revisions tracked sessions with an Mcp-Session-Id header, which meant a client's requests during a session had to be routed to the same server instance, complicating horizontal scaling behind a load balancer. The 2026-07-28 revision removes protocol-level sessions entirely, so any request can be served by any instance behind ordinary HTTP infrastructure, which is the same trade normal web services made years ago when they moved from sticky sessions to stateless request handling backed by external session stores.

Alongside statelessness, the same revision introduces a formal Extensions framework, letting capabilities like MCP Apps (server-rendered UI components a host can display) and Tasks (a primitive for long-running, asynchronous work that doesn't fit a single request-response cycle) ship on independent timelines without forcing changes to the protocol core. It also adds header-based routing, cacheable list results for tool and resource enumeration, tightened authorization aligned more closely with OAuth 2.1 and OpenID Connect patterns, and a formal Feature Lifecycle and Deprecation Policy with defined Active, Deprecated, and Removed states and a twelve-month minimum deprecation window - the kind of governance artifact that shows up once a protocol has enough enterprise adopters that breaking changes have real cost.

Implementation Patterns in Practice

The cleanest way to internalize MCP's shape is to build a small server and walk through what each piece is actually doing. The TypeScript SDK is the most common starting point for new servers, and a minimal but realistic tool-exposing server looks like this:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

const server = new McpServer({
  name: "inventory-server",
  version: "1.0.0",
});

server.registerTool(
  "lookup_inventory_item",
  {
    title: "Look up an inventory item by SKU",
    description:
      "Returns current stock level, warehouse location, and reorder " +
      "threshold for a given SKU. Use this before answering any question " +
      "about product availability.",
    inputSchema: {
      sku: z.string().min(3).max(32).describe("The product SKU, e.g. WH-1032"),
    },
  },
  async ({ sku }) => {
    const record = await inventoryDb.findBySku(sku);
    if (!record) {
      return {
        content: [{ type: "text", text: `No inventory record found for SKU ${sku}.` }],
        isError: true,
      };
    }
    return {
      content: [
        {
          type: "text",
          text: `SKU ${sku}: ${record.quantity} units at ${record.warehouse}. ` +
                `Reorder threshold: ${record.reorderAt}.`,
        },
      ],
      structuredContent: record,
    };
  }
);

const transport = new StdioServerTransport();
await server.connect(transport);

Notice that the tool description above does real work: it tells the model when to use the tool ("before answering any question about product availability"), not just what the tool technically does. This is a pattern worth internalizing early - tool descriptions are prompts, and should be engineered with the same care as any other prompt, including testing how the model behaves when multiple similarly-described tools are available simultaneously. A vague description like "gets inventory data" invites the model to guess, or to skip the tool and hallucinate an answer instead.

On the client side, a Python agent consuming this server typically looks like the following, using the official Python SDK's client session abstraction:

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def run_agent(user_query: str) -> str:
    server_params = StdioServerParameters(
        command="node",
        args=["./inventory-server/dist/index.js"],
    )
    async with stdio_client(server_params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()

            tools = await session.list_tools()
            # Tool schemas are passed to the model's function-calling
            # interface here; omitted for brevity.

            result = await session.call_tool(
                "lookup_inventory_item", {"sku": "WH-1032"}
            )
            return result.content[0].text

if __name__ == "__main__":
    print(asyncio.run(run_agent("Is WH-1032 in stock?")))

A pattern that separates production-grade servers from demo-grade ones is treating tool granularity as a deliberate design decision rather than a one-to-one mirror of an underlying API. Wrapping every REST endpoint as its own MCP tool produces a server with forty near-identical tools and no clear guidance on which to use when, which bloats the model's context and increases the odds of a wrong tool call. The more durable pattern is to design tools around tasks a model will actually be asked to perform - "reconcile invoice against purchase order," not "get_invoice" and "get_purchase_order" as two separate calls the model has to sequence itself - and to let the server do the multi-step orchestration internally where that orchestration doesn't require model judgment at each step.

Anti-Patterns and Common Pitfalls

The single most common anti-pattern in early MCP adoption is what practitioners have started calling tool sprawl: connecting a client to many servers, each contributing many tools, until the model's context window is dominated by tool schemas before the user has said anything. Every tool definition consumes tokens on every single turn of the conversation, whether or not it's relevant. Beyond the direct cost, large tool sets measurably degrade tool-selection accuracy, because the model has to discriminate between more semantically similar options under the same context budget it needs for reasoning about the actual task. The fix is not "use fewer servers" reflexively - it's curating which servers and which specific tools are active for a given agent's job, and where the ecosystem supports it, using capability negotiation and tool filtering rather than connecting everything by default.

A closely related pitfall is what should be called prompt-as-documentation drift. Because tool descriptions double as both human documentation and model-facing prompts, teams that treat them as an afterthought - copy-pasted from an internal wiki, written once and never revisited - end up with descriptions that don't match current tool behavior after a few iterations. Unlike a stale README, a stale tool description doesn't just confuse a developer; it actively misleads the model into calling the tool at the wrong moment or with wrong assumptions about its side effects, and because the model doesn't complain the way a compiler would, this drift is often invisible until something breaks downstream in production.

Security-specific anti-patterns deserve their own weight because MCP's trust model is genuinely different from a typical API integration, and teams that port over API-era assumptions get burned. The industry has converged on naming two dominant attack classes here. Tool poisoning is when malicious instructions are embedded inside a tool's metadata - its name, description, or parameter schema - text that is largely invisible to the human approving the connection but fully visible to the model reading it as context on every call. Security researchers at Invariant Labs, who first documented this class of attack in 2025, showed that these hidden instructions can direct a model to exfiltrate sensitive data or take unauthorized actions without the user ever seeing the malicious text. A specific variant researchers call a rug pull compounds this: a hosted server can change its tool descriptions after a user has already approved the connection, since most clients only prompt for consent once per tool rather than re-verifying on every change, so a tool that looked benign at approval time can later be silently rewritten into something harmful.

The second major class is cross-server prompt injection propagation, which the OWASP MCP Security Cheat Sheet frames around a specific structural fact: a single client can connect to multiple MCP servers at once, and the model sees all of their tool descriptions and tool outputs in the same context. This means a successful injection delivered through one compromised or careless server - say, one that fetches and returns untrusted web content as a tool result - can influence the model's behavior toward every other tool it has access to, not just the one that was compromised. Research on indirect prompt injection in LLM-integrated applications, including foundational work by Greshake et al., documented exactly this propagation pattern: injected instructions in retrieved content causing downstream data theft and self-propagating behavior across a tool chain. Benchmarks on the InjecAgent dataset have reported indirect injection success rates against ReAct-style agents in the range of roughly a quarter of attempts under baseline conditions, rising substantially higher when the injected instructions are reinforced - numbers that make clear this is not a theoretical edge case but a measured, exploitable weakness in naive tool-chaining architectures.

Security-Focused Best Practices

Given that threat landscape, the most important architectural decision an MCP server author can make is to stop treating tool descriptions, tool outputs, and even session identifiers as inherently trustworthy just because they arrived over an authenticated connection. Microsoft's guidance on this point is blunt and worth internalizing directly: MCP clients must validate and sanitize all data received from servers before acting on it, regardless of the apparent trust level of the connection, because a valid, authenticated session does not imply a valid or safe request. In practice, this means running returned tool outputs through the same kind of input validation you'd apply to untrusted user input - checking for embedded instructions, unexpected formatting designed to look like system messages, or content that references actions outside the tool's stated scope - rather than piping server responses straight back into model context unchecked.

Consent design deserves the same rigor as the code itself. Per-tool, per-session approval prompts that a user clicks through once and never sees again are exactly the mechanism that makes rug-pull attacks viable, so production deployments should re-verify tool definitions against a known-good hash or signature on every connection, not just the first one, and should flag or block silently changed definitions rather than treating them as routine updates. Least-privilege scoping matters just as much on the server side as it does in traditional API security: a server that connects to a database should hold credentials scoped to exactly the tables and operations its tools need, not a superuser connection string reused because it was convenient during development - because if a tool poisoning or injection attack does succeed, the blast radius should be bounded by what that specific tool's credentials can reach, not by what the underlying service account can do.

Authorization itself has matured alongside the protocol. The 2026-07-28 specification's hardening work aligns MCP's authorization flows more closely with established OAuth 2.1 and OpenID Connect patterns rather than leaving servers to invent bespoke token schemes, and Enterprise-Managed Authorization has emerged as a pattern for centralizing token issuance and revocation across many servers rather than each one managing its own auth independently - the same lesson centralized identity providers taught the rest of the industry, now being relearned specifically for agentic tool access. Teams operating MCP servers in production should also budget for continuous adversarial testing rather than a one-time security review; the Cloud Security Alliance's agentic security guidance and firms like Mindgard have both emphasized red-teaming MCP tool metadata and outputs on an ongoing basis, precisely because tool poisoning and rug-pull attacks are designed to slip past a single point-in-time audit and only reveal themselves under continuous monitoring.

Trade-offs Engineers Should Weigh Deliberately

Adopting MCP is not a free upgrade over bespoke integrations, and pretending otherwise leads to disappointment down the line. The protocol buys you standardization and reusability across hosts, but it costs you a layer of indirection: debugging a failure now means reasoning about the host, the client, the transport, and the server as four potentially independent points of failure, instead of a single function call you can step through in a debugger. Teams migrating from ad hoc function-calling integrations to MCP servers should expect to invest in observability - structured logging of tool calls, their inputs, and their outputs - specifically because the extra indirection makes "what actually happened" much harder to reconstruct after the fact without it.

The stateless-versus-stateful trade-off introduced by the 2026-07-28 revision is worth naming explicitly, because it's a real trade, not a strict improvement. Stateless servers scale trivially behind commodity load balancers and simplify deployment, which matters enormously at the scale Anthropic's own adoption numbers suggest the ecosystem is now operating at. But statelessness pushes the responsibility for anything session-like - multi-step workflows, cached intermediate results, conversational context spanning several tool calls - either back onto the client or into an external store the server author now has to build and operate. Teams building servers that genuinely need durable multi-step state, rather than servers that were merely using sessions as a convenience, should plan for that externalized state layer deliberately rather than discovering the gap mid-migration.

Key Takeaways

A Mental Model for MCP

The most useful analogy for MCP is a restaurant kitchen with a shared supply chain rather than a single client-server API pair. The host is the dining room - where the customer actually sits and makes requests. The client is the waiter, who doesn't cook anything but knows which kitchens (servers) can fulfill which kinds of orders and relays requests and results between the customer and those kitchens. Each MCP server is an independent kitchen with its own menu (tools), pantry (resources), and house specials (prompts), and critically, the waiter can bring orders to several kitchens at once for a single meal. That last detail is exactly why cross-server contamination is a real risk: if one kitchen's ingredient labels are tampered with, the waiter - who trusts the labels rather than tasting everything - can carry that contamination into how the whole meal is assembled, even though the other kitchens did nothing wrong.

This model also clarifies why statelessness matters architecturally. A stateful kitchen that remembers "table 4 ordered the soup first, so bring the same soup for their second course" is convenient until you need ten kitchens open simultaneously and any of them might handle any table's order - at which point that memory has to live somewhere everyone can see it, not inside one kitchen's head. That's precisely the shift the 2026-07-28 revision makes: memory of "what happened earlier" moves out of the individual server and into infrastructure everyone can reach, so any kitchen behind the pass can pick up any ticket.

The 80/20 of MCP Engineering

If a team can only internalize a handful of ideas from everything above, these are the ones that account for most of the practical outcome, good or bad. First, tool description quality dominates almost every downstream failure mode in agent behavior - more than model choice, more than transport selection, more than almost anything else on this list - because the description is the only information the model has about when and how to use a capability, and it's read fresh on every relevant turn. Second, treating every piece of data crossing the MCP boundary as untrusted input, symmetrically in both directions, closes the majority of the security gap that tool poisoning and cross-server injection exploit; the industry's hard-won lesson here is that authentication is not the same thing as trust. Third, keeping the active tool set small and curated per agent, rather than connecting every available server by default, solves both the context-bloat problem and a meaningful share of the tool-confusion and injection-surface problems simultaneously, because a smaller surface is both cheaper and safer. Everything else in this article - transports, elicitation, sampling, the specifics of the 2026-07-28 revision - matters for production maturity, but these three habits are where most of the return on engineering effort actually lives.

Conclusion

MCP has moved from a promising idea to load-bearing infrastructure faster than most protocols do, and the 2026-07-28 specification revision is a clear signal that the ecosystem has already hit the scaling and governance pressures that force protocols to grow up - statelessness for horizontal scale, formal extensions for capabilities like Tasks and MCP Apps, and a real deprecation policy because enterprises now depend on this not breaking under them. None of that changes the fundamentals engineers need to get right at the server level: describe tools for the model's benefit as carefully as you'd write a prompt, because that's what a description actually is; treat every byte crossing the protocol boundary, in either direction, as untrusted until proven otherwise; and keep the surface area a model can act on as small as the task actually requires. Teams that internalize those three habits will find MCP a genuine force multiplier for agentic systems. Teams that treat it as "just another API wrapper" will eventually meet tool poisoning, context bloat, or a rug-pull attack the hard way - usually in production, and usually at the worst possible time.

References