paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

MongoDB at Scale: A Senior Engineer's Guide to Data Modeling, Performance, and Production Pitfalls

From schema design and aggregation pipelines to sharding, transactions, and operational best practices - everything a professional engineer needs to wield MongoDB effectively in production.

Introduction

MongoDB occupies a peculiar position in the database ecosystem. It is simultaneously one of the most misused and most powerful storage engines available to modern engineering teams. Developers reach for it when they want to "move fast" with flexible schemas, then spend months wrestling with query performance, memory pressure, and consistency edge cases they didn't anticipate. The engineers who use it well - teams at Amadeus, eBay, and Cisco, to name a few documented cases - treat it as a first-class architectural decision, not a default escape hatch from relational thinking.

This article is not a tutorial. It assumes you already know how to insert a document and run a find query. Instead, it is a map of the decisions that separate a naive MongoDB deployment from one that holds up under production load, evolves cleanly over time, and gives operators the observability they need to diagnose problems quickly. We will cover data modeling philosophy, aggregation pipeline internals, indexing strategy, transactions, sharding, and the operational patterns that experienced teams converge on after enough production incidents.

Why MongoDB, and Why It Goes Wrong

The core value proposition of a document database is that it stores data in the shape your application already uses, eliminating the object-relational impedance mismatch that makes heavily normalized SQL schemas tedious to work with in object-oriented or functional codebases. A MongoDB document is a BSON object - essentially rich JSON with typed scalars - and the query language operates on that structure natively. There is no join syntax because, for many access patterns, you simply do not need one: related data lives together in the same document.

This design decision is also the source of MongoDB's most common failure mode. Teams accustomed to relational databases assume that the absence of a rigid schema means the absence of schema design work entirely. In practice, the opposite is true. In a relational database, the schema is enforced by the engine, and queries are written to navigate the schema. In MongoDB, the schema lives in your application code and your team's collective memory, and the engine will cheerfully store whatever you give it. Every missing discipline in data modeling shows up eventually as a production query that scans a full collection, an aggregation pipeline that runs out of the 100 MB RAM limit, or an application that silently handles five different shapes of the same document type because the schema drifted over two years of feature development.

The engineers who use MongoDB effectively understand that it shifts schema responsibility from the database to the application layer. That is a trade-off worth making in many contexts, but only if you actually take ownership of it.

Data Modeling: Embedding vs. Referencing

The fundamental modeling decision in MongoDB is whether to embed related data inside a document or to store it as a separate document and reference it by ID. This is not an arbitrary stylistic choice. It directly determines which queries are fast, which operations are atomic, and how your storage footprint grows over time.

The general heuristic is well-established in the MongoDB documentation: embed when you always access the data together, and reference when the related data is large, frequently updated independently, or shared across many documents. A blog post and its tags are a classic embedding case - they are fetched together, tags are small, and there is no meaningful scenario in which you update a tag independently of the post. Conversely, a blog post and its author are a referencing case: the author document is large, it is updated independently (profile photo, bio), and it is shared across potentially thousands of posts.

Where teams go wrong is in the unbounded array anti-pattern. Consider an e-commerce order that embeds all its events - "order placed", "payment confirmed", "shipped", "delivered" - as an array inside the order document. For most orders this is fine. But for a document that represents a subscription or a long-running workflow, that array can grow to thousands of entries, causing the document to exceed MongoDB's 16 MB BSON limit, or bloating it enough to cause significant I/O overhead on every read. The correct model for unbounded growth is a separate collection with a reference, not a growing embedded array.

Schema versioning is the other modeling discipline teams underinvest in. Because MongoDB allows mixed document shapes in the same collection, it is tempting to simply start writing new fields without migrating old documents. Over time, application code accumulates defensive checks and default fallbacks that are really compensating for an unmanaged schema migration. A practical pattern is to add an explicit schemaVersion field to every document and handle version-specific logic in a migration layer, either at read time (lazy migration) or via a background migration job. MongoDB's $set update operator and the aggregation pipeline update syntax make bulk migrations feasible without application downtime.

Indexing Strategy and Query Planning

An unindexed MongoDB collection is a collection waiting to cause an outage. Without an index, every query performs a COLLSCAN - a full collection scan - which means the query latency scales linearly with collection size. Understanding how MongoDB selects and uses indexes is one of the highest-leverage skills for production engineers.

MongoDB uses a query planner that evaluates candidate query plans using a "trial period" mechanism, running multiple plans in parallel and picking the winner based on which one reaches a threshold of returned documents first. This winning plan is then cached for a given query shape (the combination of filter predicates, sort, and projection operators). The cache is invalidated when the collection's index configuration changes or when write volume crosses a threshold. You can inspect the planner's decision with explain("executionStats"), which gives you the winning plan, the number of documents examined, the number of keys examined, and whether an in-memory sort was required.

// Use explain to inspect query performance
const result = await collection
  .find({ status: "active", createdAt: { $gte: new Date("2024-01-01") } })
  .sort({ createdAt: -1 })
  .explain("executionStats");

console.log(result.executionStats.totalDocsExamined);  // Should be close to nReturned
console.log(result.executionStats.executionTimeMillis);

Compound indexes are where most of the real performance leverage comes from. MongoDB can use a compound index to satisfy a query that filters on a prefix of the index keys, and it can also use the index to satisfy a sort without an in-memory sort stage - but only if the sort direction matches the index direction for all sorted fields. The ESR rule (Equality, Sort, Range) is the canonical guidance for compound index field ordering: put equality predicates first, then sort fields, then range predicates. This ordering maximizes the fraction of the index that can be used to both filter and sort in a single pass.

// ESR rule example: equality on status, sort on createdAt, range on amount
await collection.createIndex({ status: 1, createdAt: -1, amount: 1 });

// This query benefits from the full index
await collection
  .find({ status: "paid", amount: { $gte: 100 } })
  .sort({ createdAt: -1 })
  .toArray();

Partial indexes and sparse indexes are underused levers. A partial index only indexes documents that match a filter expression, making it significantly smaller than a full collection index. If you have a collection where 90% of documents have status: "archived" and your application only ever queries status: "active" documents, a partial index on active documents is far cheaper to maintain and fit in RAM. Similarly, a sparse index does not index documents where the indexed field is absent, which is useful for optional fields with high nullability.

Index bloat is a real operational problem. Every index you add increases the write amplification on the collection - inserts and updates must update every index. For write-heavy collections, over-indexing causes measurable throughput degradation. The $indexStats aggregation stage reports usage statistics per index and is the right tool for identifying indexes that are never used by any query.

The Aggregation Pipeline

The aggregation pipeline is MongoDB's answer to SQL's SELECT with GROUP BY, window functions, and subqueries. It processes documents through a sequence of stages, each transforming or filtering the stream. Understanding how the pipeline interacts with indexes and memory is essential for writing aggregations that work at scale.

The most important optimization principle is stage ordering: put $match and $sort stages as early as possible in the pipeline. MongoDB's query optimizer can push $match stages before $project and $addFields stages, but it cannot always push them before $lookup or $unwind stages automatically. If your pipeline unwinds an array and then filters on a field in the unwound documents, you should manually move the $match before the $unwind and add a corresponding array-level filter where possible.

// Suboptimal: unwinds all documents before filtering
const pipeline = [
  { $unwind: "$lineItems" },
  { $match: { "lineItems.sku": "ABC-123" } },
  { $group: { _id: "$customerId", total: { $sum: "$lineItems.price" } } }
];

// Better: filter at document level first to reduce unwind volume
const optimizedPipeline = [
  { $match: { "lineItems.sku": "ABC-123" } },  // Uses array index if available
  { $unwind: "$lineItems" },
  { $match: { "lineItems.sku": "ABC-123" } },  // Filter after unwind
  { $group: { _id: "$customerId", total: { $sum: "$lineItems.price" } } }
];

The 100 MB RAM limit per aggregation pipeline stage is a frequent stumbling block. By default, each stage that needs to buffer documents - $sort, $group, $bucket - is limited to 100 MB of working memory. If your pipeline exceeds this, MongoDB throws an error unless you set allowDiskUse: true. Disk-based operations are orders of magnitude slower and should be treated as a symptom of a modeling or query design problem, not a solution. The right response to hitting this limit is to reconsider whether the aggregation is doing too much work: can it be scoped to fewer documents with earlier filtering? Can the aggregation result be precomputed and cached? Can the workload be moved to a dedicated analytics replica?

$lookup is MongoDB's left outer join operation. It is powerful but has important performance characteristics to understand. A $lookup against a collection that is not indexed on the foreignField causes a full collection scan for every document in the pipeline at that stage. Always ensure the foreignField (or localField if the join direction is reversed) is indexed. The pipeline form of $lookup, introduced in MongoDB 3.6 and improved in later versions, allows you to apply a pipeline to the joined collection and filter documents before they are returned, which reduces the data transferred and processed.

// Pipeline-form $lookup with early filtering (more efficient than simple form)
const pipeline = [
  { $match: { status: "active" } },
  {
    $lookup: {
      from: "products",
      let: { productId: "$productId" },
      pipeline: [
        { $match: { $expr: { $eq: ["$_id", "$$productId"] }, inStock: true } },
        { $project: { name: 1, price: 1, _id: 0 } }
      ],
      as: "product"
    }
  }
];

Transactions: When and How to Use Them

MongoDB introduced multi-document ACID transactions in version 4.0 for replica sets and version 4.2 for sharded clusters. Before this, the only atomicity guarantee was at the single-document level. The common guidance - "if you need transactions, use PostgreSQL" - became outdated in 2018, but the correct interpretation is more nuanced: MongoDB supports transactions, but they are not free, and over-relying on them often indicates a modeling problem.

Multi-document transactions in MongoDB carry overhead. They acquire collection-level locks during the transaction and hold a snapshot of data from the transaction's start time. Long-running transactions increase the likelihood of write conflicts, which cause transaction aborts and require retry logic in your application. MongoDB's WiredTiger storage engine uses MVCC (Multi-Version Concurrency Control), so reads do not block writes, but transactions still impose coordination overhead that single-document operations do not.

// Proper transaction pattern with retry logic for transient errors
async function transferFunds(
  session: ClientSession,
  fromId: ObjectId,
  toId: ObjectId,
  amount: number
): Promise<void> {
  const maxRetries = 3;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      await session.withTransaction(async () => {
        const accounts = db.collection("accounts");

        const from = await accounts.findOne({ _id: fromId }, { session });
        if (!from || from.balance < amount) {
          throw new Error("Insufficient funds");
        }

        await accounts.updateOne(
          { _id: fromId },
          { $inc: { balance: -amount } },
          { session }
        );

        await accounts.updateOne(
          { _id: toId },
          { $inc: { balance: amount } },
          { session }
        );

        await db.collection("auditLog").insertOne({
          type: "transfer",
          fromId,
          toId,
          amount,
          timestamp: new Date()
        }, { session });
      });

      return; // Success

    } catch (err: any) {
      // Retry on transient transaction errors
      if (err.hasErrorLabel?.("TransientTransactionError") && attempt < maxRetries - 1) {
        continue;
      }
      throw err;
    }
  }
}

The practical guidance is: prefer designing your documents so that the operations you need are atomic at the single-document level, and reach for multi-document transactions only when that is genuinely impossible. The classic case is a financial transfer between two accounts - you cannot make debiting one and crediting another atomic in a single document unless you denormalize the ledger into a single document, which may not be appropriate. For many other use cases that seem to require transactions, a careful re-read of the MongoDB single-document atomicity guarantees reveals that a different document model eliminates the need.

Sharding: Architecture and Common Mistakes

Sharding is MongoDB's horizontal scaling mechanism. A sharded cluster distributes documents across multiple shards (each a replica set) using a shard key. The mongos router processes queries from the application, consults the config servers for the chunk distribution map, and routes operations to the relevant shards.

The choice of shard key is the most consequential and least reversible decision in a sharded deployment. A bad shard key causes two failure modes: hotspot sharding and scatter-gather queries. A hotspot occurs when the shard key has low cardinality or monotonically increasing values (like an auto-incrementing counter or a timestamp), causing all new writes to go to a single shard while others sit idle. Scatter-gather occurs when queries do not include the shard key, forcing the mongos to fan out the query to every shard and merge the results - effectively eliminating the performance benefit of sharding.

A good shard key has high cardinality, even distribution of writes, and alignment with the application's most common query patterns. Compound shard keys are often better than single-field keys. For a multi-tenant SaaS application, { tenantId: 1, _id: 1 } is a common effective choice: it groups all documents for a tenant on the same shard (enabling efficient tenant-scoped queries), while _id provides enough cardinality within each tenant to avoid intra-tenant hotspots.

Hashed shard keys distribute writes uniformly by hashing the shard key value, which eliminates monotonic write hotspots. The trade-off is that range queries on the hashed field become scatter-gather, because hashed values lose their natural ordering. For write-heavy workloads with no range query requirements on the shard key, a hashed key is a reasonable choice.

Chunk migrations are a source of operational overhead. MongoDB's balancer continuously monitors chunk distribution across shards and migrates chunks to achieve balance. Each migration uses I/O, network bandwidth, and CPU on the source and destination shards. For write-heavy workloads, frequent migrations can meaningfully impact throughput. A common mitigation is to pre-split chunks at collection creation time for predictable workloads, or to schedule the balancer window to avoid peak write hours.

Trade-offs and Production Pitfalls

Every technology has failure modes that are invisible until you hit them in production. MongoDB's are well-documented by teams that have operated it at scale.

The Working Set and RAM. MongoDB's WiredTiger storage engine relies heavily on the operating system's page cache to serve frequently accessed data from RAM. When your working set - the set of documents and index pages accessed by common queries - exceeds available RAM, page faults become frequent and query latency spikes sharply. Monitoring the wiredTiger.cache.pages read into cache metric and setting alerts on it is one of the most important operational practices for any MongoDB deployment. The solution is either to increase RAM (by scaling up the instance), to reduce the working set (by archiving old data or splitting hot and cold data into separate collections), or to optimize indexes so that query plans touch fewer documents and pages.

Write Concern and Read Concern Trade-offs. MongoDB's default write concern (w: 1) acknowledges a write after it has been applied to the primary, before it has been replicated to any secondaries. In a replica set, if the primary crashes before replication completes, that write is lost when the primary is elected anew. For data you cannot afford to lose, use w: "majority", which waits for acknowledgment from a majority of replica set members. The latency cost is real - typically an additional round trip to the nearest secondary - but the durability guarantee is critical for financial or transactional workloads. Similarly, readConcern: "majority" ensures that reads only return data that has been committed to a majority of replicas and will not be rolled back.

Schema Validation. MongoDB supports JSON Schema validation rules on collections via $jsonSchema. Enabling validation does not eliminate the flexibility of the document model but it does prevent accidental writes of structurally invalid documents, which is one of the main sources of the "five different document shapes in the same collection" problem mentioned earlier.

// Enforce schema validation at the collection level
await db.createCollection("orders", {
  validator: {
    $jsonSchema: {
      bsonType: "object",
      required: ["customerId", "status", "lineItems", "createdAt"],
      properties: {
        customerId: { bsonType: "objectId" },
        status: { enum: ["pending", "paid", "shipped", "cancelled"] },
        lineItems: {
          bsonType: "array",
          minItems: 1,
          items: {
            bsonType: "object",
            required: ["sku", "quantity", "price"],
            properties: {
              sku: { bsonType: "string" },
              quantity: { bsonType: "int", minimum: 1 },
              price: { bsonType: "decimal" }
            }
          }
        },
        createdAt: { bsonType: "date" }
      }
    }
  },
  validationAction: "error"
});

Connection Pool Exhaustion. Applications that open connections without bound, or that hold connections while doing slow work outside the database, quickly exhaust the connection pool and cause new requests to queue or fail. The MongoDB driver's default maxPoolSize is 100 connections per client instance. In serverless environments where each function invocation may create a new client, connection pool exhaustion is a persistent and subtle problem. The established mitigation pattern is connection reuse across invocations using module-level singletons, combined with serverSelectionTimeoutMS tuning to fail fast rather than queue indefinitely.

Best Practices for Production MongoDB

Experienced MongoDB operators converge on a set of practices that dramatically reduce the frequency of production incidents and make recovery faster when they do occur.

Use replica sets everywhere, even for development. A standalone MongoDB instance has no high availability, no automatic failover, and importantly, does not support transactions or change streams. Running a three-member replica set is the minimum viable MongoDB deployment for any workload that matters. Tools like MongoDB Atlas, or local setups with rs.initiate(), make this straightforward. Instrument at the driver level, not just at the server. MongoDB's Atlas Monitoring, mongostat, and mongotop give you server-level throughput and operation counts. But the most actionable signal - which queries are slow, and why - comes from enabling slow query logging (the slowms threshold, default 100ms) and from the MongoDB Profiler levels 1 and 2. Pair these with APM traces at the application level, so that slow queries can be correlated with specific API endpoints and user actions rather than appearing as anonymous database load. Prefer $expr over JavaScript in queries. Operators like $where and the mapReduce command execute JavaScript inside the MongoDB server process, bypassing indexes and performing poorly at scale. Modern MongoDB versions provide sufficient expressive power in the aggregation pipeline and $expr to replace virtually all $where usage. JavaScript evaluation in queries also prevents the query planner from using indexes on the evaluated field. Test your indexes with realistic data volumes. An index that appears efficient on a collection of ten thousand documents may be insufficient at ten million. The query planner's behavior changes with selectivity, and what was an IXSCAN at small scale can degrade to a COLLSCAN if index statistics become stale. Run db.collection.reIndex() and db.runCommand({ planCacheClear: "collection" }) periodically in staging environments with production-scale data to validate your indexes continuously. Design for partial availability. In a sharded or replica set cluster, individual nodes fail. Application code that does not handle MongoNetworkError, MongoServerSelectionError, and write concern errors with appropriate retry logic will surface these failures as user-facing errors. The MongoDB drivers implement retryable writes and retryable reads as options precisely because transient errors are a normal part of operating a distributed system - they should be handled transparently by the driver layer, not propagated to users.

80/20 Insight: The Three Things That Move the Needle

Across all the topics in this article, three decisions account for the majority of MongoDB performance and reliability outcomes in practice.

First, your data model must match your access patterns. Embedding vs. referencing is not a stylistic choice - it is the primary determinant of whether your queries are single-document lookups or multi-stage aggregations that fan across collections. Get this right during design, not after you have millions of documents.

Second, your indexes must be maintained with the same discipline as your schema. Add an index for every query pattern you rely on, audit unused indexes regularly, and always validate index efficiency with explain("executionStats") before deploying a new query to production.

Third, your write concern and read concern must match your durability requirements. For most production workloads, w: "majority" and readConcern: "majority" are the correct defaults. The latency cost is small; the correctness guarantee is large.

Everything else - sharding strategy, aggregation optimization, connection pooling - is important but secondary. Teams that get these three decisions right rarely have MongoDB crises. Teams that do not routinely do.

Key Takeaways

Five practical steps you can apply immediately, regardless of where you are in your MongoDB journey:

  1. Audit your collections with $indexStats and drop any index that has zero or near-zero usage. Unused indexes impose write overhead with no query benefit.
  2. Add schemaVersion to all documents and establish a migration policy for evolving your schema. Make version-handling explicit rather than implicit.
  3. Run explain("executionStats") on your top-10 most frequent queries and confirm each uses an IXSCAN, not a COLLSCAN. Resolve any COLLSCAN before going to production.
  4. Enable slow query logging at a 50ms threshold in production and review the output weekly. Most MongoDB performance regressions announce themselves as increasing slow query frequency before they become outages.
  5. Set write concern to w: "majority" for any data your application cannot afford to lose, and implement retry logic for TransientTransactionError and UnknownTransactionCommitResult in all transaction-using code paths.

Conclusion

MongoDB is a genuinely capable database for a wide range of production workloads - flexible enough to evolve with your application, powerful enough to handle significant scale, and operationally mature enough to run reliably in production. The engineers who get the most out of it are the ones who treat schema design as a first-class engineering discipline, who understand how the query planner works well enough to anticipate its decisions, and who instrument their deployments well enough to diagnose problems before users report them.

The technology's flexibility is both its greatest strength and its most common liability. Without deliberate modeling and index discipline, MongoDB will faithfully store whatever you give it and faithfully scan the entire collection to find it. With those disciplines applied, it is one of the most productive databases available for teams building applications that need to iterate quickly without sacrificing production reliability. The difference is almost entirely in the engineering practices layered on top, not in the technology itself.

References

  1. MongoDB, Inc. MongoDB Manual - Data Modeling Introduction. https://www.mongodb.com/docs/manual/core/data-modeling-introduction/
  2. MongoDB, Inc. MongoDB Manual - Indexing Strategies. https://www.mongodb.com/docs/manual/applications/indexes/
  3. MongoDB, Inc. MongoDB Manual - Aggregation Pipeline Optimization. https://www.mongodb.com/docs/manual/core/aggregation-pipeline-optimization/
  4. MongoDB, Inc. MongoDB Manual - Transactions. https://www.mongodb.com/docs/manual/core/transactions/
  5. MongoDB, Inc. MongoDB Manual - Sharding. https://www.mongodb.com/docs/manual/sharding/
  6. MongoDB, Inc. MongoDB Manual - WiredTiger Storage Engine. https://www.mongodb.com/docs/manual/core/wiredtiger/
  7. MongoDB, Inc. MongoDB Manual - Read Concern / Write Concern. https://www.mongodb.com/docs/manual/reference/read-concern/ and https://www.mongodb.com/docs/manual/reference/write-concern/
  8. Banker, Kyle et al. MongoDB in Action, 2nd ed. Manning Publications, 2016.
  9. Chodorow, Kristina. MongoDB: The Definitive Guide, 3rd ed. O'Reilly Media, 2019.
  10. MongoDB Engineering Blog. "Schema Design Patterns." https://www.mongodb.com/blog/post/building-with-patterns-a-summary
  11. MongoDB, Inc. MongoDB Manual - JSON Schema Validation. https://www.mongodb.com/docs/manual/core/schema-validation/jsonschema/
  12. MongoDB, Inc. MongoDB Manual - Compound Indexes and ESR Rule. https://www.mongodb.com/docs/manual/tutorial/equality-sort-range-rule/