paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

January 24, 2022

Probabilistic vs. Deterministic Systems: A Practical Guide for Software Architects

How to reason about certainty, reliability, and design trade-offs when building systems that must sometimes guess

Introduction

For most of software engineering's history, correctness meant the same thing: given the same input, a system produces the same output, every time. This is the deterministic worldview, and it underlies almost everything we learned to build - compilers, databases, payment processors, control systems. A function that returns different results for identical inputs was, by definition, buggy. Testing, debugging, and reasoning about programs all assumed this contract held.

That assumption is now only partially true. Distributed systems introduced non-determinism through network timing, partial failures, and concurrent state. Machine learning models introduced it more fundamentally: a language model, a recommendation engine, or a fraud-detection classifier does not compute an answer so much as it estimates one, with an inherent margin of uncertainty. Modern architectures increasingly combine both paradigms in the same system - a deterministic order-processing pipeline that calls a probabilistic recommendation service, or a rules-based access-control layer wrapping an LLM-based support agent. Understanding where each paradigm applies, and how to design the seams between them, has become a core architectural skill rather than a niche concern for the machine learning team.

Context and Problem Overview

Deterministic systems are built on a simple but powerful guarantee: identical inputs and identical starting state produce identical outputs. This property makes deterministic components easy to test (unit tests with fixed inputs are meaningful), easy to debug (a bug is reproducible), and easy to reason about formally (you can prove properties about the code). Classic examples include arithmetic operations, sorting algorithms, SQL queries against a consistent snapshot, and most business logic - tax calculations, inventory checks, state machines governing an order's lifecycle. ACID-compliant relational databases exist specifically to preserve this guarantee even under concurrent access, using isolation levels and locking to make transactions behave as if they ran one at a time.

Probabilistic systems, in contrast, produce outputs that carry uncertainty by design. This uncertainty comes from at least three distinct sources, and conflating them is a common architectural mistake. The first is statistical modeling: a machine learning classifier assigns a probability distribution over possible labels, and the "correct" answer is inherently a best estimate, not a fact. The second is randomized algorithms, such as reservoir sampling, randomized load balancing, or Monte Carlo simulations, where randomness is deliberately introduced to achieve better average-case performance or to approximate an answer that would be too expensive to compute exactly. The third, and often the most operationally troublesome, is emergent non-determinism in distributed systems - the same request can produce different outcomes depending on network latency, retry timing, clock drift, or the order in which concurrent events are processed, even though no component intended to be probabilistic.

The practical stakes of this distinction show up constantly in system design. A payment system that debits an account twice because a retry was not idempotent is not "approximately correct" - it is wrong, and the deterministic contract has been violated somewhere in the stack. An LLM-powered support agent that occasionally gives a slightly different phrasing for the same question is not violating any contract at all; that is the expected behavior of the paradigm. The engineering discipline lies in correctly classifying which parts of your system belong to which category, and in building explicit boundaries - validation layers, idempotency guarantees, fallback logic - at the points where they interact.

Deep Technical Explanation

The theoretical foundation for deterministic computing traces back to the Church-Turing model of computation: a deterministic Turing machine's next state is a strict function of its current state and input. Real-world systems approximate this ideal, and most of distributed systems engineering is an effort to preserve determinism in the face of an inherently non-deterministic physical substrate - unreliable networks, independent clocks, and concurrent processes. Techniques like Lamport clocks, vector clocks, and consensus protocols (Paxos, Raft) exist precisely to impose a deterministic, agreed-upon ordering of events on top of a fundamentally asynchronous network. When people say "distributed systems are hard", much of that difficulty is the cost of manufacturing determinism where none exists naturally.

Probabilistic systems, by contrast, embrace uncertainty as a first-class property rather than something to be eliminated. In machine learning, this shows up as calibrated confidence scores: a well-calibrated classifier that outputs "73% confidence" should be correct roughly 73% of the time across many predictions at that confidence level. This is fundamentally different from a deterministic bug, because there is no single "correct" output to compare against - there is a distribution, and the system's quality is judged by how well its outputs track that distribution over time, using metrics like precision, recall, calibration curves, or perplexity rather than pass/fail assertions. Probabilistic data structures follow a related but distinct logic: a Bloom filter, for example, is deterministic in its false-negative behavior (it never reports "not present" for something that was inserted) but probabilistic in its false positives, trading a small, mathematically bounded error rate for massive space savings. HyperLogLog and Count-Min Sketch follow the same pattern - deterministic guarantees on one side, bounded probabilistic error on the other, which is why they are usable in production systems at all: the uncertainty is quantified, not open-ended.

The generative AI wave has pushed probabilistic reasoning into places it never occupied before: application logic itself. An LLM-based agent deciding how to route a support ticket or draft a reply is not executing a lookup table; it is sampling from a learned distribution over token sequences, and the same prompt can yield different outputs depending on sampling temperature, model version, or even non-deterministic execution on GPU hardware. This means the "function" your application code is calling no longer has a stable, testable contract in the traditional sense, which is precisely why techniques like structured output validation, retrieval grounding, and evaluation harnesses have become necessary infrastructure rather than optional polish.

Implementation and Practical Examples

In practice, the most important architectural work is not choosing one paradigm over the other but designing clean, explicit boundaries between them. A deterministic core - order state machines, ledger updates, permission checks - should remain deterministic and auditable, while a probabilistic component is wrapped in validation, retries, and fallbacks that make its uncertainty explicit and bounded rather than silently propagating it downstream. The example below shows a TypeScript pattern for exactly this: an idempotent write path around a payment operation, ensuring that network retries (an inherently non-deterministic distributed systems concern) do not corrupt deterministic financial state.

// idempotent-payment.ts
// Ensures retried requests (caused by network non-determinism)
// never double-charge a deterministic ledger.

interface PaymentRequest {
  idempotencyKey: string;
  accountId: string;
  amountCents: number;
}

interface LedgerStore {
  getByIdempotencyKey(key: string): Promise<PaymentResult | null>;
  commit(request: PaymentRequest, result: PaymentResult): Promise<void>;
}

interface PaymentResult {
  status: "committed";
  transactionId: string;
  amountCents: number;
}

async function processPayment(
  request: PaymentRequest,
  ledger: LedgerStore,
  chargeProvider: (amount: number) => Promise<{ id: string }>
): Promise<PaymentResult> {
  // Check for a prior attempt with the same idempotency key first.
  // This makes retries safe even though the network call itself
  // is not deterministic (it may time out after actually succeeding).
  const existing = await ledger.getByIdempotencyKey(request.idempotencyKey);
  if (existing) {
    return existing;
  }

  const charge = await chargeProvider(request.amountCents);

  const result: PaymentResult = {
    status: "committed",
    transactionId: charge.id,
    amountCents: request.amountCents,
  };

  await ledger.commit(request, result);
  return result;
}

Wrapping a probabilistic model in a similar discipline requires a different set of tools: schema validation, confidence thresholds, and human or rule-based fallback paths. The Python example below illustrates a guardrail pattern commonly used around LLM outputs, where the model's probabilistic response is treated as a proposal that must pass a deterministic validation gate before it is allowed to affect application state.

# guarded_classification.py
# Treats an LLM's probabilistic output as untrusted input
# that must pass deterministic validation before use.

from dataclasses import dataclass
from typing import Optional
import json

VALID_CATEGORIES = {"billing", "technical", "account", "other"}
CONFIDENCE_THRESHOLD = 0.75


@dataclass
class ClassificationResult:
    category: str
    confidence: float
    raw_response: str


def parse_model_output(raw_response: str) -> Optional[ClassificationResult]:
    try:
        data = json.loads(raw_response)
        category = data["category"]
        confidence = float(data["confidence"])
    except (json.JSONDecodeError, KeyError, ValueError, TypeError):
        return None

    if category not in VALID_CATEGORIES:
        return None
    if not (0.0 <= confidence <= 1.0):
        return None

    return ClassificationResult(category, confidence, raw_response)


def route_ticket(raw_model_response: str) -> str:
    result = parse_model_output(raw_model_response)

    # Any failure to parse, or low confidence, falls back to a
    # deterministic default rather than trusting an ambiguous output.
    if result is None or result.confidence < CONFIDENCE_THRESHOLD:
        return "manual_review_queue"

    return f"{result.category}_queue"

Both examples share the same underlying principle: uncertainty is not eliminated, it is contained. The payment example contains network non-determinism behind an idempotency check so the ledger stays deterministic. The classification example contains model uncertainty behind a validation and confidence gate so downstream routing logic stays deterministic. Neither example tries to make the underlying non-deterministic process behave deterministically - that would be either impossible (the network) or counterproductive (the model's calibrated uncertainty is useful information). Instead, each defines a clear, testable contract at the boundary.

Trade-offs and Common Pitfalls

The most common architectural mistake is treating a probabilistic component as if it had a deterministic contract, usually because it is convenient to do so in the short term. Teams write unit tests that assert an LLM returns exact string output, then watch those tests become flaky as the underlying model is updated or as sampling temperature introduces natural variance. Some teams then over-correct by setting temperature to zero and calling the result "deterministic", which is misleading: even greedy decoding across model versions, hardware, or minor prompt changes can produce different outputs, and floating-point non-associativity on GPUs means bitwise reproducibility is not actually guaranteed even at temperature zero. The safer mental model is that probabilistic components should be evaluated statistically, across representative test sets with metrics like accuracy or pass rate, rather than asserted against a single expected string.

The opposite failure mode is equally damaging: introducing unnecessary non-determinism into what should be a deterministic path, usually as a side effect of distributed architecture. A classic example is relying on wall-clock timestamps to order events across services with unsynchronized clocks, producing subtly different results depending on which server processed a request first. Race conditions in concurrent code, unseeded random number generators in what should be reproducible simulations, and retries without idempotency keys are all cases where a system that should behave deterministically instead behaves probabilistically by accident, which is far worse than an intentionally probabilistic component because nobody designed for it, tested for it, or built monitoring around it. Debugging these issues is notoriously difficult precisely because they resist reproduction - the defining feature of a deterministic bug is that you can reliably trigger it again, and accidental non-determinism strips that property away.

Best Practices for Mixed Architectures

Effective system design in this space starts with explicit classification: for every component, ask whether it is meant to be deterministic, meant to be probabilistic, or accidentally non-deterministic due to an implementation detail that should be fixed. Components in the third category are technical debt, not architecture, and should generally be corrected - via idempotency keys, deterministic tie-breaking rules, seeded randomness in tests, or consensus protocols - rather than accepted as unavoidable. Components genuinely in the second category should have this fact reflected in how they are tested (statistical evaluation harnesses, golden datasets, drift monitoring) rather than forced into deterministic unit-test patterns that will produce false confidence or constant flakiness.

Observability strategy also needs to differ by paradigm. Deterministic components benefit from traditional error-rate and latency monitoring, where any deviation from expected behavior is a bug to be fixed. Probabilistic components need a second layer on top of that: model quality monitoring, including confidence score distributions, prediction drift over time, and outcome-based evaluation (did the recommendation get clicked, did the classification get overturned by a human reviewer). Tools like structured logging of model inputs and outputs, paired with periodic offline evaluation against labeled data, let teams detect quality degradation that would never show up as a conventional error or exception.

Finally, the boundary between the two paradigms deserves the same rigor as an API contract, because that is effectively what it is. Input validation, output schema enforcement, confidence thresholds, circuit breakers, and clearly defined fallback behavior are not optional hardening for probabilistic components - they are the mechanism by which the rest of the system is allowed to remain deterministic despite calling something that is not. Teams that skip this step tend to discover the hard way, in production incidents, that uncertainty they thought was contained inside a model call had quietly leaked into billing logic, access control, or data integrity elsewhere in the system.

Key Takeaways

Conclusion

Probabilistic and deterministic systems are not competing philosophies where one must be chosen over the other; they are complementary tools that solve different problems, and most production architectures today genuinely need both. The engineering skill that matters is not picking a side but drawing clear, well-tested boundaries between the two - keeping deterministic cores auditable and reproducible, keeping probabilistic components honest about their uncertainty, and never letting one masquerade as the other.

As machine learning and LLM-based components become standard parts of mainstream software architecture rather than specialized additions, this distinction will only grow in importance. The teams that build reliable systems in this new landscape will not be the ones who eliminate uncertainty - that is often impossible or counterproductive - but the ones who architect for it deliberately, with the same rigor historically reserved for deterministic correctness.

References