Building a Customer Support Chatbot: An AI Engineering Architecture Case Study

Introduction

Not every AI engineering decision looks like the previous one. Where a product category classifier is a close-ended problem that reopens classical ML as a genuine contender, a customer support chatbot sits at the opposite end of the spectrum - and working through its architecture reveals a completely different set of trade-offs. This article continues the case-study format from the product classifier scenario, applying the same systematic decision process to a harder, more open-ended problem: a chatbot that answers customer questions about orders, return policies, and product details by pulling from a company's knowledge base.

The contrast between these two scenarios is instructive precisely because they're both plausible features at the same e-commerce company, built by the same engineering team, and yet they call for almost entirely different architectures. Walking through output type, model approach, data requirements, adaptation technique, and differentiation strategy for this chatbot shows how differently these same five questions get answered once the underlying task shifts from classification to open-ended generation.

The Scenario: A Knowledge-Grounded Customer Support Assistant

The feature requirement here is a chatbot that answers customer questions about orders, return policies, and product details, pulling its answers from the company's existing knowledge base. Unlike the product classifier, there's no predefined set of correct outputs to choose from - a customer might ask "can I return this if I already opened the box," "where's my order," or "does this blender come with a warranty," and each of these deserves a different, fluent, contextually accurate response rather than a label picked from a fixed list.

This pattern - a conversational interface grounded in a company's internal documentation - is now one of the most common AI application archetypes, spanning internal IT help desks, documentation search assistants, and customer-facing support bots across industries. Recognizing this pattern matters because it immediately signals a very different architecture than the classification scenario: the core challenge here isn't choosing between a small number of valid outputs, it's generating a genuinely new, accurate, well-grounded response to a question that's never been asked in exactly that form before.

Step 1: Characterizing the Output Type

As with the classifier scenario, the first question is whether this task is close-ended or open-ended, and the answer here is unambiguous in the opposite direction: this is an open-ended task. The chatbot must generate natural language responses that could vary enormously depending on the customer's specific question, their product, their order history, and the phrasing they happen to use. There is no fixed, enumerable set of possible answers the way there was with 75 product categories - the space of valid responses is effectively unbounded, even though the underlying information being drawn on (policies, order details, product specs) is itself fixed and well-defined.

This distinction between open-ended output and fixed underlying information is worth sitting with, because it's easy to conflate the two. The company's return policy is a fixed piece of information - there is exactly one correct return window for a given product. But the way that policy gets communicated in response to "can I return this if I lost the receipt" versus "I bought this as a gift, can my friend return it" requires genuinely different, context-sensitive natural language, not a single canned template reused verbatim. This is precisely what makes the task open-ended: correctness depends on the response being accurate, helpful, and appropriately phrased for the specific question asked, not on matching one of a small number of predefined strings.

Step 2: Choosing a Model Approach

With the task identified as open-ended, the model approach question resolves far more decisively than it did for the classifier. A foundation model is the clear choice here, and classical ML genuinely isn't a serious alternative. Classical ML models - classifiers, regressors, even sequence-to-sequence models trained from scratch - are not designed for the kind of open-ended natural language generation this task requires. You need a model that can parse a wide range of phrasings for essentially the same underlying question, reason over retrieved policy text, and produce a fluent, contextually appropriate response, all without having seen that exact question before.

Building this capability from scratch would require the kind of massive pretraining investment - vast text corpora, enormous compute budgets, deep NLP expertise - that foundation models already represent. This is exactly the category of problem foundation models were built to solve: general-purpose language understanding and generation that would be prohibitively expensive and time-consuming to replicate for a single company's support use case. There's no meaningful "build vs. buy" tension here the way there was with the classifier; buying (or more precisely, adapting) an existing foundation model is overwhelmingly the sensible path, and the real engineering decision shifts entirely to how that foundation model gets adapted to this specific task.

Step 3: Working Out Data Requirements

The data requirements for this task look almost inverted compared to the product classifier. You do not need traditional labeled data to get started - there's no equivalent of "product listing paired with correct category" here, because the system isn't learning to predict a label; it's learning to generate grounded responses using information retrieved at inference time. Instead, the foundational data asset is a well-organized knowledge base: documents covering order policies, return procedures, product details, and frequently asked questions.

The data engineering work here centers on quality control rather than labeling. Outdated or contradictory policy documents need to be identified and removed, since a chatbot retrieving and confidently stating an outdated return window is worse than one that says it doesn't know. Deduplication matters too - near-identical policy documents can confuse retrieval or cause the model to cite the wrong version of a policy that's since been updated. Beyond cleaning the existing knowledge base, documents need to be prepared specifically for context retrieval: typically chunked into appropriately sized passages, embedded, and indexed in a vector store so relevant sections can be retrieved efficiently at query time rather than forcing the model to search an entire document corpus in-context.

Over time, a second data asset becomes valuable: human feedback on chatbot responses, collected through thumbs up/down signals, explicit corrections, or human review of a sample of conversations. This feedback loop is what eventually enables fine-tuning or systematic prompt improvement, but it's fundamentally different from the labeled training data a classical ML model would need, since it's collected after deployment rather than prepared upfront.

It's worth being explicit about why evaluation is harder here than for the classifier. Annotating open-ended outputs for evaluation is significantly harder than annotating a close-ended classification, because judging whether a chatbot response is helpful, accurate, and complete requires nuanced human judgment, whereas checking whether a category label is correct is comparatively mechanical. This single difference ripples through the entire evaluation strategy for this feature, a topic the next section returns to in more depth.

Step 4: Selecting an Adaptation Technique

The recommended starting point here combines two prompt-based techniques rather than relying on either alone: prompt engineering and retrieval-augmented generation (RAG). Prompt engineering establishes the assistant's tone, scope, and behavioral boundaries through clear system instructions - for instance, explicitly instructing the model to say "I don't know" rather than guessing when it lacks sufficient information, which matters enormously for a support context where a confidently wrong answer about a return policy can cause real customer friction.

RAG is the technique that actually makes this feature viable at all, not just an optimization. The underlying foundation model has no inherent knowledge of this specific company's return policies, current order statuses, or product catalog - that information simply doesn't exist in its pretraining data. RAG solves this by connecting the model to the company's knowledge base at inference time, retrieving the specific policy documents or product details relevant to a given question and inserting them into the model's context before it generates a response.

import Anthropic from "@anthropic-ai/sdk";
import { retrieveKnowledgeBaseChunks } from "./retrieval";
import { lookupOrderStatus } from "./orders";

const client = new Anthropic();

interface SupportResponse {
  answer: string;
  usedOrderLookup: boolean;
}

async function answerSupportQuestion(
  question: string,
  customerId: string
): Promise<SupportResponse> {
  // Retrieve relevant policy/product documentation
  const docs = await retrieveKnowledgeBaseChunks(question, 5);
  const context = docs.map((d, i) => `[${i + 1}] ${d.text}`).join("\n\n");

  // Optionally pull live order data if the question seems order-related
  const needsOrderData = /\border\b|\breturn\b|\bshipping\b/i.test(question);
  const orderContext = needsOrderData
    ? JSON.stringify(await lookupOrderStatus(customerId))
    : null;

  const response = await client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 500,
    system:
      "You are a customer support assistant for an e-commerce company. " +
      "Answer using ONLY the provided documentation and order data. " +
      "If you don't have enough information to answer confidently, say so " +
      "and offer to connect the customer with a human agent. Be concise " +
      "and friendly. Cite policy sources by bracketed number when relevant.",
    messages: [
      {
        role: "user",
        content:
          `Documentation:\n${context}\n\n` +
          (orderContext ? `Order data:\n${orderContext}\n\n` : "") +
          `Customer question: ${question}`,
      },
    ],
  });

  const block = response.content.find((b) => b.type === "text");
  return {
    answer: block && block.type === "text" ? block.text : "",
    usedOrderLookup: Boolean(orderContext),
  };
}

Fine-tuning is appropriately deferred here rather than reached for immediately - it can come later, once the team has accumulated enough high-quality conversation examples and has identified a specific, recurring gap that prompting and RAG haven't closed, whether that's improving response quality on a particular category of question, reducing latency, or lowering inference cost by shifting to a smaller, specialized model. This sequencing mirrors the classifier scenario's logic (start cheap, escalate on evidence) even though the underlying task and chosen techniques differ substantially.

Step 5: Thinking About Differentiation

Since many companies could plausibly build a support chatbot on the exact same foundation model, competitive advantage here - just as in the classifier scenario - doesn't come from the model itself. For this feature, it comes specifically from the application development layer, which carries even more weight here than it did for the classifier, precisely because the task is open-ended and every component of that layer has more room to differentiate good execution from mediocre execution.

Excellent prompt engineering - clear, well-tested system instructions that have been iterated on against real failure cases - determines whether the assistant reliably stays within scope, admits uncertainty appropriately, and maintains a consistent, on-brand tone. Superior context construction via RAG, meaning a clean, comprehensive, well-indexed knowledge base free of outdated or contradictory information, determines whether the model's answers are actually grounded in correct information rather than confidently wrong. A great user interface - one that handles follow-up questions gracefully, maintains conversation context, and makes it easy to escalate to a human when needed - shapes how forgiving customers are of any imperfections in the underlying responses. And rigorous evaluation, meaning systematic testing of response quality backed by a feedback loop that uses production data to continuously improve the system, is what prevents quality from silently degrading as the knowledge base and customer question patterns evolve over time.

Trade-offs and Pitfalls in This Scenario

The most consequential pitfall here is underinvesting in retrieval quality while over-investing in prompt polish. Because RAG is the mechanism that actually grounds the chatbot's answers in correct, company-specific information, a retrieval step that surfaces the wrong or outdated documents will produce confidently incorrect answers regardless of how well-crafted the surrounding prompt is. Teams sometimes spend disproportionate effort refining instructions and tone while the knowledge base itself contains stale or duplicate policy documents that no amount of prompt engineering can compensate for.

A second pitfall is treating evaluation as optional or informal because it's harder than it was for the classifier. Because judging whether a response is helpful, accurate, and complete requires genuine human judgment rather than a mechanical correctness check, it's tempting to skip building a rigorous evaluation process and rely instead on spot-checking a handful of conversations. This is a mistake precisely because the difficulty of evaluation doesn't reduce its importance - if anything, the harder a task is to evaluate, the more a team needs a deliberate process (a rubric, periodic human review, LLM-as-judge scoring, systematic tracking of escalations to human agents) to catch regressions that casual spot-checking would miss.

A third pitfall is underestimating how much the "say I don't know" instruction matters in a support context specifically. A support chatbot that confidently fabricates a return policy detail or invents an order status creates real business risk - frustrated customers, support escalations, even policy disputes - in a way that's qualitatively different from a chatbot giving a slightly awkward answer to a general knowledge question. Teams building support assistants need to treat graceful uncertainty (knowing when to say "I'm not sure, let me connect you with a human agent") as a first-class design requirement, not an afterthought bolted onto the system prompt.

Best Practices for Open-Ended Knowledge-Grounded Assistants

When building a similar knowledge-grounded conversational assistant, invest in knowledge base hygiene before investing heavily in prompt iteration - a clean, deduplicated, up-to-date knowledge base has a larger impact on answer quality than almost any amount of prompt refinement can compensate for. Build retrieval evaluation as its own discipline, separate from end-to-end response evaluation, since a problem with retrieved documents and a problem with how the model uses those documents require different fixes and are easy to conflate if you only measure final response quality.

Treat the "I don't know" instruction and its downstream behavior (offering human escalation, acknowledging uncertainty) as a tested feature in its own right, not an incidental prompt line - verify through evaluation that the model actually follows this instruction consistently across a range of genuinely ambiguous or out-of-scope questions, rather than assuming a single sentence in the system prompt guarantees the behavior. Finally, build the human feedback collection mechanism (thumbs up/down, escalation tracking, periodic conversation review) from day one, even before fine-tuning is on the roadmap, since this data becomes the foundation for every subsequent improvement cycle, from better few-shot examples to an eventual fine-tuning dataset.

Mental Model: The Librarian vs. The Encyclopedia

A useful way to contrast this scenario with the product classifier is to think of the classifier as sorting books onto one of 75 labeled shelves - a mechanical, well-defined task with a clearly correct answer for each book. The support chatbot, by contrast, is more like a knowledgeable librarian who's read the entire reference collection and can answer a patron's specific, idiosyncratically phrased question by pulling together relevant material and explaining it in their own words. The librarian isn't just retrieving a book and handing it over (that would be plain retrieval without generation) - they're synthesizing an answer tailored to exactly what was asked, which is precisely what RAG plus a capable foundation model enables, and precisely why evaluating the librarian's answer is harder than checking which shelf a book belongs on.

Key Takeaways

• Open-ended generation tasks point decisively toward foundation models, since classical ML isn't designed for fluent, context-sensitive natural language generation.
• RAG is not optional here - it's the mechanism that makes company-specific accuracy possible, since the base model has no inherent knowledge of your policies or order data.
• Data investment shifts from labeling to knowledge base quality control: deduplication, removing outdated content, and preparing documents for retrieval matter more than traditional labeled training data.
• Evaluation for open-ended tasks requires deliberate, ongoing investment precisely because it's harder than checking a classification label - difficulty is a reason to invest more, not less.
• Competitive differentiation lives entirely in the application layer: prompt quality, knowledge base quality, interface design, and evaluation rigor, since the underlying foundation model is equally available to competitors.

Conclusion

Placed side by side, the product classifier and the customer support assistant demonstrate just how differently the same five-question framework resolves depending on a single upstream characterization: close-ended versus open-ended. The classifier reopened classical ML as a genuine contender and pointed toward labeled data and feature engineering; the support assistant rules classical ML out almost entirely and points toward a knowledge base, RAG, and a fundamentally harder evaluation problem.

What both scenarios share is the underlying discipline: characterize the task honestly before choosing an architecture, start with the cheapest adaptation technique that could plausibly work, gather real evidence before escalating to something heavier, and recognize that differentiation increasingly lives in the application layer rather than in the choice of foundation model itself. Teams that internalize this framework - rather than defaulting to the same architecture regardless of task shape - consistently make better-calibrated decisions across the genuinely wide range of problems that fall under the umbrella of "AI engineering."

References

1. Huyen, Chip. AI Engineering: Building Applications with Foundation Models. O'Reilly Media, 2024.
2. Huyen, Chip. Designing Machine Learning Systems. O'Reilly Media, 2022.
3. Lewis, Patrick et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS, 2020.
4. Brown, Tom B. et al. "Language Models are Few-Shot Learners." NeurIPS, 2020.
5. Ouyang, Long et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS, 2022.
6. Anthropic. "Claude API Documentation." docs.anthropic.com.
7. Hu, Edward J. et al. "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685, 2021.
8. Sculley, D. et al. "Hidden Technical Debt in Machine Learning Systems." NeurIPS, 2015.