paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

August 06, 2026

Building an AI Knowledge Base: Taming Distributed, Multi-Format, Multi-Owner Data

A practical architecture for unifying knowledge scattered across departments, formats, and access systems into a single AI-ready system

Introduction

Every organization past a certain size ends up with the same quiet problem: its knowledge is everywhere and nowhere. Engineering keeps design docs in Confluence. Sales lives in Salesforce and Google Drive. Support tickets sit in Zendesk. Finance has spreadsheets locked behind a shared drive only three people can see. HR has policies in a document management system with its own login. None of these systems were built to talk to each other, and none of their owners were thinking about AI retrieval when they chose them. When a company decides to build an AI knowledge base - typically to power a retrieval-augmented generation (RAG) system or an internal assistant - it quickly discovers that the hard part isn't the language model. It's the plumbing.

This article is about that plumbing: how to design a system that can ingest data from distributed sources owned by different departments, each with its own format, API, update cadence, and access control model, and turn it into something an AI system can query safely and accurately. We'll walk through the architecture, the practical implementation patterns, the trade-offs you'll inevitably hit, and the practices that separate a knowledge base that actually gets used from one that quietly rots after the demo.

Context and Problem Overview

The core difficulty with enterprise knowledge bases isn't technical in the narrow sense - it's organizational complexity expressed through technical constraints. Each department's data source is a product of decisions made independently, often years apart, by teams with no shared data strategy. A support team's knowledge lives in Zendesk articles formatted as HTML with embedded screenshots. A legal team's contracts live in PDFs in a DMS with strict retention rules. An engineering team's architecture decisions live in Markdown files in Git repositories, versioned and reviewed, while the product team's roadmap lives in a Notion workspace that changes hourly. None of these systems share a schema, an authentication mechanism, or even a consistent notion of what a "document" is.

Access control makes this harder still. A single sign-on provider might authenticate users across systems, but authorization is almost never centralized. Confluence spaces have their own permission groups, SharePoint sites have their own ACLs tied to Active Directory groups, and a Postgres-backed internal tool might enforce row-level security that has nothing to do with either. If your knowledge base retrieves a chunk of text from a sensitive HR document and surfaces it to an engineer who shouldn't see it, you haven't built a convenience tool - you've built a data leak machine. Any serious design has to treat access control as a first-class architectural concern, not an afterthought bolted on after indexing.

The third dimension of the problem is ownership and lifecycle. Departments don't just produce data in different formats; they also update it at different speeds, delete it under different policies, and have different tolerances for staleness. A pricing sheet that's a week old is actively harmful. An onboarding guide that's six months old is usually fine. Treating all sources as if they decay at the same rate, or indexing everything with a single batch job once a day, ignores the reality that the data's owners have very different expectations of freshness, and your system needs to reflect that rather than impose a single uniform policy.

Deep Technical Explanation: Architecture of a Federated Ingestion System

The architecture that handles this well separates concerns cleanly into four layers: source connectors, a normalization and enrichment pipeline, a storage layer that supports hybrid retrieval, and a retrieval/serving layer that enforces access control at query time. This separation matters because each layer changes at a different rate - new connectors get added constantly, normalization logic evolves as you learn what retrieval quality actually needs, and the storage layer should be stable enough that you're not re-architecting it every time marketing adopts a new SaaS tool.

Source connectors are the adapters between each departmental system and your pipeline. A connector's job is to know how to authenticate against a specific API (OAuth 2.0 client credentials for most SaaS tools, as specified in RFC 6749), how to enumerate what's changed since the last sync, and how to extract both content and metadata - including, critically, access control metadata. For systems that support webhooks (Slack's Events API, for instance), connectors can be event-driven. For systems that only support polling (many legacy on-prem tools), you need a scheduler with sensible backoff and rate-limit handling. For database-backed sources, change data capture tools such as Debezium let you stream row-level changes without hammering the source system with full scans.

The normalization layer is where raw, heterogeneous content becomes something a retrieval system can reason about. This involves format conversion (PDF and DOCX to plain text with structure preserved, HTML stripped of boilerplate), chunking (splitting long documents into retrieval-sized units, typically 200-800 tokens depending on your embedding model and use case), and enrichment (attaching metadata like source system, department, author, last-modified timestamp, and - again - access control lists). This is also where you handle deduplication: the same policy document often exists in three places because someone copied it from Confluence into Google Drive and attached it to an email that got archived into a knowledge base tool. Without deduplication, your retrieval results fill up with near-identical chunks that crowd out genuinely different information.

The storage layer is usually a vector database (Pinecone, Weaviate, Milvus, or pgvector running on top of Postgres) storing dense embeddings alongside metadata, often combined with a traditional inverted index for keyword/BM25 search. Hybrid search - combining semantic similarity with lexical matching - consistently outperforms pure vector search for enterprise content, because a lot of what people search for in internal systems is exact terms: product codes, ticket numbers, acronyms that an embedding model may not represent precisely. Elasticsearch and OpenSearch both support this pattern natively, as does Weaviate with its built-in hybrid search feature.

Implementation Patterns: Connectors, Normalization, and Metadata in Practice

In practice, the connector layer benefits enormously from a plugin-style architecture where each source implements a common interface. This keeps the orchestration logic - scheduling, retry handling, deduplication against prior syncs - separate from the source-specific extraction logic, which is where most of the real complexity lives. Here's a simplified TypeScript interface that illustrates the pattern:

interface DocumentSource {
  sourceId: string;
  department: string;

  // Returns documents changed since the given timestamp (or cursor)
  fetchChanges(since: string | null): Promise<RawDocument[]>;

  // Translates source-specific ACLs into a normalized permission model
  resolveAccess(doc: RawDocument): Promise<AccessPolicy>;
}

interface RawDocument {
  externalId: string;
  content: Buffer;
  mimeType: string;
  lastModified: string;
  metadata: Record<string, unknown>;
}

interface AccessPolicy {
  // Normalized groups/users allowed to view this document,
  // resolved from the source system's native ACL model
  allowedPrincipals: string[];
  visibility: "public" | "restricted" | "confidential";
}

class ConfluenceSource implements DocumentSource {
  sourceId = "confluence";
  department = "engineering";

  async fetchChanges(since: string | null): Promise<RawDocument[]> {
    const cql = since
      ? `lastmodified > "${since}"`
      : `type=page`;
    const results = await confluenceClient.search(cql);
    return results.map(toRawDocument);
  }

  async resolveAccess(doc: RawDocument): Promise<AccessPolicy> {
    const restrictions = await confluenceClient.getRestrictions(doc.externalId);
    return normalizeConfluenceAcl(restrictions);
  }
}

The key design decision here is resolveAccess: every connector is responsible for translating its source system's native permission model into a normalized representation your retrieval layer can enforce consistently, regardless of whether the original ACL came from Confluence space permissions, a SharePoint site's Active Directory groups, or a database's row-level security policy.

Once documents are fetched, the normalization pipeline chunks and enriches them before embedding. A production pipeline typically looks like this in Python, using a text-splitting strategy that respects document structure rather than blindly cutting at a fixed character count:

from dataclasses import dataclass
from typing import Optional
import hashlib

@dataclass
class Chunk:
    text: str
    source_id: str
    document_id: str
    department: str
    allowed_principals: list[str]
    chunk_hash: str
    section_title: Optional[str] = None

def chunk_document(doc: RawDocument, access_policy: AccessPolicy) -> list[Chunk]:
    sections = split_by_headings(doc.content)  # structure-aware split
    chunks = []
    for section in sections:
        for piece in sliding_window(section.text, max_tokens=512, overlap=64):
            chunk_hash = hashlib.sha256(piece.encode()).hexdigest()
            chunks.append(Chunk(
                text=piece,
                source_id=doc.source_id,
                document_id=doc.external_id,
                department=doc.metadata.get("department", "unknown"),
                allowed_principals=access_policy.allowed_principals,
                chunk_hash=chunk_hash,
                section_title=section.title,
            ))
    return chunks

def deduplicate(chunks: list[Chunk], seen_hashes: set[str]) -> list[Chunk]:
    unique = [c for c in chunks if c.chunk_hash not in seen_hashes]
    seen_hashes.update(c.chunk_hash for c in unique)
    return unique

Note that allowed_principals travels with every chunk all the way into storage. This is what makes query-time enforcement possible: when the retrieval layer runs a similarity search, it can filter by the requesting user's resolved group memberships before results ever reach the language model, rather than relying on the model to decide what it should or shouldn't repeat. Most vector databases support this directly as metadata filtering - Pinecone, Weaviate, and pgvector all let you combine a similarity search with a structured WHERE-style filter in a single query, which is far more reliable than post-filtering results after retrieval.

Access Control and Ownership Across Departments

Because departments own their data and their access models independently, a knowledge base that centralizes content without centralizing - or at least federating - authorization is building on sand. The pattern that works is to treat the knowledge base as a consumer of each department's existing authorization decisions, not a replacement for them. This means syncing group memberships and permission grants from source systems regularly, not just once at initial ingestion, since people change teams, contractors lose access, and documents get restricted after the fact. Stale permission caches are one of the most common causes of real incidents in systems like this: a document was made confidential three weeks ago, but the knowledge base's copy of its ACL is still the old, more permissive one.

The NIST guidance on attribute-based access control (NIST SP 800-162) is a useful reference model here, because enterprise knowledge bases rarely map cleanly onto a single role-based scheme. A document's visibility might depend on department, seniority, project assignment, and client confidentiality all at once, which is exactly the kind of multi-attribute decision ABAC is designed to express. Rather than trying to force every source system into one unified role hierarchy, normalize each source's permissions into a set of attributes and principals, and let the retrieval layer evaluate access as a policy decision at query time. This also gives you a clean audit trail: you can log exactly which attributes justified showing a given chunk to a given user, which matters enormously the first time someone asks why the assistant surfaced something it shouldn't have.

Ownership also has a governance dimension beyond access control. Someone in each department needs to be accountable for the accuracy of what gets indexed from their systems, because an AI assistant that confidently repeats outdated or wrong information erodes trust faster than one that says "I don't know". Establishing a lightweight data contract per source - who owns it, how fresh it needs to be, what counts as sensitive - gives you a basis for prioritizing ingestion work and for knowing who to contact when a chunk turns out to be wrong. This is the same discipline Eric Evans describes in Domain-Driven Design as a bounded context: each department's data has its own internal model and its own authority, and your system should respect that boundary rather than flattening everything into one global schema prematurely.

Trade-offs and Common Pitfalls

The most common mistake in these systems is over-indexing: pulling in every available source because it's technically possible, without weighing the maintenance cost against the retrieval value. Every connector you add is an ongoing commitment - API versions change, OAuth tokens expire, rate limits shift, and source schemas evolve. A knowledge base with twenty poorly maintained connectors is worse than one with five well-maintained ones, because broken or stale connectors silently degrade answer quality in ways that are hard to detect until a user notices something wrong. It's worth resisting the instinct to connect everything on day one and instead prioritizing sources by how often they're actually referenced in the questions people ask.

A second pitfall is conflating chunking strategy across formats. A chat transcript, a legal contract, and an API reference document all have fundamentally different structures, and chunking them the same way degrades retrieval quality for all three. Chat transcripts need turn-aware chunking that preserves conversational context; contracts need clause-level boundaries so a chunk doesn't cut a legal definition in half; API docs benefit from keeping code examples intact alongside their surrounding explanation. Teams that apply one fixed-size splitter to everything often get retrieval results that are technically similar but practically useless, because the chunk boundary landed in the middle of the one sentence that mattered.

Freshness versus cost is a trade-off that's easy to underestimate. Real-time sync for every source sounds appealing, but re-embedding and re-indexing on every change is expensive at scale, both in compute and in API rate-limit budget against source systems. Most production systems settle on tiered freshness: near-real-time for high-value, fast-changing sources like support tickets or incident channels, and batched daily or weekly syncs for slower-moving sources like HR policy documents. Getting this tiering wrong in either direction either burns unnecessary compute or leaves users looking at stale answers without realizing it.

Best Practices for Production Systems

Build observability into the pipeline from day one, treating ingestion failures with the same seriousness as application errors. A connector silently failing to sync for two weeks is a worse failure mode than a connector that's never been built, because it creates false confidence that the data is current. Track per-source metrics: last successful sync time, document count drift, and chunk-level freshness, and alert when any of these drift beyond an expected threshold.

Version your embeddings and your chunking logic together, not independently. If you change your chunking strategy or swap embedding models, old and new chunks become incompatible in subtle ways - similarity scores aren't comparable across embedding model versions, so a retrieval index with chunks embedded by two different model versions produces unpredictable ranking. Treat an embedding model upgrade as a full re-index operation, and keep a migration plan for it rather than letting versions drift silently.

Favor normalized, source-agnostic metadata schemas over source-specific ones, even though it requires more upfront translation work per connector. A unified schema - department, owner, sensitivity level, allowed principals, last modified, source system - makes your retrieval and filtering logic simple and consistent, whereas special-casing logic for "if source is Confluence, check this field; if source is SharePoint, check that one" compounds into unmaintainable complexity as you add sources.

A Useful Mental Model

It helps to think of this system less like a database and more like a newsroom's wire service. A newsroom doesn't generate all its own content - it pulls in feeds from many independent sources (wire services, local stringers, press releases), each with its own format and credibility, and an editorial layer normalizes, verifies, and tags that content before it goes out under the publication's own voice. Your ingestion and normalization layer plays the editor's role: it doesn't trust every source equally, it checks freshness and provenance, and it attaches the context (who said this, when, and who's allowed to see it) that the final consumer - in this case, your retrieval system and the LLM - needs to use the information responsibly.

This analogy also clarifies why access control can't be bolted on at the end. A wire service wouldn't publish a confidential source's leaked memo just because it arrived in the same feed as public press releases; the editorial process is precisely what separates what's publishable from what isn't, source by source, before anything reaches a reader. Treating permission resolution as part of the same pipeline stage as content normalization, rather than a separate concern handled later, mirrors that same discipline and prevents the single most damaging failure mode in enterprise AI systems: an assistant that repeats something it was never supposed to see.

The 80/20 Insight

If you strip this problem down to the handful of decisions that determine most of the outcome, they are: get access control propagation right before optimizing retrieval quality, because a fast and accurate system that leaks data is a liability, not a feature; normalize metadata early so every downstream component - chunking, embedding, filtering, citation - works off one consistent schema instead of source-specific logic scattered everywhere; and prioritize connectors by actual query demand rather than source availability, since a small number of frequently-referenced sources (support docs, engineering wikis, policy documents) typically drive the overwhelming majority of useful answers. Teams that get these three decisions right early tend to avoid the rewrites that teams chasing broader source coverage or fancier retrieval algorithms usually run into later.

Key Takeaways

  1. Treat access control as a pipeline-stage concern, not a post-processing filter - resolve and attach permissions at ingestion time, and enforce them at query time via metadata filtering in your vector database.
  2. Build connectors behind a common interface so source-specific complexity (auth, pagination, format quirks) stays isolated and doesn't leak into your normalization or retrieval logic.
  3. Use format-aware, structure-respecting chunking rather than one fixed-size splitter applied uniformly across chat logs, contracts, and technical docs.
  4. Establish per-source data contracts with clear ownership so freshness expectations and sensitivity levels are explicit rather than assumed.
  5. Monitor ingestion health (sync recency, document drift, failed syncs) as rigorously as you monitor application uptime - stale data fails silently and erodes trust gradually.

Conclusion

Building an AI knowledge base from distributed, departmentally-owned data is fundamentally a systems integration problem wearing an AI costume. The embedding model and the retrieval algorithm matter, but they're the easy part relative to building reliable connectors, a normalization pipeline that respects format differences, and an access control model that faithfully reflects the permission boundaries each department already enforces. Organizations that treat this as primarily a machine learning project tend to underinvest in the ingestion and governance layers, and pay for it later in the form of stale answers, broken connectors, or - worse - an assistant that surfaces something it never should have.

The organizations that get this right treat the knowledge base as living infrastructure with ongoing operational ownership, not a one-time ETL job that runs and is forgotten. That means budgeting for connector maintenance, periodic re-indexing as embedding models improve, and continuous collaboration with the departments that own the underlying data. Done well, the result isn't just a chatbot that can answer questions - it's a system that makes an organization's collective knowledge genuinely queryable, safely, for the first time.

References