paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

August 30, 2026

Building a Self-Maintaining AI Knowledge Base for the Enterprise

Architecture patterns for ingesting, securing, and continuously refreshing knowledge from distributed, heterogeneous data sources - without manual babysitting

Introduction

Every organization past a certain size ends up with the same problem: knowledge is everywhere and nowhere. Product specs live in Confluence, support history lives in Zendesk, contracts live in SharePoint, customer records live in Salesforce, and architecture decisions live in a dozen Slack threads nobody can find again. When teams try to build an AI assistant or retrieval-augmented generation (RAG) system on top of this landscape, the hard part is rarely the language model. The hard part is building a knowledge base that accurately reflects a constantly moving set of source systems - one where documents get created, edited, archived, and deleted every hour, where access depends on who's asking, and where "freshness" means something different for a legal policy than for a Slack message.

This article is about the engineering discipline behind that problem: how to build an AI knowledge base that keeps itself in sync with its sources over time, without a human re-running an ingestion script every time something changes upstream. We'll walk through the architectural patterns that make this tractable - change data capture, incremental indexing, policy propagation, and freshness-aware retrieval - along with the pitfalls that tend to sink these systems in production. The goal isn't a single reference architecture you copy verbatim; it's a vocabulary and a set of trade-offs you can apply to your own mix of sources.

The Problem: Enterprise Knowledge Fragmentation

Enterprise knowledge is distributed across systems that were never designed to talk to each other. A typical mid-size company might have structured data in Postgres and Salesforce, semi-structured content in Jira and Zendesk, unstructured documents in SharePoint and Google Drive, and conversational data in Slack and Microsoft Teams. Each of these systems has its own API shape, its own rate limits, its own notion of "what changed," and its own authentication model. Some expose webhooks; others only support polling. Some have robust change feeds; others require you to diff full exports to figure out what's different.

Layered on top of format heterogeneity is ownership and access heterogeneity. A document in SharePoint might be visible to one department, a Salesforce record might be scoped to an account team, and a Confluence page might be restricted to a named group of editors. If your knowledge base flattens all of this into one undifferentiated vector index, you've built a data leak generator, not a knowledge base. The system has to carry access metadata through every stage of the pipeline - ingestion, chunking, embedding, storage, and retrieval - and enforce it at query time, not just at ingestion time.

The third axis of complexity is freshness. Not all content decays at the same rate. A pricing page might be stale after a week; a legal contract might be valid for years; a support ticket might be irrelevant within days of resolution. A naive full-reindex-on-schedule approach either wastes enormous compute re-embedding unchanged content, or leaves genuinely stale information in the index because the schedule wasn't tight enough. Getting freshness right requires per-source, sometimes per-document, policies - and a mechanism to detect and propagate deletions, which is the piece most systems get wrong first.

Architecture Overview

At a high level, a self-maintaining knowledge base has five layers: source connectors, a change detection/capture layer, a normalization and enrichment pipeline, a storage layer (typically a vector store plus a metadata store), and a retrieval layer that applies access control and freshness filtering at query time. The critical design decision is that these layers communicate through events, not through periodic batch jobs that re-read everything. Treating ingestion as a stream of change events - creates, updates, deletes - rather than a sequence of full snapshots is what allows the system to stay current without re-processing the entire corpus on every run.

Most production systems converge on a similar shape: a connector per source system that knows how to extract both content and change signals; a durable event log (often Kafka, a managed equivalent like AWS Kinesis or Confluent Cloud, or a simpler queue like SQS for lower-volume sources) that decouples extraction from processing; a worker pool that chunks, embeds, and writes to storage; and a metadata layer - frequently a relational database - that tracks document lineage, source IDs, content hashes, access control lists, and last-synced timestamps. This metadata layer is what makes the system idempotent and debuggable: when something looks wrong in retrieval, you can trace a chunk back to its source document, its last sync time, and the exact revision it was derived from.

Ingestion and Change Data Capture

The core enabling pattern here is change data capture (CDC) - detecting and propagating changes in source systems as they happen, rather than inferring them from periodic full scans. CDC takes different forms depending on what the source system offers. Databases like Postgres or MySQL support log-based CDC through tools like Debezium, which reads the write-ahead log or binlog and emits a stream of insert/update/delete events with minimal load on the source. SaaS systems like Salesforce, Confluence, or Zendesk typically expose either webhooks (push-based, near-real-time) or delta/changelog APIs (pull-based, often using a cursor or timestamp parameter) that you poll on an interval appropriate to that source's rate of change.

Deletes are the pattern most teams get wrong, because many APIs simply stop returning a record rather than telling you it was deleted. If your connector only processes "here's what changed," it will never learn that a document disappeared, and your knowledge base will accumulate orphaned content indefinitely. The reliable pattern is reconciliation: periodically compare the full set of IDs known to the source against the full set of IDs tracked in your metadata store, and treat anything present in your store but absent from the source as a tombstone candidate. This doesn't need to run as often as your incremental sync - daily or weekly reconciliation passes are usually enough to catch drift that webhooks or delta APIs missed.

Here's a simplified connector pattern in TypeScript that illustrates the shape: incremental pull using a stored cursor, content hashing to avoid redundant re-embedding, and emission of typed change events onto a queue.

interface SourceDocument {
  sourceId: string;
  sourceSystem: string;
  content: string;
  contentHash: string;
  accessControl: AccessControlEntry[];
  updatedAt: Date;
}

interface ChangeEvent {
  type: "upsert" | "delete";
  sourceId: string;
  sourceSystem: string;
  document?: SourceDocument;
}

async function syncConfluenceSpace(
  spaceKey: string,
  cursor: SyncCursor,
  eventBus: EventPublisher,
  metadataStore: MetadataStore
): Promise<SyncCursor> {
  const pages = await confluenceClient.getPagesUpdatedSince(
    spaceKey,
    cursor.lastSyncedAt
  );

  for (const page of pages) {
    const contentHash = hashContent(page.body.storage.value);
    const existing = await metadataStore.getBySourceId(page.id);

    // Skip re-embedding if content hasn't actually changed -
    // Confluence sometimes reports "updated" on metadata-only edits.
    if (existing && existing.contentHash === contentHash) {
      continue;
    }

    const doc: SourceDocument = {
      sourceId: page.id,
      sourceSystem: "confluence",
      content: page.body.storage.value,
      contentHash,
      accessControl: await resolveConfluencePermissions(page.id),
      updatedAt: new Date(page.version.when),
    };

    await eventBus.publish({ type: "upsert", sourceId: page.id, sourceSystem: "confluence", document: doc });
  }

  // Reconciliation pass: find IDs we track that no longer exist upstream.
  const knownIds = await metadataStore.listSourceIds("confluence", spaceKey);
  const liveIds = new Set(await confluenceClient.listAllPageIds(spaceKey));
  for (const id of knownIds) {
    if (!liveIds.has(id)) {
      await eventBus.publish({ type: "delete", sourceId: id, sourceSystem: "confluence" });
    }
  }

  return { lastSyncedAt: new Date() };
}

This pattern generalizes: every connector produces the same ChangeEvent shape regardless of source, which means everything downstream - chunking, embedding, indexing - can be source-agnostic. The connector is the only layer that needs to know Confluence's API quirks or Salesforce's SOQL syntax.

Access Control and Policy Propagation

Access control in a knowledge base is not a feature you bolt on at the end - it has to be a first-class attribute of every chunk from the moment it's ingested. The common failure mode is building a single shared vector index and trying to filter results after retrieval based on the requesting user's permissions. This is both a security risk (relevance scoring and top-k truncation can leak the existence of content even if the content itself is redacted) and a correctness problem, because by the time you've filtered, you may have already discarded the chunks the user was actually allowed to see in favor of ones they weren't.

The more robust pattern is to resolve access control lists (ACLs) at ingestion time, store them as structured metadata alongside each chunk, and apply them as a pre-filter at query time - most modern vector databases (Pinecone, Weaviate, pgvector with row-level security, Milvus) support metadata filtering that can be combined with the similarity search itself, so the filter narrows the candidate set before ranking rather than after. This means your retrieval query looks less like "find the 10 most similar chunks" and more like "find the 10 most similar chunks among those this user is authorized to see." Where source systems support group or role-based permissions (Google Workspace groups, SharePoint permission inheritance, Salesforce sharing rules), it's worth mapping those into your own normalized ACL model rather than trying to re-derive permission logic ad hoc for every source - SCIM (System for Cross-domain Identity Management, RFC 7643/7644) is a reasonable standard to lean on if you're also centralizing identity and group membership.

Propagation matters as much as resolution: if a document's permissions change upstream - someone is removed from a Confluence space, or a Salesforce sharing rule is tightened - that change needs to flow into your metadata store on roughly the same cadence as content changes, not as an afterthought during the next full resync. Treating permission changes as their own category of change event, alongside content upserts and deletes, keeps this from becoming a stale-ACL time bomb.

Freshness, Versioning, and Conflict Resolution

Freshness policy should be configured per source, and sometimes per document type within a source, rather than applied uniformly across the whole knowledge base. A sensible starting point is to classify sources into tiers: high-volatility (Slack, support tickets, ticketing systems) that might need near-real-time sync via webhooks; medium-volatility (wikis, product docs) that tolerate hourly or daily polling; and low-volatility (signed contracts, compliance policies, architecture decision records) where correctness and auditability matter more than latency, and where you might even want versioned, immutable snapshots rather than overwrite-in-place updates.

Versioning also protects you against a subtler problem: concurrent updates from the same source producing inconsistent state in your index. If a document is updated twice in quick succession and your pipeline processes the events out of order - which is a realistic possibility with distributed queues and retrying workers - you can end up indexing a stale version after a newer one, with no error raised anywhere. The fix is to track a monotonically increasing version or updatedAt timestamp from the source system itself (not your own ingestion timestamp) and have the write path to your metadata store perform a conditional update: only overwrite if the incoming version is newer than what's stored. This is the same optimistic concurrency pattern used in distributed databases, and it costs little to implement.

def upsert_chunk_with_version_check(
    metadata_store,
    chunk_id: str,
    source_updated_at: datetime,
    embedding: list[float],
    access_control: list[AccessControlEntry],
) -> bool:
    """
    Returns True if the chunk was written, False if a newer version
    already exists and this write was skipped.
    """
    existing = metadata_store.get(chunk_id)

    if existing is not None and existing["source_updated_at"] >= source_updated_at:
        # A newer or equal version is already indexed; this event
        # arrived late or out of order. Skip silently but log it.
        logger.info(f"Skipping stale write for {chunk_id}: "
                    f"existing={existing['source_updated_at']}, "
                    f"incoming={source_updated_at}")
        return False

    metadata_store.upsert(
        chunk_id,
        {
            "embedding": embedding,
            "access_control": access_control,
            "source_updated_at": source_updated_at,
            "indexed_at": datetime.utcnow(),
        },
    )
    return True

Expiration deserves equal attention to insertion. Many teams implement thorough ingestion pipelines and then never build the corresponding decay logic - so content that should have been archived six months ago still surfaces in retrieval. A simple and effective pattern is to attach a time-to-live or a "review by" date to freshness-sensitive content classes, and either down-weight or exclude chunks past that threshold at query time, while leaving an audit trail so a human can confirm whether expired content should be removed or refreshed.

Implementation Walkthrough

Tying the pieces together, a practical pipeline looks like: connectors emit ChangeEvents onto a durable queue; a worker consumes events, performs chunking and embedding, and writes to the vector store and metadata store inside the version-checked upsert described above; and a separate reconciliation job runs on a slower cadence to catch anything the event-driven path missed (deletes without explicit signals, webhooks that silently failed, permission changes that weren't captured as discrete events). Idempotency is non-negotiable at every stage - events can and will be delivered more than once in any system built on queues, so every write needs to be safe to repeat.

The chunking and embedding worker is a good place to also enforce schema consistency across sources, since different connectors will hand it content in different shapes. Normalizing early means every downstream consumer - including the retrieval layer, evaluation tooling, and any UI that shows source citations back to users - can rely on a single document and chunk schema regardless of whether the content originated in SharePoint or Slack.

async def process_change_event(event: ChangeEvent, vector_store, metadata_store, embedder):
    if event.type == "delete":
        await vector_store.delete_by_source_id(event.source_id, event.source_system)
        await metadata_store.mark_deleted(event.source_id, event.source_system)
        return

    doc = event.document
    chunks = chunk_document(doc.content, max_tokens=512, overlap_tokens=50)

    embeddings = await embedder.embed_batch([c.text for c in chunks])

    for chunk, embedding in zip(chunks, embeddings):
        chunk_id = f"{doc.source_system}:{doc.source_id}:{chunk.index}"
        written = upsert_chunk_with_version_check(
            metadata_store,
            chunk_id=chunk_id,
            source_updated_at=doc.updated_at,
            embedding=embedding,
            access_control=doc.access_control,
        )
        if written:
            await vector_store.upsert(
                id=chunk_id,
                vector=embedding,
                metadata={
                    "source_system": doc.source_system,
                    "source_id": doc.source_id,
                    "access_control": [ace.to_dict() for ace in doc.access_control],
                    "updated_at": doc.updated_at.isoformat(),
                    "text": chunk.text,
                },
            )

Pitfalls and Trade-offs

The most common failure mode is treating ingestion as a one-time project rather than an ongoing system. Teams build an impressive initial index, demo it successfully, and then discover six months later that half the content is stale or orphaned because nobody built the deletion-reconciliation or freshness-decay logic - it felt like a "nice to have" during the initial build and never got prioritized. Budgeting for the maintenance machinery from day one, even in a minimal form, is cheaper than retrofitting it onto a system that's already serving production traffic with increasingly wrong answers.

A second pitfall is under-investing in connector robustness relative to the sophistication of the retrieval and generation layers. It's common to see teams spend weeks tuning chunk sizes, embedding models, and reranking strategies while their Confluence connector silently drops pages whenever the API returns a paginated response with more than a few hundred results, or their Salesforce connector hits governor limits during business hours and fails without alerting anyone. Source connectors are unglamorous, but they're where data quality is actually determined - no amount of retrieval tuning compensates for a connector quietly missing 20% of updates.

Over-indexing is a subtler trade-off. Not every piece of content in every source system belongs in the knowledge base, and the temptation to ingest everything "just in case" tends to degrade retrieval quality by diluting the index with low-value content, while also expanding the access-control surface you have to get right. It's worth treating source onboarding as a deliberate decision with an owner, a defined freshness policy, and explicit scoping of what subset of that source is actually relevant - rather than defaulting to "connect everything." Finally, teams frequently underestimate the cost of re-embedding at scale when a model is upgraded; content-hash-based change detection helps avoid unnecessary re-embeds during normal operation, but a full model migration is still a significant, deliberately-planned batch operation, not something to trigger casually.

Best Practices

A few practices consistently separate systems that stay healthy in production from ones that quietly rot. First, make every pipeline stage idempotent and instrument it with structured logging keyed by source ID and chunk ID, so that when retrieval surfaces something wrong, you can trace it back to the exact ingestion event responsible within minutes rather than hours. Second, build the deletion-reconciliation job before you ship the initial version of the system, not after - it's the single piece of maintenance logic most likely to be skipped under deadline pressure, and its absence is the hardest kind of bug to notice because the symptom is "slightly too much stale content," not an outright failure.

Second-order practices matter too: define access control as a normalized, source-agnostic model early, even if your first one or two connectors have simple permission structures, because retrofitting ACL propagation onto an index that was built without it is substantially more painful than designing for it upfront. Treat freshness as a per-source configuration surface rather than a global constant, and expose it to the teams who own each source so they can set review cadences that reflect how that content actually changes. And invest in observability specifically for pipeline health - lag between source change and index update, failed sync counts per connector, and reconciliation drift - the same way you'd instrument any other production data pipeline, because a RAG system with silently stale data is worse than one that visibly fails, since users trust wrong answers more than they trust an error message.

Mental Model: The Knowledge Base as a Materialized View

A useful way to think about this entire system is as a materialized view over your organization's source systems, in the database sense: a derived, queryable structure that's kept consistent with its underlying sources through incremental updates rather than being recomputed from scratch. Just as a materialized view in Postgres needs a refresh strategy (full refresh, incremental refresh via triggers, or logical replication), your knowledge base needs an analogous strategy per source - and just as a stale materialized view silently returns wrong answers without erroring, a stale knowledge base silently generates wrong AI responses without any obvious failure signal.

This framing is useful because it imports a mature set of concerns that database engineers have already solved: consistency guarantees (how stale can the view be relative to the source?), refresh cost (can you refresh incrementally, or do you need a full rebuild?), and conflict resolution under concurrent writes. Borrowing this vocabulary when designing your ingestion system - rather than inventing new terminology from scratch - makes it easier to reason about trade-offs with the same rigor you'd apply to any other data consistency problem, and easier to communicate those trade-offs to engineers who haven't worked on RAG systems before but have worked on data pipelines.

The 80/20 of Self-Maintaining Knowledge Bases

If you can only invest deeply in a few things, three disproportionately determine whether this kind of system stays healthy over time. The first is reliable change detection per source - webhooks where available, delta APIs with cursors where not, and a periodic reconciliation pass to catch deletions and drift that neither mechanism reports cleanly. The second is content-hash-based idempotency combined with version-checked writes, which together make the entire pipeline safe to retry, replay, and run concurrently without producing inconsistent state. The third is treating access control as ingestion-time metadata enforced as a pre-filter at query time, rather than a post-hoc check - this single decision prevents the majority of security incidents these systems are prone to.

Everything else - chunking strategy, embedding model choice, reranking, prompt engineering for the generation step - matters for answer quality, but it operates on top of a foundation that these three patterns establish. A system with mediocre chunking but solid change detection, idempotency, and access control will degrade gracefully and stay trustworthy. A system with excellent chunking but weak foundations will produce impressive demos and unreliable production behavior.

Key Takeaways

Conclusion

Building an AI knowledge base that answers questions well on day one is a tractable engineering problem; building one that stays accurate, secure, and current on day three hundred is a genuinely different and harder problem, and it's the one that determines whether the system survives contact with a real organization. The patterns that make this work - event-driven change capture, idempotent and version-aware writes, ingestion-time access control enforcement, and per-source freshness policy - are not exotic. Most of them are adaptations of patterns the data engineering and distributed systems communities have been refining for decades, applied to the specific shape of retrieval-augmented AI systems.

The teams that get this right tend to treat the knowledge base less like a one-time ETL project and more like a long-lived distributed system with its own SLAs: lag between source change and index update, deletion propagation latency, and permission-sync accuracy are all metrics worth tracking continuously, not just checking once at launch. Investing early in connector robustness, reconciliation jobs, and access control plumbing - even when it feels like infrastructure overhead relative to the more visible work of tuning retrieval and generation - is what separates a knowledge base that quietly degrades over months from one that remains a system people can actually trust.

References