Introduction
Most teams that start building an AI assistant discover the same thing within a few weeks: the model is the easy part. A capable large language model can reason, summarize, and write code, but it knows nothing about your internal runbooks, your customer contracts, or the decision someone made in a meeting last Tuesday. Every answer it gives about your organization is either a guess or a product of whatever context you manage to place in front of it. The engineering problem, then, is not primarily about the model. It is about building the system that turns raw organizational data into knowledge the model can find, interpret, and use correctly.
This is what people now mean by an AI knowledge base. The term covers more than a vector database attached to a chatbot. It describes an architecture that ingests documents, databases, and events; normalizes them; extracts structure; stores them in representations suited to different kinds of questions; and serves them to agents under explicit constraints about freshness, permissions, and provenance. Retrieval-augmented generation (RAG) is the most widely adopted pattern in this space, and knowledge graphs are the most widely discussed complement to it. Neither is sufficient alone for most real systems, and both come with costs that are easy to underestimate.
This article walks through the problem from the ground up. We start with why naive approaches fail, then examine how RAG works in practice and where it breaks. We look at knowledge graphs as a structured alternative, then at hybrid representations that combine them. We include implementation patterns in TypeScript and Python, discuss trade-offs and pitfalls, and close with practices that hold up in production. The goal is not to crown a winner but to give you a mental model for choosing the right combination for your data and your questions.
Context and the Problem Overview
Before choosing a technique, it helps to be precise about the failure modes you are trying to avoid. An AI agent answering questions over organizational data can go wrong in at least four distinct ways. It can lack the information entirely, because the relevant document was never ingested or was ingested too late. It can have the information but fail to retrieve it, because the query and the passage use different vocabulary or because the relevant fact is buried in a long document. It can retrieve the right passage but misinterpret it, because the passage refers to "the new policy" without saying which policy or when it took effect. And it can answer from information the requesting user is not authorized to see, which is a security problem rather than a quality problem.
These failure modes call for different fixes, and conflating them is a common source of wasted effort. Missing data is an ingestion problem. Retrieval failures are an indexing and query-understanding problem. Misinterpretation is a problem of context: metadata, versioning, and the relationships between entities. Authorization failures are an access control problem that must be enforced at retrieval time, not hoped for in the prompt. A knowledge base architecture is essentially a set of decisions about which of these problems you are solving and with what machinery.
There is also a question of what kind of question users ask. Some questions are semantic and open-ended: "what have we said about data residency in enterprise contracts?" These are well suited to similarity search over text. Others are relational and precise: "which customers on the enterprise plan are using the deprecated export API and have renewal dates in Q4?" These are poorly served by similarity search, because the answer depends on joining facts across entities and applying exact filters. Many production systems fail because they treat all questions as the first kind. Understanding the mix of question types in your domain is the first real design step.
Deep Technical Explanation
Retrieval-Augmented Generation
RAG, introduced in the 2020 paper by Lewis and colleagues at Facebook AI Research, combines a parametric language model with a non-parametric memory accessed through retrieval. In the original formulation, a dense retriever selects relevant passages, and the generator conditions on them to produce output. The pattern has since been adapted into a standard production architecture that most practitioners implement in a simplified form: documents are split into chunks, each chunk is converted into an embedding vector, the vectors are stored in an index, and at query time the query is embedded and the nearest chunks are retrieved and placed in the model's context window.
The appeal is obvious. The knowledge lives outside the model, so it can be updated without retraining, it can be attributed to sources, and access control can be applied to the retrieval step. The mechanics, however, hide several decisions that determine quality. Chunking strategy matters more than most teams expect: chunks that are too small lose the context needed to interpret them, and chunks that are too large dilute the embedding and waste context budget. Splitting on fixed token counts often cuts across sentences, tables, and code blocks. Structure-aware chunking, which respects headings and document hierarchy, usually performs better, and each chunk should carry metadata such as source, section path, version, timestamp, and access labels.
Embedding model choice is the second major decision. Embeddings capture semantic similarity, which is useful for paraphrased queries but can also retrieve passages that are topically close and factually irrelevant. Dense retrieval is weak on exact identifiers such as error codes, product SKUs, and function names, where lexical matching is more reliable. This is the main reason hybrid retrieval, which combines dense vectors with sparse keyword scoring such as BM25, has become a default in serious systems. Reciprocal rank fusion is a simple and robust way to merge the two ranked lists without needing to calibrate their scores against each other.
Retrieval quality is then usually improved by a reranker, a cross-encoder model that scores each query-passage pair more accurately than the bi-encoder used for initial retrieval, at higher cost. A typical pipeline retrieves perhaps fifty candidates cheaply, reranks them, and passes the top handful to the generator. Query transformation, such as rewriting a conversational follow-up into a standalone query or generating multiple paraphrases, can also recover recall that a single embedding misses. Each of these stages is a place where the system can be measured and improved independently, which is one of the practical advantages of decomposing the pipeline.
Knowledge Graphs
A knowledge graph represents information as entities and typed relationships, typically expressed as subject-predicate-object triples. "Customer A has contract C", "contract C includes feature F", "feature F was deprecated in release R" is a small graph that can answer questions no chunk of text answers directly. The W3C's RDF and OWL standards define one formal family of such representations, and property graph models used by systems such as Neo4j offer a more application-oriented alternative. The key property is that relationships are first-class and can be traversed, filtered, and aggregated with precision.
The strength of a knowledge graph is exactness and explainability. A query like "all customers affected by the deprecation of feature F" is a traversal, and its result set can be enumerated and audited. The graph also supports multi-hop reasoning naturally: the path from a support ticket to a product component to an owning team is a sequence of edges, and the system can show that path as justification. Entity resolution, which determines that "Acme Corp", "ACME", and "Acme Corporation, Inc". refer to the same node, is central to this value, and it is often the hardest part of building the graph in the first place.
The cost is that knowledge graphs require schema design and extraction. Someone must decide which entity types and relationship types matter, and an extraction pipeline must populate them from unstructured text, typically using a combination of rules, classical NLP, and language models. LLM-based extraction has made this much more practical, but it introduces its own error profile: extracted triples can be plausible and wrong, and errors compound when the graph is used for reasoning. Graphs also struggle with the open-ended, prose-heavy knowledge that makes up most of an organization's documentation. Representing the nuance of a design rationale as triples loses much of what made it useful.
Hybrid Knowledge Representation
The most productive systems in practice do not choose between these approaches. They store the same underlying knowledge in multiple representations and route queries to the representation best suited to them, or combine evidence from several. A common arrangement keeps raw documents in object storage with rich metadata in a relational or document store, maintains a vector index over chunks for semantic retrieval, and extracts a knowledge graph of key entities and relationships that links chunks to the entities they mention. Retrieval can then start from a vector search, expand through the graph to related entities, and pull in linked passages. This pattern is sometimes called GraphRAG in the literature and tooling, though implementations vary considerably.
The advantage of this arrangement is that each store compensates for the others' weaknesses. The graph provides precise filters and relationship traversal. The vector index provides recall for vaguely phrased questions. The document store preserves the original text, which is essential for citation and for avoiding the drift that comes from answering only from extracted summaries. The metadata layer enforces permissions and freshness consistently across all of them. The cost is operational complexity: you now have multiple indexes that must be kept consistent with the source of truth, and a bug in one can produce answers that look authoritative but are assembled from stale or mismatched pieces.
A useful way to reason about the design is to ask, for each category of question, which representation should be the authority. For factual lookups on structured attributes, the relational or graph store should be authoritative and the model should be told to trust it. For explanatory questions about reasoning or history, the document text should be authoritative and the graph should only help locate it. Making these authority rules explicit in code, rather than leaving them to the model's discretion, is one of the more important habits to develop.
Implementation and Practical Examples
The following examples sketch the core of a hybrid retrieval layer. They are deliberately focused on the parts that tend to be done poorly: merging results from multiple retrievers, enforcing access control before content reaches the model, and carrying provenance through to the answer. They omit ingestion and storage details, which vary widely by stack.
The first example shows a TypeScript retrieval function that runs dense and sparse search in parallel, filters by permissions, merges results with reciprocal rank fusion, and expands candidates through a graph lookup.
type Chunk = {
id: string;
text: string;
source: string;
sectionPath: string[];
updatedAt: string;
accessGroups: string[];
};
interface Retriever {
search(query: string, k: number, filter: AccessFilter): Promise<Chunk[]>;
}
interface GraphIndex {
relatedChunkIds(chunkIds: string[], maxHops: number): Promise<string[]>;
}
interface AccessFilter {
userGroups: string[];
}
const RRF_K = 60;
function reciprocalRankFusion(lists: Chunk[][], topN: number): Chunk[] {
const scores = new Map<string, { chunk: Chunk; score: number }>();
for (const list of lists) {
list.forEach((chunk, rank) => {
const contribution = 1 / (RRF_K + rank + 1);
const existing = scores.get(chunk.id);
if (existing) {
existing.score += contribution;
} else {
scores.set(chunk.id, { chunk, score: contribution });
}
});
}
return [...scores.values()]
.sort((a, b) => b.score - a.score)
.slice(0, topN)
.map((entry) => entry.chunk);
}
export async function hybridRetrieve(
query: string,
filter: AccessFilter,
deps: { dense: Retriever; sparse: Retriever; graph: GraphIndex; fetchChunks: (ids: string[]) => Promise<Chunk[]> },
): Promise<Chunk[]> {
// Access filtering happens inside each retriever, before any content is returned.
const [denseHits, sparseHits] = await Promise.all([
deps.dense.search(query, 50, filter),
deps.sparse.search(query, 50, filter),
]);
const seeds = reciprocalRankFusion([denseHits, sparseHits], 10);
// Expand through the graph to pick up passages about linked entities.
const expandedIds = await deps.graph.relatedChunkIds(
seeds.map((c) => c.id),
1,
);
const expanded = await deps.fetchChunks(expandedIds);
// Re-check permissions on expanded content; graph edges must not bypass access control.
const allowed = expanded.filter((c) =>
c.accessGroups.some((g) => filter.userGroups.includes(g)),
);
return reciprocalRankFusion([seeds, allowed], 12);
}
The second example, in Python, shows how to assemble the prompt so that provenance and freshness are visible to the model and to the user, and so that the model is instructed to distinguish between sources rather than blending them.
from dataclasses import dataclass
from datetime import datetime
@dataclass
class RetrievedChunk:
chunk_id: str
text: str
source: str
section_path: list[str]
updated_at: datetime
def build_context(chunks: list[RetrievedChunk], max_chars: int = 12000) -> str:
"""Render retrieved chunks with stable citation IDs and freshness metadata.
Chunks are rendered in rank order and truncated by character budget.
Each block is labeled so the model can cite it and the caller can
map citations back to source documents.
"""
blocks: list[str] = []
used = 0
for chunk in chunks:
header = (
f"[{chunk.chunk_id}] source={chunk.source} "
f"section={' > '.join(chunk.section_path)} "
f"updated={chunk.updated_at.date().isoformat()}"
)
block = f"{header}\n{chunk.text.strip()}"
if used + len(block) > max_chars:
break
blocks.append(block)
used += len(block)
return "\n\n".join(blocks)
SYSTEM_INSTRUCTIONS = """You answer questions using only the provided context blocks.
Each block has an ID in square brackets. Cite the IDs you rely on.
If blocks disagree, prefer the most recently updated one and say so.
If the context does not contain the answer, say that you do not know".""
def build_messages(question: str, chunks: list[RetrievedChunk]) -> list[dict]:
context = build_context(chunks)
return [
{"role": "system", "content": SYSTEM_INSTRUCTIONS},
{
"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {question}",
},
]
A few details in these examples matter more than they appear. Permissions are enforced inside retrieval and re-checked after graph expansion, because a graph edge is a path that can reach content the user should never see. Citation IDs are stable across the pipeline, so an answer can be audited back to the exact text and version it used. Freshness is included in the context rather than left implicit, which gives the model something to reason about when sources conflict. And the system prompt instructs the model to refuse when the context is insufficient, which is a behavior that must be tested rather than assumed.
For structured questions, you would add a separate path that translates the question into a query against the relational or graph store. Some teams let the model generate such queries directly, which can work well with a constrained schema and a read-only role, but the generated query should be validated and executed under the user's identity, not a service account with broad privileges. Treating text-to-query generation as an untrusted input, with the same scrutiny applied to SQL injection, avoids a class of problems that is easy to miss in demos.
Alternatives to RAG and Graph-Centric Designs
RAG and knowledge graphs are not the only options, and sometimes they are the wrong ones. For a small, relatively static corpus, such as a product manual or a set of policy documents that fit comfortably within a model's context window, the simplest effective design may be to place the documents directly in the prompt, possibly with prompt caching to control cost. This avoids an entire retrieval pipeline and removes a class of retrieval errors, at the price of higher per-request cost and a hard ceiling on corpus size.
Fine-tuning is sometimes proposed as a way to teach a model an organization's knowledge. It is generally a poor fit for factual knowledge that changes, because the model's parameters become a stale and unverifiable snapshot with no provenance. Fine-tuning is better suited to changing behavior, style, or output format. Continued pretraining on domain text has similar limitations for facts. Where fine-tuning does help is in improving the model's use of retrieved context, such as teaching it a particular citation format, though this is an optimization on top of a retrieval system rather than a substitute for one.
Other alternatives sit closer to conventional data engineering. A well-designed relational schema with full-text search, queried through tool-calling agents, can answer a surprising share of organizational questions, especially when the data is already structured. Search engines such as Elasticsearch or OpenSearch provide mature lexical and increasingly vector-capable retrieval with rich filtering. In many organizations the best first version is an agent with a small set of well-described tools over existing systems of record, with retrieval added only where those tools fall short. The principle is to use the least complex representation that answers your actual questions.
Trade-offs and Pitfalls
The most common pitfall is building a retrieval system before measuring anything. Without an evaluation set of realistic questions paired with the passages or answers that should be produced, teams end up tuning chunk sizes and embedding models based on impressions from a few queries. Build a small labeled set early, perhaps a hundred questions drawn from real usage or from subject-matter experts, and measure retrieval recall separately from answer quality. Retrieval recall tells you whether the right material reached the model; answer quality tells you whether the model used it correctly. When answers are wrong, these two measurements point to very different fixes.
A second pitfall is stale indexes. Ingestion pipelines are easy to build and hard to keep correct. Documents get edited, deleted, or superseded, and a vector index that still contains last year's policy will happily retrieve it with high confidence. Every chunk should carry a source version and an update timestamp, deletions must propagate to every index, and reindexing should be idempotent and driven by the source of truth. Deciding how much staleness is tolerable, and monitoring for it, is an operational requirement rather than an optional refinement.
Access control is the third and most consequential pitfall. The temptation is to retrieve broadly and filter in the prompt, asking the model to omit content the user cannot see. This does not work reliably and leaks information through the model's answers and citations. Permissions must be enforced at the data layer, in every index, and re-enforced whenever content is combined through expansion or joins. Teams that implement access control only in the application UI, while the retrieval layer serves everything to the model, discover this problem when someone asks a clever question.
Graph-specific pitfalls include over-extraction, where the graph accumulates noisy entities and relationships that make traversals return irrelevant neighbors, and under-modeling, where the schema is too coarse to answer the questions users actually ask. Entity resolution errors create false connections that are hard to notice because each individual edge looks plausible. A useful discipline is to store the source passage and extraction confidence on each edge, so that suspicious paths can be inspected and bad extractions removed in bulk.
Finally, there is the cost of hybridity itself. Each additional representation adds latency, storage, indexing jobs, and failure modes. Latency compounds when retrieval stages are sequential, so parallelizing independent retrievers and setting explicit timeouts with graceful degradation matters. A system that returns a slightly less complete answer in 800 milliseconds is often more useful than one that returns a complete answer in six seconds, though the right balance depends on the application.
Best Practices
Start with the questions, not the technology. Collect the questions the system must answer, classify them by type, and decide which representation is authoritative for each class. This exercise is cheap and prevents most architectural mistakes. Keep the classification in code or configuration so that routing decisions are explicit and testable.
Treat the document as the unit of truth and everything else as a derived index. Store original sources durably, record their versions, and make every derived artifact (chunks, embeddings, graph triples) reproducible from them. When you change a chunking strategy or an extraction prompt, you should be able to rebuild the indexes without manual intervention. This principle makes experimentation safe and debugging tractable.
Make provenance a first-class feature. Every claim in a generated answer should be traceable to a source passage with a version and timestamp. Users trust answers they can verify, and engineers can diagnose failures only when they can see which context the model received. Logging the retrieved chunk IDs, scores, and final prompt for each request is inexpensive and invaluable.
Evaluate continuously. Maintain a regression suite of labeled questions, run it on every change to chunking, embedding models, prompts, or retrieval parameters, and track retrieval and answer metrics separately. Include adversarial cases: questions whose answers are not in the corpus, questions with conflicting sources, and questions that probe access boundaries. The model should decline or flag uncertainty in the first case, surface the conflict in the second, and never leak content in the third.
Enforce authorization in the data layer and test it directly. Write tests that create users with different group memberships and assert that retrieval, expansion, and tool calls return only permitted content. Treat these tests as security tests, with the same review rigor as any other access control code.
Prefer the simplest representation that meets the requirement. Add a knowledge graph when you have demonstrated that multi-hop relational questions matter and that vector retrieval fails on them. Add a reranker when recall is good but precision is poor. Each addition should be justified by a measured gap, not by the expectation that a more sophisticated architecture will perform better.
Conclusion
Building an AI knowledge base is less about choosing a single technology and more about deciding how organizational knowledge should be represented so that a model can find it, interpret it, and cite it. RAG provides a flexible and widely understood way to bring external text into the model's context. Knowledge graphs provide precision, traversal, and explainability for relational questions. Hybrid designs combine them so that each covers the other's weaknesses, at the cost of operational complexity that must be managed deliberately. Alternatives such as long-context prompting, relational data with tool calling, and conventional search engines remain strong choices for many problems, and they should be considered before building anything elaborate.
The engineering discipline that matters most is not specific to any one stack. It is the habit of starting from the questions users ask, treating source documents as the authority, enforcing permissions where data lives, measuring retrieval and answers separately, and adding complexity only where measurements justify it. Systems built this way tend to remain correct as they grow, which is the property that determines whether an AI knowledge base earns trust over time.
References
- Lewis, P., Perez, E., Piktus, A., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". Advances in Neural Information Processing Systems 33. https://arxiv.org/abs/2005.11401
- Robertson, S., & Zaragoza, H. (2009). "The Probabilistic Relevance Framework: BM25 and Beyond". Foundations and Trends in Information Retrieval, 3(4), 333-389.
- Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods". Proceedings of SIGIR 2009.
- Karpukhin, V., Oğuz, B., Min, S., et al. (2020). "Dense Passage Retrieval for Open-Domain Question Answering". EMNLP 2020. https://arxiv.org/abs/2004.04906
- W3C. "RDF 1.1 Concepts and Abstract Syntax". W3C Recommendation, 2014. https://www.w3.org/TR/rdf11-concepts/
- W3C. "OWL 2 Web Ontology Language Document Overview (Second Edition)". W3C Recommendation, 2012. https://www.w3.org/TR/owl2-overview/
- Hogan, A., Blomqvist, E., Cochez, M., et al. (2021). "Knowledge Graphs". ACM Computing Surveys, 54(4). https://arxiv.org/abs/2003.02320
- Neo4j Documentation, property graph model and Cypher query language. https://neo4j.com/docs/
- Elasticsearch Documentation, Reference for search and vector search. https://www.elastic.co/guide/
- Reimers, N., & Gurevych, I. (2019). "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks". EMNLP 2019. https://arxiv.org/abs/1908.10084
- Nogueira, R., & Cho, K. (2019). "Passage Re-ranking with BERT". arXiv:1901.04085. https://arxiv.org/abs/1901.04085
- Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.