Introduction
Most organizations have accumulated a large body of knowledge over the years: wikis, runbooks, ticket histories, design documents, policy PDFs, and chat threads that somehow contain the only record of a decision. For most of that time, the primary consumers of this material were humans. A person searching a wiki can skim several results, notice that a page is three years old, ask a colleague in a hallway, and weigh the answer against what they already know. Language models and AI agents change this relationship. They read quickly, they do not naturally judge provenance, and increasingly they act on what they retrieve. The knowledge is the same, but the consumer is very different, and that difference exposes weaknesses that were tolerable before.
The question for engineering teams is no longer whether to connect a model to internal knowledge but how to do it in a way that is consistent, secure, and maintainable. A common starting point is a script that embeds documents into a vector database and pastes the top results into a prompt. This works well enough for a demo. The cracks appear later, when three different chatbots each implement slightly different retrieval logic, when nobody can explain why a particular answer was produced, and when a contractor's chatbot quietly returns a document that was never meant to leave the legal team. These are not model problems. They are architecture problems, and they are predictable enough that they have known solutions.
This article argues that the durable answer is an access layer: a single, versioned boundary between knowledge stores and every consumer that needs them, whether that consumer is a customer-facing chatbot, an internal agent, an IDE assistant, or a batch evaluation job. We will look at the context that makes this problem hard, the internal structure of such a layer, the protocols that carry requests across it, a worked TypeScript implementation, and the pitfalls that repeatedly appear in production systems. The goal is practical guidance you can apply to your own platform, not a survey of every retrieval technique.
Context and Problem Overview
Enterprise knowledge is far more heterogeneous than the word "documents" suggests. It includes structured records such as customer accounts and configuration entries, semi-structured material like tickets and comments, long-form prose in PDFs and wiki pages, source code, and conversational transcripts. Each category has its own freshness characteristics, its own owners, and its own access rules. A design document may be visible to the entire engineering organization, while the same team's incident postmortem contains customer identifiers and is restricted to a small group. Retrieval systems tend to flatten this variety into chunks and vectors, and in the process the metadata that governed access is easily dropped or never captured in the first place. Once that happens, the index becomes a single pool of text in which every piece looks equally available.
Models add constraints of their own. Context windows are finite, and retrieval quality matters more than raw volume. Liu et al. (2023) reported that language models use information in long input contexts unevenly, with performance often degrading when relevant material sits in the middle of the input. Stuffing a prompt with fifty loosely related chunks can therefore be worse than providing five precise ones, even when the context window technically has room. There is also a security dimension. Retrieved text is untrusted input, yet a model will often treat instructions embedded in that text as if they came from the developer or user. The OWASP Top 10 for Large Language Model Applications lists prompt injection as its leading risk category, and indirect injection through documents an agent reads is one of its most practically relevant forms. A wiki page edited by anyone with write access can become an instruction channel.
Finally, the number of consumers keeps growing. Each new chatbot, agent, or integration tends to be built by a different team with different assumptions about freshness, ranking, citation format, and error handling. Without a shared boundary, every consumer re-implements the same hard problems, and every re-implementation drifts. The result is a system where the same question produces different answers depending on which interface you asked, and where nobody can say with confidence what a given user was allowed to see at a given moment. Those are the symptoms that an access layer is designed to cure.
Deep Technical Explanation: Anatomy of an Access Layer
An access layer is a service boundary that exposes a deliberately small set of operations, typically search, fetch, and source discovery, over stable contracts. Its purpose is to hide the things consumers should not depend on: which index technology is in use, how documents are chunked, where canonical content is stored, and how authorization is implemented internally. A useful analogy is the circulation desk of a library. Patrons do not wander into the stacks and reorganize the shelves themselves. They ask the desk, which knows the collection, checks membership, and returns a record with a stable identifier. The library can move books between floors, replace the catalog system, or digitize its holdings without changing how patrons ask for things. The desk is the contract, and the stacks are the implementation.
The internal components usually fall into five groups. The ingestion pipeline pulls content from source systems through connectors, normalizes it, splits it into chunks, and attaches metadata including the source system, owner, timestamps, document version, and access control information. The indexes include a lexical index, typically a BM25-style inverted index, and a dense vector index built from embeddings. The retrieval orchestrator runs queries against both, fuses the results, and optionally reranks them. An authorization enforcement point decides which candidates the requesting principal may see. A canonical document store holds the full text and serves fetch requests. Keeping these separate matters because each has different scaling characteristics and different failure modes, and because a change to one should not force a change to the others.
Hybrid retrieval deserves particular attention because it addresses a common weakness of embedding-only search. Dense embeddings, as introduced for sentence-level similarity by Reimers and Gurevych (2019), capture paraphrase and topical similarity well, but they can blur exact identifiers such as error codes, service names, or ticket numbers. Lexical search handles those precisely. Combining the two is practical, and Reciprocal Rank Fusion, described by Cormack, Clarke, and Büttcher (2009), offers a simple way to merge ranked lists without calibrating their raw scores against each other. The orchestrator can run both retrievers, fuse the lists by rank position, and then pass the top candidates to whatever authorization and filtering logic follows. The fusion method is less important than the discipline of measuring it, which we return to later.
Authorization is the most consequential design decision in the whole layer. The central principle is to enforce access against the requesting principal at retrieval time, not to bake permissions into a shared index at ingestion time and hope they remain accurate. There are two broad strategies. Pre-filtering pushes access predicates into the index query itself, so the index only considers documents the principal could see. This preserves recall under restrictive permissions but requires an index that supports efficient filtering on the relevant attributes. Post-filtering retrieves candidates first and then checks each one, which is simpler and works with any index but can leave a user with very few results when most candidates are hidden. Many production systems combine them: a coarse tenant and group filter is applied in the index, and fine-grained checks run afterward. Row-level security in PostgreSQL applies the same idea at the database layer, attaching predicates that the engine enforces on every query regardless of how the query was written.
Chunk identity is the last piece that is easy to underestimate. Every chunk must carry a reference to its parent document, the document version it came from, and a stable pointer that a consumer can cite. Chunks that lose their context perform poorly, and several techniques address this, including prepending document-level context to each chunk before embedding, a approach Anthropic described in its 2024 write-up on contextual retrieval. Whatever chunking strategy you choose, keep canonical document identifiers stable across re-chunking. If an answer cites chunk 17 of a document and the next ingestion run renumbers the chunks, the citation silently points at the wrong text. Stable document IDs and explicit version numbers prevent this class of error.
Protocols and Contracts: How Consumers Talk to the Layer
Consistency across consumers comes from contracts, and the contracts should be written down in a machine-checkable form. For tool inputs and outputs, JSON Schema is the natural choice, since it is widely supported by model tool-calling interfaces and by validation libraries in most languages. For HTTP interfaces, the OpenAPI Specification provides the same guarantees in a form that generates clients and documentation. Whichever you choose, define the contract before the implementation, and treat changes to it as API changes. Additive modifications, such as new optional fields, are generally safe within a major version. Renaming fields, changing the meaning of a score, or removing a result property breaks consumers, including agents whose behavior was tuned around the old output. A useful practice is to return explicit signals like a boolean indicating that more results may exist, rather than making consumers infer truncation from result counts.
Tool descriptions deserve the same care as schemas, because for an agent the description is the interface it uses to decide when and how to call a tool. A description such as "searches docs" gives the model little to work with. A description that states what is searched, what each result contains, that snippets are not complete documents, and that a fetch operation retrieves the full text, enables the model to plan a two-step process and to avoid pulling large amounts of text into its context unnecessarily. Treat tool descriptions as part of your API documentation, review them alongside code, and test them with real agent runs rather than assuming the model will infer intent.
The Model Context Protocol (MCP) is an open standard that Anthropic introduced in late 2024 to standardize how AI applications connect to external tools and data sources. It defines a client-server model in which servers expose capabilities such as tools, resources, and prompts, and it uses JSON-RPC 2.0 for its message format. For a knowledge access layer, MCP offers a practical benefit: you implement search and fetch once as an MCP server, and every MCP-capable client receives the same semantics without custom integration work. The protocol does not, however, solve authorization by itself. An MCP server still has to authenticate the user, enforce access rules on every operation, and handle credentials responsibly. The specification's authorization guidance builds on OAuth 2.x, and the access layer's correctness depends on how those mechanisms are wired into the retrieval logic, not on the protocol alone.
Protocol choice should follow the consumer rather than being applied uniformly. MCP suits agent clients and developer tools that already speak it. HTTP with OpenAPI suits internal services, batch jobs, and product backends that have conventional API infrastructure. Native SDKs can serve a first-party chatbot that needs tight control over streaming and latency. The critical design rule is that protocol adapters must be thin. Each adapter translates requests and responses and nothing else. All retrieval, ranking, and authorization logic lives in the core service beneath them. This is an application of the Adapter pattern described in Gamma et al.'s Design Patterns, and it keeps the behavior identical across transports. When a bug is fixed in the core service, every consumer benefits, and no consumer can quietly diverge.
Implementation: A Permission-Aware Knowledge Service
The following TypeScript sketch shows the core of a knowledge service with its dependencies expressed as narrow interfaces. The service knows nothing about MCP or HTTP. It accepts a principal, applies filters, enforces authorization, and returns bounded results. Real systems would add error types, pagination tokens, and richer metadata, but the structure reflects the decisions discussed above.
export interface Principal {
userId: string;
tenantId: string;
groups: string[];
}
export interface Document {
id: string;
title: string;
body: string;
source: string;
updatedAt: string;
tenantId: string;
allowGroups: string[];
}
export interface SearchHit {
docId: string;
title: string;
snippet: string;
source: string;
updatedAt: string;
score: number;
}
export interface SearchRequest {
query: string;
topK?: number;
sources?: string[];
updatedAfter?: string;
}
export interface SearchResponse {
hits: SearchHit[];
mayHaveMore: boolean;
}
export interface VectorIndex {
search(
query: string,
opts: { k: number; filter: Record<string, unknown> }
): Promise<Array<{ docId: string; score: number }>>;
}
export interface DocumentStore {
getMany(ids: string[]): Promise<Map<string, Document>>;
}
export interface AuthzService {
// Returns the subset of document IDs the principal may read.
filterReadable(principal: Principal, docIds: string[]): Promise<Set<string>>;
}
const MAX_TOP_K = 25;
const OVER_FETCH_FACTOR = 4;
const SNIPPET_CHARS = 400;
export class KnowledgeService {
constructor(
private readonly index: VectorIndex,
private readonly store: DocumentStore,
private readonly authz: AuthzService
) {}
async search(principal: Principal, req: SearchRequest): Promise<SearchResponse> {
const k = Math.min(req.topK ?? 10, MAX_TOP_K);
const candidateK = k * OVER_FETCH_FACTOR;
// Coarse tenant and source filters are applied inside the index (pre-filtering).
const filter: Record<string, unknown> = { tenantId: principal.tenantId };
if (req.sources?.length) filter.source = { $in: req.sources };
if (req.updatedAfter) filter.updatedAt = { $gt: req.updatedAfter };
const candidates = await this.index.search(req.query, { k: candidateK, filter });
// Fine-grained authorization is checked in one batch, avoiding per-document round trips.
const readable = await this.authz.filterReadable(
principal,
candidates.map((c) => c.docId)
);
const allowed = candidates.filter((c) => readable.has(c.docId)).slice(0, k);
const docs = await this.store.getMany(allowed.map((c) => c.docId));
const hits: SearchHit[] = allowed.flatMap(({ docId, score }) => {
const doc = docs.get(docId);
if (!doc) return [];
return [
{
docId,
title: doc.title,
snippet: doc.body.slice(0, SNIPPET_CHARS),
source: doc.source,
updatedAt: doc.updatedAt,
score,
},
];
});
// If the index returned a full candidate set, more relevant documents may exist
// beyond what this request could see or return.
return { hits, mayHaveMore: candidates.length === candidateK };
}
async fetch(principal: Principal, docId: string): Promise<Document | null> {
const readable = await this.authz.filterReadable(principal, [docId]);
if (!readable.has(docId)) return null;
const docs = await this.store.getMany([docId]);
return docs.get(docId) ?? null;
}
}
The search method makes several choices that are worth examining. It over-fetches candidates by a fixed factor because post-filtering will remove some of them, and without the extra margin a user with narrow permissions would receive suspiciously short result lists. The tenant boundary is applied as a pre-filter inside the index, so the index never considers another tenant's documents at all. Authorization runs as a single batched call, which avoids the classic N+1 pattern of checking one document at a time. Snippets are capped, which limits how much text leaves the layer per hit and gives the consumer an explicit cue that a fetch is needed for the full document.
The fetch method deserves emphasis because it is easy to omit. Agents typically search first and then retrieve full text by identifier. Authorization must be enforced again at fetch time. Never assume that because a principal was allowed to see a search result, they remain allowed to fetch it later, since permissions change, sessions outlive grants, and document IDs can be guessed or leaked. Enforcing the check on every operation also gives you a natural point to write audit records of which documents were actually read, which is often the information a security review needs.
The model-facing schema is the other half of the contract. It should constrain inputs tightly, because those constraints prevent an agent from requesting an unreasonable number of results and make invalid calls fail early with clear messages:
export const searchKnowledgeTool = {
name: "search_knowledge",
description:
"Search the internal knowledge base. Returns short snippets with document IDs. " +
"Snippets are excerpts, not complete documents; call fetch_document with an ID to read the full text. " +
"Results are limited to content the current user is permitted to read".,
inputSchema: {
type: "object",
properties: {
query: { type: "string", minLength: 3, maxLength: 500 },
topK: { type: "integer", minimum: 1, maximum: 25, default: 10 },
sources: { type: "array", items: { type: "string" }, maxItems: 10 },
},
required: ["query"],
additionalProperties: false,
},
} as const;
Operational visibility completes the implementation. Each search should emit a trace span, following OpenTelemetry conventions, that records the principal identifier, the applied filters, the number of candidates considered, the number of results returned, the document IDs returned, and the latency of each stage. Be careful with raw query text in logs, since queries often contain sensitive material; hashing or redacting them while keeping the document IDs is usually a reasonable compromise. These records are what allow you to answer the two questions that will eventually arise in every mature deployment: why did the system return this answer, and what could this user have seen at that time. Finally, avoid caching authorized results in a way that can be served to a different principal. Caching is valuable, but cache keys must include the principal's effective permissions, or the cache becomes a data leak.
Trade-offs and Pitfalls
The most common pitfall is post-filter starvation. A team builds a search pipeline that retrieves the top ten candidates and then removes the ones the user cannot see. For most users in a permissive environment this works fine. For a user restricted to a small project space, the same pipeline returns almost nothing, and the chatbot responds with "I could not find any information", which is false. The fix is some combination of pre-filtering on coarse attributes, larger candidate sets, and monitoring of result counts per principal segment. Measure the empty-result rate by permission group, not just globally, because the global average hides the users who are most affected.
Permission staleness is the second pitfall and a more serious one. Access rules live in the source systems, and a synchronization pipeline copies them into the knowledge layer on some schedule. During the gap between a permission change and its propagation, the layer may expose content that should no longer be visible, or deny content that should be. For sensitive material, the safer option is to check permissions against the source system or a frequently updated permission store at read time, accepting the latency cost. For lower-sensitivity material, a bounded synchronization window with clear documentation may be acceptable. The important part is making that decision explicitly and writing it down, rather than discovering the window during an incident.
Other pitfalls are more mechanical but just as damaging. Changing embedding models without a migration plan forces a full reindex, and if old and new vectors coexist in one index the rankings become meaningless. Store the embedding model version in index metadata and plan for dual-index migrations. Chunk-version mismatch occurs when a citation points to a chunk from a superseded document version, which is solved by stable document IDs and explicit version fields. Unbounded tool output is an easy mistake: a tool that returns full documents on every search will fill the context window and degrade answer quality, consistent with the long-context findings of Liu et al. Prompt injection through indirect channels remains a live risk, since any document that an agent reads may contain text designed to alter its behavior. The OWASP guidance is a useful checklist for these cases.
The broader trade-off is between a central layer and decentralized integration. A central access layer adds a network hop, latency, an operational component to run, and a platform team that can become a bottleneck if it is slow to respond to consumer needs. The alternative, in which every team wires its own retrieval, is faster to start and often produces three or four incompatible systems within a year, each with its own gaps in authorization and auditing. There is no universally correct answer, but the decision becomes clearer when you consider how many consumers you expect and how much you value consistent security behavior. Similarly, hybrid retrieval and reranking improve quality but cost latency and compute. Vector-only search is simpler and sometimes adequate, but it will miss exact identifiers. The right choice is the one that wins on your own evaluation set, not on a benchmark chosen by someone else.
Best Practices
Build an evaluation set before you tune anything. It should contain real queries drawn from consumers, each paired with the documents that should answer it, and it should include cases where the correct answer is "nothing in the knowledge base". Measure retrieval recall at a fixed cutoff and, where possible, the faithfulness of generated answers to the retrieved text. Run this evaluation on every change to chunking, embedding models, ranking logic, or ingestion rules. Without it, tuning becomes a matter of anecdote, and regressions reach users before anyone notices. Keep the set versioned alongside the code so that changes to expectations are reviewable.
Make the layer the only path to the data. Consumers should not hold credentials for the underlying index or document store. Enforce this through network policy and scoped service credentials, so that a chatbot cannot bypass authorization by querying the index directly. Scope credentials per consumer, which also makes it possible to revoke one integration without disrupting the others and to attribute usage in logs. These measures sound obvious, but they are frequently skipped during early development and are then difficult to retrofit once several integrations depend on direct access.
Version and deprecate deliberately. Version the public API, record the embedding model and chunking configuration in index metadata, and maintain a changelog that consumers can read. When a breaking change is unavoidable, run old and new versions in parallel with dual writes during migration, and set a date for retirement that is communicated well in advance. Treat the tool schemas and descriptions as part of the public surface, reviewing them with the same care as code, since an agent's behavior can change when its tool descriptions change.
Treat retrieved content as data, never as instructions. Delimit retrieved text clearly in prompts, state explicitly that instructions inside retrieved documents must not be followed, and never allow retrieved content to modify tool permissions or trigger write-capable actions without an explicit confirmation step from the user. Sanitize rendered output, particularly links and markup, since a model can be induced to produce content that a downstream renderer interprets. Return document identifiers with every answer so that citations are verifiable, and make the source of each claim inspectable by the user. A system whose answers can be traced to specific documents is far easier to trust and to debug than one that produces fluent text with no provenance.
Conclusion
Exposing organizational knowledge to AI agents and chatbots is less a question of which model or vector database to choose and more a question of boundaries. The access layer is the boundary that matters. It hides the implementation details that consumers should not depend on, presents a small and stable set of operations, enforces authorization on every call, and records enough information to explain its behavior afterward. Retrieval quality, security, and consistency all follow from getting that boundary right. When the boundary is missing, each consumer invents its own version of these properties, and the inconsistencies accumulate until nobody can say with confidence what the system knows or who is allowed to know it.
The components themselves are not exotic. Hybrid retrieval, rank fusion, pre- and post-filtering, JSON Schema contracts, OpenAPI definitions, and MCP servers are established tools, and the engineering discipline around them, including evaluation sets, scoped credentials, versioned contracts, and trace-level observability, is the same discipline that mature backend systems have always required. What is new is the consumer. An agent reads faster than a person, trusts retrieved text more than it should, and acts on what it finds. That makes the old practices more important, not less, and it rewards teams that treat the access layer as long-lived infrastructure rather than as a prototype that happens to work.
If you are starting from scratch, begin with a single consumer, write the contract before the code, enforce authorization at retrieval time, and build the evaluation set before you optimize anything. Add consumers only after the first one is measurable and its behavior is understood. The models and protocols will continue to change, and the access layer will outlast many of them. Designing it as a stable, auditable boundary is the investment that keeps the rest of the system adaptable.
References
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arXiv:2005.11401.
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
- Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval.
- Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP-IJCNLP 2019. arXiv:1908.10084.
- Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4).
- Anthropic. Model Context Protocol (MCP) Specification. https://modelcontextprotocol.io
- JSON-RPC Working Group. JSON-RPC 2.0 Specification. https://www.jsonrpc.org/specification
- OpenAPI Initiative. OpenAPI Specification v3.1.0. https://spec.openapis.org/oas/v3.1.0
- JSON Schema organization. JSON Schema Specification. https://json-schema.org
- OWASP Foundation. OWASP Top 10 for Large Language Model Applications. https://owasp.org/www-project-top-10-for-large-language-model-applications/
- Hardt, D. (Ed.). (2012). The OAuth 2.0 Authorization Framework. RFC 6749. Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc6749
- PostgreSQL Global Development Group. Row Security Policies. PostgreSQL Documentation. https://www.postgresql.org/docs/current/ddl-rowsecurity.html
- Fowler, M. (2002). Patterns of Enterprise Application Architecture. Addison-Wesley.
- Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1994). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley.
- Kleppmann, M. (2017). Designing Data-Intensive Applications. O'Reilly Media.
- Anthropic. (2024). Introducing Contextual Retrieval. https://www.anthropic.com/news/contextual-retrieval
- OpenTelemetry project. OpenTelemetry Documentation. https://opentelemetry.io/docs/