RAG Architecture for B2B SaaS Products: What Engineering Teams Get Wrong in Production

Retrieval-augmented generation (RAG) is now the standard architecture for embedding large language model capabilities into B2B SaaS products. The pattern is well-understood at the tutorial level: embed your documents, store them in a vector database, retrieve relevant chunks at query time, pass them to the model as context. It works in demos. It works on clean, general-purpose datasets. It tends to fail in ways that take months to diagnose when you build it into a real product serving real enterprise customers.

This post covers the three engineering decisions that most frequently determine whether a production RAG system works reliably — and the one discipline that almost every team skips until it becomes a crisis.

Why Naive RAG Fails in B2B SaaS

The tutorial RAG pattern assumes a reasonably homogeneous corpus of documents in natural language: documentation, knowledge base articles, product descriptions. Most B2B SaaS products do not look like this.

Enterprise business data is domain-specific, semi-structured, and inconsistent. Financial data has its own terminology. Legal documents have clause numbering and defined terms that reference each other non-linearly. Operations data has codes, identifiers, and abbreviations that mean nothing to a general-purpose embedding model but are critical to the domain expert using your product.

Two problems compound each other. First, embedding models trained on general-purpose web corpora do not represent domain-specific terminology well. The cosine similarity between a customer's query and the correct document chunk is lower than it should be — retrieval fails even when the document is there. Second, the chunking strategies used in tutorials (fixed window, 512 tokens, small overlap) are tuned for coherent narrative text. Applied to structured or semi-structured business data, they cut across entity boundaries, separate headers from their content, and produce chunks that contain partial information that looks relevant but is not.

The consequence is a system that returns plausible-sounding but incorrect answers with high confidence — the worst failure mode for enterprise software, where trust is the product.

The Multi-Tenant Isolation Problem

B2B SaaS products serve multiple customers. In RAG, this introduces a data isolation requirement that is easy to underestimate.

A naive implementation stores all customer documents in a single vector index and filters by metadata at query time. This works until the metadata filter logic has a bug, a query construction error, or an edge case in the retrieval pipeline — at which point documents from one tenant appear in the context for another tenant's query. In a consumer product, this might be a privacy annoyance. In B2B, especially in regulated industries, it is a security incident that ends enterprise deals.

The right architecture treats tenant isolation at the schema level, not the filter level. Either separate vector stores per tenant (operationally costly at scale but architecturally clean), or namespaced collections within a multi-tenant vector store with explicit access control enforced at the infrastructure layer — not just in application code. The isolation boundary must be explicit in the schema design from the first day of the build. Retrofitting it later is significantly more expensive than getting it right at the start.

The Three Engineering Decisions That Determine RAG Quality

1. Embedding Model Selection

Not all embedding models perform equally across domains. General-purpose embedding models (such as those trained predominantly on web text) handle common English prose well. They handle legal contracts, financial statements, medical records, and operations logs less well.

Before committing to an embedding model, run retrieval quality checks against a sample of your actual domain data. The metric that matters is not benchmark performance on standard datasets — it is whether the model retrieves the right chunks for the queries your users actually ask.

Domain-specific fine-tuned embedding models exist for several verticals and are worth evaluating if your data is highly specialised. For many B2B applications, a hybrid approach — starting with a strong general model and fine-tuning on a domain-specific subset if retrieval quality is insufficient — is more tractable than training from scratch.

2. Chunk Strategy

Chunking is the decision most teams make once and never revisit. It deserves more care than it usually gets.

Fixed-window chunking (split every N tokens with M overlap) is the default and the worst option for most B2B data. It destroys document structure, cuts entity descriptions in half, and separates context from the content that depends on it.

Three alternatives perform better depending on the data structure:

Semantic chunking splits on content meaning rather than token count — at paragraph boundaries, sentence boundaries, or topic shifts detected by a smaller classifier. For narrative text like documentation and support articles, this reliably outperforms fixed windows.

Document-structure-aware chunking uses the document's own structural markers — headings, table rows, clause numbers — as chunk boundaries. This is the right approach for structured business documents where the structure is meaningful: financial reports, contracts, operations manuals.

Hierarchical indexing stores both fine-grained chunks (for retrieval) and larger parent chunks (for context). The small chunks are retrieved by similarity; the larger parent provides the surrounding context for the model. This reduces the risk of retrieving a snippet without the context that makes it interpretable.

The right chunk strategy is data-dependent. The wrong approach is to pick one and never measure whether it works.

3. Retrieval Ranking

Pure vector similarity retrieval — find the N most similar chunks by cosine distance — has a well-known failure mode: it finds chunks that are semantically similar to the query but not the ones that answer it. Keyword overlap matters for many business queries, and pure vector retrieval ignores it.

Hybrid retrieval combines vector similarity with keyword search (typically BM25 or a similar sparse retrieval method). The combination consistently outperforms either method alone on domain-specific business queries. Most production-grade vector stores support hybrid retrieval natively; use it by default.

Re-ranking on top of hybrid retrieval — using a cross-encoder model to re-score the top-K retrieved candidates — adds another layer of quality at the cost of latency. For latency-sensitive applications, re-ranking the top 10–20 candidates rather than the full retrieved set keeps the latency acceptable while still improving precision.

The Missing Discipline: Evaluation

The single most common cause of RAG quality problems reaching production is the absence of an evaluation harness.

Teams build RAG, test it manually with a handful of queries, and ship it. Quality problems surface when customers complain — at which point the team has no systematic way to understand whether a change made things better or worse. Every tuning decision (different embedding model, different chunk size, different retrieval strategy) is made by feel rather than by measurement.

This is solvable and the upfront cost is lower than it appears.

A minimal eval loop requires three components: a reference set of queries drawn from real or realistic user behaviour (not developer-imagined edge cases), expected answers or relevant document IDs for each query, and an automated pipeline that runs the retrieval system against the reference set and reports a quality score.

The quality score can be as simple as recall at K — did the correct chunk appear in the top K retrieved results? For answer quality, LLM-as-judge evaluation (using a model to score the generated answer against the expected answer) is an imperfect but practical proxy that scales better than human evaluation.

The critical property of the eval loop is that it runs automatically on every change. Not as a quarterly exercise. Not when someone remembers to run it. As part of the deployment pipeline, the same way you run unit tests.

This connects directly to the advice in our earlier post on shipping LLM features that survive production — evaluation is not a phase that comes after building, it is the mechanism that makes confident iteration possible.

What Production RAG Actually Requires at Build Time

Pulling the above together: a production RAG system for a B2B SaaS product needs to be designed with four properties from day one.

Tenant isolation at the schema level. Not a filter added later — a hard boundary enforced at the infrastructure layer. Every design decision that involves storing customer data in a shared index needs to be re-examined against this requirement.

A domain-appropriate embedding and chunking strategy. Not the tutorial default. Tested against a sample of real customer data before the architecture is committed.

Hybrid retrieval as the default. Pure vector similarity is a starting point for evaluation, not a production retrieval strategy. Most B2B query patterns benefit from keyword signal alongside semantic similarity.

An evaluation harness built before the first production rollout. Not after the first customer complaint. The reference query set can be small — 50 to 100 well-chosen queries covers a surprising amount of quality signal. The infrastructure to run the eval automatically is a one-day build.

Working with Cursopic on RAG Product Builds

Cursopic's Intelligent Software practice builds LLM-powered products and ML pipelines for engineering teams who need to ship to real users — not just demonstrate capabilities.

When we work on RAG systems, we instrument retrieval quality from the first sprint, enforce tenant data isolation in the schema design rather than the application layer, and build the evaluation harness before we cut production traffic. We use a reference eval set drawn from real domain queries where possible, and from synthetic domain-representative queries where real data is not yet available.

If you are building a RAG feature into a B2B product and want a second opinion on the architecture before you commit it, or if you are debugging a RAG system that is producing worse results in production than it did in development, a short technical call can identify the most likely failure point quickly.

Contact us to start that conversation, or read more about how we approach intelligent software builds on our services page.

Cursopic builds cloud-native platforms, secure infrastructure, and intelligent software for ambitious engineering teams. We ship to real users and leave behind systems you own and understand.