Building Reliable RAG Pipelines: More Than a Vector Database

Part 4 of the Info2K Data Engineering for AI Series

Retrieval-Augmented Generation has become one of the most common approaches for connecting Large Language Models with organisational information.

The basic idea appears straightforward:

1. Collect documents.

2. Divide them into chunks.

3. Generate embeddings.

4. Store those embeddings in a vector database.

5. Retrieve relevant chunks for a user’s question.

6. Ask a language model to generate an answer.

This can be enough to create an impressive demonstration.

However, it is not enough to build a reliable production system.

A production RAG pipeline must also answer much harder questions:

• Which documents are authoritative?

• Has a document been replaced or withdrawn?

• Can the current user access the retrieved information?

• Did parsing preserve headings, tables, lists, and context?

• Does each chunk retain its relationship to the source?

• Is the retrieved content relevant to the question?

• Can the answer be traced back to supporting evidence?

• What happens when the source document changes?

• How do we measure whether retrieval is improving?

• What should the system do when reliable evidence is unavailable?

These are not only language-model questions.

They are questions of Data Engineering, information retrieval, security, governance, evaluation, and operations.

This article is Part 4 of the Info2K Data Engineering for AI series.

In Part 1, we examined the hidden Data Engineering behind successful AI projects. Part 2 defined what AI-ready data actually means. Part 3 compared ETL and ELT for modern AI systems.

Now we will follow the data through a complete RAG pipeline: from source documents to a grounded answer.

What Is RAG?

RAG stands for:

Retrieval-Augmented Generation

A conventional language model generates a response using patterns learned during training and the information supplied in its prompt.

However, the model may not contain:

• Current organisational policies

• Private business information

• Recently updated product material

• Internal procedures

• Customer or case information

• Current clinical guidance

• Domain-specific documentation

RAG introduces a retrieval step.

Before asking the model to generate an answer, the system searches an approved knowledge source for relevant information. The retrieved passages are then supplied to the model as grounding context.

User Question
      ↓
Search Approved Knowledge
      ↓
Retrieve Relevant Evidence
      ↓
Add Evidence to the Prompt
      ↓
Generate a Grounded Answer
      ↓
Return Answer with Sources

AWS describes RAG as a pattern that retrieves external, domain-specific information before generation, allowing a model to use current enterprise knowledge without placing that knowledge directly into the model’s trained parameters. AWS Prescriptive Guidance

This can make responses more relevant and easier to verify.

But RAG does not automatically guarantee accuracy.

It gives the model access to retrieved information. The quality of the final answer still depends on the quality of the complete pipeline.

The Simple RAG Diagram Hides Most of the Work

RAG is frequently represented like this:

Documents
    ↓
Vector Database
    ↓
Language Model
    ↓
Answer

A production architecture is more likely to look like this:

Approved Sources
       ↓
Ingestion and Validation
       ↓
Parsing and Content Extraction
       ↓
Classification and Deduplication
       ↓
Chunking and Context Preservation
       ↓
Metadata and Permission Enrichment
       ↓
Embeddings and Search Indexes
       ↓
Query Understanding
       ↓
Permission-Aware Retrieval
       ↓
Ranking and Context Assembly
       ↓
Language Model
       ↓
Grounded Answer with Evidence
       ↓
Evaluation, Monitoring and Feedback

Every stage can improve or damage the final answer.

If the wrong document is ingested, retrieval can find the wrong information perfectly.

If parsing loses a table heading, the retrieved numbers may no longer have meaning.

If chunks are too small, important context may be separated.

If chunks are too large, retrieval may return excessive irrelevant information.

If permissions are not enforced during retrieval, the model may receive content the user is not authorised to see.

If changed documents are not refreshed, the system may continue providing outdated answers.

The vector database is therefore only one component of the architecture.

Stage 1: Start with Approved Knowledge Sources

The first step is not generating embeddings.

It is deciding which information is allowed to become part of the knowledge base.

Potential sources may include:

• Policies

• Procedures

• Technical documentation

• Product manuals

• Knowledge-base articles

• Research material

• Contracts

• Customer records

• Clinical guidelines

• Referral documents

• Case notes

• Operational databases

Not every available source should be indexed.

Before ingestion, organisations should establish:

• Ownership

• Authority

• Intended audience

• Information classification

• Retention requirements

• Effective and expiry dates

• Permitted uses

• Review responsibilities

A folder containing hundreds of documents is not automatically a trustworthy knowledge base.

Some documents may be:

• Drafts

• Duplicates

• Superseded versions

• Incomplete

• Incorrectly classified

• Written for a different audience

• Outside the approved scope

• Missing an identifiable owner

A reliable RAG system begins with an intentional source boundary.

Stage 2: Ingestion Must Preserve Provenance

Ingestion transfers information from approved sources into the RAG processing environment.

This sounds like a straightforward copying task, but ingestion establishes the lineage required to operate the system later.

For every document or record, the pipeline should capture information such as:

• Source system

• Source identifier

• Document identifier

• Document version

• Owner

• Ingestion time

• Source modification time

• Content hash

• Classification

• Effective date

• Expiry date

• Access restrictions

The content hash can help detect whether a document has changed.

The source identifier allows transformed chunks to be traced back to the original material.

The version and effective dates help distinguish current information from historical information.

Without this provenance, it becomes difficult to:

• Refresh changed content

• Remove deleted information

• Identify duplicate documents

• Investigate incorrect answers

• Demonstrate where an answer originated

• Rebuild the index consistently

Ingestion should also validate files before processing.

Controls may include:

• Approved file types

• File-size limits

• Malware scanning

• Encryption

• Corruption detection

• Sensitive-data detection

• Source authentication

• Quarantine of rejected files

AWS recommends applying security controls at multiple stages of the RAG pipeline, including validation during ingestion, access control in storage, metadata filtering during retrieval, and output controls during inference. AWS Prescriptive Guidance

Stage 3: Parsing Is Part of Data Quality

A RAG system rarely searches the original document directly.

The document is first converted into a representation the retrieval system can process.

For a simple text file, this may be easy.

For a complex PDF, presentation, spreadsheet, scanned form, or clinical document, parsing can be much harder.

Important information may be encoded through:

• Headings

• Columns

• Tables

• Lists

• Captions

• Footnotes

• Page boundaries

• Diagrams

• Reading order

• Visual relationships

A parser may successfully extract every word while still destroying the meaning.

Consider a table containing product limits:

Product A    20 units
Product B    50 units
Product C    10 units

If the parser separates the values from their labels, the extracted text may still contain all six elements, but no longer preserve which limit belongs to which product.

Parsing quality should therefore be evaluated, not assumed.

Useful checks include:

• Was the reading order preserved?

• Were headings associated with their sections?

• Were tables represented accurately?

• Was repeated header or footer noise removed?

• Was optical character recognition reliable?

• Were document boundaries retained?

• Were images or diagrams important to interpretation?

AWS guidance notes that raw documents can create RAG problems through poor structure, missing metadata, ambiguous terminology, redundancy, and insufficient domain context. Clear headings and contextual structure can improve how information is retrieved and interpreted. AWS Prescriptive Guidance

Stage 4: Classification and Deduplication

Before chunking and indexing, the system should understand what each document represents.

Classification may identify:

• Document type

• Topic

• Business unit

• Owner

• Audience

• Sensitivity

• Jurisdiction

• Product

• Customer

• Patient

• Case

• Effective status

Classification helps retrieval select the correct information for the user and use case.

Deduplication is equally important.

If the same policy exists in several folders, retrieval may return multiple copies. This can crowd out other useful evidence and give the repeated text excessive influence.

Version management is more complex than exact duplicate detection.

Two documents may have different filenames and slightly different content while representing different versions of the same policy.

The pipeline should determine:

• Which version is authoritative?

• Should historical versions remain searchable?

• Under what circumstances?

• What happens when a document is withdrawn?

• Can every indexed chunk be linked to its version?

A reliable system does not treat every document as equally current and authoritative.

Stage 5: Chunking Must Preserve Meaning

Language models and retrieval systems usually work with smaller passages rather than entire documents.

The process of dividing a document into these passages is called chunking.

A simple implementation may divide text every fixed number of characters or tokens.

This is easy to implement, but it can separate information from the context required to understand it.

For example:

Chunk 1:
Employees may work remotely for up to three days per week...

Chunk 2:
...except staff handling restricted on-site systems.

If only the first chunk is retrieved, the generated answer may omit the exception.

Chunking should consider the structure and meaning of the document.

Possible strategies include:

• Fixed-length chunking

• Paragraph-based chunking

• Section-based chunking

• Sentence-aware chunking

• Semantic chunking

• Parent-and-child chunking

• Overlapping chunks

• Document-type-specific chunking

There is no universally correct chunk size.

Smaller chunks may improve retrieval precision but lose surrounding context.

Larger chunks may preserve context but introduce unrelated information and consume more of the model’s context window.

Chunking should therefore be treated as an engineering parameter that must be tested against realistic questions.

Stage 6: Metadata Makes Retrieval Contextual

Embeddings represent semantic similarity, but similarity alone is not enough.

A user asking about leave policy may retrieve a semantically similar policy from:

• Another country

• Another business unit

• An expired version

• A confidential HR repository

• A different employment category

Metadata helps the system constrain retrieval to appropriate information.

Useful metadata may include:

• Source

• Document type

• Version

• Owner

• Department

• Effective date

• Expiry date

• Region

• Language

• Product

• Customer

• Security classification

• Permitted roles

• Retention status

Metadata can support:

• Permission filtering

• Version filtering

• Time-based filtering

• Department filtering

• Tenant isolation

• Source prioritisation

• Result explanation

AWS guidance specifically recommends attaching metadata such as department and information classification to indexed content, then applying filters based on the user’s identity, role, business unit, or clearance. AWS Prescriptive Guidance

Metadata is not decorative information surrounding the vector.

It actively determines what the AI system is allowed to retrieve.

Stage 7: Embeddings and Indexes Must Be Reproducible

Embeddings convert content into numerical representations that support semantic search.

However, the pipeline must preserve more than the vector.

Each indexed unit should remain connected to:

• The original source

• The document version

• The chunk text

• The chunk position

• The embedding model and version

• The parsing configuration

• The chunking configuration

• The metadata

• The access rules

• The ingestion run

This becomes important when:

• The embedding model changes

• The chunking strategy changes

• A parser is improved

• A document is withdrawn

• Permissions change

• An incorrect result must be investigated

• The complete index must be rebuilt

An embedding without provenance is difficult to govern.

A production RAG system should be able to answer:

Which source content created this indexed vector, using which processing configuration?

If it cannot answer that question, maintaining the system will become increasingly difficult.

Stage 8: Retrieval Is More Than Vector Similarity

A user’s question may pass through several retrieval steps.

These can include:

• Query rewriting

• Keyword search

• Vector search

• Hybrid search

• Metadata filtering

• Reranking

• Diversity selection

• Date filtering

• Permission filtering

• Context assembly

Vector search is valuable for finding semantically related material.

Keyword search may be better for:

• Exact identifiers

• Product codes

• Policy numbers

• Medication names

• Technical terms

• Abbreviations

Hybrid retrieval combines semantic and lexical techniques.

Reranking can then assess the retrieved candidates more carefully and choose the best evidence for the final prompt.

The important distinction is:

Retrieval should find the most relevant authorised evidence, not merely the most similar text.

A semantically similar chunk can still be:

• Outdated

• Unauthorised

• Incomplete

• From the wrong customer

• From the wrong jurisdiction

• From a non-authoritative source

This is why retrieval logic must combine similarity with metadata, permissions, authority, and context.

Stage 9: Permissions Must Be Enforced Before Generation

A common RAG security mistake is enforcing access only in the application interface.

For example, the user interface may hide a confidential document, while the retrieval service can still search its chunks.

If unauthorised content is retrieved and placed into the model’s prompt, the security boundary has already failed, even if the final interface attempts to hide part of the answer.

Permissions should operate at retrieval time.

The system should determine:

• Who is asking?

• What roles do they have?

• Which organisation or tenant do they belong to?

• Which classifications can they access?

• Do time-based restrictions apply?

• Does the source contain row-level or document-level permissions?

AWS recommends a defence-in-depth approach that includes ingestion validation, encrypted storage, metadata and role-based filtering during retrieval, and guardrails during inference. It also identifies indirect prompt injection and malicious knowledge-base content as RAG-specific risks. AWS Prescriptive Guidance

RAG security is therefore a pipeline property, not a single filter added at the end.

Stage 10: The Model Needs Evidence, Not Just Text

After retrieval, the selected chunks are assembled into the model’s context.

The model should receive enough information to distinguish:

• Source content

• User instructions

• System instructions

• Document metadata

• Citation identifiers

• Conflicting evidence

• Missing evidence

The application should also define what the model should do when the retrieved evidence is insufficient.

A trustworthy response may be:

“I could not find enough approved information to answer this question.”

That is often better than a fluent but unsupported answer.

Depending on the use case, the response may need to include:

• Source title

• Source link

• Document version

• Effective date

• Quoted supporting section

• Confidence or limitation statement

• Recommendation for human review

Source attribution improves transparency and allows users to verify whether the answer reflects the underlying material. AWS highlights provenance and source attribution as important for verification, auditing, troubleshooting, and user trust. AWS Prescriptive Guidance

Citations do not automatically make an answer correct.

But they make unsupported claims easier to identify and help qualified users evaluate the response.

Stage 11: Evaluate Retrieval Separately from Generation

A RAG system can fail in two fundamentally different ways.

Retrieval Failure

The system does not retrieve the evidence required to answer the question.

Generation Failure

The correct evidence is retrieved, but the model misinterprets, ignores, or contradicts it.

These problems require different solutions.

Changing the prompt will not fix missing evidence.

Changing the embedding model may not fix a generation instruction that encourages speculation.

Evaluation should therefore examine both stages.

Retrieval Evaluation

Useful questions include:

• Did the system retrieve the required evidence?

• How many retrieved chunks were relevant?

• Did irrelevant content displace useful evidence?

• Were authoritative sources ranked above weaker sources?

• Did permission filtering work?

• Did date and version filters work?

Answer Evaluation

Useful questions include:

• Is the answer grounded in the retrieved context?

• Does it address the user’s question?

• Is it complete?

• Does it introduce unsupported claims?

• Are the citations correct?

• Does it acknowledge uncertainty?

• Does it follow domain and safety requirements?

Microsoft’s RAG evaluation guidance separates retrieval-process evaluation from final-response evaluation. It identifies retrieval relevance, groundedness, response relevance, and completeness as distinct dimensions. Microsoft Foundry documentation

This separation helps teams diagnose the real bottleneck.

Build an Evaluation Dataset Before Production

A RAG system should be tested using realistic questions and expected evidence.

An evaluation dataset may include:

• Common questions

• Difficult questions

• Ambiguous questions

• Questions with no approved answer

• Questions requiring multiple documents

• Questions involving exceptions

• Questions about outdated information

• Permission-sensitive questions

• Adversarial questions

• Domain-specific terminology

For each test case, the team may record:

• Expected source

• Relevant passages

• Acceptable answer

• Unacceptable claims

• Required limitations

• Expected permission behaviour

Automated evaluation can help compare configurations at scale.

Human review remains important, especially in high-trust domains.

AWS’s healthcare guidance recommends evaluating components systematically and distinguishes response relevance, context precision, and faithfulness. AWS Prescriptive Guidance

The objective is not merely to obtain one accuracy score.

It is to understand how the system behaves across realistic and risky situations.

Refresh, Deletion and Change Management

A production knowledge base is never static.

Documents are:

• Added

• Revised

• Reclassified

• Replaced

• Deleted

• Expired

The RAG pipeline must propagate these changes.

When a source document changes, the system may need to:

1. Detect the new version.

2. Reprocess the content.

3. Generate new chunks.

4. Apply current metadata and permissions.

5. Create new embeddings.

6. update the search index.

7. Remove or archive previous chunks.

8. Validate the refreshed results.

Deletion is especially important.

Removing a document from the source system does not automatically remove:

• Extracted text

• Chunks

• Embeddings

• Search-index entries

• Cached answers

• Evaluation records

The system requires a traceable deletion process across every derived representation.

This is another reason provenance matters.

Without a relationship between the source document and its downstream assets, reliable refresh and deletion become difficult.

Monitor the RAG Pipeline in Production

Evaluation before release is necessary, but it is not sufficient.

Production behaviour changes over time.

New questions appear.

Source content evolves.

Permissions change.

Retrieval patterns drift.

Users discover weaknesses that were not represented in the original evaluation set.

Monitoring may include:

• Ingestion failures

• Parsing failures

• Index freshness

• Document and chunk counts

• Duplicate rates

• Retrieval latency

• Empty retrievals

• Frequently retrieved sources

• Permission-filter failures

• Citation coverage

• Groundedness

• User feedback

• Cost and token consumption

Feedback should be connected to investigation and improvement.

If a user reports a poor answer, the team should be able to inspect:

• The original question

• Query transformations

• Applied filters

• Retrieved chunks

• Source versions

• Final prompt

• Model response

• Evaluation results

Observability turns an unexplained AI failure into a diagnosable engineering problem.

A Healthcare Example

Imagine a RAG assistant that helps a healthcare professional find information across:

• Clinical guidelines

• Local care pathways

• Medication policies

• Referral procedures

• Patient documents

• Discharge summaries

A query might ask:

“What is the recommended follow-up, considering this patient’s latest discharge summary?”

The pipeline must do more than retrieve semantically similar text.

It may need to confirm that:

• The user can access the patient information.

• The record belongs to the correct patient.

• The latest discharge summary is identified.

• Older summaries are not treated as current.

• The applicable guideline is current.

• Local and national guidance are distinguished.

• Medication information is not taken from a discontinued section.

• Retrieved evidence is shown to the clinician.

• The answer is presented as decision support, not an autonomous decision.

Clinical text also contains:

• Negation

• Uncertainty

• Abbreviations

• Historical conditions

• Family history

• Temporal relationships

• Corrected results

These details can be lost during parsing or chunking.

Healthcare therefore reinforces a central principle:

RAG reliability depends on domain-aware data preparation, retrieval, governance and human oversight, not simply on model capability.

Common RAG Mistakes

Mistake 1: Indexing Everything

More documents do not automatically create better answers.

Unapproved, duplicated, outdated, or irrelevant material can reduce retrieval quality.

Mistake 2: Using One Chunking Strategy for Every Document

Policies, tables, clinical notes, presentations, and technical manuals have different structures.

Their chunking strategies may need to differ.

Mistake 3: Treating Metadata as Optional

Without metadata, the system cannot reliably filter by authority, version, date, region, audience, tenant, or permission.

Mistake 4: Testing Only the Final Answer

A good answer does not reveal whether the correct evidence was retrieved consistently.

Retrieval and generation should be evaluated separately.

Mistake 5: Ignoring Deletions

Removing a source document without removing its chunks and embeddings leaves obsolete information searchable.

Mistake 6: Applying Security Only in the Interface

Access controls must apply before content is supplied to the language model.

Mistake 7: Assuming RAG Eliminates Hallucination

RAG can provide better grounding, but the model can still misinterpret evidence, combine sources incorrectly, or make unsupported claims.

Mistake 8: Improving the Model Before Investigating Retrieval

Replacing the language model may produce little benefit if the actual problem is parsing, chunking, metadata, filtering, or ranking.

A Practical RAG Readiness Checklist

Before moving a RAG system into production, ask:

Sources

• Are all indexed sources approved?

• Is authority and ownership clear?

• Are versions and effective dates available?

Ingestion

• Can changed and deleted content be detected?

• Are invalid or unsafe files quarantined?

• Is source provenance captured?

Parsing

• Are headings, tables, lists, and reading order preserved?

• Has extraction quality been evaluated?

Chunking

• Does chunking preserve enough context?

• Has it been tested using realistic questions?

• Is the strategy appropriate for each document type?

Metadata

• Can results be filtered by version, date, region, audience, and classification?

• Is metadata complete and reliable?

Security

• Are permissions enforced during retrieval?

• Can malicious or unauthorised content enter the knowledge base?

• Are retrieval and generation events auditable?

Retrieval

• Are semantic, keyword, hybrid, and reranking approaches evaluated?

• Does retrieval prioritise authoritative evidence?

• Can the system identify that no reliable answer exists?

Generation

• Is the answer grounded in the supplied evidence?

• Are sources visible and verifiable?

• Are uncertainty and limitations communicated?

Operations

• Can the index be reproduced?

• Are refresh and deletion reliable?

• Are quality, latency, cost, failures, and user feedback monitored?

• Is there a clear owner for remediation?

If these questions cannot be answered, the system may be demonstration-ready, but not production-ready.

RAG Is a Data Product

A reliable RAG system should be treated as a maintained data and AI product.

It requires:

• Approved sources

• Repeatable pipelines

• Versioned processing

• Data-quality controls

• Metadata

• Security

• Evaluation datasets

• Monitoring

• Ownership

• Continuous improvement

The language model may be the most visible component.

But much of the reliability comes from the engineering around it.

A vector database can store embeddings.

It cannot determine by itself whether information is current, authoritative, correctly parsed, permission-appropriate, or sufficient to support an answer.

That responsibility belongs to the complete RAG pipeline.

Continue the Info2K Data Engineering for AI Series

This article is Part 4 of the Info2K Data Engineering for AI series.

Part 1: The Hidden Data Engineering Behind Successful AI Projects

Why reliable AI depends on pipelines, quality, governance, metadata, retrieval, and monitoring.

Part 2: What Does AI-Ready Data Actually Mean?

Why clean data alone is insufficient, and why context, relevance, governance and operational reliability matter.

Part 3: ETL vs ELT for Modern AI Systems

How transformation timing and location affect control, flexibility, scalability, and AI readiness.

Part 4: Building Reliable RAG Pipelines

Why production RAG requires governed sources, parsing, chunking, metadata, secure retrieval, evidence, evaluation, and operations.

Next, Part 5: Data Quality for AI

How measurable quality controls, validation, lineage, monitoring and ownership reduce unreliable AI outcomes.

Future parts will explore:

• Healthcare NLP

• Responsible AI

• Cloud Data Platforms

• Modern Data Engineering

The purpose of this series is to explain how raw information becomes reliable, governed, contextual, and usable knowledge.

Because a reliable AI answer does not begin with the prompt.

It begins with the information pipeline behind it.

About Info2K

Info2K: Information to Knowledge

Info2K shares practical insights across Data Engineering, AI Engineering, Cloud Data Platforms, Digital Health, and responsible technology.

Our focus is on connecting technical architecture with context, governance, operational reliability, and real-world outcomes.

Website: https://info2k.co.nz

LinkedIn: https://www.linkedin.com/company/info2k