What Software Engineers Get Wrong About RAG

Every week, another tutorial promises to show you how to build a Retrieval-Augmented Generation application in thirty minutes.

The demonstration works. A question is entered, several passages are retrieved, and the model produces a fluent answer. It feels like a complete application.

But it is not.

It is a useful proof of concept: enough to demonstrate the basic mechanism and test whether the idea has potential. The difficulty begins when that prototype is expected to support real users, changing information, access controls, unpredictable questions, measurable quality and dependable operation.

Confusing a successful demonstration with a production system is where many RAG projects begin to fail.

The central misconception

RAG Is Primarily a Software Engineering Problem

The largest misconception about RAG is not about embeddings, vector databases or language models. It is the belief that RAG is primarily an AI problem.

Of course, AI components are central to the solution. Embedding models represent meaning, retrieval algorithms locate potentially useful evidence, and large language models generate responses. But none of these components operates in isolation.

A useful RAG application is an information system composed of interacting subsystems. It ingests and transforms data, maintains indexes, enforces permissions, retrieves and ranks evidence, constructs context, invokes models, evaluates outputs and exposes behaviour to monitoring and operational control.

Once RAG is viewed in this way, the engineering questions become impossible to ignore. What are the subsystem boundaries? What contracts exist between them? How does data move through the pipeline? How are failures detected? How is quality measured? How does the system evolve?

Misconception 01

RAG Is Just Vector Search with an LLM Attached

It is easy to see why this view is common. Many introductory examples present RAG as a short sequence: create embeddings, place them in a vector store, retrieve the nearest chunks, and pass those chunks to a model.

Vector search is important, but similarity is not the same as relevance. A retrieved passage may be semantically close to the question while being outdated, incomplete, unauthorised, duplicated or inappropriate for the user's task.

Real retrieval may need to combine semantic search with keyword search, metadata constraints, source authority, document recency, user permissions, query rewriting and reranking. It may also need to detect that no sufficiently reliable evidence exists.

This changes the design problem. Retrieval becomes a subsystem with explicit inputs, outputs, quality thresholds and failure modes. It needs test data, versioned configuration, traceability and observable behaviour.

Misconception 02

A Working Demonstration Is a Working System

A demonstration is deliberately small and controlled. It proves that a concept is possible. A production system must prove that the concept remains useful under scale, change, uncertainty and failure.

Consider a company assistant built over internal policies. The initial prototype uses fifty clean documents and performs well in front of the project team. Six months later, the source collection has doubled. Some policies have been replaced, others conflict, and nobody can say with certainty which versions were indexed.

The model has not necessarily become worse. The system around the model has failed to manage the information lifecycle.

Production readiness therefore includes ingestion schedules, idempotent processing, version tracking, deletion handling, index rebuilding, consistency during updates, security boundaries and graceful failure. These concerns are not peripheral to RAG. They are part of RAG.

Misconception 03

Better Models Can Compensate for Poor Engineering

When quality is disappointing, teams often change the most visible component. They adopt a larger language model, switch embedding models or migrate to another vector database.

Sometimes that is justified. Model selection affects capability, latency and cost. But a more capable generator cannot reliably recover evidence that was never retrieved. A stronger embedding model cannot correct missing metadata, obsolete content or an ingestion process that silently failed.

The end-to-end answer is constrained by the quality of the complete pipeline. If relevant information is absent, fragmented across poor chunks or ranked below irrelevant material, the language model is being asked to reason from an inadequate evidence set.

A powerful model working from poor evidence is still part of a poorly engineered RAG system.

This is why system-level diagnosis matters. Before replacing a model, teams should be able to inspect the query, retrieval candidates, filtering decisions, ranking scores, selected context, generated answer and citations. Without this trace, model replacement can become an expensive form of guessing.

Misconception 04

Evaluation Can Wait Until the Application Is Finished

Many RAG projects are evaluated through informal demonstrations: several questions are asked, the answers appear plausible, and the system is judged to be working.

Plausibility is not a reliable quality measure. Fluent answers can be incomplete, unsupported or based on the wrong source. A system can produce an impressive response while failing at the task it was built to perform.

Evaluation should begin early because it shapes architectural decisions. A representative set of questions and expected evidence allows engineers to compare chunking strategies, retrieval methods, rerankers, prompts and models. It also creates a baseline against which later changes can be tested.

Measures such as Recall@k and Precision@k can help assess retrieval. Answer correctness, faithfulness, citation accuracy, abstention quality, latency and user task completion may also matter. The exact metrics depend on the application, but the principle does not.

Misconception 05

Prompt Engineering Is System Architecture

Prompt design matters. Instructions influence how the model uses context, formats answers, handles uncertainty and cites sources. Prompts should be versioned, tested and observed.

But a prompt is not an architecture.

Architecture defines responsibilities, boundaries, dependencies and flows. It determines where source data originates, how it is validated, how user permissions are enforced, how components fail, where state is stored, how behaviour is observed, and how the system can be changed without destabilising everything around it.

A carefully written prompt cannot compensate for stale indexes, missing documents, weak retrieval, absent access controls or an inability to explain why a particular source was selected.

The prompt is better understood as one configurable interface between the application and the model. It is an important component, but it remains a component.

A better way to think

Software Engineers Already Have Much of What RAG Needs

The encouraging conclusion is that software engineers are not starting from zero. Many of the capabilities required to build dependable RAG systems are already familiar.

• Architecture: decomposing complex behaviour into coherent subsystems.
• Interfaces: defining contracts between ingestion, retrieval, generation and evaluation.
• Data engineering: managing quality, lineage, versioning and change.
• Testing: replacing intuition with repeatable evidence.
• Observability: exposing internal decisions and detecting failure.
• Security: enforcing identity, permissions and information boundaries.
• Operations: planning for scale, latency, cost, recovery and evolution.

AI introduces probabilistic behaviour and new forms of uncertainty, but it does not invalidate established engineering principles. It increases the need to apply them deliberately.

The most productive question is therefore not, “How quickly can we connect a vector database to an LLM?” It is:

How do we engineer an information system whose use of AI is measurable, dependable and appropriate for its purpose?

Final thoughts

The Future Belongs to Better-Engineered AI Systems

Retrieval-Augmented Generation has made it possible to build valuable applications over private, specialised and rapidly changing information. The underlying idea is powerful, and a simple demonstration can be built remarkably quickly.

That accessibility is a strength. It is also a source of confusion. The ease of assembling a prototype can hide the depth of engineering required to create a system that users and organisations can depend on.

Software engineering transformed web applications, distributed systems and cloud platforms. AI systems will be no different. The organisations that succeed will not necessarily be those with access to the largest models. They will be those that understand their problems, manage their information, measure quality and build systems designed to evolve.

Retrieval-Augmented Generation is not the engineering challenge. Engineering Retrieval-Augmented Generation is.