When RAG Is Not Enough: How to Decide Between Semantics, Ontology, and Knowledge Graphs 

Most RAG systems already in production are answering questions they cannot reliably answer, and the team running them usually doesn’t find out until a query depends on a relationship the pipeline never represented.  RAG can retrieve relevant passages, but relevance alone does not guarantee that the model interprets them consistently or combines them correctly.  Different documents may use the same term in different ways, describe related entities without explicitly connecting them, or provide only fragments of the information needed to answer a question. For example, the same part number can be referred to as SKU in one system and material ID in another.  

When the model must reconcile those fragments or apply business rules, or maintain context across multiple steps, gaps in the retrieved evidence and ambiguity in its interpretation can lead to unreliable answers. 

What Comes After RAG?

RAG is suitable for retrieving information from unstructured documents, but many enterprise questions also depend on structured data and relationships between entities.  

For example, a question such as “Which customers bought products affected by the latest supplier recall?” may require the system to connect customer records, orders, products, and supplier information.  

This is where semantics, ontologies, knowledge graphs, and context engineering can extend a RAG architecture. Each addresses a different need:  

• Semantics helps interpret meaning 
• Ontologies define domain concepts and relationships 
• Knowledge graphs represent connected entities for querying and traversal 
• Context engineering determines what information and instructions the model receives at each step.

The Question That Gets Lost

The common response is to treat all four as simultaneously necessary. That assumption has two costs: teams that move forward build more than the problem requires, and teams that hesitate do not move at all. In both cases, the question that gets lost is the only one that matters: which failure mode are you actually trying to solve?  

For example, a knowledge graph layered on top of an unresolved semantic-consistency problem can still return confidently wrong answers. It adds another component to the stack without addressing the original failure. 

Semantics, ontology, knowledge graphs, and context engineering address four different failure modes, not escalating severities of the same failure mode. They get conflated because they occupy overlapping territory in data management.  

Stacking all of them without first diagnosing which capability is needed introduces unnecessary governance, maintenance, and operational overhead.     

Vendors reinforce this by bundling several of these capabilities under unified product terminology, making the undifferentiated stack look like the safe default. However, this doesn’t make a misdiagnosis go away; rather, it adds more components to maintain without addressing the original failure.  

What Each Layer Actually Does

It is critical to understand that semantics, ontology, knowledge graphs, and context engineering address different architectural constraints. Choosing between them starts with understanding which part of the system each one changes. So, let’s understand them in detail:

The Decision Framework

The choice between semantics, ontologies, and knowledge graphs should follow the requirements of the query, not a presumed progression toward greater architectural sophistication.  

A team does not need all four simply because its RAG application has reached enterprise scale. The useful question is: what must the system understand, connect, or retain that its current architecture cannot handle reliably?  

Let’s look at four supply-chain scenarios where that distinction becomes clearer.  

Where Teams Get This Wrong

Getting the framework right on paper does not prevent teams from misapplying it in production. The same three misdiagnoses recur: teams commit to a layer without tying it to the specific failure mode it needs to solve. The mismatch surfaces later as unnecessary cost, integration effort, and operational complexity. 

The Graph-First Reflex

Teams introduce a knowledge graph before confirming that their queries require relationship traversal. The result is extra integration and maintenance work without a measurable improvement in answer quality. The graph becomes another system to maintain rather than a solution to the retrieval problem. 

The Ontology-as-Data Fallacy

A team defines domain concepts and relationships and treats that definition as equivalent to having complete, accurate, queryable knowledge. The ontology may be logically consistent, but the application still fails to answer reliably when the underlying facts are missing, outdated, or disconnected. 

The Context-Window Cure-All

A team responds to unreliable answers by supplying more text without investigating whether the underlying failure sits in retrieval, entity relationships, data quality, or interpretation. More context adds noise and consumes token budget while leaving the underlying failure untouched. 

MathCo in Practice: When the Architecture Looked Right and Still Failed

The misdiagnoses above are easier to recognize in real engagements. We’ve seen this play out in our own work. At MathCo, we have worked with two cases where the architecture looked right on paper, but the system still failed to solve the problem it was built for:  

CASE 1 – When the graph was there but never used

A global commercial real-estate firm came to us to build a weekly executive briefing that connected market signals with its portfolio data. 

On the paper, the setup looked solid. There was a knowledge graph, internal data, and a claim-scoring pipeline that filtered tens and thousands of signals into a quality-checked set.  

However, the output looked more like a news summary rather than an intelligence briefing. When we traced the retrieval layer, we found that the system was using semantic search against the graph, while the Cypher queries were mostly running aggregations instead of following relationships across multiple hops. 

The graph had 3,000+ quality-scored claims, property-level portfolio data, and competitor and customer relationships. But because the queries weren’t traversing those connections, the system wasn’t getting the deeper context the graph was built to provide.  

So, we rebuilt retrieval around the explicit Cypher traversal and split the reasoning process into six stages. We also replaced the single LLM prompt structure with smaller prompts, each handling one task.  

CASE 2 – When the ontology was added too early

This is a pattern we have seen across different engagements. Teams often start by building a detailed ontology before even testing what the system actually needs to retrieve. This means spending weeks defining entities, relationships, and constraints.  

But then, real queries reveal a different problem, which is that the same concept is described differently across the source systems. A query might refer to a “customer,” while one source called it an “account” and another used a different identifier altogether. 

An ontology doesn’t fix that by itself. The first step is semantic alignment: mapping those different terms and identifiers to the same underlying concept. 

For such cases, we scaled the ontology back and focused on the retrieval issue first. The approach was simple: start with the retrieval problem and then decide how much ontology is actually needed.  

How to Start

Before adding another layer to a RAG system, the first step is to establish where the current architecture falls short. 

• Trace the failure: Take a set of queries the system gets wrong and inspect the retrieval results, source data, entity resolution, and reasoning steps. Determine whether the application is struggling with retrieval, semantic consistency, domain representation, connected-data queries, or runtime context. 
• Choose the smallest architectural change:  A layer earns its place because a requirement demands it, not because it is available. So, test whether improved semantic retrieval resolves the issue. If the system needs consistent domain definitions, consider an ontology. If it must traverse connected facts, evaluate a knowledge graph.  
• Validate against real workloads. Compare the revised architecture with the existing system using representative queries. Check answer quality alongside latency, integration effort, data maintenance, and operating cost. 

A sophisticated stack is not inherently a successful one. The architecture that survives contact with production is the one built to the requirement that was diagnosed, not the one that deployed the most available capability.