Tom Hundley
Vector databases have become the default foundation for retrieval-augmented generation (RAG) systems. And for good reason: they are fast to deploy, scale well with unstructured data, and provide genuinely useful semantic search capabilities. When you ask a question, a vector database can find passages that mean something similar to your query, even if they do not share exact keywords.
But there is a problem. Semantic similarity is not the same as relevance. And relevance often depends on relationships that vector embeddings cannot capture.
Consider this query against an enterprise knowledge base: "What impact did our Q3 product launch have on customer satisfaction in the healthcare vertical?" A pure vector search might return documents about product launches, other documents about customer satisfaction, and perhaps some healthcare-related content. What it struggles to do is connect the dots: find the specific product launched in Q3, trace its relationship to customer feedback, and filter all of this by industry segment.
This is where knowledge graphs enter the picture. Not as a replacement for vector search, but as a complementary layer that captures the structured relationships that embeddings miss.
Vector embeddings work by converting text into high-dimensional numerical representations. Documents with similar meanings cluster together in this embedding space. When you search, the system finds vectors that are "close" to your query vector using distance metrics like cosine similarity.
This approach excels at handling the ambiguity of natural language. A search for "revenue growth" will find documents about "sales increases" or "expanding income" even if those exact phrases never appear. The semantic understanding is real and valuable.
But embeddings have fundamental limitations:
Relationship blindness. The vectorization process compresses meaning into a single point in space. In doing so, it loses the explicit connections between entities. A document mentioning both "Project Alpha" and "Q3 deadline" gets embedded as a single vector. The relationship between the project and the deadline, the fact that one constrains the other, is not preserved in a queryable way.
Multi-hop reasoning failures. When answering a question requires following a chain of connections (A relates to B, B relates to C, therefore A has some relationship to C), vector search struggles. Each hop introduces potential errors, and the system has no explicit path to traverse.
Context collapse. Two documents can be semantically similar but contextually opposite. "The project succeeded" and "The project failed" might embed relatively close together because they share vocabulary and structure. The crucial difference is in the relationship between subject and outcome, which is precisely what embeddings flatten.
Explainability gaps. When a vector search returns results, it is difficult to explain why those results are relevant. The path from query to answer passes through an opaque transformation into and out of embedding space. For applications requiring auditability, this is a significant limitation.
These are not theoretical concerns. Organizations building production RAG systems encounter these limitations regularly, particularly when dealing with domain-specific knowledge where relationships carry essential meaning.
A knowledge graph represents information as a network of nodes (entities) and edges (relationships). Rather than flattening meaning into vectors, it preserves the explicit structure of how concepts connect.
The node might be "Project Alpha." The edge might be "has_deadline." The connected node might be "2024-03-31." This triple, subject-predicate-object, is the atomic unit of knowledge graph representation. Simple on its own, but powerful when millions of such triples interconnect to form a queryable web of knowledge.
For AI systems, knowledge graphs provide several capabilities that vector search cannot:
Relationship-aware retrieval. When you query a knowledge graph, you can specify the type of relationship you care about. "Find all projects managed by this person" is a precise query with a precise answer. No semantic approximation required.
Multi-hop traversal. Following chains of relationships is natural in a graph. "Find customers who purchased products from suppliers located in regions affected by this supply chain disruption" requires multiple hops: customer to product, product to supplier, supplier to region, region to disruption. A graph database handles this directly.
Logical inference. Some relationships can be inferred from others. If A reports to B, and B reports to C, then A is in C's organization. Knowledge graphs can make these inferences explicit and queryable.
Transparent reasoning paths. When a graph-based system returns an answer, it can show the exact path it traversed to reach that conclusion. This explainability is essential for high-stakes domains like healthcare, legal, and financial services where decisions must be auditable.
The challenge, of course, is that knowledge graphs require structured data. Someone or something must define the entities, identify the relationships, and maintain the graph as knowledge evolves. This is not a trivial undertaking.
In 2024, Microsoft Research released GraphRAG as an open-source project addressing a specific weakness in traditional RAG: the inability to answer questions that require understanding across an entire dataset.
The GraphRAG methodology works in two phases. During indexing, an LLM processes the source documents to extract entities and their relationships, building a knowledge graph automatically from unstructured text. It also identifies "communities" of related entities and generates summaries at different levels of the graph hierarchy.
During retrieval, this graph structure enables two types of search that baseline RAG cannot perform well:
Local search retrieves information about specific entities and their immediate relationships. When you ask about a particular project, person, or concept, local search can pull in the relevant subgraph of connected information.
Global search synthesizes information across the entire knowledge graph. The question "What are the main themes in this dataset?" is fundamentally a summarization task, not a retrieval task. GraphRAG uses the pre-computed community summaries to answer such questions without needing to retrieve and process every document.
Microsoft's research indicates that GraphRAG can provide more comprehensive answers than baseline RAG while using significantly fewer tokens during the query phase, with dynamic community selection achieving up to 77% cost reduction compared to static global search. The structured graph representation enables more targeted retrieval rather than pulling in semantically similar but potentially irrelevant content.
The trade-off is computational cost. Building the knowledge graph requires multiple LLM calls per document chunk: extracting entities, identifying relationships, and generating summaries. This indexing cost is substantially higher than simple vector embedding.
LightRAG, which emerged in late 2024, attempts to address this cost issue through a dual-level retrieval approach that achieves comparable accuracy with significant reductions in token usage during retrieval, with LightRAG achieving less than 100 tokens compared to GraphRAG's 600-10,000+ tokens during retrieval.
In practice, the most effective production systems do not choose between vector search and knowledge graphs. They combine both in hybrid architectures that leverage the strengths of each approach.
Pattern 1: Vector-first with graph enrichment. The system performs an initial vector search to quickly identify semantically relevant documents. These results are then passed to a graph layer that enriches the context with relationship information. If the vector search returns documents about "Project Alpha," the graph layer can pull in related entities: the project manager, the timeline, the dependencies, the stakeholders.
This pattern is relatively easy to add to existing RAG systems. The vector search remains the primary retrieval mechanism, with the graph providing contextual augmentation.
Pattern 2: Graph-first with vector fallback. For highly structured domains, the primary query runs against the knowledge graph. When the graph query returns results, vector search can find additional relevant content within that focused context. When the graph query returns nothing (perhaps the entities in question are not yet in the graph), the system falls back to pure vector search.
This pattern requires a more mature knowledge graph but provides better precision for queries that map well to the graph structure.
Pattern 3: Parallel retrieval with fusion. Both vector search and graph queries run simultaneously. A fusion layer combines the results, weighting them based on query characteristics. Questions with identifiable entities and relationship keywords lean toward graph results; more abstract or exploratory questions lean toward vector results.
This pattern offers flexibility but adds complexity to the retrieval pipeline. The fusion logic becomes a critical component that requires tuning for specific use cases.
Pattern 4: Graph-structured vector search. Some systems embed the graph structure itself into vector space, creating representations that capture both semantic meaning and relational context. This is an active area of research rather than a settled production pattern.
Not every RAG system needs a knowledge graph. The added complexity, cost, and maintenance burden must be justified by genuine improvements in retrieval quality. Here is a practical decision framework:
Favor pure vector search when:
Consider adding knowledge graphs when:
A hybrid approach makes sense when:
The honest reality is that most organizations starting their AI journey should begin with vector search. It provides real value quickly and teaches important lessons about retrieval quality, embedding strategies, and user query patterns. Knowledge graphs should enter the picture when vector search limitations become evident in your specific use case.
If you decide to pursue a hybrid approach, several practical challenges await:
Knowledge graph construction. Building the graph is not trivial. You can use LLMs to extract entities and relationships from documents (as GraphRAG does), but this requires careful prompt engineering and quality validation. Extracted relationships can be noisy or incorrect. Domain experts may need to review and refine the schema and the extracted content.
Neo4j offers an LLM Knowledge Graph Builder that automates much of this process, supporting multiple LLM providers and various document formats. Other tools exist, but expect to invest significant effort in tuning for your specific domain.
Schema design. What entities matter in your domain? What relationships connect them? Schema design requires domain expertise and affects everything downstream. An overly rigid schema misses important connections; an overly flexible schema becomes difficult to query meaningfully.
Graph maintenance. Unlike vector embeddings (which can be regenerated from source documents relatively easily), knowledge graphs accumulate semantic decisions. When the underlying documents change, you must decide: regenerate affected portions of the graph? Apply incremental updates? Maintain versioning?
Query complexity. Graph query languages (like Cypher for Neo4j) have learning curves. Your application layer must translate user intent into appropriate graph queries, vector searches, or combinations of both. This translation logic is non-trivial.
Latency considerations. Graph traversals can be slower than vector similarity searches, particularly for complex multi-hop queries on large graphs. Caching, query optimization, and thoughtful index design become important for production performance.
Cost and resources. Graph databases add infrastructure costs. LLM-based graph construction adds significant API costs during indexing. The ongoing maintenance requires engineering attention. Budget accordingly.
We are at an interesting moment in the evolution of AI retrieval systems. Vector search has proven its value and will remain foundational. Knowledge graphs have demonstrated their potential but have not yet achieved the same ease of adoption.
The emerging pattern is clear: production AI systems will increasingly combine both approaches. The question is not whether to use knowledge graphs, but when and how to integrate them effectively.
For organizations building AI applications today, my practical recommendations are:
Start with vector search. Get a working RAG system in production. Learn what queries work well and where retrieval quality falls short. These failure patterns will inform whether and how to add graph capabilities.
Identify relationship-heavy use cases. Look for queries that your users want to ask but that vector search cannot answer well. If these queries involve following connections between entities, a knowledge graph may help.
Experiment with hybrid approaches. Tools like Neo4j's graph + vector capabilities, Amazon Neptune's GraphRAG support in Amazon Bedrock, and the Microsoft GraphRAG project provide entry points for experimentation without building everything from scratch.
Plan for maintenance. Before committing to a knowledge graph, understand who will maintain it. Graph construction is not a one-time effort. If you cannot resource the ongoing work, the graph will degrade in value over time.
Measure what matters. Define retrieval quality metrics specific to your use case. A/B test vector-only against hybrid approaches. Let the data guide your architecture decisions rather than hype cycles.
The goal is not to have the most sophisticated architecture. The goal is to retrieve the right information so your AI system can provide useful answers to the people who depend on it. Sometimes that requires knowledge graphs. Sometimes it does not. The skill is knowing the difference.
Discover more content:
