Retrieval Augmented Generation (RAG) was supposed to help with the accuracy of Large Language Models (LLMs), making it possible to venture into the realm of agentic AI, among other things. And it has helped with accuracy, to some extent. But the level of inaccuracy in many major RAG implementations is still unnecessarily high, regardless of industry. 

Take the telecom industry, for example. The GSM Association (GSMA), a longtime trade association with over 1,000 mobile operator members around the globe, asserted the following in a May 2025 article (emphasis mine):

Many organisations exploring AI, especially using Retrieval-Augmented Generation (RAG) for knowledge extraction from technical documents, encounter a frustrating performance plateau. Accuracy often stalls around 75%, even with sophisticated system design. For an industry where precision is paramount – whether in network diagnostics or customer support – a 25% error rate is unacceptable. It undermines trust, introduces operational risks, and ultimately limits the return on AI investments.

This accuracy ceiling exists because generic models lack the deep semantic understanding required. They might recognise keywords but fail to grasp the nuanced context and relationships specific to telecoms technologies and operations.

The GSMA attributes the 25% error rate to a lack of domain-specific language (DSL) model training. The fact that these models haven’t been trained on telecom industry-specific jargon (for example) is certainly a factor, but it’s hardly the only one. 

In fact, there are many different factors, regardless of industry, that prevent quality, heterogeneous knowledge and instance data from being effectively utilized together as input for LLMs. 

The best inputs available are often datasets that are either incomplete or fragmented data. Whatever ends up on the input side of an LLM’s black box is generally not what it needs to be.

How to avoid this outcome and make your initiative a success story? Understand at the beginning the nature and the depths of the potential pitfalls you’ll encounter. Here are two recommendations for starters.

1. Vector-Only RAGs Are Never Enough

Although useful, ordinary vector embeddings on their own are not sufficient to meet enterprise accuracy requirements, because they’re probabilistic in nature and aren’t optimized for cost-effective, thoroughgoing disambiguation or entity linking. 

In other words, vector databases alone won’t help companies achieve true contextual computing, a necessary precondition for accuracy. Full digitization, after all, is about building sufficiently relationship rich, accurate contexts to begin. Each context is a type of semantic digital twin designed to interoperate with the other contexts of the business.  LLMs must learn the precise meaning within each context and how those contexts interrelate. Otherwise, accuracy will suffer.

A large research organization commissioned Graphwise to develop a functioning AI assistant for their policy documents with the help of our GraphRAG system. Our system delivered a greater than 95 percent answer success rate, outperforming vector-only RAG by more than ten percentage points.

By contrast with a vector-only approach, a complete Graph RAG semantic layer approach such as the one Graphwise offers allows rich, articulated graph embeddings and similarity search via its support for Elasticsearch at far lower (total cost of ownership.  (See Sumit Pal, “Lower Your Large Language Model Costs with Graphwise GraphDB“ for more information.)

2. Sidestepping the Not Invented Here (NIH) Problem

Like traditional software development teams, AI teams are most comfortable doing things the way they’ve done them before. Semantic Graph RAG is a novel concept they may well not be familiar with. As a result, even if you’ve effectively piloted a graph database or a content or knowledge management suite doesn’t mean you’ll succeed in an implementation. If the AI team didn’t propose the solution to begin with, they may not be receptive to implementing it.

Two success factors loom large in such a scenario: Proactive leadership support and harnessing the true power of both the AI team and the teams that intuitively grasp the nature of contextualization and disambiguation. Data and knowledge management teams and those from the business units who work through domain-specific terminology and taxonomy issues should be engaged cooperatively with the AI team.   

MIT Sloan Management Review completed a four-year study of a LEGO Open Innovation initiative and published the results in 2023. That study focused on an earlier Lego Ideas consumer product idea effort and funded, crowdsourced inspiration from a 2019 acquisition of Bricklink. 

LEGO’s management approach was to encourage collaboration between consumer Lego Ideas innovators, who’d win the opportunity to design and build new products, and Bricklinks consumer-led product development efforts, which had a lesser chance of lesser approval. Aloso in the mix were design SMEs from internal product development.  The design SMEs effectively played a product vetting role. All the teams involved had vested responsibilities. All stayed committed and collaborating, despite the fact that some creators were denied chances at commercialization. (See https://sloanreview.mit.edu/article/lego-takes-customers-innovations-further/ for more information.)

Hybrid Semantic Layer Success with Human in the Loop Organizational Change

Successful semantic layers aren’t just probabilistic, and they aren’t built in relative isolation from the rest of the business. Though organizations can harness significant amounts of automation, they are built with an extensive amount of human collaboration and hard, informed work.

Those who have a visceral understanding of business process and the legacy app environment are essential to the success of a RAG initiative. And data and knowledge management teams have a crucial role to play as well, one that involves the data and content assets and what constitutes quality, discoverable and reusable assets. 

As Graphwise’s “Unlocking Enterprise Potential: The Strategic Power of a Semantic Layer” puts it, “a semantic layer, especially with well-defined ontologies and metadata management, enforces data consistency by clearly defining data definitions and relationships, thus enhancing overall data quality.” It’s that clear path to consistency and the platform’s involvement of different departments, roles. talents and mentalities that separates the Graphwise Graph RAG product from others.