I worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN".

It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.

It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.

I think people also overestimate the need for full text search when the one doing the querying is an LLM. If your underlying data is structured records, like a customer database, while humans might not have time or skills to figure out that when they want to search by phone number they need to do a join from the contacts table to the users table and normalize the phone number to look up first, making it best to just surface phone numbers as part of the data that is full/text-indexed… an agent is quite happy to handcraft the right SQL to find records that match on a specific field, given the right SKILLS.md and schema information. Turning fuzzy searches into exact DB lookups is a great way LLMs can augment users.

(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)

I indexed thousands of documents into a SQLite database with an FTS5 index, plugged it into DeepSeek v4 Flash and got better much better results than any other commercial solutions my company has tried in the past.

The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.

RAG is art. I have a very straight forward setup that is highly modular.

RAG is routing and decision making.

I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.

I agree with you.

Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.

It maintains itself.

Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.

I found my niche with RAG selling and I build them myself.

I take pride in them.

So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.

And there are regulated businesses where certain information is required to be always factual correct - pricing for example.

Vector search becomes a liability for this use case.

So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.

I love this kind of stuff.

And there is personal information etc.

Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.

And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.

It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.

I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.

RAG gives me dbCORE vibes so to say.

Good luck and fun with your RAG systems.

Re: the rube goldberg machine of diminishing returns

https://www.anthropic.com/engineering/contextual-retrieval

This is from two years ago, but I think it's still SotA?

And you get bm25 for free with so many modern setups! I do still love to experiment with tuning semantic search for your specific corpus via various kinds of embeddings, but bm25 is hard to beat.

Can you elaborate? We have technicians searching in different languages. Also our knowledge base is often in different languages. I just don't see how full text search can work? Maybe in a problem space like a wiki where people always know what to search for?

I thought text search was always the first thing you try, then fuzzy search, then you go for RAG

How long have "large scale RAG systems" really existed in the first place? I'm always surprised at this, given how new all this really is, relatively speaking.

> people vastly underestimate full text search

It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.

RAG only makes sense if you have an LLM review the results, pick the most relevant ones and iterate further if there's a need running another query and repeating the process. Raw dump of vector search (even with reranking) is asking for troubles (or rather weird user questions like 'why this crap popped up in the results?')

This depends on your users and doc corpus. On finance docs and with users that use jargon and acronyms heavily, BM25 can fail on trivial queries... you end up encoding a masters in economics into the query re-writing logic.

Double hard if you're dealing with private market finance customers with their own ideas on what "common" terms mean. I tried to replace the embedding/ingestion pipeline multiple times and nothing I tried was better over a large amount of documents. Performance sucked, the agent was re-writing and re-trying queries over and over until it found what it wanted, and vector search with a little work up front was worlds better (though it was expensive)

On my last go at making my own rag i still got better results by collecting the data and uploading to a project in open(butclosed)ai. My own rag, used by an agent was giving poorer results, and even the agent prefered (derailed)to not use it and look for the info itself rather than using the rag

I believe the second Suggestion solves 95% of my problems. I want a system where i can describe my search and the system generated 5-15 keywords for a query

Yes, and don’t forget, LLMs are very good at tagging, so it’s not even that painful to backfill the corpus.

have you also explored graph based mechanisms to bypass some of these challenges - especially around noise generated due to similarity?

I don't know what level of quality is required for this site but RAG is trash its just trash. its magic beans.

> With search, and by extension RAG, the principle of shit in, shit out applies

Similar to SEO on marketing pages, we started rewriting product docs around the idea that it will be consumed by a RAG. Mostly by putting a lot of focus on well structured headlines, thinking more carefully about technical terminology vs common human-language questions, occasionally using variations of keywords in the text, etc. This applies to pure LLM consumption too, not just hybrid search.

Once you start tracking what users are asking you learn to adapt the documentation around it. And LLMs can also suggest improvements by comparing questions vs search results vs LLM responses.

> RAG is basically good old information retrieval with LLMs doing the querying.

No - rag is doing search before you call the llm to give it context from some corpus like your helpdesk articles.

Your brain is incredibly adept at pattern recognition; it doesn't focus on LLM-generated text for the same reason it doesn't stare at wallpaper.

We've all learnt that it's not really communication, and so can be dispensed with.

It’s largely because LLMs are reaching for many different types of adjectives or verbs in the same sentence, in a jarring way. While embedding it in a confidently declarative sentence. Everything sounds like some profound insight, dialed to an 11, but written as poetry. Especially those headings. With the short sentences.

I'm Becoming AI-Blind: https://news.ycombinator.com/item?id=49386699

the biggest giveway is actually not the writing style, but the content

"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model

"This is where the real trade-off lives – not cost, but speed."

That is where I stopped

Everything that is generate from a LLM is shit, I don't know why people continue using it. I'm waiting for this bubble to explode once for all so we can return doing things in the sane way.

In the same boat here.

The enshittification of the web, now powered by AI.

The audience for this piece is already very familiar with RAG. I don't want articles discussing e.g. OLED screens telling me what the acronym is - that would be a sign that the article is far below the level that I need.

For those times you need to Red Amber Green your BM25

This has Claude written all over it.

"Recipe 4: On-The-Fly Embedding (The Fresh Data Play)

The insight If your data changes frequently, why pay to re-embed everything?"

This reads like every Claude generated presentation I've seen.

> embed everything the first time

This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”

I'm sorry is this ironic or not? doesn't sounds simple at all

Yep, lock into some vendor from day 1. Great idea!

Embeddings are reasonably simple, but it’s a journey to get there, and I am very proud of the dog-heavy explainer I wrote on them: https://sgnt.ai/p/embeddings-explainer/

I think it's VERY project specific. If you are looking for anything technical at all, then keyword search almost always does better (in my experience). I'd actually recommend starting with keyword search, and then expanding with embeddings after you have a better idea of what your users are trying to determine.

How is pgvector with Sentence Transformers, a CPU-only embedding model, compared to a model hosted by OpenAI?

[flagged]

That sounds more like semantic search and vector db.

RAG is simply fetching external data (retrieval) and adding it to LLM context (augmenting) prior to generating a final response.

Any time LLMs do a grep or a web search to answer the query, it’s RAG. Many people use vector db for their own RAG implementation bc of the semantic search benefits.

It seems not many RAG compare themselves across the same benchmarks. https://ggozad.github.io/haiku.rag/ Does an ok job. The part I don’t see being discuss is the whole RL agents writing code to perform RAG queries. It’s one thing haiku-rag does that’s interesting and would like to know what other RAG have that agentic querying with benchmarks

author here - that's an amazing idea. would be an insanely large article though - maybe will write up a series

cursor still uses embeddings and theyve found it works better than just grep

https://cursor.com/blog/semsearch

Ok? I'm not seeing how that is interesting, you're exclusively focusing on coding which requires precise substring locations. Google is basically almost entirely driven by embedding models now.

Don't leave us hanging! How do you set it?

You are right, now I noticed "Real talk" and "Why this is underrated" and I can't unsee it.

The whole embedding thing which converts “tokens” to vectors, which you then store in a vector database so that you can later query by vector distance, seems to be LLM specific technology, no? As far as I know the vectors look a lot like the weights in a LLM itself which is why the vector search also works with some level of intelligence.

It's just information retrieval packaged as something new.

[flagged]

Thanks for the nice article.

If you're looking for feedback, I'd suggest adding a short demo at the end. It would be nice to see you send it a prompt that says, "hey, remember this" and then tell it to recall that memory. Or show what the memories look like on their way to the model. Are the memories added to the context on every turn or only once per conversation?

author here - apologies for it not being clear. the idea here is:

step 1: sparse index retrieval (FTS/BM25) - say with k = 10 step 2: re-rank the 10 records using embeddings

the difference in this approach is during step 2, you convert text to embeddings on the fly - when you're running the retrieval pipeline, meaning you don't need to have all of your corpus pre-embedded in a vector db

How would you use a big LLM for retrieval?

author here - thanks, I'm honored you would use my agents :)

author here - thank you!

Text search is not ideal for that. I such cases embedding works better