TF-IDF: Why Rare Words Matter in Search

TF-IDF is one of the oldest techniques in information retrieval, yet its intuition remains useful in modern RAG systems. Understanding it helps explain retrieval, ranking, and why some words tell us more about a document than others.

What is TF-IDF?

TF stands for Term Frequency, and IDF stands for Inverse Document Frequency. Together, they measure how important a term is to a document relative to a collection of documents.

Frequent in one document + rare across the corpus = informative.

Why do we need it?

Words such as “the”, “is”, and “and” appear in many documents and often do little to distinguish them. In a particular collection, terms such as “transformer”, “retrieval”, or “embeddings” may be more informative. How informative they are depends on that collection.

Term frequency

TF measures how often a term appears in a document. Consider this sentence:

RAG uses retrieval. Retrieval improves grounding.

retrieval = 2
RAG       = 1
grounding = 1

A higher frequency can give a term a higher TF score. But frequency alone is not enough: if “data” appears in every document, it may do little to help distinguish one from another.

Inverse document frequency

IDF measures how rare a term is across the documents. Its basic intuition is:

IDF(term) ≈ log(total documents / documents containing term)

Common terms receive lower IDF weights; rarer terms receive higher ones. Implementations may use smoothing or other variations.

Putting them together

TF-IDF = TF × IDF

A term receives a high weight when it appears frequently within a document but occurs in relatively few documents across the corpus.

Suppose a collection has 1,000 documents. “the” appears in 990, while the phrase “vector database” appears in 15. If we index that phrase and compare equal term frequencies, the rarer phrase receives a higher TF-IDF weight.

From TF-IDF to BM25

BM25 builds on related term-weighting ideas, adding term-frequency saturation and document-length normalization. Repeating a term has diminishing returns, and document length is taken into account. Understanding TF-IDF makes these ranking choices easier to grasp.

Why it matters for RAG

Modern RAG systems can combine keyword retrieval with semantic retrieval. TF-IDF offers a useful foundation for understanding the keyword side and why rare, specific terms can matter so much.

Sometimes, the best way to understand cutting-edge AI is to study the classic search algorithms that came before it.