##  [TF–IDF Weighting](/tf-idf-weighting-0) 

 Definition

A family of lexical weighting schemes that assign to each term in a document a numerical score equal to a function of the term's frequency in that document (TF) multiplied by a function of the term's rarity across a corpus (IDF). Common formulations use raw or normalized term frequency and IDF = log(N / df) or variants; TF–IDF quantifies a term's importance for distinguishing a document within a corpus under a bag‑of‑words model.

 

 

 

 

 

 





## Principle

Principle

TF–IDF increases for terms that are frequent in a specific document but infrequent across the corpus: high TF indicates local importance, high IDF indicates global specificity; their product highlights discriminative terms while down‑weighting common words.

 

 

 

 

 





## Demonstration

Demonstration

Illustrative scenario → Corpus size N = 1000 documents. Term t appears in df = 10 documents, and in the target document appears tf = 4 times. Using IDF = log(N/df) = log(1000/10) = log(100) ≈ 4.605 (natural log) and raw TF = 4, TF–IDF ≈ 4 × 4.605 ≈ 18.42. Different TF or IDF normalizations yield different numerical values but the same relative ordering principle.

 

 

 

 

## Misapplication

Misapplication

Treating TF–IDF as a semantic or probabilistic model of meaning (it is a lexical weighting heuristic), or using raw TF without length normalization which biases long documents. Also incorrect is assuming IDF is stable on very small or dynamically changing corpora — df estimates can be noisy.

 

 

 

 

 





## Consequence

Consequence

TF–IDF produces sparse, interpretable feature vectors useful for lexical retrieval, document ranking, and many classical NLP tasks; its effectiveness depends on corpus representativeness and preprocessing (stemming, stopword removal, tokenization) and it can be outperformed by semantic embeddings for meaning-based matching.

 

 

 

 

## Reversal

Reversal

When semantic matching beyond lexical overlap is required (synonymy, paraphrase, contextual meaning), neural embeddings or methods using external knowledge often outperform TF–IDF. When the corpus is tiny or extremely dynamic, IDF estimates may be unreliable and alternative weighting or smoothing may be necessary.

 

 

 

 

 





## Boundary

Boundary

Clearly within: bag‑of‑words retrieval and classification tasks where lexical frequency and corpus rarity are informative. Boundary case: short text fragments where TF is low and IDF variance is high. Clearly outside: tasks requiring deep semantic understanding, cross-lingual matching without translation, or contextualized token representations.

 

 

 

 

 





## Semantic Tension

Semantic Tension

Lexical Specificity ↔ Semantic Generalization — TF–IDF prioritizes rare lexical signals that distinguish documents, which can conflict with goals of capturing semantic equivalence across different surface forms.

 

 

 

 

 





## Synthesis

Synthesis

TF–IDF is a pragmatic, corpus‑based discriminator: it is best used when lexical distinctiveness is a reasonable proxy for relevance or class membership, and it should be combined or replaced by semantic methods when surface-form mismatch dominates the task.