##  [Heaps' Law](/heaps-law-1) 

 Definition

An empirical relationship observed in text corpora that the number of distinct word types V(N) grows sublinearly with the total token count N and is commonly modelled as V(N) ≈ K · N^β with 0 < β < 1 and constant K depending on language and tokenization. It characterizes diminishing rates of new vocabulary discovery as a corpus grows.

 

 

 

 

 

 





## Principle

Principle

Because new tokens increasingly repeat existing types as corpora expand, incremental addition of text yields progressively fewer novel types; therefore vocabulary size should be forecasted with a sublinear growth model rather than a linear extrapolation.

 

 

 

 

 





## Demonstration

Demonstration

Illustrative scenario → Indexing a news-article collection: after processing 1 million tokens the indexer observes 70,000 distinct types; after doubling tokens, type growth is noticeably less than double, consistent with V(N) ≈ K·N^β. Recognition → Indexers use a fitted K and β to estimate dictionary size and plan storage and stemming. Action → Tokenization rules and stopword lists are refined to control index growth. Consequence → Indexing and compression budgets are set realistically, and experimental designs account for lower marginal vocabulary discovery.

 

 

 

 

## Misapplication

Misapplication

Assuming a single universal β or treating the relation as exact for any tokenization. The error is ignoring that β and K vary with language, genre, morphological richness, preprocessing (e.g., lemmatization), and corpus heterogeneity.

 

 

 

 

 





## Consequence

Consequence

Properly used, Heaps’ law guides capacity planning for indexes, vocabulary thresholds in search systems, and expectations for corpus annotation effort; misused, it leads to under‑ or overprovisioning when model parameters are not empirically estimated for the specific corpus and preprocessing pipeline.

 

 

 

 

## Reversal

Reversal

In artificially generated or adversarially constructed token streams (e.g., many unique identifiers) or corpora with extreme morphological productivity and no normalization, β may approach 1 and vocabulary may grow nearly linearly, so Heaps’ sublinearity assumption fails.

 

 

 

 

 





## Boundary

Boundary

Clearly within: large natural language corpora with standard tokenization and minimal synthetic identifiers. Boundary case: agglutinative languages without morphological normalization where apparent type growth is higher. Clearly outside: datasets composed mainly of randomly generated unique strings or database keys.

 

 

 

 

 





## Semantic Tension

Semantic Tension

Completeness (exhaustively capturing all types for lexicographic purposes) ↔ Practicality (indexing and storage limits that require token filtering or normalization).

 

 

 

 

 





## Synthesis

Synthesis

Heaps’ Law is a practical, parametrized expectation about vocabulary discovery that must be empirically fitted and interpreted relative to tokenization and linguistic morphology; it predicts diminishing returns but does not determine which tokens are meaningful for downstream tasks.