How to use the zipf and heaps law text fit
- Paste your text into the box. A few hundred words or more gives a steadier fit than a sentence or two.
- Choose how many ranks the Zipf fit should use. All words is the plain version. The top 20, 50 or 100 ranks leave out the long tail of words that appear once.
- Read the Zipf exponent and the Heaps K and beta in the summary, with the R squared of each fit to show how well a straight line describes the text.
- Check the table of the most frequent words, then copy or download the full ranked list as a tab-separated file.
Formula
Zipf: ln(count) = a - s x ln(rank), exponent s
Heaps: V(n) = K x n^beta, fitted as ln V = ln K + beta x ln n
Both lines are found by ordinary least squares
R squared = share of the variation the line explains
Words are lowercased and split at anything that is not a letter or digit. An apostrophe inside a word, as in don't, stays part of the word. Heaps uses every position in the text, so V(n) is the number of different words among the first n.
Counts of 100, 50, 33, 25 and 20
Suppose the five most frequent words appear 100, 50, 33, 25 and 20 times. If Zipf's law held exactly the counts would be 100, 50, 33.3, 25 and 20, so the line of ln(count) against ln(rank) has a slope of about -1.
The least squares fit gives a slope of -1.0009, so the Zipf exponent is 1.0009 and R squared is 0.9999. The tool reports the exponent as a positive number, which is the s in count proportional to 1 / rank^s.
What the two laws describe
Zipf's law is an observation about how word counts fall off. The most frequent word in a long English text appears about twice as often as the second, three times as often as the third and so on, so a plot of count against rank on log scales is close to a straight line. The exponent is often near 1, but it changes with the language, the kind of text and the length.
Heaps' law is about vocabulary. As a text grows, new words keep appearing, but more slowly each time. The number of different words grows like K times the length raised to a power beta below 1. For English text beta is often around 0.4 to 0.6, and K is often between 10 and 100. These are descriptions of what texts tend to do, not rules that every text must follow.
Reading the fit
A high R squared means a straight line describes the points well, not that the text is special. Short texts give noisy results, because the long tail of words that appear once makes a staircase on the log-log chart. Limiting the fit to the top ranks gives a cleaner line and a different exponent, so quote the setting you used when you compare texts.
Hapax legomena, the words used exactly once, are counted in the summary. A text with a very large share of them is usually short or full of rare terms. To compare two writers, use passages of similar length and the same ranks setting.
Assumptions and limits
- Words are lowercased, so The and the count together. Numbers made of digits count as words.
- A hyphenated word splits into two words, and an apostrophe inside a word keeps it together.
- The fit uses ordinary least squares on the logarithms, not a maximum likelihood estimate, so exponents can differ from those in research papers.
- Nothing is uploaded. The text stays in your browser and up to 300,000 characters are read.
Frequently asked questions
Why is my Zipf exponent not 1?
Real texts only approximate the law. The exponent depends on the language, the topic, the length of the text and how many ranks you fit, so it is often near 1 but can sit noticeably above or below it.
What does Heaps beta mean?
It is the power in V = K x n^beta. A beta of 0.5 means the vocabulary grows with the square root of the text length, so doubling the text adds about 41% more different words.
Should I fit all ranks or only the top ones?
The tail of words that appear once or twice forms flat steps that pull the line down. Fitting the top 100 or 50 ranks gives a line that follows the frequent words, while all ranks uses everything.
Can I use this for other languages?
Yes for languages that separate words with spaces. Text without spaces, such as Chinese or Japanese, needs a word splitter first, because this tool would see a whole sentence as one word.