How to use the lexical diversity calculator
- Paste the text you want to measure. The longer it is, the more reliable the numbers.
- Set the window size for the moving average, 25 by default. Use about 50 for longer texts.
- Read the type-token ratio and the other measures. The moving average is the one to compare between texts of different lengths.
- Compare texts only with the same measure and, ideally, similar lengths.
Formula
TTR = types / tokens
Guiraud = types / sqrt(tokens)
Herdan C = ln(types) / ln(tokens)
Maas a-squared = (ln tokens - ln types) / (ln tokens)^2
MATTR = average of types / window over every window of that many words
Tokens are all words and types are the different words, ignoring capital letters. The moving average slides a window one word at a time over the text and averages the ratio found in each.
A short passage about a fox and a dog
The starting text has 39 words and 25 different words, so the type-token ratio is 25 / 39 = 0.641. Guiraud is 25 / sqrt(39) = 4.0032 and Herdan C is ln 25 / ln 39 = 0.8786. Maas a-squared is 0.0331. Twenty-one of the 25 words are used only once, which is 84% of the types.
With a window of 25 words, the moving average type-token ratio is 0.6587, the mean of 15 window ratios. A text of "a a a a" has 4 words and 1 type, a ratio of 0.25. These figures are checked against a separate Python calculation that counts the distinct words in each window.
Why length matters
The type-token ratio falls as a text gets longer, because common words repeat while new words appear more slowly. A short text will always look more varied than a long one by this measure alone. Guiraud, Herdan and Maas try to reduce that effect with a logarithm or a square root, but they do not remove it.
The moving average type-token ratio avoids the problem by always looking at windows of the same size, so texts of different lengths can be compared. It needs at least as many words as the window.
What the numbers can and cannot tell you
These measures describe how repetitive the vocabulary of one text is. They do not measure quality, difficulty or the skill of the writer, and different genres and languages have different normal values. A technical text may repeat key terms on purpose. Use them to compare drafts of the same piece, or texts of similar kind and length. See also the Word Frequency Counter, the Word Counter, the Longest Word Finder and the Repeated Word Finder.
Assumptions and limits
- Words are runs of letters and numbers with apostrophes and hyphens allowed inside. Capital letters are ignored.
- The text can be up to 1,000,000 characters. No stemming is done, so walk and walked are different words.
- Words used once are counted as a share of the different words.
- The default window of 25 is an example. Choose a window that suits your text length.
Frequently asked questions
Which measure should I use to compare two texts?
Use the moving average type-token ratio with the same window for both texts. It is the least affected by length. Use the plain ratio only for texts of almost the same length.
What does a high value mean?
More different words for the length of the text, so less repetition. Whether that is good depends on the purpose. A children's story or a legal contract may be repetitive on purpose.
Are common words included?
Yes. Every word counts, including the, and and of. The page does not remove stop words, because removing them changes what is being measured.
Why does the moving average say it needs more words?
The window cannot be larger than the text. Lower the window size, or add more text, so that at least one full window fits.