How to use the tf-idf calculator
- Paste your documents into the box, one document per line.
- Choose a term frequency: the raw count, the count divided by the length of the document, or 1 plus the natural log of the count.
- Choose an inverse document frequency and tick the box if each document row should be scaled to length 1.
- Read the table of the highest weighted words in each document, and copy or download the full matrix as a tab-separated file.
Formula
weight(t, d) = tf(t, d) x idf(t)
tf: count, count / length of d, or 1 + ln(count)
idf smooth: ln((1 + N) / (1 + df)) + 1
idf plain: ln(N / df) + 1, idf classic: ln(N / df)
N is the number of documents and df is the number of documents that contain the word. With scaling on, each row is divided by its Euclidean length.
Red apple red
Take three documents: "red apple red", "green apple" and "red grape". Red appears in 2 of the 3 and apple in 2 of the 3, so with the plain idf each has ln(3/2) + 1 = 1.4055.
In the first document red appears twice, so its weight is 2 x 1.4055 = 2.8109, and apple appears once, so its weight is 1.4055. Grape and green are not in that document, so their weights there are 0.
Why a word gets a high weight
A word scores high in a document when it appears often there but in few other documents. A word that appears in every document, such as the, has an idf close to its lowest value and adds little. A word that appears in only one document gets the largest idf, so it stands out. That makes the matrix useful for finding the terms that characterize each document in a small collection.
The three idf choices differ only in how they treat the edges. The classic form gives 0 to a word that is in every document, the plain form adds 1 so such a word still keeps a little weight, and the smooth form adds one to both counts so no ratio divides by zero.
Choices that change the numbers
Raw counts favor long documents, because a long document simply has more of every word. Dividing by the length or taking a log reduces that. Scaling each row to length 1 makes rows comparable, and it is what a cosine similarity comparison expects.
Weights depend on the collection. Add or remove a document and every idf can change. For a real search or classification task use a large collection and a stop word list, and treat the output here as a way to understand or check the method.
Assumptions and limits
- Words are lowercased and split at anything that is not a letter or digit. An apostrophe inside a word keeps it together.
- No stop words are removed and no stemming is done, so run and running are different words.
- Up to 200 documents and 5,000 different words are shown.
- Nothing is uploaded. The documents stay in your browser.
Frequently asked questions
Which idf should I choose?
The smooth form matches the default in scikit-learn. The plain form is the textbook form with 1 added. The classic form can give 0, which removes words found in every document.
Why is a word weight 0 in a document?
Either the word does not appear in that document, or you used the classic idf and the word appears in all of them.
What does scaling to length 1 do?
It divides each document row by the square root of the sum of its squared weights, so the rows can be compared by cosine similarity.
Can I use a single document?
Yes with the smooth idf. With the other two every word is in every document, which gives zero or constant weights, so add more documents.