How to use the text chunk splitter with overlap
- Paste the text. Choose whether to split by words, characters or sentences.
- Set the chunk size, which is the most units in one chunk, and the overlap, which is how many units each chunk repeats from the one before. The overlap must be smaller than the size.
- For characters, leave Keep whole words ticked to avoid cutting a word in half.
- Read the numbered chunks with their start and end positions, then copy the table or download the CSV.
Formula
step = size - overlap
chunk k covers units [k x step, k x step + size)
number of chunks = ceil((total - size) / step) + 1, for total > size
In words and sentences mode a unit is a whole word or sentence. In characters mode the end of a chunk can move back to the previous space so that no word is cut, which makes a chunk a little shorter than the size.
A 72 word text in chunks of 20 with 5 repeated
The sample text has 72 words. With a size of 20 and an overlap of 5, the step is 15, so the chunks start at words 1, 16, 31, 46 and 61. That makes ceil((72 - 20) / 15) + 1 = 5 chunks, and the last one holds the final 12 words.
Each chunk shares its first five words with the end of the one before. The Starts at and Ends at columns give character positions, so the text between those positions in your original is exactly the chunk. In characters mode without snapping, joining the first chunk to every later chunk with its overlap removed rebuilds the original exactly.
Why overlap helps
When a long text is cut into pieces that are read separately, a sentence at the edge loses the context that came just before it. Repeating a few words or a sentence at the start of the next piece keeps that context. Too much overlap wastes space and repeats content, so keep it to a small share of the size.
The page counts characters, words and sentences. It does not count tokens, which depend on the model that will read the text, so leave a margin if a limit is set in tokens.
Sentence splitting and its limits
Sentences are found by looking for a full stop, question mark or exclamation mark followed by a space. Abbreviations such as Dr. and numbers such as 3.5 can look like the end of a sentence in some texts, so check the chunks before relying on them. Line breaks and spacing inside a chunk are kept as they were in your text.
To split text into numbered posts for a short-message limit use the Tweet Thread Splitter, and to tighten the writing first try the Filler Word and Hedge Finder.
Assumptions and limits
- Words are runs of characters that are not spaces, so a hyphenated word counts as one.
- Text is limited to 500,000 characters and the result to 2,000 chunks.
- Positions count characters from 0 in the text as the browser holds it, with line breaks counted as one character.
- Sizes are not token counts.
- Nothing typed here leaves your browser.
Frequently asked questions
What overlap should I use?
A common choice is about 10 to 20 percent of the size. The right value depends on how much context a piece needs, so try a few and read where the edges fall.
Why is a chunk shorter than the size I set?
In characters mode with whole words, a chunk ends before a word that would be cut, so it can be shorter. In sentences mode, a chunk holds up to the size in sentences, and the last one holds whatever is left.
Can the chunks be put back together?
Yes. Take the first chunk, then add each later chunk without its first overlap units. The start and end columns also let you check each piece against your original text.
Does it count tokens for a language model?
No. Token counts depend on the model, and they cannot be computed exactly here. Use a smaller size to leave a safety margin when a limit is in tokens.