How to use the log retention storage calculator
- Enter the average number of events per second and the average size of one event in bytes.
- Enter how many days you will keep the logs, the compression ratio you expect and how many copies are kept.
- Open the advanced options to add index overhead as a share of the raw size.
- Read the total storage in decimal and binary units, the storage per day and per 30 days, and the table of sizes.
Formula
raw per day = events per second x event size x 86,400 x (1 + overhead)
stored per day = raw per day / compression ratio x copies
total = stored per day x days kept
A day has 86,400 seconds. Decimal units use 1,000 (KB, MB, GB, TB) and binary units use 1,024 (KiB, MiB, GiB, TiB).
A busy service for 30 days
2,500 events per second of 450 bytes make 1,125,000 bytes a second, which is 97,200,000,000 bytes a day, or 111.78 GB with 15% index overhead. That is 216,000,000 events a day.
Compressing 8 to 1 and keeping 2 copies gives 111.78 / 8 x 2 = 27.95 GB a day, so 30 days need 838.35 GB, which is 780.77 GiB in binary units.
Making the estimate realistic
Event rates swing during the day, so use an average over a busy week, not an off-peak hour. The size of an event should include any fields your logging system adds. Compression depends strongly on content; repetitive text compresses far better than logs with random ids, so take a real sample and measure its ratio instead of trusting a rule of thumb.
Leave room above the figure for growth and for housekeeping work that needs free space. To size disks from the result, use the RAID Calculator, and for video use the CCTV Storage Calculator. The Access Log Summarizer can help you find the event rate and size from a real log file.
Tiers and retention rules
Many systems keep recent logs on fast storage and older ones on cheaper storage. Calculate each tier separately by entering its days kept and its own compression ratio. Rules about how long logs must be kept come from your own policies and laws, which this page does not know.
Assumptions and limits
- Event rate and size are averages and stay the same all day.
- Compression, copies and index overhead apply to every event equally.
- Overhead is added to the raw size before compression is applied.
- Days kept can be from 1 to 36,500 and copies must be a whole number from 1 to 100.
- The example figures are made up for illustration, not measurements of any system.
Frequently asked questions
What compression ratio should I use?
Measure it. Compress a day of your own logs and divide the raw size by the compressed size. Use 1 if you want no compression in the estimate.
Why show both GB and GiB?
Disk sellers use powers of 1,000 and many operating systems show powers of 1,024. A terabyte of disk shows as about 931 GiB, so mixing them leads to wrong plans.
Does the estimate include the free space a system needs?
No. Add a margin for compaction, growth and recovery on top of the figure.
How should I count a replicated cluster?
Use the number of copies that are stored in total, including the original. A setup with one primary and one replica has 2 copies.