DownloadTime
Engineering Analysis

The small-files penalty: why your fast drive feels slow

Every file pays a toll

Opening, writing, closing, and logging each file costs time regardless of size — a 1 KB file costs nearly as much overhead as a 1 MB one. A million tiny files can run 10× slower than the same bytes in one archive, which is why code repositories and photo libraries are the slowest transfers per gigabyte in existence.

The archive trick

Zipping a directory of many small files into one archive converts a penalty-riddled transfer into a full-speed stream — then extracts at the destination at local speed. It is the oldest transfer optimization in computing and still the right answer for move-the-library jobs. Compression is a bonus; the file-count reduction is the win.

Estimating like a professional

Professional migrations estimate in three numbers: interface rate, drive rate (the slower governs), and file profile multiplier. The storage calculator here implements exactly that — and the discipline transfers: any transfer estimate that ignores the file profile is a guess wearing a number.

Advertisement