Pack as pack can

Back in the days where data were transported on disks capable of holding a mere 360 kB, file compression was a vital technology and widely used. In the age of terabyte harddisks, file compression appears somewhat outdated, even obsolete, since storage space is cheap and available in abundance. Even more important, most modern file formats utilize compression (both lossy and lossless) anyway. Pictures, for example, are stored as png or jpeg, documents as pdf or odt (which is nothing but a zipped xml container), music as mp3, and movies as mpeg4. Very little is gained when trying to compress these files once more, so why bother at all?

Well, I can only speak for myself. I use LaTeX for publications and presentations, so all my texts are also stored as such: plain text. Graphics are included as pure ASCII encoded encapsulated post scripts, and thus as plain text. Data are plain text anyway, Mathematica scripts are, and shell or Python scripts also. All of my important files are plain ASCII. Now, plain text can be compressed very efficiently, and since I usually synchronize my computers via the net, I welcome small sizes since the bandwidth I'm commanding is far from infinite.

Size, however, is only one aspect: the compression and decompression time should not exceed the saving in time obtained when transmitting the compressed file. Otherwise, the undisputed champion of text compression, paq8, would win single-handedly. 😉

For a comparison, I've selected a realistic example: the project folder of this paper, which is roughly 200 MB in size (precisely 215336960 bytes). This folder contains all files needed in the course of the project, including many whose size cannot be reduced significantly by further compression (such as figures in pdf format). I've archived the folder using 'tar -a -cvf test.tar paper/' since several of the file compressors do not archive themselves.

The following table displays all relevant data: the name of the contender in the first, the size of the compressed tar archive in the second, and the compression and decompression time in the third and fourth column, respectively. I've used the default setting unless otherwise noted. The first numbers in the latter columns stem from my dual core desktop system (e6600, 2.4 GHz, 8 GB RAM), the second ones in brakets from a 24 core workstation (opteron 8431, 2.4 GHz, 64 GB RAM). Draw your own conclusions (but note that only pigz and pbzip are fully parallelized).

Ok, ok: I use nanozip. And yes, nanozip is also available for Windows, even with a GUI. 😉

Maximum compression reference (noncompetetive):

¹ with -m0=BZIP2 -mmt
² with -cdP -t6
³ with -t7z -m0=lzma -mx=9 -mfb=64 -ms=on -md=32m (upon popular request 😉 )