Surprisingly good results compressing scientific images

[ThisIsFineTM]

I did a cursory test with all the compression methods I had immediately available for compressing a huge scientific image file and found that bzip3 did spectacularly well in both ratio and speed.

Image source: https://astrogeology.usgs.gov/search/map/enceladus-cassini-global-mosaic-100m-schenk

296M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.zpaq (-m5)
307M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.bz3 (-b256)
316M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.bz3
330M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.xz (-9)
331M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.lzip (-9)
360M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.br (-Z)
368M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.rzip (-9)
368M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.bz2
372M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.zst (-22 --long)
409M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.gz (-9)
409M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.zip (-9)
419M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.orz (-2)
495M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif.lz4
495M Enceladus_Cassini_mosaic_global_100m_schenk2024.tif

Figured I'd pass along the good result :)

[iczelia]

Very cool, thank you!

[ThisIsFineTM]

...
...
...

So yeah, um, bzip3 is just rediculously outperforming everything else for numerical data sets. I got ahold of a friend's data set from a particle physics experiment. These are the results for compressing with all available threads on the system:

58M  data.zpaq (-m5, 87 sec)
64M  data.bin.bz3 (4 sec)
76M  data.xz (-9, 54 sec)
98M  data.bin.zst (--long -22, 97 sec)
513M data.bin

I need to speak to some colleagues about this. I'll be back :)

[alerque]

Don't forget to benchmark decompression separate from compression. Those two plus compression ratio are three major metrics that can be optimized for and different algorithms tend to excel at different things. After those three then you get into other aspects like streaming, chunk sizes, determinism, etc.

[ThisIsFineTM]

@alerque I did, and the multithreaded compression and decompression were of comparable times for bzip3. These were using the default settings. I did some other tests with -b256 and -b511, but the speed penalty wasn't worth the negligible improvement in size for my particular data.

[ThisIsFineTM]

I did a little bit more experimentation, mainly comparing bz3 to zstd. I haven't had much time to dive into how bz3 works, but it seems like bz3 really accels at compressing unstructured raw data. I tested with some hdf5 files, and zstd was able to pull ahead in final ratio (albeit at -22), but individual raw blobs of data had bz3 consistently blowing everything else out of the water.

If I had to guess, I would say the blockwise chunking bz3 does works exceptionally well when the blocks are fundamentally the same flavor of data, and the structure around blobs in containers interferes with getting the better ratios. Since zstd has a comparably huge search window, it can compress the structured portions through deduplication more effectively so that's why it can squeeze out slightly better ratios in the hdf5 tests I ran.

[iczelia]

@ThisIsFineTM you can use lrzip or rzip to preprocess the data before applying bzip3 to find these redundancies. @pete4abw's lrzip-next has native support for bzip3. https://github.com/pete4abw/lrzip-next

[ThisIsFineTM]

I did some cursory experimentation with lrzip, and it did help some, but not enough to change the relative final compression ratios of the different algorithms/settings for the test data I used.

[RokerHRO]

I did a cursory test with all the compression methods I had immediately available for compressing a huge scientific image file and found that bzip3 did spectacularly well in both ratio and speed.
[…]
Figured I'd pass along the good result :)

And… how this shall be fixed?

SCNR

L.