SQLite Archiver

View Ticket
Ticket Hash: bd983680e5810db5a6ef1d856248ebd2b1d3ab5c
Title: Feature Request: XZ, Zstd, and/or no compression options
Status: Open Type: Feature_Request
Severity: Minor Priority:
Subsystem: Resolution:
Last Modified: 2020-01-16 18:29:19
Version Found In: 4824e7389653a46f
User Comments:
anonymous added on 2020-01-16 18:29:19:

We have a potential use case (trying to consolidate 1000's of tiny files, which many parallel filesystems like GPFS have problems with) into sqlar archives, and there would likely be thousands of sqlar archives (it's storing science work on our HPC cluster).

It would be useful if sqlar supported xz (for space), Zstd (for speed), or no compression options, at least on a per-sqlar archive basis.

As a stretch goal, it would be useful if the compression was specifiable on a per-file basis, and the sqlar command would give options for changing the compression of a file or archive. That way, researchers could store their immediate results with no compression/Zstd for speed, and when spare cycles are available we could run a script to recompress the archive with xz to reduce long-term storage usage.