Compression algorithm operating on pre-built dictionary as data structure

Question

I was fairly sure this is a common use case, but after half day of googling I have to fill a question.

I would really like an algorithm, that I can run on a corpus of data to determine dictionary (as a data structure), and then use that dictionary to compress newly arriving data very fast and efficient thanks to that dictionary.

For example, I would run it on 10,000 messages totalling 10MB to determine a dictionary data structure, share this dictionary between all parties and then exchange messages while enjoying very fast and strong compression.

Is there something of that sort? IBM DB2 does exactly that but I doubt they open-sourced the approach. zlib allows one to pass dictionary, but it is raw byte array which will need to be processed for every message, and there's no method of generating said byte array.

The idea of holding a data structure in memory is to avoid any overhead of per-message processing.

Bonus points for Java implementations.

Answer 1

Eventually I was pointed to Zstd compression that allow supplying your own (shared) dictionary. There are Java bindings with capability of dictionary training based on samples.

It is able to outperform my own algorithm with shared dictionary as small as 512 bytes:

(s is number of samples, d is dictionary length, l is compression level)