Flume agent - can I specify compression like gzip or bz2?

Question

Is it possible to specify a compression option on a Flume agent so that the data is transferred to the collector in a compressed format? I know there are compression options on the collector level, but it would be also extremely useful to be able to transfer compressed data from the agent, too.

Thank you!

Answer 1

Flume has the gzip and gunzip sink decorators that you can use. Put the gzip decorator on your agent sink and the gunzip decorator on your collector sink, and transferring compressed data between the agent and collector should Just Work.