Simple Flume agent hat some lag when logging to the console

Question

I have a simple Flume agent with the following configuration:

agent.sources = http-source
agent.sinks = logger-sink
agent.channels = logger-channel


# HTTP Source
###############################
agent.sources.http-source.type =  org.apache.flume.source.http.HTTPSource
agent.sources.http-source.channels = logger-channel
agent.sources.http-source.port = 81


# Logger Sink
###############################
agent.sinks.logger-sink.type = logger
agent.sinks.logger-sink.channel = logger-channel


# Channel
###############################

agent.channels.logger-channel.type = memory
agent.channels.logger-channel.capacity = 1000

The only thing that the Flume agent does, it to receive the HTTP POST request through the HTTP Source and logs the events using the Logger Sink.

The problem that I have is as follows: sometimes when I send the HTTP POST request to the Flume agent, It takes 1-5 second till I see the logs in the console. This is only the case for the first message being sent after starting the Flume agent. After sending several messages I see the logs immediately in the console.

My question is: is it a warm-up issues in Flume? It seems that if I do not send any message for a while, again I will have some delay seeing the logs in the console.

Notice that I start the Flume agent in a Docker container using a docker-compose file.

Answer 1

First check if the issue persists with the latest Flume agent available (release 1.7), using a recent image like mrwilson/docker-flume.
You can compare its docker-compose.yml with yours.

An image like gilt/docker-flume is older and still in 1.5.