Can Apache Storm spouts communicate with each other?

Question

I have a directory which another process throws files into.

Our current implementation of Storm reads this directory and selects the oldest file and opens a reader to the file. This reader is held as a field within the spout so when nextTuple() is called, a single line is output from the file. Once the spout has finished reading it closes the reader and opens a new reader to a new file.

To increase the throughput an idea was to have multiple spouts reading multiple files at once, as these spouts will be fighting over the same files in the same directory, is there a way to communicate between spouts so they can negotiate on which files to read? (Or have an overall manager which allocates files to spouts).

The directory and files are stored and read from HDFS.

Answer 1

I think out of the box there is no way to make two spout communicate together. However, you should try https://github.com/ptgoetz/storm-signals

There is a BaseSignalSpout that relies on zookeeper to send messages between storm components.

Hope this help!