Flume - Solr Integration

Question

This is my scenario.

Input JSON data flows to Flume and it needs to be indexed and stored into Solr in near real time. I am using the latest CDH release.

I did not find the documentation complete. It is disconnected at places.

Can you please point me in the right direction here?

• Should i use the Morphilines sink and say dont do any transformations? Or if i don't want any transformations to the json, could i use some other sink just to directly write to Solr?
• Pointers to documentation with clear steps
• Or please list what you would do in step-by-step form (of course, very high level).

Answer 1

Here a few pointers:

1) If your Source is RabbitMQ, JMSSource is the right component.

2) Short of writing you own SOLR custom Source, I would recommend using MorphlineSOLRSink, with no transformation. It works like a charm. If you don have to transform or modify things, you can either implement your own custom Handler oder use an interceptor. Keep in mind that amount of SOLR records must corresponds to the amount of event your source receives.