Solr DataImport: managing configuration for different environments (development/staging/production)

Question

Configuration files for SolR DataImportHandler contain the SQL queries to run against the database, how to map the resulting columns to SolR document fields and, of course, the parameters needed to connect to the database.

In our project, the database connection parameters (specifically the database passwords) change from environment to environment, which forces us to maintain one slightly different copy of the whole configuration XML file for each environment.

Is there a way to configure the database connection parameters (specially the passwords) separately from the SQL statements and declarations, so each configuration is maintained Once and Only Once?

Answer 1

Actually this can be done using standard solr configuration.

First, you need to define your data sources in solrconfig.xml [See Adding Datasource in Solrconfig]

Secondly, you can externalize the DIH configurations into a separate file using XInclude

I use this approach both to use local configuration files and to centralize the connections across different cores.

---

Example: In solrconfig.xml, add:

<xi:include href="../../common-config/local.dih.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />

local.dih.xml will look something like:

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
    <str name="config">data-config.xml</str>
    <lst name="datasource">
      <str name="name">mongo</str>
      <str name="type">MongoDataSource</str>
      <str name="database">myMongoDb</str>
    </lst>
    <lst name="datasource">
      <str name="name">psql</str>
      <str name="driver">org.postgresql.Driver</str>
      <str name="type">JdbcDataSource</str>
      <str name="url">jdbc:postgresql://localhost:5432/myPsqlDb</str>
      <str name="user">dbUser</str>
      <str name="password">dbPassword</str>
    </lst>
  </lst>
</requestHandler>  

Answer 2

this is a known issue in Solr.

If you look at the Solr doc, or at the Solr Entreprise Server book, they say you can use a core1.properties with key=value and use key in your xml config files...but in my experience it does not work. I have tried it several ways, there are open questions on the solr mailing list about this.

So you have to resort to ugly workarounds (like using a template xml file and replacing #password# with the real password etc).