Deploying topologies to different remote clusters?

Question

I am developing topologies for storm (storm-project.net). I have 2 remote clusters: staging and production.

I have two storm.yaml files on the client (my laptop that I write code on) that point to the different remote clusters, production.storm.yaml and staging.storm.yaml.

Unfortunately, the only way I know to switch between them is to change a symlink to ~/.storm/storm.yaml before deploying topologies remotely via the "storm jar" command. This is error prone and creates a dependency in the project source tree to something at a rather arbitrary place in the client's file tree.

There has to be a better way. "storm list --config staging.storm.yaml" will give me info about the staging cluster, but I can't find a comparable flag to set the conf file with "storm jar". Or perhaps there is an environment variable like "STORM_HOME" that I can set?

Answer 1

You don't need to maintain separate .yaml files. You can override the nimbus.host config by using the -c command line option:

storm jar -c nimbus.host=nimbus.example.com my-storm-jar.jar com.example.MyTopology

Answer 2

I also had the same problem. What I did was to write a Makefile for deploying my Storm topologies. Where different goals would create different symlinks. Something like:

export STORM_PATH=/opt/storm
export PROJECT_PATH=/project/path

compile:
    cd $(PROJECT_PATH)
    mvn compile

package:
    cd $(PROJECT_PATH)
    mvn package

deploy-staging: package
    ln -s $(STORM_PATH)/conf/staging.storm.yaml $(STORM_PATH)/conf/storm.yaml
    storm jar $(PROJECT_PATH)/target/project.jar my.project.Topology myTopology

deploy-production: package
    ln -s $(STORM_PATH)/conf/production.storm.yaml $(STORM_PATH)/conf/storm.yaml
    storm jar $(PROJECT_PATH)/target/project.jar my.project.Topology myTopology

So to deploy a topology to production you can go:

make deploy-production

and the same with staging. You'll see the topology is packaged as well (if you use Maven). You can also have the different yaml files as part of your repository and have the symlinks just point into your repository. Alternatively, you can have the yaml files on your deployment machines and they can be different for different deployments. Always with the same name.