How to debug Apache Storm in Eclipse?

Question

We can generate storm jar using particular parameter. However, if we need to debug this project (actually a far) locally as well as remotely?

If it is simple jar, that we can debug. However, here we are deploying jar using following command: storm jar project.jar main_class_name

Not sure how can we deploy storm topology, so that we can do storm project in debugging mode?

Please find updated yaml file as below:

 

    # Licensed to the Apache Software Foundation (ASF) under one
    # or more contributor license agreements.  See the NOTICE file
    # distributed with this work for additional information
    # regarding copyright ownership.  The ASF licenses this file
    # to you under the Apache License, Version 2.0 (the
    # "License"); you may not use this file except in compliance
    # with the License.  You may obtain a copy of the License at
    #
    # http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.

    ########### These MUST be filled in for a storm configuration
    # storm.zookeeper.servers:
    #     - "server1"
    #     - "server2"
    # 
    # nimbus.host: "nimbus"
    # 
    # 
    # ##### These may optionally be filled in:
    #    
    ## List of custom serializations
    # topology.kryo.register:
    #     - org.mycompany.MyType
    #     - org.mycompany.MyType2: org.mycompany.MyType2Serializer
    #
    ## List of custom kryo decorators
    # topology.kryo.decorators:
    #     - org.mycompany.MyDecorator
    #
    ## Locations of the drpc servers
    # drpc.servers:
    #     - "server1"
    #     - "server2"

    ## Metrics Consumers
    # topology.metrics.consumer.register:
    #   - class: "backtype.storm.metric.LoggingMetricsConsumer"
    #     parallelism.hint: 1
    #   - class: "org.mycompany.MyMetricsConsumer"
    #     parallelism.hint: 1
    #     argument:
    #       - endpoint: "metrics-collector.mycompany.org"

    worker.childopts:"-agentlib:jdwp=transport=dt_socket,server=y,address=8999,suspend=n"

Answer 1

If you want to remote debug, you need to enable debugging in worker JVMs. See here for the correct Java flags: Remote debugging a Java application

Add this flags in all/shared supervisor storm.yaml to entry worker.childopts (it might be necessary to restart Storm cluster). You need to make sure, that only a single worker is stared per host! Otherwise, two JVMs want to open the same port and one will fail of course.

In Eclipse, choose Run -> Debug Configuration and create a new Remote Java Application. Specify the host and port (port must be the same as specified in worker.childopts) of the running worker JVM you want to debug.

Answer 2

To enable debugging across the various supervisor VMs, all you have to do is edit the storm.yaml file and update the worker.childopts entry to something like:

worker.childopts: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5%ID%"

If you had your supervisors on ports 6700,6701,6702 etc., this would create a debug port on 56700,56701,56702 etc.