Spring Boot + Eureka Server + Hystrix with Turbine: empty turbine.stream

Question

I'm trying to run Spring Boot (with Spring Cloud) + Eureka Server + Hystrix Dashboard and Turbine stream, but I run into a problem I couldn't find any solution so far. I use Spring Boot 1.2.1.RELEASE and Spring Cloud 1.0.0.RC2. Here is what I have:

The first instance is running Eureka server and Hystrix dashboard:

@Configuration
@EnableAutoConfiguration
@EnableEurekaServer
@EnableHystrixDashboard
@EnableDiscoveryClient
class Application {

    public static void main(String[] args) {
        SpringApplication.run Application, args
    }
}

Here you can find build.gradle for that instance - https://gist.github.com/wololock/570272ad7cf2d14a4d3c

Eureka server is running ok, I can see registered instances on eureka server dashboard, I can also use LoadBalancer to get the URL to registered instance using its id. So far everything is ok.

I have a few instances that are run with @EnableHystrix annotation and use @HystrixCommand to define which methods have to be monitored by Hystrix. When I pass URL to hystrix.stream of single instance to Hystrix dashboard, I can see it running with no problem.

I have also separate Turbine server, not complicated one:

@EnableAutoConfiguration
@EnableTurbine
@Configuration
@EnableDiscoveryClient
class Application {

    public static void main(String[] args) {
        SpringApplication.run Application, args
    }
}

Here you can find build.gradle for Turbine server instance - https://gist.github.com/wololock/ff0d855b8a890232851e

It uses very simple configuration, build mostly on the one provided by sample turbine app - https://github.com/spring-cloud-samples/turbine

info:
  component: Turbine

endpoints:
  restart:
    enabled: true
  shutdown:
    enabled: true

turbine:
  appConfig: pdf-creator-service

InstanceDiscovery:
  impl: io.spring.platform.netflix.turbine.EurekaInstanceDiscovery

server:
  port: 8989

management:
  port: 8990

eureka:
  instance:
    leaseRenewalIntervalInSeconds: 10
  client:
    serviceUrl:
      defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/

After running those instances in order:

1. eureka server
2. turbine server
3. discover client instance,

I have 2nd and 3rd instance registered in eureka server, turbine server log says, that there is one instance up:

[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:35:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:36:04.162] boot - 20495  INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0

Calling hystrix.stream from a single instance works, e.g. curl http://localhost:8885/hystrix.stream returns:

data: {"type":"HystrixCommand","name":"post","group":"PdfController","currentTime":1423223614259,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":8000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}

data: {"type":"HystrixCommand","name":"generate","group":"WkHtmlToPdfGenerator","currentTime":1423223614259,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":8000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}

data: {"type":"HystrixThreadPool","name":"PdfController","currentTime":1423223614259,"currentActiveCount":0,"currentCompletedTaskCount":4,"currentCorePoolSize":10,"currentLargestPoolSize":4,"currentMaximumPoolSize":10,"currentPoolSize":4,"currentQueueSize":0,"currentTaskCount":4,"rollingCountThreadsExecuted":0,"rollingMaxActiveThreads":0,"propertyValue_queueSizeRejectionThreshold":5,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"reportingHosts":1}

But when I attach turbine.stream to hystrix dashboard, I get nothing. Logs say:

[2015-02-06 12:42:48.922] boot - 24816  INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:42:48.922] boot - 24816  INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:42:48.922] boot - 24816  INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:42:48.922] boot - 24816  INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineStreamServlet: FilterCriteria: []
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineStreamServlet: StatsType filters: []
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineStreamingConnection: Relevance config: []
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineStreamingConnection: Relevance metrics config: {}
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- ClusterMonitor: Registering event handler for cluster monitor: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.237] boot - 24816  INFO [XNIO-2 task-4] --- TurbineDataDispatcher: 

Just added and starting handler tuple: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.238] boot - 24816  INFO [XNIO-2 task-4] --- AggDataFromCluster: Per handler dispacher started for: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.238] boot - 24816  INFO [XNIO-2 task-4] --- ClusterMonitor: All event handlers for cluster monitor: [StreamingHandler_637572ab-acda-4bf4-81cd-6a658adb73eb, StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b, StaticListener_For_Aggregator, StreamingHandler_5ec12ee8-3fcd-4a6f-9006-d2a6ecc309d0, StreamingHandler_72d7b9e2-ad98-42a0-9ac3-abe4aa57cc7a]
[2015-02-06 12:43:26.238] boot - 24816  INFO [XNIO-2 task-4] --- ClusterMonitor: Starting up the cluster monitor for default_agg

If I do curl http://localhost:8989/turbine.stream I get only:

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223006935}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223010935}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223013936}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223017936}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223020937}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223024937}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223028938}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223032938}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223036938}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223039939}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223043939}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223046940}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223050940}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223054941}

My question is: do I missed something in setting it up and running? Previously I tried to manage this by having eureka and turbine server in single instance, but in that case turbine couldn't even find the registered application in eureka using proper application name. I made a progress after splitting eureka and turbine, but it still does not work correctly.

I will be grateful for any suggestion. If you need more information, just let me know, I might miss something important.

UPDATE 20150209

Following Dave's suggestion I applied small changes in application.yml file of turbine-server. Now the file contains only:

info:
  component: Turbine

turbine:
  appConfig: pdf-creator-service
  clusterNameExpression: 'default'

server:
  port: 8989

management:
  port: 8990

eureka:
  instance:
    leaseRenewalIntervalInSeconds: 10
  client:
    serviceUrl:
      defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/

But it doesn't make turbine.stream working. After turbine-server gets aware of registered client in eureka server, it fails with given exception:

[2015-02-09 21:25:03.516] boot - 4808  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-09 21:25:03.516] boot - 4808  INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-09 21:25:03.516] boot - 4808  INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-09 21:25:03.520] boot - 4808 ERROR [Timer-0] --- EurekaInstanceDiscovery: Failed to fetch instances for app: pdf-creator-service, retrying once more
org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Property or field 'default' cannot be found on object of type 'com.netflix.appinfo.InstanceInfo' - maybe not public?
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:226)
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:93)
    at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:81)
    at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:120)
    at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:242)
    at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getClusterName(EurekaInstanceDiscovery.java:183)
    at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.marshallInstanceInfo(EurekaInstanceDiscovery.java:141)
    at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getInstancesForApp(EurekaInstanceDiscovery.java:123)
    at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getInstanceList(EurekaInstanceDiscovery.java:88)
    at com.netflix.turbine.discovery.InstanceObservable.getInstanceList(InstanceObservable.java:327)
    at com.netflix.turbine.discovery.InstanceObservable.access$500(InstanceObservable.java:66)
    at com.netflix.turbine.discovery.InstanceObservable$1.run(InstanceObservable.java:258)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)

I tried writting 'default' as well as default, but the result is unfortunately the same.

Solution

Thanks Dave Syer for finding the proper solution. Basically what did the trick was adding:

turbine:
    clusterNameExpression: new String("default")

to application.yml file in turbine server application instance. It might look weird, I didn't believe it will work, but it does. Now when I call my hystrix client application I get the proper information in hystrix.stream that is served by this application and in turbine.stream of Turbine server as well. My current application.yml in turbine server looks as follows:

info:
  component: Turbine

turbine:
  clusterNameExpression: new String("default")
  appConfig: pdf-creator-service

server:
  port: 8989

management:
  port: 8990

eureka:
  instance:
    leaseRenewalIntervalInSeconds: 10
  client:
    serviceUrl:
      defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/

turbine.appConfig holds the information about the hystrix clients (by their IDs). To add another client to your turbine server you will have to simply put another instance id, coma separated with the previous one. And that's all folks :)

Answer 1

It works for me if I add some configuration for the cluster, e.g.

turbine:
  appConfig: customers,stores
  clusterNameExpression: new String('default')

Turbine has to know how to construct the "cluster" name (an aggregation key for sets of applications). The default is to use the appname, so if you don't set the clusterNameExpression you need to use a query param in the stream URL, e.g. /turbine.stream?cluster=CUSTOMERS (uppercased appname).

Answer 2

In my case I had :7979/mock.stream as client for testing turbine app. The turbine checks time of event (timeOfEvent) so it will show only the current date time events.

It can be turned off using

turbine.InstanceMonitor.eventStream.skipLineLogic.enabled = false