Production considerations for Spring Cloud, Spring Data Redis & Eureka

Question

I have a Spring Cloud micro services application spanning 4 server types: A security gateway, two UI servers and a REST APIs server. Each one of these will run on its own VM in a production environment: 4 server instances of the REST server and 2 instances of each other server.

The system is expected to serve around 30,000 users.

The service discovery is provided by Eureka. I have two Eureka servers for failover.

The shared HTTP session is provided by Spring Session & Spring Data Redis, using @EnableRedisHttpSession annotation on the participating servers.

I decided to go with a 3 VMs setup for Redis ("Example 2: basic setup with three boxes" at this URL: http://redis.io/topics/sentinel).

Each VM will run a Redis server and a Redis sentinel process (one of the Redis servers will be the master, two instances will be slaves)

This all works great on development machines and System Test machines, mostly running all processes on the same server.

I am now moving towards running performance testing on production-like environments, with multiple VMs. I would like some feedback and recommendations from developers who already have similar Spring Cloud setups in production:

• What edge cases should I look for?
• Are there any recommended configuration settings? My setup is shown below.
• Are there configuration settings that might work well in testing environments but become serious issues in production environments?
• In my specific scenario, I would also like a solution that would purge old data from Redis, since it only exists to save session information. If for some reason spring would not cleanup the session data on session expiration (for example the server was killed abruptly), I would like to have some cleanup of the really old data. I read about the LRU/Caching mechanism on Redis but it does not seem to have some guarantee with regards to time, only when some data size is reached.

Here is a configuration of my master Redis server. The slaves are pretty much the same, just different ports and indicating they are slaveof the master:

daemonize no

port 6379
dbfilename "dump6379.rdb"
dir "/Users/odedia/Work/Redis/6379"
pidfile "/Users/odedia/Work/Redis/redis6379.pid"
#logfile "/Users/odedia/Work/Redis/redis6379.log"

tcp-backlog 511
timeout 0
tcp-keepalive 60
loglevel notice
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
slave-serve-stale-data yes
slave-read-only no
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events "gxE"
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

Here is a Redis sentinel configuration:

port 5000
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 5000
sentinel config-epoch mymaster 59

And here is the application.yml for the Eureka server:

server:
  port: 1111 

eureka:
  instance:
    hostname: localhost
  client:
    serviceUrl: 
      defaultZone: https://${eureka.instance.hostname}:${server.port}/eureka/
    registerWithEureka: false #Dont register yourself with yourself...
    fetchRegistry: false
  server:
    waitTimeInMsWhenSyncEmpty: 0

spring:
  application:
    name: eureka

And here is the application.yml for the gateway server, which is responsible for the Zuul-based routing:

# Spring properties
spring:
  application:
   name: gateway-server  # Service registers under this name
  redis:
    sentinel:
      master: mymaster
      nodes: 127.0.0.1:5000,127.0.0.1:5001,127.0.0.1:5002

server:
  port: 8080

security:
  sessions: ALWAYS 


zuul: 
 retryable: true #Always retry before failing
 routes:
   ui1-server: /ui1/** 
   ui2-server: /ui2/** 
   api-resource-server: /rest/** 

# Discovery Server Access
eureka:
  client:
    serviceUrl:
      defaultZone: https://localhost:1111/eureka/ 
  instance:
    hostname: localhost
  metadataMap:
        instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}


hystrix:
  command:
    default:
      execution:
        isolation:
          strategy: THREAD
          thread:
            timeoutInMilliseconds: 40000 #Timeout after this time in milliseconds

ribbon:
  ConnectTimeout: 5000 #try to connect to the endpoint for 5 seconds.
  ReadTimeout: 50000 #try to get a response after successfull connection for 5 seconds
  # Max number of retries on the same server (excluding the first try)
  maxAutoRetries: 1
  # Max number of next servers to retry (excluding the first server)
  MaxAutoRetriesNextServer: 2

Answer 1

I wrote an article following my experience in production with Spring Data Redis, it is available here for those interested.

https://medium.com/@odedia/production-considerations-for-spring-session-redis-in-cloud-native-environments-bd6aee3b7d34