Menu
We have a akka application running in production in a non clustered mode ( doesn't akka clustering ). We are trying to dockerize this application and running into an issue. We are using docker host networking and for various other reasons we are not yet ready to use other types of networking.
Application binds to 0.0.0.0 interface on port 9080 and does a simple http healthcheck on itself. This healthcheck is failing because we are seeing a connection reset error. We are seeing this only inside docker container.
During startup time application gets bound to 0:0:0:0:0:0:0:0:9080. 2018-04-13T08:29:54.921-0700 [INFO ] - Bound to /0:0:0:0:0:0:0:0:9080
akka logs:
2018-04-13T08:34:25.311-0700 [DEBUG] akka.io.TcpListener [15:34:25.311UTC] - New connection accepted
2018-04-13T08:34:25.312-0700 [DEBUG] akka.io.SelectionHandler [15:34:25.312UTC] - now supervising Actor[akka://appname/system/IO-TCP/selectors/$d/12#394551476]
2018-04-13T08:34:25.312-0700 [DEBUG] akka.io.TcpIncomingConnection [15:34:25.312UTC] - started (akka.io.TcpIncomingConnection@1dafecc7)
2018-04-13T08:34:25.312-0700 [DEBUG] akka.io.TcpIncomingConnection [15:34:25.312UTC] - now watched by Actor[akka://appname/system/IO-TCP/selectors/$d#992175932]
2018-04-13T08:34:25.312-0700 [DEBUG] akka.io.TcpIncomingConnection [15:34:25.312UTC] - now watched by Actor[akka://appname/user/IO-HTTP/listener-0/88#-1187840999]
2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.SelectionHandler [15:34:25.313UTC] - now supervising Actor[akka://appname/system/IO-TCP/selectors/$h/11#789402148]
2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpOutgoingConnection [15:34:25.313UTC] - started (akka.io.TcpOutgoingConnection@62c77639)
2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpOutgoingConnection [15:34:25.313UTC] - now watched by Actor[akka://appname/system/IO-TCP/selectors/$h#227381172]
2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpOutgoingConnection [15:34:25.313UTC] - Attempting connection to [/0.0.0.0:9080]
2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpListener [15:34:25.313UTC] - New connection accepted
2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpOutgoingConnection [15:34:25.313UTC] - Connection established to [/0.0.0.0:9080]
2018-04-13T08:34:25.314-0700 [DEBUG] akka.io.TcpOutgoingConnection [15:34:25.314UTC] - now watched by Actor[akka://appname/user/IO-HTTP/group-0/52#1518564728]
2018-04-13T08:34:25.314-0700 [DEBUG] akka.io.SelectionHandler [15:34:25.314UTC] - now supervising Actor[akka://appname/system/IO-TCP/selectors/$h/12#1641348988]
2018-04-13T08:34:25.314-0700 [DEBUG] akka.io.TcpIncomingConnection [15:34:25.314UTC] - started (akka.io.TcpIncomingConnection@65bac45)
2018-04-13T08:34:25.314-0700 [DEBUG] akka.io.TcpIncomingConnection [15:34:25.314UTC] - now watched by Actor[akka://appname/system/IO-TCP/selectors/$h#227381172]
2018-04-13T08:34:26.327-0700 [DEBUG] akka.io.TcpIncomingConnection [15:34:26.327UTC] - now watched by Actor[akka://appname/user/IO-HTTP/listener-0/89#-1079090421]
2018-04-13T08:34:26.327-0700 [DEBUG] akka.io.TcpIncomingConnection [15:34:26.327UTC] - stopped
2018-04-13T08:34:26.327-0700 [DEBUG] akka.io.TcpOutgoingConnection [15:34:26.327UTC] - Closing connection due to IO error java.io.IOException: Connection reset by peer
2018-04-13T08:34:26.327-0700 [DEBUG] akka.io.TcpOutgoingConnection [15:34:26.327UTC] - stopped
2018-04-13T08:34:26.328-0700 [DEBUG] akka.io.TcpIncomingConnection [15:34:26.328UTC] - stopped
Spray logs:
2018-04-13T08:35:39.168-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$b/20#-595408603]
2018-04-13T08:35:45.296-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$d/19#721242322]
2018-04-13T08:35:51.372-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$c/16#1510321611]
2018-04-13T08:35:57.436-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$g/22#1994752981]
2018-04-13T08:36:03.622-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$f/20#-1450294853]
Healthcheck logs
2018-04-13T08:32:53.356-0700 [INFO ] - [BEGIN] Health check
2018-04-13T08:32:53.356-0700 [DEBUG] - Memory : SUCCESS
2018-04-13T08:32:54.378-0700 [DEBUG] - HTTP : FAILURE, spray.can.Http$ConnectionException: ErrorClosed(Connection reset by peer)
2018-04-13T08:32:54.378-0700 [ERROR] - spray.can.Http$ConnectionException: ErrorClosed(Connection reset by peer)
at spray.can.client.HttpHostConnectionSlot.reportDisconnection(HttpHostConnectionSlot.scala:228)
at spray.can.client.HttpHostConnectionSlot$$anonfun$connected$1.applyOrElse(HttpHostConnectionSlot.scala:161)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at spray.can.client.HttpHostConnectionSlot.aroundReceive(HttpHostConnectionSlot.scala:33)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:487)
at akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at akka.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorCellInstrumentation.scala:66)
at akka.actor.ActorCell.invoke(ActorCell.scala:483)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-04-13T08:32:54.378-0700 [INFO ] - [END ] Status: Failed (1022 ms)
Dockerfile:
FROM java-base:0.2.3
EXPOSE 9080 9081
ENV MAIN_CLASS="akka.kernel.Main" \
ADMIN_PORT="9080" \
HEALTHCHECK_URI="/admin/health" \
TEMPLATES="/app/templates/override.settings.tmpl:/app/conf/override.settings /app/templates/environment.conf.tmpl:/app/conf/environment.conf" \
JMX_PORT="9081" \
APP_NAME="appname" \
APP_VERSION="1.34.0" \
CLASSPATH="/app/conf" \
ENV_FILES="/app/conf/override.settings" \
INSTRUMENTATION_PATH="/app/ext" \
JVM_MISC_OPTS="-Dpid=1 \
-Dlog4j.configuration=file:///app/conf/log4j.xml \
-javaagent:\${INSTRUMENTATION_PATH}/aspectjweaver-1.7.4.jar \
-Dlog4j.configuration=file:///app/conf/log4j.xml \
-Dlogs.dir=/app/logs \
-Dakka.home=/app \
-Dakka.kernel.quiet=false \
-Dconfig.file=/app/conf/environment.conf" \
MAIN_CLASS_ARGS=appname.Boot
ADD ext /app/ext
ADD deps /app/libs
ADD templates/ /app/templates/
Docker-compose file:
version: '3.3'
services:
appname:
container_name: appname
image: appname:1.34.0
network_mode: host
restart: unless-stopped
ulimits:
nofile:
soft: 65535
hard: 65535
environment:
UID:
SDLC_ENVIRONMENT:
REGION:
PARTITION:
TAGS:
INTERNAL_IP:
CONSUL_ADDRESS:
DOCKER_GID:
MAIN_CLASS_ARGS: appname.Boot
volumes:
- "./logs:/app/logs"
Akka Config:
2018-04-17T11:39:17.753-0700 [DEBUG] akka.event.EventStream [18:39:17.713UTC] - Default Loggers started
2018-04-17T11:39:17.755-0700 [INFO ] akka.actor.ActorSystemImpl [18:39:17.738UTC] - {
"akka" : {
"actor" : {
"creation-timeout" : "20s",
"debug" : {
"autoreceive" : "off",
"event-stream" : "off",
"fsm" : "off",
"lifecycle" : "off",
"receive" : "off",
"router-misconfiguration" : "off",
"unhandled" : "off"
},
"default-dispatcher" : {
"attempt-teamwork" : "on",
"default-executor" : {
"fallback" : "fork-join-executor"
},
"executor" : "default-executor",
"fork-join-executor" : {
"parallelism-factor" : 16,
"parallelism-min" : 8,
"task-peeking-mode" : "FIFO"
},
"mailbox-requirement" : "",
"shutdown-timeout" : "1s",
"thread-pool-executor" : {
"allow-core-timeout" : "on",
"core-pool-size-factor" : 3,
"core-pool-size-max" : 64,
"core-pool-size-min" : 8,
"keep-alive-time" : "60s",
"max-pool-size-factor" : 3,
"max-pool-size-max" : 64,
"max-pool-size-min" : 8,
"task-queue-size" : -1,
"task-queue-type" : "linked"
},
"throughput" : 1,
"throughput-deadline-time" : "0ms",
"type" : "Dispatcher"
},
"default-mailbox" : {
"mailbox-capacity" : 1000,
"mailbox-push-timeout-time" : "10s",
"mailbox-type" : "akka.dispatch.UnboundedMailbox",
"stash-capacity" : -1
},
"deployment" : {
"default" : {
"dispatcher" : "",
"mailbox" : "",
"nr-of-instances" : 1,
"resizer" : {
"backoff-rate" : 0.1,
"backoff-threshold" : 0.3,
"enabled" : "off",
"lower-bound" : 1,
"messages-per-resize" : 10,
"pressure-threshold" : 1,
"rampup-rate" : 0.2,
"upper-bound" : 10
},
"routees" : {
"paths" : []
},
"router" : "from-code",
"tail-chopping-router" : {
"interval" : "10 milliseconds"
},
"virtual-nodes-factor" : 10,
"within" : "5 seconds"
}
},
"dsl" : {
"default-timeout" : "5s",
"inbox-size" : 1000
},
"guardian-supervisor-strategy" : "akka.actor.DefaultSupervisorStrategy",
"mailbox" : {
"bounded-deque-based" : {
"mailbox-type" : "akka.dispatch.BoundedDequeBasedMailbox"
},
"bounded-queue-based" : {
"mailbox-type" : "akka.dispatch.BoundedMailbox"
},
"requirements" : {
"akka.dispatch.BoundedDequeBasedMessageQueueSemantics" : "akka.actor.mailbox.bounded-deque-based",
"akka.dispatch.BoundedMessageQueueSemantics" : "akka.actor.mailbox.bounded-queue-based",
"akka.dispatch.DequeBasedMessageQueueSemantics" : "akka.actor.mailbox.unbounded-deque-based",
"akka.dispatch.MultipleConsumerSemantics" : "akka.actor.mailbox.unbounded-queue-based",
"akka.dispatch.UnboundedDequeBasedMessageQueueSemantics" : "akka.actor.mailbox.unbounded-deque-based",
"akka.dispatch.UnboundedMessageQueueSemantics" : "akka.actor.mailbox.unbounded-queue-based"
},
"unbounded-deque-based" : {
"mailbox-type" : "akka.dispatch.UnboundedDequeBasedMailbox"
},
"unbounded-queue-based" : {
"mailbox-type" : "akka.dispatch.UnboundedMailbox"
}
},
"provider" : "akka.actor.LocalActorRefProvider",
"reaper-interval" : "5s",
"router" : {
"type-mapping" : {
"balancing-pool" : "akka.routing.BalancingPool",
"broadcast-group" : "akka.routing.BroadcastGroup",
"broadcast-pool" : "akka.routing.BroadcastPool",
"consistent-hashing-group" : "akka.routing.ConsistentHashingGroup",
"consistent-hashing-pool" : "akka.routing.ConsistentHashingPool",
"from-code" : "akka.routing.NoRouter",
"random-group" : "akka.routing.RandomGroup",
"random-pool" : "akka.routing.RandomPool",
"round-robin-group" : "akka.routing.RoundRobinGroup",
"round-robin-pool" : "akka.routing.RoundRobinPool",
"scatter-gather-group" : "akka.routing.ScatterGatherFirstCompletedGroup",
"scatter-gather-pool" : "akka.routing.ScatterGatherFirstCompletedPool",
"smallest-mailbox-pool" : "akka.routing.SmallestMailboxPool",
"tail-chopping-group" : "akka.routing.TailChoppingGroup",
"tail-chopping-pool" : "akka.routing.TailChoppingPool"
}
},
"serialization-bindings" : {
"[B" : "bytes",
"java.io.Serializable" : "java"
},
"serialize-creators" : "off",
"serialize-messages" : "off",
"serializers" : {
"bytes" : "akka.serialization.ByteArraySerializer",
"java" : "akka.serialization.JavaSerializer"
},
"typed" : {
"timeout" : "5s"
},
"unstarted-push-timeout" : "10s"
},
"daemonic" : "off",
"extensions" : [],
"home" : "/app",
"io" : {
"pinned-dispatcher" : {
"executor" : "thread-pool-executor",
"thread-pool-executor" : {
"allow-core-pool-timeout" : "off",
"allow-core-timeout" : "off"
},
"type" : "PinnedDispatcher"
},
"tcp" : {
"batch-accept-limit" : 10,
"direct-buffer-pool-limit" : 1000,
"direct-buffer-size" : "128 KiB",
"file-io-dispatcher" : "akka.actor.default-dispatcher",
"file-io-transferTo-limit" : "512 KiB",
"finish-connect-retries" : 5,
"management-dispatcher" : "akka.actor.default-dispatcher",
"max-channels" : 256000,
"max-received-message-size" : "unlimited",
"nr-of-selectors" : 10,
"register-timeout" : "5s",
"selector-association-retries" : 10,
"selector-dispatcher" : "akka.io.pinned-dispatcher",
"trace-logging" : "off",
"windows-connection-abort-workaround-enabled" : "off",
"worker-dispatcher" : "akka.actor.default-dispatcher"
},
"udp" : {
"direct-buffer-pool-limit" : 1000,
"direct-buffer-size" : "128 KiB",
"management-dispatcher" : "akka.actor.default-dispatcher",
"max-channels" : 4096,
"nr-of-selectors" : 1,
"receive-throughput" : 3,
"received-message-size-limit" : "unlimited",
"select-timeout" : "infinite",
"selector-association-retries" : 10,
"selector-dispatcher" : "akka.io.pinned-dispatcher",
"trace-logging" : "off",
"worker-dispatcher" : "akka.actor.default-dispatcher"
},
"udp-connected" : {
"direct-buffer-pool-limit" : 1000,
"direct-buffer-size" : "128 KiB",
"management-dispatcher" : "akka.actor.default-dispatcher",
"max-channels" : 4096,
"nr-of-selectors" : 1,
"receive-throughput" : 3,
"received-message-size-limit" : "unlimited",
"select-timeout" : "infinite",
"selector-association-retries" : 10,
"selector-dispatcher" : "akka.io.pinned-dispatcher",
"trace-logging" : "off",
"worker-dispatcher" : "akka.actor.default-dispatcher"
}
},
"jvm-exit-on-fatal-error" : "on",
"kernel" : {
"quiet" : "true"
},
"log-config-on-start" : "on",
"log-dead-letters" : 10,
"log-dead-letters-during-shutdown" : "on",
"logger-startup-timeout" : "5s",
"loggers" : [
"akka.event.slf4j.Slf4jLogger"
],
"loglevel" : "DEBUG",
"scheduler" : {
"implementation" : "akka.actor.LightArrayRevolverScheduler",
"shutdown-timeout" : "5s",
"tick-duration" : "10ms",
"ticks-per-wheel" : 512
},
"stdout-loglevel" : "WARNING",
"version" : "2.3.16"
},
"awt" : {
"toolkit" : "sun.awt.X11.XToolkit"
},
"java" : {
"version" : "1.8.0_121",
"vm" : {
"info" : "mixed mode",
"name" : "Java HotSpot(TM) 64-Bit Server VM",
"specification" : {
"name" : "Java Virtual Machine Specification",
"vendor" : "Oracle Corporation",
"version" : "1.8"
},
"vendor" : "Oracle Corporation",
"version" : "25.121-b13"
}
}
"pid" : "1",
"spray" : {
"can" : {
"client" : {
"chunkless-streaming" : "off",
"connecting-timeout" : "10s",
"idle-timeout" : "610s",
"max-encryption-chunk-size" : "1m",
"parsing" : {
"header-cache" : {
"Content-MD5" : 0,
"Date" : 0,
"If-Match" : 0,
"If-Modified-Since" : 0,
"If-None-Match" : 0,
"If-Range" : 0,
"If-Unmodified-Since" : 0,
"User-Agent" : 32,
"default" : 12
},
"illegal-header-warnings" : "off",
"incoming-auto-chunking-threshold-size" : "infinite",
"max-chunk-ext-length" : 256,
"max-chunk-size" : "1m",
"max-content-length" : "8m",
"max-header-count" : 64,
"max-header-name-length" : 64,
"max-header-value-length" : "8k",
"max-response-reason-length" : 64,
"max-uri-length" : "2k",
"ssl-session-info-header" : "off",
"uri-parsing-mode" : "strict"
},
"proxy" : {
"http" : "default",
"https" : "default"
},
"reaping-cycle" : "250 ms",
"request-header-size-hint" : 512,
"request-timeout" : "605s",
"response-chunk-aggregation-limit" : "5m",
"ssl-tracing" : "off",
"user-agent-header" : "spray-can/1.3.2"
},
"connection-dispatcher" : "akka.actor.default-dispatcher",
"host-connector" : {
"client" : {
"chunkless-streaming" : "off",
"connecting-timeout" : "10s",
"idle-timeout" : "60 s",
"max-encryption-chunk-size" : "1m",
"parsing" : {
"header-cache" : {
"Content-MD5" : 0,
"Date" : 0,
"If-Match" : 0,
"If-Modified-Since" : 0,
"If-None-Match" : 0,
"If-Range" : 0,
"If-Unmodified-Since" : 0,
"User-Agent" : 32,
"default" : 12
},
"illegal-header-warnings" : "on",
"incoming-auto-chunking-threshold-size" : "infinite",
"max-chunk-ext-length" : 256,
"max-chunk-size" : "1m",
"max-content-length" : "8m",
"max-header-count" : 64,
"max-header-name-length" : 64,
"max-header-value-length" : "8k",
"max-response-reason-length" : 64,
"max-uri-length" : "2k",
"ssl-session-info-header" : "off",
"uri-parsing-mode" : "strict"
},
"proxy" : {
"http" : "default",
"https" : "default"
},
"reaping-cycle" : "250 ms",
"request-header-size-hint" : 512,
"request-timeout" : "20 s",
"response-chunk-aggregation-limit" : "1m",
"ssl-tracing" : "off",
"user-agent-header" : "spray-can/1.3.2"
},
"idle-timeout" : "30 s",
"max-connections" : 100,
"max-redirects" : 0,
"max-retries" : 0,
"pipelining" : "off"
},
"host-connector-dispatcher" : "akka.actor.default-dispatcher",
"listener-dispatcher" : "akka.actor.default-dispatcher",
"manager-dispatcher" : "akka.actor.default-dispatcher",
"parsing" : {
"header-cache" : {
"Content-MD5" : 0,
"Date" : 0,
"If-Match" : 0,
"If-Modified-Since" : 0,
"If-None-Match" : 0,
"If-Range" : 0,
"If-Unmodified-Since" : 0,
"User-Agent" : 32,
"default" : 12
},
"illegal-header-warnings" : "off",
"incoming-auto-chunking-threshold-size" : "infinite",
"max-chunk-ext-length" : 256,
"max-chunk-size" : "1m",
"max-content-length" : "8m",
"max-header-count" : 64,
"max-header-name-length" : 64,
"max-header-value-length" : "8k",
"max-response-reason-length" : 64,
"max-uri-length" : "2k",
"ssl-session-info-header" : "off",
"uri-parsing-mode" : "strict"
},
"server" : {
"automatic-back-pressure-handling" : "on",
"back-pressure" : {
"noack-rate" : 10,
"reading-low-watermark" : "infinite"
},
"bind-timeout" : "1s",
"chunkhandler-registration-timeout" : "500 ms",
"chunkless-streaming" : "off",
"default-host-header" : "",
"idle-timeout" : "610s",
"max-encryption-chunk-size" : "1m",
"parsing" : {
"header-cache" : {
"Content-MD5" : 0,
"Date" : 0,
"If-Match" : 0,
"If-Modified-Since" : 0,
"If-None-Match" : 0,
"If-Range" : 0,
"If-Unmodified-Since" : 0,
"User-Agent" : 32,
"default" : 12
},
"illegal-header-warnings" : "off",
"incoming-auto-chunking-threshold-size" : "infinite",
"max-chunk-ext-length" : 256,
"max-chunk-size" : "1m",
"max-content-length" : "8m",
"max-header-count" : 64,
"max-header-name-length" : 64,
"max-header-value-length" : "16k",
"max-response-reason-length" : 64,
"max-uri-length" : "2k",
"ssl-session-info-header" : "off",
"uri-parsing-mode" : "relaxed-with-raw-query"
},
"parsing-error-abort-timeout" : "2s",
"pipelining-limit" : 1,
"raw-request-uri-header" : "off",
"reaping-cycle" : "250 ms",
"registration-timeout" : "1s",
"remote-address-header" : "on",
"request-chunk-aggregation-limit" : "1m",
"request-timeout" : "605s",
"response-header-size-hint" : 512,
"server-header" : "BlueJeans Proxy",
"ssl-encryption" : "off",
"ssl-tracing" : "off",
"stats-support" : "on",
"timeout-handler" : "",
"timeout-timeout" : "2 s",
"transparent-head-requests" : "on",
"unbind-timeout" : "1s",
"verbose-error-messages" : "off"
},
"settings-group-dispatcher" : "akka.actor.default-dispatcher"
}
"version" : "1.3.2"
}
}
2018-04-17T11:39:20.980-0700 [DEBUG] akka.io.TcpListener [18:39:20.979UTC] - Successfully bound to /10.4.4.178:9080
Versions: akka - 2.3.0 spray - 1.3.2 running on alpine linux inside a docker container.
Any pointers would help. Thanks in advance.
517
In the docker-compose file, define port mapping under appname.
ports:
9080:9080
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
