Menu
I'm working to install and run Keycloak 20.0.2 in my own network. I'm hosted on AWS and running on EC2 linux instances with a direct-install (no kubernetes)
As I have two instances, I need to enable clustering on Keycloak to ensure there are no issues with user sessions between the two nodes.
The problem I'm facing is that neither node appears to recognize the other node. I've also taken a look into the S3 bucket in question and no records have been written to it
I was able to install the jgroups-aws-2.0.1.Final.jar to the providers folder. I set up a custom cache-ispn.xml and custom-jgroups-ec2.xml file
The cache-ispn.xml file looks like this
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2019 Red Hat, Inc. and/or its affiliates
~ and other contributors as indicated by the @author tags.
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
xmlns="urn:infinispan:config:11.0">
<!-- custom stack goes into the jgroups element -->
<jgroups>
<stack-file name="ec2" path="/home/keycloak/config/custom-jgroups-ec2.xml"/>
</jgroups>
<cache-container name="keycloak">
<transport lock-timeout="60000" stack="ec2"/>
<local-cache name="realms">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<local-cache name="users">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<distributed-cache name="sessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="authenticationSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="clientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="loginFailures" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<local-cache name="authorization">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<replicated-cache name="work">
<expiration lifespan="-1"/>
</replicated-cache>
<local-cache name="keys">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="3600000"/>
<memory max-count="1000"/>
</local-cache>
<distributed-cache name="actionTokens" owners="2">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="-1" lifespan="-1" interval="300000"/>
<memory max-count="-1"/>
</distributed-cache>
</cache-container>
</infinispan>
The /home/keycloak/config/custom-jgroups-ec2.xml looks like this
<!--
Based on tcp.xml but with new aws.S3_PING.
-->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:org:jgroups"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP bind_port="7800"
recv_buf_size="${tcp.recv_buf_size:5M}"
send_buf_size="${tcp.send_buf_size:5M}"
max_bundle_size="64K"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="8"
thread_pool.keep_alive_time="5000"/>
<aws.S3_PING region_name="us-east-1"
bucket_name="my_test_proj-keycloak"
bucket_prefix="production"/>
<MERGE3 min_interval="10000"
max_interval="30000"/>
<FD_SOCK/>
<FD_ALL timeout="30000" interval="5000"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK2 use_mcast_xmit="false"
discard_delivered_msgs="true"/>
<UNICAST3/>
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
max_bytes="4M"/>
<pbcast.GMS print_local_addr="true" join_timeout="2000"
view_bundling="true"/>
<MFC max_credits="2M"
min_threshold="0.4"/>
<FRAG2 frag_size="60K"/>
</config>
Next, I ran the build command
/home/keycloak/current/bin/kc.sh build --cache=ispn --cache-config-file=/home/keycloak/config/cache-ispn.xml --db=postgres --health-enabled=true --metrics-enabled=true
Finally, I loaded my env files and custom config files and ran the following
env "$(cat /home/keycloak/config/keycloak.service.env)" /home/keycloak/current/bin/kc.sh --config-file=/home/keycloak/config/keycloak.conf start --optimized
The output on both servers looks effectively like this
Appending additional Java properties to JAVA_OPTS: -Djgroups.s3.bucket=my_test_proj-keycloak -Djgroups.s3.access_key=**REDACTED_ACCESS_KEY** -Djgroups.s3.secret_access_key=**REDACTED_SECRET_KEY**
2023-01-08 13:36:35,304 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: ohmvision.com, Strict HTTPS: true, Path: /auth, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: 443, Proxied: true
2023-01-08 13:36:39,048 WARN [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2023-01-08 13:36:41,474 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-01-08 13:36:41,500 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-01-08 13:36:41,541 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-01-08 13:36:41,604 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-01-08 13:36:42,137 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.10.Final
2023-01-08 13:36:42,392 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-01-08 13:36:42,392 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2023-01-08 13:36:42,536 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2023-01-08 13:36:42,539 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20.00MB, but the OS only allocated 212.99KB
2023-01-08 13:36:42,541 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1.00MB, but the OS only allocated 212.99KB
2023-01-08 13:36:42,543 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25.00MB, but the OS only allocated 212.99KB
2023-01-08 13:36:44,553 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) web1-11583: no members discovered after 2003 ms: creating cluster as coordinator
2023-01-08 13:36:44,565 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [web1-11583|0] (1) [web1-11583]
2023-01-08 13:36:44,571 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `web1-11583`, physical addresses are `[11.159.1.137:54366]`
2023-01-08 13:36:45,554 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: web1-11583, Site name: null
2023-01-08 13:36:47,108 INFO [io.quarkus] (main) Keycloak 20.0.2 on JVM (powered by Quarkus 2.13.3.Final) started in 15.837s. Listening on: http://0.0.0.0:7000
2023-01-08 13:36:47,108 INFO [io.quarkus] (main) Profile prod activated.
2023-01-08 13:36:47,109 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, smallrye-metrics, vault, vertx]
684
asked Nov 23 '25 15:11
Keycloak: 20.0.3
To implement default "ec2" Infinispan stacks protocol you should:
./providers dirjgroups-aws-2.0.1.Final.jar (stacks protocol)aws-java-sdk-core-1.12.410.jar (access to AWS creds, etc.)aws-java-sdk-s3-1.12.410.jar (access to S3, etc.)joda-time-2.12.2.jar (is used)--cache-stack=ec2 (no --cache-config-file option!)bin/kc.[sh|bat] build --cache-stack=ec2By me it's working) GL & HF!
Logs:
2023-02-21 09:47:28,062 INFO [org.infinispan.server.core.transport.EPollAvailable] (keycloak-cache-init) ISPN005028: Native Epoll transport not available, using NIO instead: java.lang.ExceptionInInitializerError
2023-02-21 09:47:28,453 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-02-21 09:47:28,472 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-02-21 09:47:28,506 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-02-21 09:47:28,987 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-02-21 09:47:29,086 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.10.Final
2023-02-21 09:47:29,270 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-02-21 09:47:29,290 WARN [org.jgroups.stack.Configurator] (keycloak-cache-init) NATIVE_S3_PING has been deprecated; please upgrade to a newer version of the protocol
2023-02-21 09:47:30,181 INFO [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) using Amazon S3 ping in region us-east-1 with bucket 'my-jgroups-s3-bucket-test' and prefix ''
2023-02-21 09:47:30,937 INFO [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) found bucket my-jgroups-s3-bucket-test
2023-02-21 09:48:04,101 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [ip-10-68-49-170-40943|3] (2) [ip-10-68-49-170-40943, ip-10-68-49-190-31671]
2023-02-21 09:48:04,111 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `ip-10-68-49-190-31671`, physical addresses are `[10.68.49.190:7800]`
2023-02-21 09:48:07,087 INFO [io.quarkus] (main) Keycloak 20.0.3 on JVM (powered by Quarkus 2.13.6.Final) started in 78.156s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
2023-02-21 09:48:07,087 INFO [io.quarkus] (main) Profile prod activated.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With