Currently setting up a keycloak cluster in standalone-ha mode, to be able to run on docker swarm. In keycloak, the user sessions are cached in an embedded infinispan store and infinispan can be configured to be a distributed cache across the cluster.
I have also set the owner to be 2, but the problem is that.. during scale-down, there is a possibility for the user-sessions to be lost, If both the owners containing the cache are killed during scale-down.
I have also read about Infinispan Redis cache store, but I am not sure how to configure this.
Question 1: Is it possible to configure Keycloak Infinispan to user a Redis Store ?
Question 2: If this is not possible, is there a way that one could overcome this problem ?
Any suggestions would be helpful.
Keycloak is designed for high availability and multi-node clustered setups. The current distributed cache implementation is built on top of Infinispan, a high-performance, distributable in-memory data grid. All available cache options are build options, so they need to be applied to a build of Keycloak before starting.
Infinispan replicates cache entries on a subset of nodes in a cluster and assigns entries to fixed owner nodes. Infinispan requests read operations from owner nodes to ensure it returns the correct value. Infinispan evicts stale data from all nodes whenever operations modify entries in the cache.
To clear the realm or user cache, go to the Red Hat Single Sign-On admin console Realm Settings→Cache Config page. On this page you can clear the realm cache or the user cache. This will clear the caches for all realms and not only the selected realm.
Infinispan is an open-source in-memory data grid that offers flexible deployment options and robust capabilities for storing, managing, and processing data. Infinispan provides a key/value data store that can hold all types of data, from Java objects to plain text.
Due to this PR https://github.com/keycloak/keycloak/commit/056ba75a72b1595ca9fa471f5693201fd5b2c7ae by default (Keycloak latest release 6.0.1) the Infinispan Connection SPI
which uses InfinispanChangelogBasedTransaction.java
has a very particular use of a CacheDecorator.java
which will skipCacheStore
. This means that no matter if you configure a store with persistence the store will be ignored.
In order to achieve what you want, besides configuring the store, you would have to customize most of the SPIs here https://github.com/keycloak/keycloak/tree/master/model/infinispan/src/main/resources/META-INF/services in order to make sure that Keycloak will be using the cache store.
This will also not be easy since there are a lot of perks involved into the process, for example, since Keycloak is using the Marshaller of Jboss, if you customize this SPIs you would have to bring most of the org.keycloak.models.sessions.infinispan
package and register your module to make sure that Wildfly will be able to see the entities to marshall.
Another thing is that you should, with Redis, configure most of the caches pointing to one common database, except the authenticationSessions
which cannot be in the same database as sessions
, otherwise, there will be conflicts like the RootAuthenticationSesssionEntity
being found but expected to be SessionEntityWrapper
.
To resume, the process will be painfull, but if you want to dare and do it, this is how I achieved it:
private Configuration getRedisConfiguration(int database) {
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.persistence()
.passivation(false)
.addStore(RedisCacheStoreConfigurationBuilder.class)
.ignoreModifications(false)
.fetchPersistentState(false)
.purgeOnStartup(false)
.preload(false)
.shared(true)
.addProperty("host", System.getenv("REDIS_HOST"))
.addProperty("port", System.getenv("REDIS_PORT"))
.addProperty("database", String.valueOf(database));
return cb.build();
}
The RedisCacheStoreConfigurationBuilder
that you see there is basically a stripped-down version of the original store but I don't need Sentinel or Server mode I just want to connect to a host, port, and database.
Then I basically copied the org.keycloak.models.sessions.infinispan
removing everything related to remove cache, and instead of using normally the cache without the decorator to skipCacheStore.
Let me know if I can help with something, I will most prepare a post that instructs more detailed how to do this, involving also a repository that will contain the codes that I am talking about. Please let me know more if someone is still trying this.
When expecting to scale large systems dynamically, it is expected to avoid such a constraint to register a list of available nodes in configurations. So Redis nodes discovery is a benefit here.
Infinispan itself supports Redis Store as an extension implementing both SPI and adding XML entities to ease configuration:
http://infinispan.org/docs/cachestores/redis/
https://github.com/infinispan/infinispan-cachestore-redis
http://infinispan.org/docs/stable/user_guide/user_guide.html#custom_cache_stores
But this extension is not supported (yet) in WildFly infinispan subsystem - as Keycloak relies on WildFly.
So I expect the following tasks to get Infinispan Redis Store available for WildFly, and so Keycloak:
Create jboss module for infinispam redis store jar - see modules/system/layers/base/org/infinispan/
Create a "Custom Cache Store" factory able to instanciate Redis Store objects (store, servers and connection pool) from WildFly configuration key/value properties. This has to be added to WildFly as a jboss module too
Use WildFly infinispan subsystem "local-cache" "custom" to configure this factory with attributes class
and properties
: https://wildscribe.github.io/WildFly/11.0/subsystem/infinispan/cache-container/local-cache/store/custom/index.html
Work is in progress and it is possible code and configuration will be published.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With