Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Keycloak - Infinispan Redis cache store

Currently setting up a keycloak cluster in standalone-ha mode, to be able to run on docker swarm. In keycloak, the user sessions are cached in an embedded infinispan store and infinispan can be configured to be a distributed cache across the cluster.

I have also set the owner to be 2, but the problem is that.. during scale-down, there is a possibility for the user-sessions to be lost, If both the owners containing the cache are killed during scale-down.

I have also read about Infinispan Redis cache store, but I am not sure how to configure this.

Question 1: Is it possible to configure Keycloak Infinispan to user a Redis Store ?

Question 2: If this is not possible, is there a way that one could overcome this problem ?

Any suggestions would be helpful.

like image 565
tux Avatar asked Aug 25 '17 12:08

tux


People also ask

What is Keycloak Infinispan?

Keycloak is designed for high availability and multi-node clustered setups. The current distributed cache implementation is built on top of Infinispan, a high-performance, distributable in-memory data grid. All available cache options are build options, so they need to be applied to a build of Keycloak before starting.

How does Infinispan cache work?

Infinispan replicates cache entries on a subset of nodes in a cluster and assigns entries to fixed owner nodes. Infinispan requests read operations from owner nodes to ensure it returns the correct value. Infinispan evicts stale data from all nodes whenever operations modify entries in the cache.

How do I clear my Keycloak cache?

To clear the realm or user cache, go to the Red Hat Single Sign-On admin console Realm Settings→Cache Config page. On this page you can clear the realm cache or the user cache. This will clear the caches for all realms and not only the selected realm.

What is Infinispan server?

Infinispan is an open-source in-memory data grid that offers flexible deployment options and robust capabilities for storing, managing, and processing data. Infinispan provides a key/value data store that can hold all types of data, from Java objects to plain text.


2 Answers

Due to this PR https://github.com/keycloak/keycloak/commit/056ba75a72b1595ca9fa471f5693201fd5b2c7ae by default (Keycloak latest release 6.0.1) the Infinispan Connection SPI which uses InfinispanChangelogBasedTransaction.java has a very particular use of a CacheDecorator.java which will skipCacheStore. This means that no matter if you configure a store with persistence the store will be ignored.

In order to achieve what you want, besides configuring the store, you would have to customize most of the SPIs here https://github.com/keycloak/keycloak/tree/master/model/infinispan/src/main/resources/META-INF/services in order to make sure that Keycloak will be using the cache store.

This will also not be easy since there are a lot of perks involved into the process, for example, since Keycloak is using the Marshaller of Jboss, if you customize this SPIs you would have to bring most of the org.keycloak.models.sessions.infinispan package and register your module to make sure that Wildfly will be able to see the entities to marshall.

Another thing is that you should, with Redis, configure most of the caches pointing to one common database, except the authenticationSessions which cannot be in the same database as sessions, otherwise, there will be conflicts like the RootAuthenticationSesssionEntity being found but expected to be SessionEntityWrapper.

To resume, the process will be painfull, but if you want to dare and do it, this is how I achieved it:

  • Introduced a custom InfinispanConnectionProviderFactory in order to have full capability to use infinispan configuration and then configure my containers like:
private Configuration getRedisConfiguration(int database) {
    ConfigurationBuilder cb = new ConfigurationBuilder();
    cb.persistence()
      .passivation(false)
      .addStore(RedisCacheStoreConfigurationBuilder.class)
      .ignoreModifications(false)
      .fetchPersistentState(false)
      .purgeOnStartup(false)
      .preload(false)
      .shared(true)
      .addProperty("host", System.getenv("REDIS_HOST"))
      .addProperty("port", System.getenv("REDIS_PORT"))
      .addProperty("database", String.valueOf(database));

    return cb.build();
  }

The RedisCacheStoreConfigurationBuilder that you see there is basically a stripped-down version of the original store but I don't need Sentinel or Server mode I just want to connect to a host, port, and database.

Then I basically copied the org.keycloak.models.sessions.infinispan removing everything related to remove cache, and instead of using normally the cache without the decorator to skipCacheStore.

Let me know if I can help with something, I will most prepare a post that instructs more detailed how to do this, involving also a repository that will contain the codes that I am talking about. Please let me know more if someone is still trying this.

like image 178
Guilherme Oliveira Avatar answered Sep 19 '22 14:09

Guilherme Oliveira


When expecting to scale large systems dynamically, it is expected to avoid such a constraint to register a list of available nodes in configurations. So Redis nodes discovery is a benefit here.

Infinispan itself supports Redis Store as an extension implementing both SPI and adding XML entities to ease configuration:

  • http://infinispan.org/docs/cachestores/redis/

  • https://github.com/infinispan/infinispan-cachestore-redis

  • http://infinispan.org/docs/stable/user_guide/user_guide.html#custom_cache_stores

But this extension is not supported (yet) in WildFly infinispan subsystem - as Keycloak relies on WildFly.

So I expect the following tasks to get Infinispan Redis Store available for WildFly, and so Keycloak:

  • Create jboss module for infinispam redis store jar - see modules/system/layers/base/org/infinispan/

  • Create a "Custom Cache Store" factory able to instanciate Redis Store objects (store, servers and connection pool) from WildFly configuration key/value properties. This has to be added to WildFly as a jboss module too

  • Use WildFly infinispan subsystem "local-cache" "custom" to configure this factory with attributes class and properties: https://wildscribe.github.io/WildFly/11.0/subsystem/infinispan/cache-container/local-cache/store/custom/index.html

Work is in progress and it is possible code and configuration will be published.

like image 21
Yves Martin Avatar answered Sep 21 '22 14:09

Yves Martin