Export all users from KeyCloak

Question

I have a specific use case in which we want to ask Keycloak for all the users and the groups and roles for each user, on a daily basis. For reconciliation purposes with other internal systems.

Currently we are using the provided Keycloak endpoints in the UsersResource for this. But we see that performance slows down after each call to a point we can't use this solution anymore. There are more then 30K users in the realm.

We've also seen that Keycloak can export the database, but only on system boot (I guess for migration purposes). Given that we want to extract all the users on a daily basis we cannot use this.

Are there some known functionalities or workarounds?

Answer 1

you need in your docker-compose-yml to bind your folder, not just the realm json file, like this:

keycloak:
    image: jboss/keycloak:8.0.1
    container_name: "keycloak"
     volumes:
      - ./realms/:/tmp/
    environment:
      - KEYCLOAK_USER=admin
      - KEYCLOAK_PASSWORD=admin
      - KEYCLOAK_IMPORT=/tmp/realm-export.json -Dkeycloak.profile.feature.upload_scripts=enabled 

where realms is your folder beside the yaml file. At this point you can run docker-compose up -d with your basic realm-export.json as always, go in your admin panel, adding users with credentials and roles, and then with this command you will able to export the entire configuration:

docker exec -it keycloak /opt/jboss/keycloak/bin/standalone.sh -Djboss.socket.binding.port-offset=100 -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=singleFile -Dkeycloak.migration.realmName=ed-realm -Dkeycloak.migration.usersExportStrategy=REALM_FILE -Dkeycloak.migration.file=/tmp/export.json

You will see in your realms folder that a new file will be created, and it will contain the entire configuration, so you can run docker-compose down, replace your old file with this new and run again docker-compose up as many time you want, and redoing the process when you will change your realm again.

Answer 2

I have done it with an parallel starting container via docker, which connects to the existing keycloak db.

Please use the same Version of the container keycloak as the real keacloak has. Because of db schema differences between versions.

EXPORT

docker run --rm\
    --name keycloak_exporter\
    -v /tmp:/tmp/keycloak-export:Z\
    -e POSTGRES_DATABASE=keycloak\
    -e POSTGRES_PASSWORD=PASSOWRD_PLEASE\
    -e POSTGRES_USER=keycloak\
    -e DB_VENDOR=POSTGRES\
    -e POSTGRES_PORT_5432_TCP_ADDR=postgresql.local\
    jboss/keycloak:3.4.3.Final\
    -Dkeycloak.migration.action=export\
    -Dkeycloak.migration.provider=dir\
    -Dkeycloak.migration.dir=/tmp/keycloak-export\
    -Dkeycloak.migration.usersExportStrategy=SAME_FILE\
    -Dkeycloak.migration.realmName=therealm

IMPORT

docker run --rm\
    --name keycloak_importer\
    -v /tmp:/tmp/keycloak-import:Z\
    -e POSTGRES_DATABASE=keycloak_dest\
    -e POSTGRES_PASSWORD=PASSOWRD_DEST_PLEASE\
    -e POSTGRES_USER=keycloak\
    -e DB_VENDOR=POSTGRES\
    -e POSTGRES_PORT_5432_TCP_ADDR=postgresql2.local\
    jboss/keycloak:3.4.3.Final\
    -Dkeycloak.migration.action=import\
    -Dkeycloak.migration.provider=dir\
    -Dkeycloak.migration.dir=/tmp/keycloak-import\
    -Dkeycloak.migration.strategy=IGNORE_EXISTING\
    -Dkeycloak.migration.usersExportStrategy=SAME_FILE\
    -Dkeycloak.migration.realmName=therealm

Possible config options: https://github.com/keycloak/keycloak-documentation/blob/master/server_admin/topics/export-import.adoc