Ensuring spring boot and liquibase receive and handle SIGTERM

Question

Currently running SpringBoot applications in a containerised environment (ECS) and I've observed scenarios in which the container gets terminated during start-up and while it's still holding the Liquibase changelock.

This leads to issues in all containers that are spun afterwards and ends up requiring manual intervention.

Is it possible to ensure that if the process receives a SIGTERM, it will gracefully handle termination and release the lock?

I've already ensured that the container is receiving the signals by enabling via InitProcessEnabled (in the CloudFormation template) and use of "exec java ..." as a java agent we use does gracefully shutdown on this circumstances.

Answer 1

Heyo,

As mentioned in the GitHub issue I have a workaround. A solution is yet to be implemented.

You can manually register a shutdown hook before running spring boot.. That hook should assure that the Termination is postponed until liquibase is done.

package dang;

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;


@EnableJpaRepositories
@SpringBootApplication
public class DangApplication {
  public static void main(String[] args) throws InterruptedException {
    Thread thread = new GracefulShutdownHook();
    Runtime.getRuntime().addShutdownHook(thread);

    new SpringApplicationBuilder(DangApplication.class)
            .registerShutdownHook(true)
            .logStartupInfo(true)
            .build()
            .run();
    Runtime.getRuntime().removeShutdownHook(thread);
  }
}

And the hook:

package dang;

import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;

import java.util.Map;

@Slf4j
public class GracefulShutdownHook extends Thread {
  @SneakyThrows
  @Override
  public void run() {


    super.run();
    log.info("Shutdown Signal received.. Searching for Liquibase instances!");
    boolean liquibaseIsRunning = true;
    while (liquibaseIsRunning) {

      Map<Thread,StackTraceElement[]> stackTraces = Thread.getAllStackTraces();
      for(Map.Entry<Thread, StackTraceElement[]> entry : stackTraces.entrySet()) {
        StackTraceElement[] stackTraceElements = entry.getValue();
        for (StackTraceElement stackTraceElement : stackTraceElements) {
          if (stackTraceElement.getClassName().contains("liquibase") && stackTraceElement.getMethodName().contains("update")) {
            try {
              log.warn("Liquibase is currently updating");
              entry.getKey().join();
              liquibaseIsRunning = false;
            } catch (InterruptedException e) {
              log.error("Shutdown Hook was interrupted.. Fatal databaselock may be imminent", e);
              if (Thread.interrupted()) {
                throw e;
              }
            }
          }
        }
      }
    }
  }
}

EDIT

After implementing my workaround a contributor of liquibase shared a different solution (It's actually the same solution just through Spring functionality) which is much better than what I did:

package dang;

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;


@EnableJpaRepositories
@SpringBootApplication
public class DangApplication {
  public static void main(String[] args) throws InterruptedException {
    new SpringApplicationBuilder(DangApplication.class)
            .initializers(ConfigurableApplicationContext::registerShutdownHook) // Registers application hook before liquibase executes.
            .logStartupInfo(true)
            .build()
            .run();
  }
}