Deadlock in Spring Boot Application's @PostConstruct Method

Question

I'm using Spring TaskScheduler to schedule tasks (obviously...) on start of application.

The TaskScheduler is created in my SpringConfig:

@Configuration
@EnableTransactionManagement
public class SpringConfig {

    @Bean
    public TaskScheduler taskScheduler() {
        return new ThreadPoolTaskScheduler();
    }

}

The spring boot application is started in my Main.class and schedules tasks @PostConstruct

@SpringBootApplication
@ComponentScan("...")
@EntityScan("...")
@EnableJpaRepositories("... .repositories")
@EnableAutoConfiguration
@PropertySources(value = {@PropertySource("classpath:application.properties")})
public class Main {

    private final static Logger LOGGER = LoggerFactory.getLogger(Main.class);

    private static SpringApplication application = new SpringApplication(Main.class);

    private TaskScheduler taskScheduler;

    private AnalysisCleaningThread cleaningThread;

    @Inject
    public void setCleaningThread(AnalysisCleaningThread cleaningThread) {
        this.cleaningThread = cleaningThread;
    }

    @Inject
    public void setTaskScheduler(TaskScheduler taskScheduler) {
        this.taskScheduler = taskScheduler;
    }

    public static void main(String[] args)
            throws Exception {

        try {

            //Do some setup

            application.run(args);

        } catch (Exception e) {

            LOGGER.error(e.getMessage(), e);
        }
    }


    @PostConstruct
    public void init()
            throws Exception {

        //Do some setup as well

        ScheduledFuture scheduledFuture = null;
        LOGGER.info("********** Scheduling one time Cleaning Thread. Starting in 5 seconds **********");
        Date nowPlus5Seconds = Date.from(LocalTime.now().plus(5, ChronoUnit.SECONDS).atDate(LocalDate.now()).atZone(ZoneId.systemDefault()).toInstant());
        scheduledFuture = this.taskScheduler.schedule(this.cleaningThread, nowPlus5Seconds);




        while (true) {
           //Somehow blocks thread from running
           if (scheduledFuture.isDone()) {
               break;
           }
           Thread.sleep(2000);
        }



        //schedule next periodic thread

}

The application must wait for the thread to complete because its task is to clean dirty database entries after an unexpected application shutdown. The next task is picking up the cleaned entries and processes them again. The cleaning thread is implemented as follows:

@Named
@Singleton
public class AnalysisCleaningThread implements Runnable {

    private static Logger LOGGER = LoggerFactory.getLogger(AnalysisCleaningThread.class);

    private AnalysisService analysisService;

    @Inject
    public void setAnalysisService(AnalysisService analysisService) {
        this.analysisService = analysisService;
    }

    @Override
    public void run() {
        List<Analysis> dirtyAnalyses = analysisService.findAllDirtyAnalyses();
        if(dirtyAnalyses != null && dirtyAnalyses.size() > 0) {
            LOGGER.info("Found " + dirtyAnalyses.size() + " dirty analyses. Cleaning... ");
            for (Analysis currentAnalysis : dirtyAnalyses) {
                //Reset AnalysisState so it is picked up by ProcessingThread on next run
                currentAnalysis.setAnalysisState(AnalysisState.CREATED);
            }
            analysisService.saveAll(dirtyAnalyses);
        } else {
            LOGGER.info("No dirty analyses found.");
        }
    }

}

I put a breaking point on first line of run method and on the second line. If I use ScheduledFuture.get(), the first line is called, which then calls a JPA repository method, but it never returns... It doesn't generate a query in the console...

If I use ScheduledFuture.isDone() the run method isn't invoked at all...

EDIT:

So i dug further into that problem and this is what i found out where it stops working:

1. I used scheduledFuture.get() to wait for the task completion
2. First line of code in run() method of AnalysisCleaningThread gets called which should call the service to retrieve a List of Analysis
3. CglibAopProxy is called to intercept the method
4. ReflectiveMethodInvocation -> TransactionInterceptor -> TransactionAspectSupport -> DefaultListableBeanFactory -> AbstractBeanFactory is called to search and match the PlatformTransactionManager bean by type
5. DefaultSingletonBeanRegistry.getSingleton is called with beanName "main" and at line 187 synchronized(this.singletonObjects) the application pauses and never continues

From my point of view it seems like this.singletonObjects is currently in use so the thread cannot continue somehow...

Answer 1

So I've done a lot of research since that problem occured and finally found a solution to my rare case.

What I first noticed was, that without future.get(), the AnalysisCleaningThread did run without any problem but the run method took took like 2 seconds for the first line to execute so I thought there must be something going on in the background before the database call could finally be made.

What I found out through debugging in my original question edit was, that the application stops at a synchronize block synchronized(this.singletonObjects) in the DefaultSingletonBeanRegistry.getSingleton method on line 93 so something must be holding that lock object. And it actually stopped at that line as soon as the iterating method calling DefaultSingletonBeanRegistry.getSingleton passed "main" as parameter "beanName" into getSingleton.

BTW that method (or better chain of methods) get's called in order to obtain an instance of the PlatformTransactionManager bean to make that service (database) call.

My first thought then was, that it must be a deadlock.

Final Thoughts

From my understanding the bean is still not finally ready within its lifecycle (still in its @PostConstruct init() method). As spring tries to a aquire an instance of the platform transaction manager in order to make the database query, the application deadlocks. It actually deadlocks because while iterating over all bean names to find the PlatformTansactionManager, it also tries to resolve the "main" bean which currently is waiting because of the future.get() in its @PostConstruct method. Therefore it cannot obtain an instance and is waiting forever for the lock to be released.

Solution

As I didn't want to put that code in another class because the Main.class is my entry point, I started to look for a hook which starts the tasks after the application has fully started up.

I stumpled upon the @EventListener which in my case listens for ApplicationReadyEvent.class and voilà, it works. Here is my code solution.

@SpringBootApplication
@ComponentScan("de. ... .analysis")
@EntityScan("de. ... .persistence")
@EnableJpaRepositories("de. ... .persistence.repositories")
@EnableAutoConfiguration
@PropertySources(value = {@PropertySource("classpath:application.properties")})
public class Main {

    private final static Logger LOGGER = LoggerFactory.getLogger(Main.class);

    private static SpringApplication application = new SpringApplication(Main.class);

    private TaskScheduler taskScheduler;

    private AnalysisProcessingThread processingThread;

    private AnalysisCleaningThread cleaningThread;


    @Inject
    public void setProcessingThread(AnalysisProcessingThread processingThread) {
        this.processingThread = processingThread;
    }

    @Inject
    public void setCleaningThread(AnalysisCleaningThread cleaningThread) {
        this.cleaningThread = cleaningThread;
    }

    @Inject
    public void setTaskScheduler(TaskScheduler taskScheduler) {
        this.taskScheduler = taskScheduler;
    }

    public static void main(String[] args)
            throws Exception {

        try {

            //Do some setup

            application.run(args);

        } catch (Exception e) {

            LOGGER.error(e.getMessage(), e);
        }
    }

    @PostConstruct
    public void init() throws Exception {

        //Do some other setup

    }

    @EventListener(ApplicationReadyEvent.class)
    public void startAndScheduleTasks() {
        ScheduledFuture scheduledFuture = null;
        LOGGER.info("********** Scheduling one time Cleaning Thread. Starting in 5 seconds **********");
        Date nowPlus5Seconds = Date.from(LocalTime.now().plus(5, ChronoUnit.SECONDS).atDate(LocalDate.now()).atZone(ZoneId.systemDefault()).toInstant());
        scheduledFuture = this.taskScheduler.schedule(this.cleaningThread, nowPlus5Seconds);
        try {
            scheduledFuture.get();
        } catch (InterruptedException | ExecutionException e) {
            LOGGER.error("********** Cleaning Thread did not finish as expected! Stopping thread. Dirty analyses may still remain in database **********", e);
            scheduledFuture.cancel(true);
        }
  }
}

Summary

Executing a spring data repository call from a @PostConstruct method can - under rare circumstances - deadlock if the method annoted with @PostConstruct does not end before spring can aquire the PlatformTransactionManager bean to execute the spring data repository query. It won't matter if its an endless loop or a future.get() method... . It also only occurs if the method, which iterates over all registered beanNames and finally calls DefaultSingletonBeanRegistry.getSingleton in order to find the PlatformTransactionManager bean, calls getSingleton with the bean name which currently is in the @PostConstruct method. If it finds the PlatformTransactionManager before that, then it won't happen.