java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider when flink boot up

Question

I'm running Flink 1.11 on k8s cluster and getting the following error when trying to update the log4j-console.properties file:

Starting Task Manager
Enabling required built-in plugins
Linking flink-s3-fs-hadoop-1.11.1.jar to plugin directory
Successfully enabled flink-s3-fs-hadoop-1.11.1.jar
sed: couldn't open temporary file /opt/flink/conf/sedl2dH0X: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedPLYAzY: Read-only file system
/docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml: Permission denied
sed: couldn't open temporary file /opt/flink/conf/sede0G5LW: Read-only file system
/docker-entrypoint.sh: 120: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting taskexecutor as a console application on host flink-taskmanager-c765c947c-qx68t.
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ser/FilterProvider
    at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:158)
    at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:69)
    at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:102)
    at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:77)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
    at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<clinit>(TaskManagerRunner.java:89)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
    at java.base/java.lang.ClassLoader.loadClass(Unknown Source)

my log4j-console.properties:

rootLogger.level = INFO
#rootLogger.appenderRef.console.ref = ConsoleAppender

appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n

appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.topic = test
appender.kafka.layout.type = JsonLayout
appender.kafka.layout.complete = false
appender.kafka.additional1.type = KeyValuePair
appender.kafka.additional1.key=app
appender.kafka.additional1.value=TEST
appender.kafka.additional2.type = KeyValuePair
appender.kafka.additional2.key=subsystem
appender.kafka.additional2.value=TEST
appender.kafka.additional3.type = KeyValuePair
appender.kafka.additional3.key=deployment
appender.kafka.additional3.value=TEST
appender.kafka.property.bootstrap.servers=***

rootLogger.appenderRef.console.ref = STDOUT
rootLogger.appenderRef.kafka.ref = Kafka

Im using "flink:1.11.1-scala_2.11-java11" docker image and validated that all log4j2 dependencies are in the classpath.

I have also tried to create a new docker image from the above base image and add to it the missing dependency and yet nothing happened.

Answer 1

I too suffered from this bug. The issue here is that when the task manager and job managers start they are running with a modified classpath, not the JAR that you've built via your build system.

See the constructFlinkClassPath in the flink source code. To prove this out, revert the JSON logging pattering and check out the classpath in the tm/jm logs on startup. You'll notice that your JAR isn't on the classpath.

To fix this issue you need to provide the dependencies (in this case you'll need jackson-core jackson-annotations and jackson-databind) to the lib folder within the tm/jm nodes (the lib folder is included by default in the flink classpath).

If you are using docker, you can do this when you build the container (RUN wget...).