Could not resolve substitution to a value: ${akka.stream.materializer} in AWS Lambda

Question

I have a java application, in which I'm using the Flink Api. So basically what I'm trying to do with the code, is to create two Datasets with few records and then register them as two tables along with the necessary fields.

 DataSet<Company> comp = env.fromElements(
                new Company("Aux", 1),
                new Company("Comp2", 2),
                new Company("Comp3", 3));

        DataSet<Employee> emp = env.fromElements(
                new Employee("Kula", 1),
                new Employee("Ish", 1),
                new Employee("Kula", 3));


        tEnv.registerDataSet("Employee", emp, "name, empId");
        tEnv.registerDataSet("Company", comp, "cName, empId");

And then I'm trying to join these two tables using the Table API:

Table anotherJoin = tEnv.sql("SELECT Employee.name, Employee.empId, Company.cName FROM " +
                "Employee RIGHT JOIN Company on Employee.empId = Company.empId");

And I'm just printing the results on the console. This works perfectly locally on my machine. I created a fat-jar by using the maven-shade-plugin with the dependencies and I'm trying to execute it in AWS Lambda.

So when I try to execute it there, I'm being thrown with the following exception (I'm posting only the first few lines):

reference.conf @ file:/var/task/reference.conf: 804: Could not resolve substitution to a value: ${akka.stream.materializer}: com.typesafe.config.ConfigException$UnresolvedSubstitution com.typesafe.config.ConfigException$UnresolvedSubstitution: reference.conf @ file:/var/task/reference.conf: 804: Could not resolve substitution to a value: ${akka.stream.materializer} at com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:111) at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179) at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)

I extracted the jar before executing it in Lambda, and happened to see all the dependencies were there. I can't figure out where is it going wrong?

Any help could be appreciated.

Answer 1

You need to add this code in the pom -> maven-shaded-plugin -> configuration section:

<transformers>
    <!-- append default configs -->
    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
        <resource>reference.conf</resource>
    </transformer>
</transformers>

Answer 2

Was able to finally, figure this out, and it was some major version issues within my pom. I then downgraded all the dependencies to Flink 1.3.2 and also added <relocations> within the shade plugin. It works now. I'm attaching the entire pom, so that it would help someone someday:

<build>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.7.0</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>


       <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.1.0</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>com.ink.FlinkLambdaTest.FlinkToLambda</mainClass>
                </transformer>
                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                  <resource>reference.conf</resource>
                </transformer>
              </transformers>
              <relocations>
                <relocation>
                  <pattern>org.codehaus.plexus.util</pattern>
                  <shadedPattern>org.shaded.plexus.util</shadedPattern>
                  <excludes>
                    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                  </excludes>
                </relocation>
              </relocations>
            </configuration>
          </execution>
        </executions>
      </plugin>  
    </plugins>
  </build>

<dependencies>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-table_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>

    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-java</artifactId>
      <version>1.3.2</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-clients_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>

    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-scala_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>

    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-scala_2.10</artifactId>
      <version>1.3.2</version>
    </dependency>

    </dependencies>

Make sure to change the main class with yours.