AWS Lambda handler throws a ClassCastException with Scala generics

Question

I'm building an AWS lambda function with their POJO handler, but abstracting over the RequestHandler interface results in an erased type. When this happens AWS can't cast to the input type of my lambda function:

java.util.LinkedHashMap cannot be cast to com.amazonaws.services.lambda.runtime.events.SNSEvent: java.lang.ClassCastException
java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to com.amazonaws.services.lambda.runtime.events.SNSEvent

The following code works when uploaded to AWS:

import com.amazonaws.services.lambda.runtime._
import com.amazonaws.services.lambda.runtime.events.SNSEvent

// Only working version
class PojoTest1 extends Handler1[SNSEvent]{
  override def handleRequest(input: SNSEvent, context: Context): Unit =
    println(s"message: ${input.getRecords.get(0).getSNS.getMessage}")
}

trait Handler1[Event] extends RequestHandler[Event, Unit]{
  override def handleRequest(input: Event, context: Context): Unit
}

Now because I'm using Scala I'm abstracting away the Java RequestHandler with generic traits. The following is a minified example of what doesn't work:

// Doesn't work
class PojoTest2 extends Handler2[SNSEvent]{
  override def act(input: SNSEvent): Unit =
    println(s"message: ${input.getRecords.get(0).getSNS.getMessage}")
}

trait Handler2[Event] extends RequestHandler[Event, Unit]{
  def act(input: Event): Unit
  override def handleRequest(input: Event, context: Context): Unit = act(input)
}

When I run javap PojoTest1.class this is the method that makes everything work:

public void handleRequest(com.amazonaws.services.lambda.runtime.events.SNSEvent, com.amazonaws.services.lambda.runtime.Context);

When I run javap PojoTest2.class You can see from this signature that the type of SNSEvent has been erased to Object:

public void handleRequest(java.lang.Object, com.amazonaws.services.lambda.runtime.Context);

This looks exactly like the issue described in SI-8905. Unfortunately the posted workaround doesn't seem to work either:

// Doesn't work
abstract class Handler3[T] extends Handler2[T]

class PojoTest3 extends Handler3[SNSEvent]{
  override def act(input: SNSEvent): Unit =
    println(s"message: ${input.getRecords.get(0).getSNS.getMessage}")
}

Even extending an abstract class directly doesn't yield better results:

// Doesn't work
class PojoTest4 extends Handler4[SNSEvent]{
  override def act(input: SNSEvent): Unit =
    println(s"message: ${input.getRecords.get(0).getSNS.getMessage}")
}

abstract class Handler4[Event] extends RequestHandler[Event, Unit] {
  def act(input: Event): Unit
  override def handleRequest(input: Event, context: Context): Unit = act(input)
}

When I use javap on any of the classes that don't work I still get the same method signature with the erased type.

I'm using Scala 2.12.7, sbt 1.1.2, and sbt-assembly 0.14.8.

I'm looking for any sort of work around to deal with this.

Answer 1

Note: I don't work for Amazon or Sun/Oracle so parts of the answer is a speculation.

I think there is a fundamental conflict between JVM type erasure, how AWS tries to work it around and what you are trying to do. I also don't think that the bug you referenced is relevant. I think the behavior is the same for Java.

AFAIU from the AWS point of view the problem looks like this: there is a stream of events of different types and a bunch of handlers. You need to decide which events a given handler can handle. The obvious solution is to look at the signature of the handleRequest method and use the argument's type. Unfortunately JVM type system doesn't really supports generics so you have to look for the most specific method (see further) and assume that that method is the real deal.

Now assume you develop a compiler that targets JVM (Scala or Java, further examples will be in Java to show that this is not a Scala-specific issue). Since JVM doesn't support generics you have to erasure your types. And you want to erase them to the most narrow type that covers all possible arguments so you are still type-safe at the JVM level.

For the RequestHandler.handleRequest

public O handleRequest(I input, Context context);

the only valid type erasure is

public Object handleRequest(Object input, Context context);

because I and O are unbound.

Now assume you do

public class PojoTest1 implements RequestHandler<SNSEvent, Void> {
    @Override
    public Void handleRequest(SNSEvent input, Context context) {
        // whatever
        return null;
    }
}

At this point you say that you have a handleRequest method with this non-generic signature and the compiler has to respect it. But at the same time it has to respect your implements RequestHandler as well. So what the compiler has to do is to add a "bridge method" i.e. to produce a code logically equivalent to

public class PojoTest1 implements RequestHandler {
    // bridge-method
    @Override
    public Object handleRequest(Object input, Context context) {
        // call the real method casting the argument
        return handleRequest((SNSEvent)input, context);
    }

    // your original method
    public Void handleRequest(SNSEvent input, Context context) {
        // whatever
        return null;
    }
}

Note how your handleRequest is not really an override of the RequestHandler.handleRequest. The fact that you also have Handler1 doesn't change anything. What is really important is that you have an override in your non-generic class so the compiler has to generate a non-generic method (i.e. a method with not erased types) in your final class. Now you have two methods and AWS can understand that the one that takes SNSEvent is the most specific one so it is represents your real bound.

Now assume you do add your generic intermediate class Handler2:

public abstract class Handler2<E> implements RequestHandler<E, Void> {
    protected abstract void act(E input);

    @Override
    public Void handleRequest(E input, Context context) {
        act(input);
        return null;
    }
}

At this point the return type is fixed but the argument is still an unbound generic. So compiler has to produce something like this:

public abstract class Handler2 implements RequestHandler {
    protected abstract void act(Object input);

    // bridge-method
    @Override
    public Object handleRequest(Object input, Context context) {
        // In Java or Scala you can't distinguish between methods basing
        // only on return type but JVM can easily do it. This is again
        // call of the other ("your") handleRequest method
        return handleRequest(input, context);
    }

    public Void handleRequest(Object input, Context context) {
        act(input);
        return null;
    }
}

So now when we come to

public class PojoTest2 extends Handler2<SNSEvent> {
    @Override
    protected void act(SNSEvent input) {
        // whatever
    }
}

you have overridden act but not handleRequest. Thus the compiler doesn't have to generate a specific handleRequest method and it doesn't. It only generates a specific act. So the generated code looks like this:

public class PojoTest2 extends Handler2 {
    // Bridge-method
    @Override
    protected void act(Object input) {
        act((SNSEvent)input); // call the "real" method
    }

    protected void act(SNSEvent input) {
        // whatever
    }
}

Or if you flatten the tree and show all (relevant) methods in PojoTest2, it looks like this:

public class PojoTest2 extends Handler2 {

    // bridge-method
    @Override
    public Object handleRequest(Object input, Context context) {
        // In Java or Scala you can't distinguish between methods basing
        // only on return type but JVM can easily do it. This is again
        // call of the other ("your") handleRequest method
        return handleRequest(input, context);
    }

    public Void handleRequest(Object input, Context context) {
        act(input);
        return null;
    }

    // Bridge-method
    @Override
    protected void act(Object input) {
        act((SNSEvent)input); // call the "real" method
    }

    protected void act(SNSEvent input) {
        // whatever
    }
}

Both of the handleRequest methods accept just Object as a parameter and this is what AWS has to assume. Since you didn't override the handleRequest method in PojoTest2 (and not having to do so is the whole point of your inheritance hierarchy), the compiler didn't produce a more specific method for it.

Unfortunately I don't see any good workaround for this problem. If you want AWS to recognize the bound of the I generic parameter, you have to override handleRequest at the place in hierarchy where this bound becomes really known.

You may try do something like this:

// Your _non-generic_ sub-class has to have the following implementation of handleRequest:
// def handleRequestImpl(input: EventType, context: Context): Unit = handleRequestImpl(input, context)
trait UnitHandler[Event] extends RequestHandler[Event, Unit]{
     def act(input: Event): Unit

     protected def handleRequestImpl(input: Event, context: Context): Unit = act(input)
}

The benefit of this approach is that you can still put some additional wrapping logic (such as logging) into your handleRequestImpl. But still this will work only by convention. I see no way to force developers to use this code in the correct way.

If the whole point of your Handler2 is just bind the output type O to Unit without adding any wrapping logic, you can just do this without renaming the method to act:

trait UnitHandler[Event] extends RequestHandler[Event, Unit]{
     override def handleRequest(input: Event, context: Context): Unit
}

In such way your sub-classes still will have to implement handleRequest with specific types bound to Event and compiler will have to produce specific methods there so the issue will not happen.

Answer 2

As @SergGr said, there are no real generics in the JVM. All types are replaced with their bounds or objects.

This answer has a different take on how to achieve the creation of custom abstract handlers which doesn't involve using the AWS RequestHandler.

The way I have solved this is by using context bounds and ClassTag like this:

abstract class LambdaHandler[TEvent: ClassTag, TResponse<: Any] {

  def lambdaHandler(inputStream: InputStream, outputStream: OutputStream, context: Context): Unit = {
    val json = Source.fromInputStream(inputStream).mkString
    log.debug(json)
    val event = decodeEvent(json)
    val response = handleRequest(event, context)
    // do things with the response ...
    outputStream.close()
  }

  def decodeEvent(json: String): TEvent = jsonDecode[TEvent](json)
}

where jsonDecode is a function that turns the String event to the expected TEvent. In the following example I use json4s but you can use any de/serialization method you want:

def jsonDecode[TEvent: ClassTag](json: String): TEvent = {
  val mapper = Mapper.default
  jsonDecode(mapper)
}

In the end, you will be able to write functions like this

// AwsProxyRequest and AwsProxyResponse are classes from the com.amazonaws.serverless aws-serverless-java-container-core package
class Function extends LambdaHandler[AwsProxyRequest, AwsProxyResponse] {
  def handleRequest(request: AwsProxyRequest, context: Context): AwsProxyResponse = {
    // handle request and retun an AwsProxyResponse
  }
}

Or custom SNS handlers where TEvent is the custom type of the SNS message:

// SNSEvent is a class from the com.amazonaws aws-lambda-java-events package
abstract class SnsHandler[TEvent: ClassTag] extends LambdaHandler[TEvent, Unit]{
  override def decodeEvent(json: String): TEvent = {
    val event: SNSEvent = jsonDecode[SNSEvent](json)
    val message: String = event.getRecords.get(0).getSNS.getMessage
    jsonDecode[TEvent](message)
  }
}

If you use this method, straight out of the box, you will quickly realize that there are a large number of edge cases deserializing the JSON payloads because there are inconsistencies in the types that you get from AWS events. Therefore, you will have to fine tune the jsonDecode method to suit your needs.

Alternatively, use an existing library that takes care of these steps for you. There is one library that I know of for Scala (but have not used) called aws-lambda-scala or you can take a look at the full implementation of my LambdaHandler in GitHub