Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Generic Java serialization/deserialization using Kryo

I am trying to serialize and deserialize objects of a custom class (say, SomeClass, having a default no-args constructor) to a byte[] array, using Kryo 2.19 and the default serializer (FieldSerializer).

Serialization seems to work OK, but I get various exceptions in deserialization, depending on the actual implementation of SomeClass.

The code looks something like this:

SomeClass object = getObject(); // Create and populate a new object of SomeClass

Kryo kryo = new Kryo();
FieldSerializer<?> serializer = new FieldSerializer<SomeClass>(kryo, SomeClass.class);
kryo.register(SomeClass.class, serializer);

ByteArrayOutputStream stream = new ByteArrayOutputStream();
Output output = new Output(stream);

kryo.writeObject(output, object);

output.close(); // Also calls output.flush()

byte[] buffer = stream.toByteArray(); // Serialization done, get bytes

// Deserialize the serialized object.
object = kryo.readObject(new Input(new ByteArrayInputStream(buffer)), SomeClass.class);

An example of the exceptions I am getting is:

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.MethodVisitor, but class was expected
    at com.esotericsoftware.reflectasm.ConstructorAccess.insertConstructor(ConstructorAccess.java:89)
    at com.esotericsoftware.reflectasm.ConstructorAccess.get(ConstructorAccess.java:70)
    at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1009)
    at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1059)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:228)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:217)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629)

It seems that parameterized types are problematic to deserialize. To test this assumption , here is a parameterized implementation of SomeClass and getObject():

class SomeClass<T extends Serializable>
{
    private final T[] elements;

    private final int first;
    private final int second;

    private SomeClass()
    {
        this.elements = null;
        this.first  = 0;
        this.second = 0;
    }

    private SomeClass(T[] elements, int first, int second)
    {
        this.elements = elements;
        this.first = first;
        this.second = second;
    }
}

SomeClass<?> getObject()
{
    String[] elements = new String[] {"This is a test", "one"};

    return new SomeClass<String>(elements, 1, 2);
}

This serializes fine, but deserialization throws the following exception (observe how the first letter of the string is not reported in the exception cause):

Exception in thread "main" com.esotericsoftware.kryo.KryoException: Unable to find class: his is a test
Serialization trace:
elements (net.cetas.parserserver.data.report.SourceDataReporter$SomeClass)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:132)
    at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:109)
    at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:613)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:724)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:338)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:293)
    at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:702)
    at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:521)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629)

If the above class is implemented without parameterization (i.e., the elements array declared as a String[]), deserialization works as expected.

Any ideas?

like image 738
PNS Avatar asked Aug 09 '12 09:08

PNS


People also ask

What is KRYO in Java?

Kryo is a fast and efficient object graph serialization framework for Java. The goals of the project are speed, efficiency, and an easy to use API. The project is useful any time objects need to be persisted, whether to a file, database, or over the network.

What is KRYO serialization in spark?

KryoSerializer") . This setting configures the serializer used for not only shuffling data between worker nodes but also when serializing RDDs to disk. The only reason Kryo is not the default is because of the custom registration requirement, but we recommend trying it in any network-intensive application.

How can we achieve serialization and deserialization in Java?

For serializing the object, we call the writeObject() method of ObjectOutputStream class, and for deserialization we call the readObject() method of ObjectInputStream class. We must have to implement the Serializable interface for serializing the object.

Can we use KRYO serializer in Pyspark?

Kryo serialization: Spark can also use the Kryo library (version 4) to serialize objects more quickly.


1 Answers

Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.MethodVisitor, but class was expected

Please check which version of ASM are you use in your project. MethodVisitor is an interface in ASM up to 3.3. After 4.0 version this is a class.

Links:

  • http://asm.ow2.org/asm33/javadoc/user/org/objectweb/asm/MethodVisitor.html
  • http://asm.ow2.org/asm40/javadoc/user/org/objectweb/asm/MethodVisitor.html
like image 178
Pawel Avatar answered Sep 21 '22 11:09

Pawel