Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kryo serialization dependent on Java version?

When deserializing a serialized object (from a file) using Kryo, I get the following exception:

java.lang.ExceptionInInitializerError
    (...)
Caused by: com.esotericsoftware.kryo.KryoException: (...)
Serialization trace: (...)
    at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:786)
    at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:143)
    at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:21)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:682)
    (...)
Caused by: java.lang.IndexOutOfBoundsException: Index: 1582, Size: 2
    at java.util.ArrayList.rangeCheck(Unknown Source)
    at java.util.ArrayList.get(Unknown Source)
    at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
    at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:830)
    at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)
    at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
    ... 27 more

My hypothesis is that the serialized format is not properly understood when deserializing (ie. it changed). The Kryo version for serializing and deserializing was the same. The java version could have been different at the time of serialization: could this be an explanation?

If not, any other hints on what my generate such exceptions are more than welcome!

Many thanks, Thomas

UPDATE: as suggested, hereby the class that is being deserialized from the file

The main class deserialized is HashMap<Integer, PreflopEhsVO> where the custom class definitions (child and parent) are:

public class PreflopEhsVOExtended extends PreflopEhsVO{
    private int numbValues = 0;

    public synchronized void addValue(PreflopEhsVO values){
        if (numbValues == 0) this.valuesPerNumbOpp = values.valuesPerNumbOpp;
        else{
            //Weighted avg
            for (int i=0; i<this.valuesPerNumbOpp.length; ++i) this.valuesPerNumbOpp[i] = (this.valuesPerNumbOpp[i] * numbValues + values.valuesPerNumbOpp[i]) / (float) (numbValues + 1);
            ++numbValues;
        }
    }

    public PreflopEhsVOExtended(PreflopEhsVO values) {
        this.valuesPerNumbOpp = values.valuesPerNumbOpp;
        this.numbValues = 1;
    }
}

public class PreflopEhsVO {
    public float[] valuesPerNumbOpp = new float[9];

    public PreflopEhsVO(){
    }

    public PreflopEhsVO(float[] valuesPerNumbOpp) {
        this.valuesPerNumbOpp = valuesPerNumbOpp;
    }
}
like image 698
Tom Avatar asked Nov 21 '15 20:11

Tom


People also ask

What is Kryo in Java?

Kryo is a Java serialization framework with a focus on speed, efficiency, and a user-friendly API. In this article, we’ll explore the key features of the Kryo framework and implement examples to showcase its capabilities. 2. Maven Dependency The first thing we need to do is to add the kryo dependency to our pom.xml:

Why should I switch to Kryo serialization?

As you can see, switching to Kryo resulted in much smaller byte arrays, so therefore in the same amount of memory, one keep much more map entries. So if you get the chance, try switching to Kryo, or one of the many other serialization libraries, to see how much your application can benefit from it.

How do I use the @defaultserializer annotation with Kryo?

We can also use the @DefaultSerializer annotation to let Kryo know we want to use the PersonSerializer each time it needs to handle a Person object. This helps avoid the call to the register () method:

What version of Kryo should I upgrade to?

In Kryo 5.1.0, a direct dependency on the unversioned JAR accidentally made it into the POM for the versioned artifact (see #825 ). If you are using the versioned artifact, upgrading to 5.1.1 is recommended.


1 Answers

Kryo in general is not stable across different JVMs unless you are very careful about how you register types.

When Kryo encounters a type it hasn't seen before it increments a counter and registers the type to that value. It uses these values in the object graph as an alias for the type (an optimization for performance). If the execution order is slightly different on the second JVM (whether or not the version is the same), then Kryo will end up with different registries of aliased types. This prevents the serialized blobs from being deserialized.

The solution to this is to turn on "strict" mode in kryo and manually register the classes with explicit ids.

  kryo.register(SomeClass.class, 1);
  kryo.register(AnotherClass.class, 2);

This will ensure that the classes get registered to the same stable ids on both JVMs. If you do this, you should be able to get stable cross JVM serialization.

Another possible problem could be a change in the fields of the class pre and post serialization. The only practical way to handle changing fields is with the TaggedFieldSerializer

like image 110
Russell Cohen Avatar answered Sep 18 '22 17:09

Russell Cohen