Java Serialization - Automatically handling changed fields?

Question

UPDATE MAY 20: I should have mentioned that the objects in question do have "serialVersionUID" set (same value in both old & new), but serialization fails before readObject() is called with the following exception:

Exception in thread "main" java.io.InvalidClassException: TestData; incompatible types for field number

I'm also now including an example down below.

I'm working with a large application that sends serialized objects (implements Serializable, not Exernalizable) from client to server. Unfortunately, we now have a situation where an existing field has changed type altogether, which breaks the serialization.

The plan is to upgrade the server side first. From there, I have control over the new versions of the serialized objects, as well as the ObjectInputStream to read the (at first old) objects coming from the clients.

I had at first thought of implementing readObject() in the new version; however, after trying it out, I discovered that validation fails (due to incompatible types) prior to the method ever being called.

If I subclass ObjectInputStream, can I accomplish what I want?

Even better, are there any 3rd-party libraries that do any sort of serialization 'magic'? It would be really interesting if there were any tools/libraries that could convert a serialized object stream into something like an array of HashMaps...without needing to load the objects themselves. I'm not sure if it is possible to do that (convert a serialized object to a HashMap without loading the object definition itself), but if it is possible, I could imagine a tool that could convert a serialized object (or stream of objects) to a new version, using, say, a set of Properties for conversion hints/rules, etc...

Thanks for any suggestions.

Update May 20 - example source below - The field 'number' in TestData changes from an 'int' in the old version to a 'Long' in the new version. Note readObject() in new version of TestData is not called, because an exception is thrown before it ever gets there:

Exception in thread "main" java.io.InvalidClassException: TestData; incompatible types for field number

Below is the source. Save it to a folder, then create sub-folders 'old' and 'new'. Put the 'old' version of the TestData class in the "old" folder, and the new version in "new". Put the "WriteIt" and "ReadIt" classes in the main (parent of old/new) folder. 'cd' to the 'old' folder, and compile with: javac -g -classpath . TestData.java ...then do the same thing in the 'new' folder. 'cd' back to the parent folder, and compile WriteIt/ReadIt:

javac -g -classpath .;old WriteIt.java

javac -g -classpath .;new ReadIt.java

Then run with:

java -classpath .;old WriteIt  //Serialize old version of TestData to TestData_old.ser

java -classpath .;new ReadIt  //Attempt to read back the old object using reference to new version

[OLD version]TestData.java

import java.io.*;


public class TestData
    implements java.io.Serializable
{
    private static final long serialVersionUID = 2L;

    public int number;
}

[NEW version] TestData.java

import java.io.*;


public class TestData
    implements java.io.Serializable
{
    private static final long serialVersionUID = 2L;

    public Long number; //Changed from int to Long


    private void readObject(final ObjectInputStream ois)
        throws IOException, ClassNotFoundException
    {
        System.out.println("[TestData NEW] readObject() called...");
        /* This is where I would, in theory, figure out how to handle
         * both the old and new type. But a serialization exception is
         * thrown before this method is ever called.
         */
    }
}

WriteIt.java - Serialize old version of TestData to 'TestData_old.ser'

import java.io.*;

public class WriteIt
{
    public static void main(String args[])
        throws Exception
    {
        TestData d = new TestData();

        d.number = 2013;

        ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("TestData_old.ser"));

        oos.writeObject(d);

        oos.close();

        System.out.println("Done!");
    }
}

ReadIt.java - Attempt to de-serialized old object into new version of TestData. readObject() in new version is not called, due to exception beforehand.

import java.io.*;

public class ReadIt
{
    public static void main(String args[])
        throws Exception
    {
        ObjectInputStream ois = new ObjectInputStream(new FileInputStream("TestData_old.ser"));

        TestData d = (TestData)ois.readObject();

        ois.close();

        System.out.println("Number = " + d.number);
    }
}

Answer 1

Your immediate problem seems to be that you have not defined fixed the serialVersionUID strings for your classes. If you don't do that, the object serialization and deserialization code generates the UIDs based on the representational structure of the types being sent and received. If you change the type at one end and not the other, the UIDs don't match. If you fix the UIDs, then the reader code will get past the UID check, and your readObject method will have a chance to "work around" the differences in serialized data.

Apart from that, my suggestion would be:

• Try to change all of the clients and servers at the same time so that you don't need to use readObject / writeObject hacks

• If that's not practical, try to move away from using Java serialized objects in anything where there is a possibility of version matches as your system evolves. Use XML, JSON, or something else that is less sensitive to "protocol" or "schema" changes.

• If that's not practical, version your client/server protocol.

I doubt that you will get any traction by subclassing the ObjectStream classes. (Take a look at the code ... in the OpenJDK codebase.)

Answer 2

You can continue to use serialization even when changes to classes will make the old serialized data incompatible with the new class, if you implement Externalizable and write an extra field indicating the version before writing the class data. This allows readExternal to handle old versions in the way that you specify. I know of no way to automatically do this, but using this manual method may work for you.

Here are the modified classes that will compile and will run without throwing an exception.

old/TestData.java

import java.io.*;

public class TestData
implements Externalizable
{
    private static final long serialVersionUID = 1L;
    private static final long version = 1L;

    public int number;

    public void readExternal(ObjectInput in)
    throws IOException, ClassNotFoundException {
        long version = (long) in.readLong();
        if (version == 1) {
            number = in.readInt();
        }
    }
    public void writeExternal(ObjectOutput out)
    throws IOException {
        out.writeLong(version); // always write current version as first field
        out.writeInt(number);
    }
}

new/TestData.java

import java.io.*;

public class TestData
implements Externalizable
{
    private static final long serialVersionUID = 1L;
    private static final long version = 2L; // Changed

    public Long number; //Changed from int to Long

    public void readExternal(ObjectInput in)
    throws IOException, ClassNotFoundException {
        long version = (long) in.readLong();
        if (version == 1) {
            number = new Long(in.readInt());
        }
        if (version == 2) {
            number = (Long) in.readObject();
        }
    }

    public void writeExternal(ObjectOutput out)
    throws IOException {
        out.writeLong(version); // always write current version as first field
        out.writeObject(number);
    }
}

These classes can be run with the following statements

$ javac -g -classpath .:old ReadIt.java
$ javac -g -classpath .:old WriteIt.java
$ java -classpath .:old WriteIt
Done!
$ java -classpath .:old ReadIt
Number = 2013
$ javac -g -classpath .:new ReadIt.java
$ java -classpath .:new ReadIt
Number = 2013

Changing a class to implement Exernalizable instead of Serializable will result in the following exception.

Exception in thread "main" java.io.InvalidClassException: TestData; Serializable incompatible with Externalizable
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:634)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
    at ReadIt.main(ReadIt.java:10)

In your case, you'd have to change the old version to implement Externalizable first, restart the server and clients at the same time, then you could upgrade the server before the clients with incompatible changes.

Answer 3

after trying it out, I discovered that validation fails (due to incompatible types) prior to the method ever being called

That's because you haven't defined a serialVersionUID. That's easy to fix. Just run the serialver tool on the old version of the classes, then paste the output into the new source code. You may then be able to write your readObject() method in a compatible way.

In addition, you could look into writing readResolve() and writeReplace() methods, or defining serialPersistentFields in a compatible way, if that's possible. See the Object Serialization Specification.

See also the valuable suggestions by @StephenC.

Post your edit I suggest you change the name of the variable along with its type. That counts as a deletion plus an insertion, which is serialization-compatible, and will work find as long as you don't want the old int number read into the new Long number (why Long instead of long?). If you do need that, I suggest leaving int number there and adding a new Long/long field with a different name, and modifying readObject() to set the new field to 'number' if it isn't already set by defaultReadObject().