How do I convert from ByteBuffer to Avro bytes?

Question

I have an avro schema that includes the following as one of the fields

{
  "name" : "currency",
  "type" : ["null","bytes"],
  "logicalType": "decimal",
  "precision": 9,
  "scale": 4
},

I ran the avro-tools jar to create the java file to represent the schema. That produced the property that looks like: public java.nio.ByteBuffer currency;

Elsewhere in my code, I will be working with currency values in BigDecimal types.

How do I convert a BigDecimal value to the expected ByteBuffer as I create an instance of this class? Can I just use the ByteBuffer.toByteArray() or do I need to do anything special to ensure it is compatible with avro (and other tools such as Impala that may be reading the data)?

Answer 1

Let's start with a disclaimer. While the "Logical Types" section appeared in the specification circa 2014, it is not yet supported by any Avro Java release.

You can decide to declare a schema which complies with the specification and push the right bytes into the field, but Avro Java won't help you (it is exactly like if you had omitted the logical type related fields).

How do I convert a BigDecimal value to the expected ByteBuffer

The documentation states:

A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two's-complement representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified using an attribute.

Which can be translated in Java as (copy pasted from Avro 1.8.0-rc2):

public ByteBuffer toBytes(BigDecimal value, Schema schema, LogicalType type)
{
    int scale = ((LogicalTypes.Decimal) type).getScale();
    if (scale != value.scale()) {
        throw new AvroTypeException("Cannot encode decimal with scale " +
          value.scale() + " as scale " + scale);
    }

    return ByteBuffer.wrap(value.unscaledValue().toByteArray());
}

You can read BigDecimal & BigInteger's Javadoc to check that value.unscaledValue().toByteArray() complies with the specification.

In a similar way, you can deserialize the field using the following code: return new BigDecimal(new BigInteger(bytes), scale);

Should you use logical types ?

As said in preamble, if you are using Avro 1.7 nothing will come for free. You have to write your own (de)serializer, code generation & reflect don't support this construct. The only reason to use it, is to comply with the specification and hope that future Avro releases will make your life easier.

Avro 1.8.0-rc2 contains some code to support logical types and introduce new logical types. It seems that (de)serializers are provided for all logical types (see Conversion and Conversions) and that conversions have been plugged into GenericData. It means that you will receive a BigDecimal instance when you ask the value of the field. ReflectData seems also able to produce the expected schema if you properly annotate the field (but AFAIK no dedicated annotation have been created for logical types).

However, it is unclear to me if avro-compiler / codegen has been updated to support logical types.