How do I use Serde to (de)serialize arrays greater than 32 elements, such as [u8; 128]?

Question

I have a struct containing a byte array that I would like to serialize and deserialize to and from binary, but it only works for arrays up to 32 elements.

Here is my minimal example code

main.rs:

#[macro_use]
extern crate serde_derive;
extern crate serde;
extern crate bincode;

use bincode::{serialize, deserialize, Infinite};

const BYTECOUNT: usize = 32; // 33 and more does not work, I need 128
type DataArr = [u8; BYTECOUNT];

#[derive(Serialize, Deserialize, Debug)]
struct Entry {
    number: i64,
    data: DataArr
}

fn main() {
    let mut my_entry = Entry { number: 12345, data: [0; BYTECOUNT] };
    my_entry.data[4] = 42;

    // Convert the Entry to binary.
    let serialized: Vec<u8> = serialize(&my_entry, Infinite).unwrap();
    println!("serialized = {:?}", serialized);

    // Convert the binary representation back to an Entry.
    let deserialized: Entry = deserialize(&serialized).unwrap();
    println!("deserialized = {:?}", deserialized);
}

Cargo.toml:

[package]
name = "array_serialization_test"
version = "0.1.0"

[dependencies]
serde = "*"
serde_derive = "*"
bincode = "*"

output:

serialized = [57, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
deserialized = Entry { number: 12345, data: [0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 

How can I make it work for 128 elements in the array? Can I somehow manually extend array_impls! in my user code? Is there an alternative approach?

I think this question is different from How do I map a C struct with padding over 32 bytes using serde and bincode? because I actually need the content of the array, since it is not just used for padding. Also I would like to know if I can extend array_impls! on my code.

Answer 1

Currently there is no way for Serde to provide Serialize and Deserialize impls that work for every array size. This is blocked on const generics which is being worked on and will hopefully land in nightly later in 2018.

For now you can define your own "big array" helper that can serialize and deserialize arrays of any specific sizes used in your crate. Fields for which you want to use the big array helper will need to be tagged with #[serde(with = "BigArray")] or else Serde will look for non-existent Serialize and Deserialize impls.

#[macro_use]
extern crate serde_derive;

extern crate serde;
extern crate bincode;

mod big_array;
use big_array::BigArray;

const BYTECOUNT: usize = 128;
type DataArr = [u8; BYTECOUNT];

#[derive(Serialize, Deserialize)]
struct Entry {
    number: i64,
    #[serde(with = "BigArray")]
    data: DataArr
}

fn main() {
    let mut my_entry = Entry { number: 12345, data: [0; BYTECOUNT] };
    my_entry.data[4] = 42;

    // Convert the Entry to binary.
    let serialized: Vec<u8> = bincode::serialize(&my_entry).unwrap();
    println!("serialized = {:?}", serialized);

    // Convert the binary representation back to an Entry.
    let deserialized: Entry = bincode::deserialize(&serialized).unwrap();
    println!("deserialized = {} {:?}", deserialized.number, &deserialized.data[..]);
}

The big array helper can be defined in src/big_array.rs as follows. Maybe this would make a good crate by itself if you would like to own it!

use std::fmt;
use std::marker::PhantomData;
use serde::ser::{Serialize, Serializer, SerializeTuple};
use serde::de::{Deserialize, Deserializer, Visitor, SeqAccess, Error};

pub trait BigArray<'de>: Sized {
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
        where S: Serializer;
    fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
        where D: Deserializer<'de>;
}

macro_rules! big_array {
    ($($len:expr,)+) => {
        $(
            impl<'de, T> BigArray<'de> for [T; $len]
                where T: Default + Copy + Serialize + Deserialize<'de>
            {
                fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
                    where S: Serializer
                {
                    let mut seq = serializer.serialize_tuple(self.len())?;
                    for elem in &self[..] {
                        seq.serialize_element(elem)?;
                    }
                    seq.end()
                }

                fn deserialize<D>(deserializer: D) -> Result<[T; $len], D::Error>
                    where D: Deserializer<'de>
                {
                    struct ArrayVisitor<T> {
                        element: PhantomData<T>,
                    }

                    impl<'de, T> Visitor<'de> for ArrayVisitor<T>
                        where T: Default + Copy + Deserialize<'de>
                    {
                        type Value = [T; $len];

                        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                            formatter.write_str(concat!("an array of length ", $len))
                        }

                        fn visit_seq<A>(self, mut seq: A) -> Result<[T; $len], A::Error>
                            where A: SeqAccess<'de>
                        {
                            let mut arr = [T::default(); $len];
                            for i in 0..$len {
                                arr[i] = seq.next_element()?
                                    .ok_or_else(|| Error::invalid_length(i, &self))?;
                            }
                            Ok(arr)
                        }
                    }

                    let visitor = ArrayVisitor { element: PhantomData };
                    deserializer.deserialize_tuple($len, visitor)
                }
            }
        )+
    }
}

big_array! {
    40, 48, 50, 56, 64, 72, 96, 100, 128, 160, 192, 200, 224, 256, 384, 512,
    768, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
}

Answer 2

I think this is just waiting on const generics support in Serde.