How to convert C# datetimes into Python datetimes with protobuf-net?

Question

I've serialized an array of C# DateTime.Now objects using protobuf-net and opened the file in Python with the proto file(struct_pb2.py) of its message structure.

// The struct definition in C#

    [ProtoContract]
    public struct struct_tick
    {
        [ProtoMember(1)]
        public uint[] arr_currentIndex;

        [ProtoMember(2)]
        public string[] arr_currentType;

        [ProtoMember(3)]
        public DateTime[] arr_currentTime;   // <- for DateTime.Now objects

        public struct_tick(byte size_index, byte size_type, byte size_time)
        {
            arr_currentIndex = new uint[size_index];
            arr_currentType = new string[size_type];
            arr_currentTime = new DateTime[size_time];
        }
    }

syntax = "proto3";

import "google/protobuf/timestamp.proto";

message struct_tick 
{
  repeated uint32 arr_currentIndex  = 1;
  repeated string arr_currentType   = 2;
  repeated google.protobuf.Timestamp arr_currentTime = 3;
}

It opens well in Python, but how can I convert C# datetime objects into, for example, Python datetime objects correctly? What makes me confused is that what's shown in the file's datetime array.

>>> struct = struct_pb2.struct_tick()
>>> with open(filename, "rb") as f:
>>>     struct.ParseFromString(f.read())

>>> lst_time = [ time for time in struct.arr_currentTime]
>>> lst_time[:5]
[seconds: 1573571465
nanos: 335624000
, seconds: 1573571465
nanos: 335624000
, seconds: 1573571465
nanos: 335624000
, seconds: 1573571465
nanos: 335624000
, seconds: 1573571465
nanos: 335624000
]

>>> lst_time[-5:]
[seconds: 1573571465
nanos: 337636000
, seconds: 1573571465
nanos: 337636000
, seconds: 1573571465
nanos: 337636000
, seconds: 1573571465
nanos: 337636000
, seconds: 1573571465
nanos: 337636000
]

When I read the same file in C#, the very first datetime object represents 21:59:39.7450630, and the very end object represents 23:38:50.3848014, which means there is a gap of almost 2 hours. But what's shown in Python is totally different.

Is it because I used DateTime when serializing and used google.protobuf.Timestamp when deserializing? How can I convert them correctly?

ADDED

This is the callback function that receives data and writes it into an array in its struct:

static void Handler_Real(string szCode)
{
    string code          = REAL_TR.GetData("Out", "sym");
    string currentType   = REAL_TR.GetData("Out", "cg");
    DateTime currentTime = DateTime.Now;

    if (dic_codeToTRstructCounter[code] == arraySize)
    {
        dic_codeToTRstructCounter[code] = 0;

        using (var fileStream = new FileStream(dic_codeToTRfilename[code], FileMode.Append))
        {
            Serializer.Serialize(fileStream, dic_codeToTRstruct[code]);
        }
    }

    dic_codeToTRstruct[code].arr_currentIndex[dic_codeToTRstructCounter[code]]  = dic_codeToTRCounter[code];
    dic_codeToTRstruct[code].arr_currentType[dic_codeToTRstructCounter[code]]   = currentType;
    dic_codeToTRstruct[code].arr_currentTime[dic_codeToTRstructCounter[code]]   = currentTime;

    dic_codeToTRstructCounter[code] += 1;
    dic_codeToTRCounter[code] += 1;
}

This is how I read the file in C#, which works fine.

using (var fileStream = new FileStream("file.bin", FileMode.Open))
{
    struct_tick str = Serializer.Deserialize<struct_tick>(fileStream);

    for (int startNum = 0; startNum < str.arr_currentIndex.Length; startNum++)
    {
        string str_print = string.Format("{0}, {1}, {3}", str.arr_currentIndex[startNum],
                                                          str.arr_currentType[startNum],
                                                          str.arr_currentTime[startNum].ToString("HH:mm:ss.fffffff"));
        listBox1.Items.Add(str_print);
    }
}

This is how I read the same file in Python

def convert_TRtoArray(filename):
    struct = struct_pb2.struct_TR()
    with open(filename, "rb") as f:
        struct.ParseFromString(f.read())

    lst_currentIndex  = [  indexNum for indexNum in struct.arr_currentIndex  ]
    lst_currentType   = [  type for type  in struct.arr_currentType ]
    lst_currentTime   = [  time for time in struct.arr_currentTime ]

And when I check lst_currentTime, although the C# program had received data for 2 hours, the values are like the above.

Answer 1

The easiest way to do this is to use:

[ProtoMember(..., DataFormat = DataFormat.WellKnown)]

on your DateTime / TimeSpan usages; this makes protobuf-net use the "well known" timestamp and duration layouts instead of the default format. The reason it doesn't do this automatically is that they (timestamp/duration) didn't exist back when protobuf-net first had to decide on some layout for them.

Note, however, that adding this is a breaking change: the layouts are not compatible.

If you need to use the legacy layout, they are described/defined in bcl.proto.

Alternatively, in protobuf-net v3, there are explicit types for timestamp / duration, if you prefer.

---

Given the discussion (comments), frankly it looks like you are either not populating the array correctly, or are leaking that array between contexts. Testing locally, it looks fine - the following demo shows it working (with your struct_tick), including a breakdown of the binary data that shows seconds/nanos that look correct.

using ProtoBuf;
using System;
using System.IO;

class Program
{

    static void Main()
    {
        var payload = new struct_tick(2, 2, 2);
        payload.arr_currentIndex[0] = 12;
        payload.arr_currentIndex[1] = 14;
        payload.arr_currentType[0] = "abc";
        payload.arr_currentType[1] = "def";
        payload.arr_currentTime[0] = new DateTime(2019, 11, 12, 21, 59, 39, 745, DateTimeKind.Utc);
        payload.arr_currentTime[1] = new DateTime(2019, 11, 12, 23, 38, 50, 385, DateTimeKind.Utc);

        var ms = new MemoryStream();
        Serializer.Serialize(ms, payload);
        var hex = BitConverter.ToString(ms.GetBuffer(), 0, (int)ms.Length);
        Console.WriteLine(hex);
        ms.Position = 0;
        var clone = Serializer.Deserialize<struct_tick>(ms);
        foreach(var time in clone.arr_currentTime)
            Console.WriteLine(time); // writes 12/11/2019 21:59:39 and 12/11/2019 23:38:50

        // hex is 08-0C-08-0E-12-03-61-62-63-12-03-64-65-66-
        //        1A-0C-08-CB-D6-AC-EE-05-10-C0-98-9F-E3-02-
        //        1A-0C-08-8A-85-AD-EE-05-10-C0-C4-CA-B7-01
        // 08-0C = field 1, 12
        // 08-0E = field 1, 14
        // 12-03-61-62-63 = field 2, "abc"
        // 12-03-64-65-66 = field 2, "def"
        // 1A-OC = field 3, length 12
        //    08-CB-D6-AC-EE-05 = field 1, 1573595979 = seconds
        //    10-C0-98-9F-E3-02 = field 2, 745000000 = nanos
        // 1A-OC = field 3, length 12
        //    08-8A-85-AD-EE-05 = field 1, 1573601930 = seconds
        //    10-C0-C4-CA-B7-01 = field 2, 385000000 = nanos
    }
}

[ProtoContract]
public struct struct_tick
{
    [ProtoMember(1)]
    public uint[] arr_currentIndex;

    [ProtoMember(2)]
    public string[] arr_currentType;

    [ProtoMember(3, DataFormat = DataFormat.WellKnown)]
    public DateTime[] arr_currentTime;   // <- for DateTime.Now objects

    public struct_tick(byte size_index, byte size_type, byte size_time)
    {
        arr_currentIndex = new uint[size_index];
        arr_currentType = new string[size_type];
        arr_currentTime = new DateTime[size_time];
    }
}

Answer 2

It is not good practice to use DateTime in binary serializable streams like Protobuf. It is better to convert DateTime values to something else, like epoch time or even string, and then use serialization on the epoch time or string values. Otherwise, you'll not be able to restore the content of this field correctly.