Reading data from Dukascopy tick binary file

Question

I have downloaded the Dukascopy tick data and I have decompressing it with easylzma library. The original compressed binary file is EURUSD/2010/00/08/12h_ticks.bi5 (EURUSD/2010/ian/8/12h) After decompressing we get the following format:

+-------------------------+--------+-------+
|           time          |  Bid   |   Ask |
+-------------------------+--------+-------+
000003CA 00022EC0 00022EB6 40CCCCCD 41180000
000004F5 00022EB6 00022EB1 4099999A 404CCCCD

(You can download original compressed file from: EURUSD/2010/00/08/12h_ticks.bi5. After decompressing it with lzma we get the file: 12h_ticks)

Reading the binary file:

int ii1;
int ii2;
int ii3;
float ff1;
float ff2;
ifstream in("12h_ticks",ofstream::binary);
in.read((char*)(&ii1), sizeof(int));
in.read((char*)(&ii2), sizeof(int));
in.read((char*)(&ii3), sizeof(int));
in.read((char*)(&ff1), sizeof(float));
in.read((char*)(&ff2), sizeof(float));
std::cout << " ii1=" << ii1 << std::endl;
std::cout << " ii2=" << ii2 << std::endl;
std::cout << " ii3=" << ii3 << std::endl;
std::cout << " ff1=" << ff1 << std::endl;
std::cout << " ff2=" << ff2 << std::endl;
in.close();

I get the following result:

ii1=-905773056
ii2=-1070726656
ii3=-1238498816
ff1=-4.29492e+08
ff2=8.70066e-42

What is wrong? I can't read data from binary file. Please help me.

Answer 1

The data appears to be stored in big endian format in the file. You'll need to convert it to little endian when you load it.

#include <iostream>
#include <fstream>
#include <algorithm>

template<typename T>
void ByteSwap(T* p)
{
    for (int i = 0;  i < sizeof(T)/2;  ++i)
        std::swap( ((char *)p)[i], ((char *)p)[sizeof(T)-1-i] );
}

int main()
{
    int ii1;
    int ii2;
    int ii3;
    float ff1;
    float ff2;
    std::ifstream in("12h_ticks",std::ofstream::binary);
    in.read((char*)(&ii1), sizeof(int));
    in.read((char*)(&ii2), sizeof(int));
    in.read((char*)(&ii3), sizeof(int));
    in.read((char*)(&ff1), sizeof(float));
    in.read((char*)(&ff2), sizeof(float));

    ByteSwap(&ii1);
    ByteSwap(&ii2);
    ByteSwap(&ii3);
    ByteSwap(&ff1);
    ByteSwap(&ff2);

    std::cout << " ii1=" << ii1 << std::endl;
    std::cout << " ii2=" << ii2 << std::endl;
    std::cout << " ii3=" << ii3 << std::endl;
    std::cout << " ff1=" << ff1 << std::endl;
    std::cout << " ff2=" << ff2 << std::endl;
    in.close();
    return 0;
}

This gives the result:

ii1=970
ii2=143040
ii3=143030
ff1=6.4
ff2=9.5

I grabbed the ByteSwap function from here if you want to read more about that subject. How do I convert between big-endian and little-endian values in C++?