Reading .dat file in python

Question

I have a .dat file which I don't have any idea about how it was created and what delimiter was used and any details about it. I just have its corresponding mdf and csv file. Thats all. Is there any way in python to read this .dat file?

Few ways that I have tried:

file = "736_2_PerformanceCurve_(23_0C)_(13_5V).dat"
datContent = [i.strip().split() for i in open(file, encoding='latin1').readlines()]
datContent

which gives output

[['|CF,2,1,1;|CK,1,3,1,1;'],
 ['|NO,1,7,1,0,,0,;'],
 ['|NL,1,10,1252,0x407;'],
 ['|CT,1,41,0,6,Bench#,24,Korrosionstest', '15A046-01,0,;'],
 ['|CT,1,30,0,11,StartOfTest,8,06/30/17,0,;'],
 ['|CT,1,58,0,10,ResultPath,36,c:\\korrosionstest\\daten\\#170161-OR02,0,;'],
 ['|CT,1,59,0,11,GraphicPath,36,c:\\korrosionstest\\daten\\#170161-OR02,0,;'],
 ['|CT,1,31,0,15,GraphicBaseName,5,736_2,0,;'],
 ['|CT,1,26,0,10,PartNumber,5,736_2,0,;'],
 ['|CT,1,31,0,9,VA-Nr.', 'GS,11,170161-OR02,0,;'],
 ['|CT,1,62,0,9,VA-Nr.',
  'CC,42,TO_ENV_2017_G2_C1_Platform_CC-122164-03-08,0,;'],
 ['|CT,1,24,0,6,Tester,8,Behrendt,0,;'],
 ['|CT,1,32,0,15,Test', 'Department,6,GS/ETR,0,;'],
 ['|CG,1,5,1,1,1;'],
 ['|CD,1,16,1E-2,1,1,s,0,0,0;'],
 ['|NT,1,27,30,', '6,2017,14,25,15.8050001;'],
 ['|CC,1,3,1,1;'],
 ['|CP,1,16,1,2,4,16,0,0,1,0;'],
 ['|Cb,1,33,1,0,1,1,0,11718,0,11718,1,5E-3,0,;'],
 ['|CR,1,30,1,6.103888176768602E-3,0,1,1,A;'],
 ['|CN,1,28,0,0,0,16,ai_iB1_Strom_ECU,0,;'],
 ['|CG,1,5,1,1,1;'],
 ['|CD,1,16,1E-2,1,1,s,0,0,0;'],
 ['|NT,1,27,30,', '6,2017,14,25,15.8050001;'],
 ['|CC,1,3,1,1;'],
 ['|CP,1,16,2,2,4,16,0,0,1,0;'],
 ['|Cb,1,37,1,0,2,1,11718,11718,0,11718,1,5E-3,0,;'],
 ['|CR,1,30,1,3.662332906061161E-3,0,1,1,V;'],
 ['|CN,1,31,0,0,0,19,ai_iB1_Spannung_UBB,0,;'],

The corresponding csv file for the same

[![enter image description here][1]][1]

EDIT :

from asammdf import MDF
dat_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.dat"
mdf_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.mdf"
df = mdf.to_dataframe()
mdf = MDF(mdf_file)
df.head()

which gives me [![enter image description here][2]][2]

How do I read the same data from the dat file, is there any library or code for the same?

Answer 1

If I look at the file it looks for me like a specific format.

One data block starts with a | and ends with a ;. In the data block the data are splitted with ,. Basically it's like a CSV but the newline is ;.

Now with the help of regex you can read this data like this:

import re

with open("resources/input.dat") as f:
    lines = f.readlines()
    text = "".join(lines)

regex = r"\|(.*?);"
matches = re.finditer(regex, text, re.MULTILINE | re.DOTALL)


data = []

for matchNum, match in enumerate(matches, start=1):
    for group in match.groups():
        data.append(group.split(","))

for d in data:
    print(d)

Input

|CF,2,1,1;|CK,1,3,1,1;
|NO,1,7,1,0,,0,;
|CT,1,41,0,6,Bench,24,Korrosionstest', '15A046-01,0,
otherline_data;

Output

['CF', '2', '1', '1']
['CK', '1', '3', '1', '1']
['NO', '1', '7', '1', '0', '', '0', '']
['CT', '1', '41', '0', '6', 'Bench', '24', "Korrosionstest'", " '15A046-01", '0', '\notherline_data']

As you can see even if the data block doesn't end at a new line, you still get the data until the defined end mark ;.

Edit

I downloaded your .dat file. As you can see after line 1133 there are strange characters that doesn't make sense at all. This characters or rather bytes are probably the information you need to process the data in the beginning. Basically it looks like some compressed data with the needed background information I informed you in the comment.

FAMOS has the knowledge to interpret that byte string and therefore can present you with the data as it is intented. How to interpret this? Ask the source where you get the data or find it in the FAMOS code.

I don't think somebody here can answer you this. And I don't know how. This is too specific and therefore it is better to go where you get the data.

A snippet from the .dat file: (In total 32404 lines and only 1133 with data)