Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why is fread inserting carriage returns (\r) into data.table?

I am reading data from a .csv file using data.table::fread on a Windows 10 computer. The data reads in properly through read.csv; however, when I use fread to read in the data, all of the final columns in each row of the resulting data.table ends in a \r, presumably indicating a carriage return. This causes numeric fields to be given a character datatype. (Instead of a numeric literal 4.53, a row-ending cell will contain a character literal 4.53\r.)

Why is this bug occurring? Is there a way to directly resolve this through the function call of fread?

Update

I get the following when the verbose = TRUE parameter is used

Input contains no \n. Taking this to be a filename to open
File opened, filesize is 0.000001 GB.
Memory mapping ... ok
Detected eol as \n only (no \r afterwards), the UNIX and Mac standard.
Positioned on line 1 after skip or autostart
This line is the autostart and not blank so searching up for the last non-blank ... line 1
Detecting sep ... ','
Detected 7 columns. Longest stretch was from line 1 to line 13
Starting data input on line 1 (either column names or first row of data). First 10 characters: subjectNum
All the fields on line 1 are character fields. Treating as the column names.
Count of eol: 13 (including 1 at the end)
Count of sep: 72
nrow = MIN( nsep [72] / ncol [7] -1, neol [13] - nblank [1] ) = 12
Type codes (   first 5 rows): 1131414
Type codes: 1131414 (after applying colClasses and integer64)
Type codes: 1131414 (after applying drop or select (if supplied)
Allocating 7 column slots (7 - 0 dropped)
Read 12 rows. Exactly what was estimated and allocated up front
   0.000s (  0%) Memory map (rerun may be quicker)
   0.001s ( 33%) sep and header detection
   0.000s (  0%) Count rows (wc -l)
   0.002s ( 67%) Column type detection (first, middle and last 5 rows)
   0.000s (  0%) Allocation of 12x7 result (xMB) in RAM
   0.000s (  0%) Reading data
   0.000s (  0%) Allocation for type bumps (if any), including gc time if triggered
   0.000s (  0%) Coercing data already read in type bumps (if any)
   0.000s (  0%) Changing na.strings to NA
   0.003s        Total
like image 667
Bob Avatar asked Jun 16 '16 22:06

Bob


1 Answers

If you have a file that looks like x="a\n1\r\n2\r\n", then fread(x) will give the result described:

     a
1: 1\r
2: 2\r

This occurs because the end-of-line indicators are inconsistent across lines.

I have heard of this happening to others, but I'm not sure where it comes from or whether there is a better way to address it than "fixing" the file, probably with a command-line tool.

like image 105
Frank Avatar answered Sep 18 '22 09:09

Frank