How to find/fix files with MIXED line endings (0x0d 0x0d 0x0a)

Question

I know I can "probably" fix them by using "flip -u" (cygwin flip) which basically removes one of the 0xd's leaving the file with DOS style line endings (0x0d 0x0a) (of course, technically speaking this might be considered a bug!).

But the other side of it is that i'd like to do this selectively, ensuring that what I'm fixing really is a "non-binary" file and EXPLICITLY replacing the 0x0d 0x0d 0x0a sequence with 0x0d 0x0a... not running a buggy program that appears to do what I want (and possibly more).

Note that grep -P '\x0d\x0d\x0a' and grep -P '\x0d\x0d' do not find these lines.

Although people say that grep -P 'x0d\x0a' is properly finding line endings, I'd have to surmise that something else is going on since it can't match the other patterns in a file with mixed line endings (0x0d 0x0d 0x0a).

Answer 1

Here's an easy way to identify the files that contain mixed line endings:

cat -A $FILE | grep '\^M\^M\$'

The -A implies -v and -E which includes line endings and other hidden characters. For example, let's create a testfile. I'll use the actual text to represent fairly closely with the line endings you'll see:

$ od -x test1.txt 
0000000 6464 2061 0d20 0a0d 6464 6161 2020 0d0d
0000020 0a0a 6164 2020 0a0d
0000030

Now let's see what cat gives us:

$ cat -vE test1.txt
dda  ^M^M$
ddaa  ^M^M$
$
da  ^M$

cat is indeed showing us the CRs and LFs (though the LFs don't show up on the same line -- and justifiably so), so now we can find them:

find /path -yourPredicatesOfInterest -print | while read fn ; do
    cat -A $fn | grep '\^M\^M\$' > /dev/null 2>&1 && echo "$fn contains multiple CR CR LFs"
done