Git cant diff or merge .cs file in utf-16 encoding

Question

A friend and I were working on the same .cs file at the same time and when there's a merge conflict git points out there's a conflict but the file isnt loaded with the usual "HEAD" ">>>" stuff because the .cs files were binary files. So we added numerous things (*.cs text and so on)- to our .gitattributes file to make git treat it as a text file which didnt work.

Thats when we realized that git could diff other .cs files and just not this one. The reason for that is because its in unicode encoding as it contains some chinese characters.

So how do we make git diff or merge files that are in utf-16 or utf-8 format?

The furstrating thing is that if i push, gitlab shows exactly whats different. So I dont get how git can diff on the server but not with bash.

Answer 1

I had a similar problem with *.rc files for a c++ project and found the best way to solve this is to use git's smudge and clean filters to store everything in the repository as utf-8, and then convert to utf-16 when writing to the working directory.

This way everything git does such as diffs, merges or whatever will work on the utf8 text without issue, but your working copy will have utf16, which will keep visual studio happy.

To configure this, make sure you're using a version of git that has iconv available (the most recent versions of msysgit do) and add the following to your ~/.gitconfig file:

[filter "utf16"]
    clean = iconv -f utf-16le -t utf-8
    smudge = iconv -f utf-8 -t utf-16le
    required

Then in your .gitattributes file add:

*.rc filter=utf16
resource.h filter=utf16

If you already have existing files in utf16 stored as binary, then you need to remove them from the repository and re-add them.

git rm --cached <names-of-utf16-files>
git commit -am "removed utf16 files"
git add <names-of-utf16-files>
git commit -am "added utf16 files as utf8"

And now everything should work.