Why does Mercurial think my SQL files are binary?

Question

I just scripted out my SQL Server stored procs, table definitions, etc using SQL Server Management Studio, and tried to add them to my Mercurial source control repository. They got added just fine, but now when I change and diff them, Mercurial calls them "binary files" and doesn't give me a proper unified diff.

I thought the encoding might be a problem, so I tried regenerating the scripts and specifying ANSI for the text file output, but I get the same behavior. I can view them just fine in notepad without any odd-looking characters showing up. Why does Mercurial think these files are binary?

Otherwise, if someone can recommend a good tool for scripting out a SQL Server database that might not cause this issue, that might work, too.

Answer 1

I've run into this problem because SQL Server Management Studio saves the files as Unicode. The first two bytes (most of the time) of a Unicode text file define the encoding. Most newer text editors (e.g. Notepad) handle this transparently.

The first two bytes are probably where your problem is. They may look like ÿþ. Or FF FE in hex.

On the "Save" button on the Save dialog is a pick list. Choose "Save with Encoding..." and select "US-ASCII-Codepage20127". I believe this setting is sticky and will remain for future saves.

Answer 2

According to the docs, it's considered binary iff there are null bytes in the file. SQL files shouldn't have null bytes, so I would check that first (try looking in a hex editor). I assume you do know you can force diff to treat it as text

Answer 3

Andrew is right; it's a NUL byte somewhere (my guess would be a Byte Order Mark at the start inserted by a rude editor tool). Don't worry about it though, unlike SVN or CVS Mercurial doesn't handle binary vs. text differently at all. It displays them different when you do 'hg log', but they're not handled at all differently.

Upcoming mercurial releases special case BOMs and don't let them trigger the "user probably doesn't want to see a diff of this on console" behavior.

Answer 4

I ran into this when editing a file of stored procedures from SQL Server on linux and using git. Git thought it was a binary file because the file from SQL Server was UTF-16, and therefore contained NULs. My fix for this was emacs, which lets you change the encoding to UTF-8.