Searching for all Unicode variation of hyphens in Python

Question

I have been trying to extract certain text from PDF converted into text files. The PDF came from various sources and I don't know how they were generated.

The pattern I was trying to extract was a simply two digits, follows by a hyphen, and then another two digits, e.g. 12-34. So I wrote a simple regex \d\d-\d\d and expected that to work.

However when I test it I found that it missed some hits. Later I noted that there are at least two hyphens represented as \u2212 and \xad. So I changed my regex to \d\d[-\u2212\xad]\d\d and it worked.

My question is, since I am going to extract so many PDF that I don't know what other variations of hyphen are out there, is there any regex expression covering all "hyphens", and hopefully looks better than the [-\u2212\xad] expression?

Answer 1

The solution you ask for in the question title implies a whitelisting approach and means that you need to find the chars that you think are similar to hyphens.

You may refer to the Punctuation, Dash Category, that Unicode cateogry lists all the Unicode hyphens possible.

You may use a PyPi regex module and use \p{Pd} pattern to match any Unicode hyphen.

Or, if you can only work with re, use

[\u002D\u058A\u05BE\u1400\u1806\u2010-\u2015\u2E17\u2E1A\u2E3A\u2E3B\u2E40\u301C\u3030\u30A0\uFE31\uFE32\uFE58\uFE63\uFF0D]

You may expand this list with other Unicode chars that contain minus in their Unicode names, see this list.

A blacklisting approach means you do not want to match specific chars between the two pairs of digits. If you want to match any non-whitespace, you may use \S. If you want to match any punctuation or symbols, use (?:[^\w\s]|_).