Python unicode.splitlines() triggers at non-EOL character

Question

Triyng to make this in Python 2.7:

>>> s = u"some\u2028text"
>>> s
u'some\u2028text'
>>> l = s.splitlines(True)
>>> l
[u'some\u2028', u'text']

\u2028 is Left-To-Right Embedding character, not \r or \n, so that line should not be splitted. Is there a bug or just my misunderstanding?

Answer 1

\u2028 is LINE SEPARATOR, left-to-right embedding is \u202A:

>>> import unicodedata

>>> unicodedata.name(u'\u2028')
'LINE SEPARATOR'

>>> unicodedata.name(u'\u202A')
'LEFT-TO-RIGHT EMBEDDING'

---

The list of codepoints considered linebreaks is easy (not that easy to find though) to see in python source (python 2.7, comments by me):

/* Returns 1 for Unicode characters having the line break
 * property 'BK', 'CR', 'LF' or 'NL' or having bidirectional
 * type 'B', 0 otherwise.
 */
int _PyUnicode_IsLinebreak(register const Py_UNICODE ch)
{
    switch (ch) {
    // Basic Latin
    case 0x000A:    // LINE FEED
    case 0x000B:    // VERTICAL TABULATION
    case 0x000C:    // FORM FEED
    case 0x000D:    // CARRIAGE RETURN
    case 0x001C:    // FILE SEPARATOR
    case 0x001D:    // GROUP SEPARATOR
    case 0x001E:    // RECORD SEPARATOR

    // Latin-1 Supplement
    case 0x0085:    // NEXT LINE

    // General punctuation
    case 0x2028:    // LINE SEPARATOR
    case 0x2029:    // PARAGRAPH SEPARATOR
        return 1;
    }
    return 0;
}

Answer 2

U+2028 is LINE SEPARATOR. Both U+2028 and U+2029 (PARAGRAPH SEPARATOR) should be treated as newlines, so Python is doing the right thing.

Of course it is sometimes perfectly reasonable to want to split on a non-standard list of newline characters. But you can't do that with splitlines. You will have to use split—and, if you need the additional features of splitlines, you'll have to implement them yourself. For example:

return [line.rstrip(sep) for line in s.split(sep)]