Emoji in R [UTF-8 encoding]

Question

I'm trying to make an emoji analysis on R. I have stored some tweets where there are emojis.

Here is one of the tweet that I want to analyze :

> tweetn2
[1] "Programme du week-end: \xed\xa0\xbd\xed\xb2\x83\xed\xa0\xbc \xed\xbe\xb6\xed\xa0\xbc 
    \xed\xbd\xbb\xed\xa0\xbc\xed\xbd\xbb\xed\xa0\xbc \xed\xbd\xbb\xed\xa0\xbc\xed\xbd\xbb"

To be sure that I have "UTF-8":

> Encoding(tweetn2)
[1] "UTF-8

" Now when I'm trying to recognize some characters, it's not working fine

> grepl("\\xed",tweetn2)
[1] FALSE

or

> grepl("xed",tweetn2)
[1] FALSE

But it seems that emojis "\xed\xa0\xbd" are not "UTF-8" encoding because I get an error message when I write :

> str(tweetn2)
Error in str.default(tweetn2) : invalid multibyte string, element 1

I find a kind of solution by using iconv( ) function and "ASCII" encoding there :
http://www.r-bloggers.com/emoticons-decoder-for-social-media-sentiment-analysis-in-r/

But I want to keep using "UTF-8" for my analysis because it works well with french special letters (à, é, è, ê, ë, û, etc.. )

So do you have an idea how I can get above it?

Thanks

Answer 1

The string is invalid UTF-8, as indicated. What you have there is UTF-16 encoded with UTF-8. So \xED\xA0\xBD is the high surrogate U+D83D, -- and \xED\xB2\x83 is the low surrogate U+DC83

If you apply the magical High,Low -> Codepoint formula, you'll end up with the actual codepoint:

(0xD83D - 0xD800) * 0x400 + 0xDC83 - 0xDC00 + 0x10000 = 0x1F483

You'll see this is the dancer emoji. Unfortunately I don't have a suggestion for you, as I'm not that familiar with R. But I can say you'd certainly want to get yourself in a position where this data is double encoded! Hope that helps bump you along the correct direction.