Split and count emojis and words in a given string in Python

Question

For a given string, I'm trying to count the number of appearances of each word and emoji. I did it already here for emojis that consists only from 1 emoji. The problem is that a lot of the current emojis are composed from a few emojis.

Like the emoji 👨‍👩‍👦‍👦 consists of four emojis - 👨‍ 👩‍ 👦‍ 👦, and emojis with human skin color, for example 🙅🏽 is 🙅 🏽 etc.

The problem boils down to how to split the string in the right order, and then counting them is easy.

There are some good questions that addressed the same thing, like link1 and link2 , but none of them applies to the general solution (or the solution is outdated or I just can't figure it out).

For example, if the string would be hello 👩🏾‍🎓 emoji hello 👨‍👩‍👦‍👦, then I'll have {'hello':2, 'emoji':1, '👨‍👩‍👦‍👦':1, '👩🏾‍🎓':1} My strings are from Whatsapp, and all were encoded in utf8.

I had many bad attempts. Help would be appreciated.

Answer 1

Huge thanks Mark Tolonen. Now in order to count words and emojis and words in a given string, I'll use emoji.UNICOME_EMOJI in order to decide what is an emoji and what is not (from the emoji package), and then remove from the string the emojis.

Currently not an ideal answer, but it works and I'll edit if it will be changed.

import emoji
import regex
def split_count(text):
    total_emoji = []
    data = regex.findall(r'\X',text)
    flag = False
    for word in data:
        if any(char in emoji.UNICODE_EMOJI for char in word):  
            total_emoji += [word] # total_emoji is a list of all emojis

    # Remove from the given text the emojis
    for current in total_emoji:
        text = text.replace(current, '') 

    return Counter(text.split() + total_emoji)


text_string = "👹👹👹👹👹here hello world hello👨‍👩‍👦‍👦🙅🏽"    
final_counter = split_count(text_string)

Output:

final_counter
Counter({'hello': 2,
         'here': 1,
         'world': 1,
         '👨\u200d👩\u200d👦\u200d👦': 1,
         '👹': 5,
         '🙅🏽': 1})

Answer 2

Use the 3rd party regex module, which supports recognizing grapheme clusters (sequences of Unicode codepoints rendered as a single character):

>>> import regex
>>> s='👨‍👩‍👦‍👦🙅🏽'
>>> regex.findall(r'\X',s)
['👨\u200d👩\u200d👦\u200d👦', '🙅🏽']
>>> for c in regex.findall('\X',s):
...     print(c)
... 
👨‍👩‍👦‍👦
🙅🏽

To count them:

>>> data = regex.findall(r'\X',s)
>>> from collections import Counter
>>> Counter(data)
Counter({'👨\u200d👩\u200d👦\u200d👦': 1, '🙅🏽': 1})