How to count the number of unique characters in a file?

Question

Given a file in UTF-8, containing characters in various languages, how can I obtain a count of the number of unique characters it contains, while excluding a select number of symbols (e.g.: "!", "@", "#", ".") from this count?

Answer 1

Here’s a bash solution. :)

bash$ perl -CSD -ne 'BEGIN { $s{$_}++ for split //, q(!@#.) }
                     $s{$_}++ || $c++ for split //;
                     END { print "$c\n" }' *.utf8

Answer 2

In Python:

import itertools, codecs

predicate = set('!@#.').__contains__
unique_char_count = len(set(itertools.ifilterfalse(
                      predicate, itertools.chain.from_iterable(codecs.open(filename, encoding="UTF-8")))))

When you iterate over a file, you get lines. chain joins them together, so iterating over it you get characters. ifilterfalse eliminates the characters that meet the condition, with the condition defined as membership in a set of the disallowed characters.

Without itertools:

import codecs
disallowed = set('!@#.')
unique_char_count = len(set(char for line in codecs.open(filename, encoding="UTF-8") for char in line 
                              if char not in disallowed))

Using set operations:

import codecs
unique = set()
any(unique.update(line) for line in codecs.open(filename, encoding="UTF-8"))
unique.difference_update('!@#.')
unique_char_count = len(unique)