Optimizing Haskell text processing

Question

I'm writing some simple character-counting routines in Haskell, storing the stats in a new datatype:

data Stat = Stat {
    stChars    :: !Int,
    stVowels   :: !Int,
    stPairEL   :: !Int,
    stWords    :: !Int
}

I'm running this over hundreds or thousands of plain-text files, each about 50K--100K.

tabulateFile :: FilePath -> IO Stat
tabulateFile path = do
  putStrLn path
  contents <- L.readFile path
  return $! tabulateText ' ' contents defaultStat

Instead of using fold-left, I'm using primitive recursion so I can keep around the previous character.

tabulateText :: Char -> L.ByteString -> Stat -> Stat
tabulateText lastChr bs stat =
  case U.uncons bs of
    Nothing -> stat
    Just (chr, newBs) ->
      tabulateText lchr newBs (countChar lastChr lchr stat)
        where lchr = toLower chr

{-# INLINE countChar #-}
countChar :: Char -> Char -> Stat -> Stat
countChar !lastChr !chr !(Stat stChars stVowels stPairEL stWords) =
  Stat
    (stChars  + 1)
    (stVowels + (countIf $ isVowel chr))
    (stPairEL + (countIf (lastChr == 'e' && chr == 'l')))
    (stWords  + (countIf ((not $ isLetter lastChr) && isLetter chr)))

isVowel :: Char -> Bool
isVowel c = Set.member c vowels

vowels = Set.fromAscList ['a', 'e', 'i', 'o', 'u', ...] -- rest of vowels elided

Right now, it's more than twice as slow as running cat * | wc, but my instinct tells me that the file I/O should outweigh the CPU time needed by a good margin. Simply using cat * | wc processes about 20MB/s with a hot cache, but using my Haskell program (compiled with -O) runs at less than 10MB/s, even after some basic optimization. Profiling tells me that most of the time is spent in tabulateText and countChar.

Is there something I'm missing that I could optimize here?

Edit: Complete file pasted to http://hpaste.org/74638

Answer 1

You should provide the imports so someone can compile the code. However, there's several things here that look likely:

• Compile with -O2 -funbox-strict-fields (to get the benefit of strict fields)
• tabulateText should be strict in lastChr, and stat
• Set.member seems like a very expensive way to do equality comparisons. Use a jump table.

E.g.

isSpaceChar8 :: Char -> Bool
isSpaceChar8 c =
    c == ' '     ||
    c == '\t'    ||
    c == '\n'    ||
    c == '\r'    ||
    c == '\f'    ||
    c == '\v'    ||
    c == '\xa0'

which will inline and optimize very well.

Not sure what countIf does, but it loosk bad. I suspect its an if and you return 0? How about:

Stat
   (a + 1)
   (if isVowel c then a + 1 else a)
   ...

Then look at the core.