Using sepBy string in Attoparsec

Question

I'm trying to separate a string by either ",", ", and" and "and", and then return whatever was in between. An example of what I have so far is as follows:

import Data.Attoparsec.Text

sepTestParser = nameSep ((takeWhile1 $ inClass "-'a-zA-Z") <* space)
nameSep p = p `sepBy` (string " and " <|> string ", and" <|> ", ")

main = do
  print $ parseOnly sepTestParser "This test and that test, this test particularly."

I would like the output to be ["This test", "that test", "this test particularly."]. I have a vague sense that what I'm doing is wrong, but I can't quite work out why.

Answer 1

^{Note: This answer is written in literate Haskell. Save it as Example.lhs and load it in GHCi or similar.}

The thing is, sepBy is implemented as:

sepBy p s = liftA2 (:) p ((s *> sepBy1 p s) <|> pure []) <|> pure []

This means that the second parser s will be called after the first parser has succeeded. This also means, that if you were to add whitespace to the class of characters, that, you would end up with

["This test and that test","this test particularly"]

since and is now parseable by p. This isn't easy to fix: you would need to look ahead as soon as you hit a space, and check if after an arbitrarily number of spaces an "and" follows, and if it does, stop parsing. Only then a parser written with sepBy will work.

So lets write a parser that takes words instead (the rest of this answer is literate Haskell):

> {-# LANGUAGE OverloadedStrings #-}
> import Control.Applicative
> import Data.Attoparsec.Text
> import qualified Data.Text as T
> import Control.Monad (mzero)

> word = takeWhile1 . inClass $ "-'a-zA-Z"
> 
> wordsP = fmap (T.intercalate " ") $ k `sepBy` many space
>   where k = do
>           a <- word
>           if (a == "and") then mzero
>                           else return a

wordsP now takes multiple words until it either hits something, that's not a word, or a word that equals "and". The returned mzero will indicate a parsing failure, at which another parser can take over:

> andP = many space *> "and" *> many1 space *> pure()
> 
> limiter = choice [
>     "," *> andP,
>     "," *> many1 space *> pure (),
>     andP
>   ]

limiter is mostly the same parser you've already written, it's the same as the regex /,\s+and|,\s+|\s*and\s+/.

Now we can actually use sepBy, since our first parser doesn't overlap with the second anymore:

> test = "This test and that test, this test particular, and even that test"
>
> main = print $ parseOnly (wordsP `sepBy` limiter) test

The result is ["This test","that test","this test particular","even that test"], just as we wanted. Note that this particular parser doesn't preserve whitespace.

So whenever you want to create a parser with sepBy, make sure that both parsers don't overlap.