How to resolve FParsec error "The combinator 'many' was applied to a parser that succeeds without consuming..."

Question

I have a parser that seems straight-forward enough. I added this sub-parser to the end to give info about general parsing errors since all the other sub-parsers failed -

/// Read the rest of a line as an error.
let readError =
    parse {
        let! restOfLineStr = restOfLine true
        return makeViolation ("Read error on: " + restOfLineStr + ".") }

/// Read an expression.
do readExprRef :=
    choice
        [attempt readBoolean
         attempt readCharacter
         attempt readString
         attempt readInt
         attempt readError] // just now added this sub-parser, and get the issue

However, once I add readError as a choice, I get the dreaded FParsec error about stream consumption at run-time - The combinator 'many' was applied to a parser that succeeds without consuming input and without changing the parser state in any other way. I don't understand why I get this since I do use the parsed rest of the line to create a used error (here 'violation') structure.

Can someone help me understand this? Am I going about signaling parser errors to the user in the wrong way? If not, how could I fix this?

Thank you kindly for your help!

* More detail *

Here's some more code that may be relevant -

/// The expression structure.
type Expr =
| Violation of Expr
| Boolean of bool
| Character of char
| String of string
| Int of int

/// Make a violation from a string.
let makeViolation str = Violation (String str)

/// Read whitespace character as a string.
let spaceAsStr = anyOf whitespaceChars |>> fun chr -> string chr

/// Read a line comment.
let lineComment = pchar lineCommentChar >>. restOfLine true

/// Read a multiline comment.
/// TODO: make multiline comments nest.
let multilineComment =
    between
        (pstring openMultilineCommentStr)
        (pstring closeMultilineCommentStr)
        (charsTillString closeMultilineCommentStr false System.Int32.MaxValue)

/// Read whitespace text.
let whitespace = lineComment <|> multilineComment <|> spaceAsStr

/// Skip any white space characters.
let skipWhitespace = skipMany whitespace

/// Skip at least one white space character.
let skipWhitespace1 = skipMany1 whitespace

/// Read a boolean.
let readBoolean = 
    parse {
        do! skipWhitespace
        let! booleanValue = readStr trueStr <|> readStr falseStr
        return Boolean (booleanValue = trueStr) }

/// Read a character.
let readCharacter =
    parse {
        // TODO: enable reading of escaped chars
        do! skipWhitespace
        let! chr = between skipSingleQuote skipSingleQuote (manyChars (noneOf "\'"))
        return Character chr.[0] }

/// Read a string.
let readString =
    parse {
        // TODO: enable reading of escaped chars
        do! skipWhitespace
        let! str = between skipDoubleQuote skipDoubleQuote (manyChars (noneOf "\""))
        return String str }

/// Read an int.
let readInt =
    parse {
        do! skipWhitespace
        let! value = pint32
        let! _ = opt (skipString intSuffixStr)
        do! notFollowedByLetterOrNameChar
        do! notFollowedByDot
        return Int value }

I dunno. Maybe the issue is that it's already at the end of the stream once it tries to run the readError parser. Would that make restOfLine consume no input, not even whitespace?

* Conclusion *

It turns out that the approach to error reporting with a readError parser is wrong. The correct approach is to use a 'till end' parser like so -

/// Read the end of input.
let readEndOfInput = skipWhitespace >>. eof

// Read multiple exprs.
let readExprs = many readExpr

// Read exprs until the end of the input.
let readExprsTillEnd = readExprs .>> readEndOfInput

Now I just run readExprsTillEnd when I need to get all the exprs in an input stream.

Thanks again, Gustavo!

Answer 1

Thanks for the additional code you posted, unfortunately I was unable to reproduce the error. But why don't you try to remove the last attempt? I think it makes no sense and maybe is causing a problem.

do readExprRef :=
    choice
        [attempt readBoolean
         attempt readCharacter
         attempt readString
         attempt readInt
         readError]

I'm not an FParsec expert but I think the last parser of a choice should not be an attempt.

UPDATE:

The readError parser succeeds even consuming no input, if at some point you have a call to readExpr as parameter of a many it would never ends. I mean if you call

run (many readError) "" ;;

You'll get that error message because many will continue applying that parser until it fails, but it will never fail.

Have a look at the restOfLine function specification at http://www.quanttec.com/fparsec/reference/charparsers.html#members.restOfLine it warns you about this.

Now there are many ways you can solve it, but I would say you will have to reconsider the way you handle the parser errors.

One thing you can do is take out the readError function and then when you call the readExpr parser you call it this way

let readExprs = many readExpr .>> eof

doing so you enforce the eof and if there is something not handled by the parsers in the choice before the eof, FParsec will automatically generate a nice error message for you.

And if you want to handle that error, have a look at http://www.quanttec.com/fparsec/users-guide/customizing-error-messages.html