How to determine if a string is English sentence or code?

Question

Consider the following two strings, the first one is code, the second one is English sentence (phrase to be precise). How can I detect that the first one is code and the second is not.

1. for (int i = 0; i < b.size(); i++) {
2. do something in English (not necessary to be a sentence).

I'm thinking about counting special characters (such as "=", ";", "++", etc ), and set if to some threshold. Are there any better ways to do this? Any Java libraries?

Note that the code may not parsable, because it is not a complete method/statement/expression.

My assumption is that English sentences are pretty regular, it most likely contains only ",", ".", "_", "(", ")", etc. They do not contains something like this: write("the whole lot of text");

Answer 1

You can try the OpenNLP sentence parser. It returns the n best parses for a sentence. For most English sentences it returns at least one. I believe, that for most code snippets it won't return any and hence you can be quite sure it is not an English sentence.

Use this code for parsing:

    // Initialize the sentence detector
    final SentenceDetectorME sdetector = EasyParserUtils
            .getOpenNLPSentDetector(Constants.SENTENCE_DETECTOR_DATA);

    // Initialize the parser
    final Parser parser = EasyParserUtils
            .getOpenNLPParser(Constants.PARSER_DATA_LOC);

    // Get sentences of the text
    final String sentences[] = sdetector.sentDetect(essay);

    // Go through the sentences and parse each
    for (final String sentence : sentences) {
        // Parse the sentence, produce only 1 parse
        final Parse[] parses = ParserTool.parseLine(sentence, parser, 10);
        if (parses.length == 0) {
            // Most probably this is code
        }
        else {
            // An English sentence
        }
    }

and these are the two helper methods (from EasyParserUtils) used in the code:

public static Parser getOpenNLPParser(final String parserDataURL) {
    try (final InputStream isParser = new FileInputStream(parserDataURL);) {
        // Get model for the parser and initialize it
        final ParserModel parserModel = new ParserModel(isParser);
        return ParserFactory.create(parserModel);
    }
    catch (final IOException e) {
        e.printStackTrace();
        return null;
    }
}

and

public static SentenceDetectorME getOpenNLPSentDetector(
        final String sentDetDataURL) {
    try (final InputStream isSent = new FileInputStream(sentDetDataURL)) {
        // Get models for sentence detector and initialize it
        final SentenceModel sentDetModel = new SentenceModel(isSent);
        return new SentenceDetectorME(sentDetModel);
    }
    catch (final IOException e) {
        e.printStackTrace();
        return null;
    }
}

Answer 2

Look into lexical analysis and parsing (same as if you were writing a compiler). You might not even need a parser if you're not requiring full statements.

Answer 3

The basic idea is to convert the string to a set to tokens. For example, the code line above may become "KEY,SEPARATOR,ID,ASSIGN,NUMBER,SEPARATOR,...". And then we can use simple rules to separate code from English.

check out the code here

Answer 4

You could use a Java parser or create one using the BNF but the issue here is that you said the code may not be parsable so it will fail.

My advice : use some custom regexp to detect special patterns in the code. Use as many as possible to have a good success rate.

Some examples :

• for\s*\( (for loop)
• while\s*\( (while loop)
• [a-zA-Z_$][a-zA-Z\d_$]*\s*\( (constructor)
• \)\s*\{ (begin of a block / method)
• ...

Yes it's a long shot but looking at what you want, you don't have many possibility.