C# regex.split method is adding empty string before parenthesis

Question

I have some code that tokenizes a equation input into a string array:

string infix = "( 5 + 2 ) * 3 + 4";
string[] tokens = tokenizer(infix, @"([\+\-\*\(\)\^\\])");
foreach (string s in tokens)
{
   Console.WriteLine(s);
}

Now here is the tokenizer function:

public string[] tokenizer(string input, string splitExp)
        {
            string noWSpaceInput = Regex.Replace(input, @"\s", "");
            Console.WriteLine(noWSpaceInput);
            Regex RE = new Regex(splitExp);
            return (RE.Split(noWSpaceInput));
        }

When I run this, I get all characters split, but there is an empty string inserted before the parenthesis chracters...how do I remove this?

//empty string here

(

5

+

2

//empty string here

)

*

3

+

4

Answer 1

I would just filter them out:

public string[] tokenizer(string input, string splitExp)
{
    string noWSpaceInput = Regex.Replace(input, @"\s", "");
    Console.WriteLine(noWSpaceInput);
    Regex RE = new Regex(splitExp);
    return (RE.Split(noWSpaceInput)).Where(x => !string.IsNullOrEmpty(x)).ToArray();
}

Answer 2

What you're seeing is because you have nothing then a separator (i.e. at the beginning of the string is(), then two separator characters next to one another (i.e. )* in the middle). This is by design.

As you may have found with String.Split, that method has an optional enum which you can give to have it remove any empty entries, however, there is no such parameter with regular expressions. In your specific case you could simply ignore any token with a length of 0.

foreach (string s in tokens.Where(tt => tt.Length > 0))
{
   Console.WriteLine(s);
}