Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

.NET Regular expressions on bytes instead of chars

I'm trying to do some parsing that will be easier using regular expressions.

The input is an array (or enumeration) of bytes.

I don't want to convert the bytes to chars for the following reasons:

  1. Computation efficiency
  2. Memory consumption efficiency
  3. Some non-printable bytes might be complex to convert to chars. Not all the bytes are printable.

So I can't use Regex.

The only solution I know, is using Boost.Regex (which works on bytes - C chars), but this is a C++ library that wrapping using C++/CLI will take considerable work.

How can I use regular expressions on bytes in .NET directly, without working with .NET strings and chars?

Thank you.

like image 504
brickner Avatar asked Jun 12 '10 13:06

brickner


People also ask

What is difference [] and () in regex?

[] denotes a character class. () denotes a capturing group. [a-z0-9] -- One character that is in the range of a-z OR 0-9. (a-z0-9) -- Explicit capture of a-z0-9 .

What does '$' mean in regex?

$ means "Match the end of the string" (the position after the last character in the string). Both are called anchors and ensure that the entire string is matched instead of just a substring.

What is \s in regex C#?

You regex is basically saying: . // Match any character * // Zero or more times \s // Matches any white space including space, tab, form-feed, etc. * // Those white spaces, tabs etc.

How can you negate characters in a set?

Typing a caret after the opening square bracket negates the character class. The result is that the character class matches any character that is not in the character class. Unlike the dot, negated character classes also match (invisible) line break characters.


1 Answers

There is a bit of impedance mismatch going on here. You want to work with Regular expressions in .Net which use strings (multi-byte characters), but you want to work with single byte characters. You can't have both at the same time using .Net as per usual.

However, to break this mismatch down, you could deal with a string in a byte oriented fashion and mutate it. The mutated string can then act as a re-usable buffer. In this way you will not have to convert bytes to chars, or convert your input buffer to a string (as per your question).

An example:

//BLING
byte[] inputBuffer = { 66, 76, 73, 78, 71 };

string stringBuffer = new string('\0', 1000);

Regex regex = new Regex("ING", RegexOptions.Compiled);

unsafe
{
    fixed (char* charArray = stringBuffer)
    {
        byte* buffer = (byte*)(charArray);

        //Hard-coded example of string mutation, in practice you would
        //loop over your input buffers and regex\match so that the string
        //buffer is re-used.

        buffer[0] = inputBuffer[0];
        buffer[2] = inputBuffer[1];
        buffer[4] = inputBuffer[2];
        buffer[6] = inputBuffer[3];
        buffer[8] = inputBuffer[4];

        Console.WriteLine("Mutated string:'{0}'.",
             stringBuffer.Substring(0, inputBuffer.Length));

        Match match = regex.Match(stringBuffer, 0, inputBuffer.Length);

        Console.WriteLine("Position:{0} Length:{1}.", match.Index, match.Length);
    }
}

Using this technique you can allocate a string "buffer" which can be re-used as the input to Regex, but you can mutate it with your bytes each time. This avoids the overhead of converting\encoding your byte array into a new .Net string each time you want to do a match. This could prove to be very significant as I have seen many an algorithm in .Net try to go at a million miles an hour only to be brought to its knees by string generation and the subsequent heap spamming and time spent in GC.

Obviously this is unsafe code, but it is .Net.

The results of the Regex will generate strings though, so you have an issue here. I'm not sure if there is a way of using Regex that will not generate new strings. You can certainly get at the match index and length information but the string generation violates your requirements for memory efficiency.

Update

Actually after disassembling Regex\Match\Group\Capture, it looks like it only generates the captured string when you access the Value property, so you may at least not be generating strings if you only access index and length properties. However, you will be generating all the supporting Regex objects.

like image 70
14 revs Avatar answered Oct 02 '22 14:10

14 revs