How can I encode Azure storage table row keys and partition keys?

Question

I'm using Azure storage tables and I have data going in to the RowKey that has slashes in it. According to this MSDN page, the following characters are disallowed in both the PartitionKey and RowKey:

• The forward slash (/) character

• The backslash () character

• The number sign (#) character

• The question mark (?) character

• Control characters from U+0000 to U+001F, including:

• The horizontal tab (\t) character

• The linefeed (\n) character

• The carriage return (\r) character

• Control characters from U+007F to U+009F

I've seen some people use URL encoding to get around this. Unfortunately there's a few glitches that can arise from this, such as being able to insert but unable to delete certain entities. I've also seen some people use base64 encoding, however this also can contain disallowed characters.

How can I encode my RowKey efficiently without running in to disallowed characters, or rolling my own encoding?

Answer 1

Updated 18-Aug-2020 for (new?) issue with '+' character in Azure Search. See comments from @mladenb below for background. Of note, the documentation page referenced does not exclude the '+' character.

When a URL is Base64 encoded, the only character that is invalid in an Azure Table Storage key column is the forward slash ('/'). To address this, simply replace the forward slash character with another character that is both (1) valid in an Azure Table Storage key column and (2) not a Base64 character. The most common example I have found (which is cited in other answers) is to replace the forward slash ('/') with the underscore ('_').

private static String EncodeUrlInKey(String url)
{
    var keyBytes = System.Text.Encoding.UTF8.GetBytes(url);
    var base64 = System.Convert.ToBase64String(keyBytes);
    return base64.Replace('/','_').Replace('+','-');
}

When decoding, simply undo the replaced character (first!) and then Base64 decode the resulting string. That's all there is to it.

private static String DecodeUrlInKey(String encodedKey)
{
    var base64 = encodedKey.Replace('-','+').Replace('_', '/');
    byte[] bytes = System.Convert.FromBase64String(base64);
    return System.Text.Encoding.UTF8.GetString(bytes);
}

Some people have suggested that other Base64 characters also need encoding. According to the Azure Table Storage docs this is not the case.