HttpClient decodes encoded Url?

Question

I'm trying to figure out some inconsistencies in how HttpClient handles URLs.

I have the following test code:

public async Task TestHttpClient()
{
    var baseUrl = "https://api.twitter.com/1.1/search/tweets.json";
    //var query = "(cafe OR boulangerie)";
    var query = "(café OR boulangerie)";

    var url = baseUrl + $"?q={Uri.EscapeDataString(query)}";

    var httpClient = new HttpClient();
    var response = await httpClient.GetAsync(url);

    await response.Content.ReadAsStringAsync();
}

The code won't actually work, since we need authentication and other stuff for Twitter searches. But it demonstrates my problem.

The variable url will have the following value: https://api.twitter.com/1.1/search/tweets.json?q=%28caf%C3%A9%20OR%20boulangerie%29

However, looking at the request in Fiddler, I can see that what is actually sent is: https://api.twitter.com/1.1/search/tweets.json?q=(caf%C3%A9%20OR%20boulangerie)

So all of a sudden, the parentheses are no longer encoded. This matters in my case, because I use the encoded query string to calculate a signature that I use to authenticate against twitter. So my signature will have percent encoded parentheses and the request won't, so Twitter throws an error and tells me the authentication fails.

What is interesting is that if I send the query with a regular e instead of é then the parentheses are encoded in the request! Like this: https://api.twitter.com/1.1/search/tweets.json?q=%28cafe%20OR%20boulangerie%29

I suppose this is some kind of bug with HttpClient? Can I work around this somehow?

Answer 1

So this turned out to be a difference in how Uri encodes and decodes urls with and without unicode chars in them: https://github.com/dotnet/corefx/issues/15865.

The solution for me was to parse the contents of Uri.AbsoluteUri (which encodes the url in the same, inconsistent, way) and use that when calculating a signature for the authentication. Instead of using Uri.EscapeDataString as I was dpoing previously.