Percent-encoded slash ("/") is decoded before the request dispatch

Question

I have an URL containing several slash characters (/) as a part of the filename (not the URL). But when I send http request, the percent-encoded %2F is translated to / before the request dispatch, therefore generating a wrong URL.

How can I make a literal http request ignoring the percent-encoded values in PowerShell?

Actual URL used (Chromium browser):

https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%2F292817%2Fchrome-win32.zip?generation=1409504089694000&alt=media

---

I have tried Invoke-WebRequest cmdlet:

Invoke-WebRequest -Uri $ChromeUrl -OutFile $FilePath -Verbose

VERBOSE: GET https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64/292817/chrome-win32.zip?generation=1409504089694000&alt=media with 0-byte payload1`

Not found error.

Also tried WebClient's DownloadFile method:

$wclient = New-Object System.Net.WebClient
$wclient.DownloadFile($ChromeUrl, $FilePath)

Returns 404 due to wrong URL requested again.

---

Workaround 1 (successful)

Reflection-based workarounds provided by briantist and Tanuj Mathur are both working great. The latter one:

$UrlFixSrc = @" 
using System;
using System.Reflection;

public static class URLFix 
{ 
    public static void ForceCanonicalPathAndQuery(Uri uri)
    {
        string paq = uri.PathAndQuery;
        FieldInfo flagsFieldInfo = typeof(Uri).GetField("m_Flags", BindingFlags.Instance | BindingFlags.NonPublic);
        ulong flags = (ulong) flagsFieldInfo.GetValue(uri);
        flags &= ~((ulong) 0x30);
        flagsFieldInfo.SetValue(uri, flags);
    }
} 
"@ 

Add-Type -TypeDefinition $UrlFixSrc-Language CSharp
[URLFix]::ForceCanonicalPathAndQuery([URI]$ChromeUrl)

Invoke-WebRequest -Uri $ChromeUrl -OutFile $FilePath -Verbose

VERBOSE: GET https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%2F292640%2Fchrome-win32.zip?generation=1409351584147000&alt=media

Workaround 2 (successful)

More clean solution (offered by Tanuj Mathur), but requires access to system files, is by adding a config file %SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe.config with the following content:

<?xml version="1.0" encoding="utf-8" ?> 
 <configuration> 
   <uri>
     <schemeSettings>
      <add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes" />
      <add name="https" genericUriParserOptions="DontUnescapePathDotsAndSlashes" />
     </schemeSettings>
   </uri>
</configuration>

Corresponding modifications has to be done in powerhsell_ise.exe.config for it to work in ISE.

Workaround 3 (failed)

I thought its a System.URI class constructor problem that is called upon implicit casting, which translates the escaped values. Tried an overloaded variant Uri ([String]uriString, [Boolean]dontEscape). But there was no difference. The same outcome with or without dontEscape argument.

$uri = new-object System.Uri($ChromeUrl, $true)
$uri | Format-List OriginalString, AbsoluteUri

  OriginalString : https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%2F292817%2Fchrome-win32.zip?generation=1409504089694000&alt=media
  AbsoluteUri    : https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64/292817/chrome-win32.zip?generation=1409504089694000&alt=media

---

Workaround 4 (failed)

Also tried to fool URI parser by replacing percent character with its percent-encoded value %25. But then it ignored everything completely.

Invoke-WebRequest -Uri $ChromeUrl.Replace('%', '%25') -OutFile $DownloadPath -Verbose

VERBOSE: GET https://www.googleapis.com/download/storage/v1/b/chromium-browser-continuous/o/Win_x64%252F292817%252Fchrome-win32.zip?generation=1409504089694000&alt=media with 0-byte pa yload

---

Workaround 5 (not implemented)

The only way I found that requests URL properly is through Internet Explorer instance.

$ie = New-Object -ComObject InternetExplorer.Application                
$ie.Visible = $true
$ie.Silent = $false
$ie.Navigate2($ChromeUrl)

But then I don't know how to automate the 'Saves as' button click and save it to desired path. Also, even if implemented, I don't feel like this is a good solution. What happens when IE is already running or uninstalled from the system?

Answer 1

I've been playing around with your code for the last few hours, and it's a doozy. The given code and it's variants all pass when run in the Powershell ISE, but fail on the Powershell console. The issue itself seems to be the one documented on Microsoft Connect here.

Interestingly, as per user Glenn Block's answer on a related issue, this bug was fixed in .NET Framework 4.5. You can check the version of the .NET framework being used by your Powershell by running the command $PSVersionTable. As long as the CLRVersion value is of the form 4.0.30319.x, where x > 1700, then you are running v4.5 of the framework.

I'm running Powershell v4.0 on .NET framework 4.5 on my machine, so that explains why Powershell ISE shows the correct behaviour, but I was not able to figure out why Powershell console does not. I verified the .NET assemblies loaded by both, and they seem to be the same.

As things stand, we have two options. One is to use reflection and set a private field on the .Net class to prevent this behaviour (as outlined in this answer). The other is to use the workaround listed in the Microsoft Connect issue. This involves the following steps:

1. Go to your Powershell install folder (this was "C:\Windows\System32\WindowsPowerShell\v1.0\" on my machine). This folder should have the file powershell.exe in it.
2. Create a new text file in this folder, and name it powershell.exe.config

3. Open this file in a text editor, and paste the following text into it: <?xml version="1.0" encoding="utf-8" ?> <configuration> <uri> <schemeSettings> <add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes" /> <add name="https" genericUriParserOptions="DontUnescapePathDotsAndSlashes" /> </schemeSettings> </uri> </configuration>

4. Save this file. Close ALL running instances of Powershell.

5. Start a new instance of Powershell. This will cause Powershell to detect the config file you created and parse it. The config entries basically tell the .NET libraries to disable the automatic unescaping of HTTP and HTTPS uri's.
6. Run your script. You should no longer see the issue with the Uris.

Answer 2

If you're going to use PowerShell you can also do Workaround 1 in pure PowerShell:

function UrlFix([Uri]$url) {
    $url.PathAndQuery | Out-Null
    $m_Flags = [Uri].GetField("m_Flags", $([Reflection.BindingFlags]::Instance -bor [Reflection.BindingFlags]::NonPublic))
    [uint64]$flags = $m_Flags.GetValue($url)
    $m_Flags.SetValue($url, $($flags -bxor 0x30))
}

UrlFix $ChromeUrl
Invoke-WebRequest -Uri $ChromeUrl -OutFile $FilePath -Verbose