ConvertTo-Json and ConvertFrom-Json with special characters

Question

I have a file containing some properties which value of some of them contains escape characters, for example some Urls and Regex patterns.

When reading the content and converting back to the json, with or without unescaping, the content is not correct. If I convert back to json with unescaping, some regular expression break, if I convert with unescaping, urls and some regular expressions will break.

How can I solve the problem?

Minimal Complete Verifiable Example

Here are some simple code blocks to allow you simply reproduce the problem:

Content

$fileContent = 
@"
{
    "something":  "http://domain/?x=1&y=2",
    "pattern":  "^(?!(\\`|\\~|\\!|\\@|\\#|\\$|\\||\\\\|\\'|\\\")).*"
}
"@

With Unescape

If I read the content and then convert the content back to json using following command:

$fileContent | ConvertFrom-Json | ConvertTo-Json | %{[regex]::Unescape($_)}

The output (which is wrong) would be:

{
    "something":  "http://domain/?x=1&y=2",
    "pattern":  "^(?!(\|\~|\!|\@|\#|\$|\||\\|\'|\")).*"
}

Without Unescape

If I read the content and then convert the content back to json using following command:

$fileContent | ConvertFrom-Json | ConvertTo-Json 

The output (which is wrong) would be:

{
    "something":  "http://domain/?x=1\u0026y=2",
    "pattern":  "^(?!(\\|\\~|\\!|\\@|\\#|\\$|\\||\\\\|\\\u0027|\\\")).*"
}

Expected Result

The expected result should be same as the input file content.

Answer 1

I decided to not use Unescape, instead replace the unicode \uxxxx characters with their string values and now it works properly:

$fileContent = 
@"
{
    "something":  "http://domain/?x=1&y=2",
    "pattern":  "^(?!(\\`|\\~|\\!|\\@|\\#|\\$|\\||\\\\|\\'|\\\")).*"
}
"@

$fileContent | ConvertFrom-Json | ConvertTo-Json | %{
    [Regex]::Replace($_, 
        "\\u(?<Value>[a-zA-Z0-9]{4})", {
            param($m) ([char]([int]::Parse($m.Groups['Value'].Value,
                [System.Globalization.NumberStyles]::HexNumber))).ToString() } )}

Which generates the expected output:

{
    "something":  "http://domain/?x=1&y=\\2",
    "pattern":  "^(?!(\\|\\~|\\!|\\@|\\#|\\$|\\||\\\\|\\'|\\\")).*"
}

Answer 2

tl;dr

The problem does not affect PowerShell (Core) 6+ (the install-on-demand, cross-platform PowerShell edition), which uses a different implementation of the ConvertTo-Json and ConvertFrom-Json cmdlets, based on Newtonsoft.JSON (whose direct use is shown in r3verse's answer), as of Powershell 7.2. There, your sample roundtrip command works as expected.

Only ConvertTo-Json in Windows PowerShell is affected (the bundled-with-Windows PowerShell edition whose latest and final version is 5.1). But note that the JSON representation - while unexpected - is technically correct.

A simple, but robust solution focused only on unescaping those Unicode escape sequences that ConvertTo-Json unexpectedly creates - namely for & ' < > - while ruling out false positives:

# The following sample JSON with undesired Unicode escape sequences for `& < > '`, was
# created with Windows PowerShell's ConvertTo-Json as follows:
#   ConvertTo-Json "Ten o'clock at <night> & later. \u0027 \\u0027"
# Note that \u0027 and \\u0027 are NOT Unicode escape sequences and must not be
# interpreted as such.
# The *desired* JSON representation - without the unexpected escaping - would be:
#   "Ten o'clock at <night> & later. \\u0027 \\\\u0027"
$json = '"Ten o\u0027clock at \u003Cnight\u003e \u0026 later. \\u0027 \\\\u0027"'

[regex]::replace(
  $json, 
  '(?<=(?:^|[^\\])(?:\\\\)*)\\u(00(?:26|27|3c|3e))', 
  { param($match) [char] [int] ('0x' + $match.Groups[1].Value) },
  'IgnoreCase'
)

The above outputs the desired JSON representation, without the unnecessary escaping:

"Ten o'clock at <night> & later. \\u0027 \\\\u0027"

---

Background information:

ConvertTo-Json in Windows PowerShell unexpectedly represents the following ASCII-range characters by their Unicode escape sequences in JSON strings:

• & (Unicode escape sequence: \u0026)
• ' (\u0027)
• < and > (\u003c and \u003e)

There's no good reason to do so (these characters only require escaping in HTML/XML text).

However, any compliant JSON parser - including ConvertFrom-Json - converts these escape sequences back to the characters they represent.

In other words: While the JSON text created by Windows PowerShell's ConvertTo-Json is unexpected and can impede readability, it is technically correct and - while not identical - equivalent to the original representation in terms of the data it represents.

---

Fixing the readability problem:

As an aside: While [regex]::Unescape(), whose purpose is to unescape regexes only, also converts Unicode escape sequences to the characters they represent, it is fundamentally unsuited to selectively unescaping Unicode sequences JSON strings, given that all other \ escapes must be preserved in order for the JSON string to remain syntactically valid.

While your answer works well in general, it has limitations (aside from the easily corrected problem that a-zA-Z should be a-fA-F to limit matching to those letters that are valid hex. digits):

• It doesn't rule out false positives, such as \\u0027 or \\\\u0027 (\\ escapes \, so that the u0027 part becomes a verbatim string and must not be treated as an escape sequence).

• It converts all Unicode escape sequences, which presents two problems:

  • Escape sequences representing characters that require escaping would also be converted to the verbatim character representations, which would break the JSON representations with \u005c, for instance, given that the character it represents, \, requires escaping.

  • For non-BMP Unicode characters that must be represented as pairs of Unicode escape sequences (so-called surrogate pairs), your solution would mistakenly try to unescape each half of the pair separately.

For a robust solution that overcomes these limitations, see this answer (surrogate pairs are left as Unicode escape sequences, Unicode escape sequences whose characters require escaping are converted to \-based (C-style) escapes, such as \n, if possible).

However, if the only requirement is to unescape those Unicode escape sequences that Windows PowerShell's ConvertTo-Json unexpectedly creates, the solution at the top is sufficient.

Answer 3

If you don't want to rely on Regex (from @Reza Aghaei's answer), you could import the Newtonsoft JSON library. The benefit is the default StringEscapeHandling property which escapes control characters only. Another benefit is avoiding the potentially dangerous string replacements you would be doing with Regex.

This StringEscapeHandling is also the default handling of PowerShell Core (version 6 and up) because they started to use Newtonsoft internally since then. So another alternative would be to use ConvertFrom-Json and ConvertTo-Json from PowerShell Core.

Your code would look something like this if you import the Newtonsoft JSON library:

[Reflection.Assembly]::LoadFile("Newtonsoft.Json.dll")

$json = Get-Content -Raw -Path file.json -Encoding UTF8 # read file
$unescaped = [Newtonsoft.Json.Linq.JObject]::Parse($json) # similar to ConvertFrom-Json

$escapedElementValue = [Newtonsoft.Json.JsonConvert]::ToString($unescaped.apiName.Value) # similar to ConvertTo-Json
$escapedCompleteJson = [Newtonsoft.Json.JsonConvert]::SerializeObject($unescaped) # similar to ConvertTo-Json

Write-Output "Variable passed = $escapedElementValue"
Write-Output "Same JSON as Input = $escapedCompleteJson"