JSON empty array

Question

I am trying to parse some JSON that is returned from a REST web service. The return from the get() call is a TStringStream. I'm using dbxjson to work with the data. To make things easier to demonstrate here, I've created a test project that reproduces the error without calling the web service (uses a text file for the web service output instead). Here's the code:

var SL : TStringStream;
  LJsonObj : TJSONObject;
begin
  SL := TStringStream.Create;
  try
    SL.LoadFromFile('output.txt');
    LJsonObj := TJSONObject.ParseJSONValue(TEncoding.ASCII.GetBytes(SL.DataString), 0) as TJSONObject;
  finally
    SL.Free;
  end;
end;

Sometimes the phone_numbers array in this JSON data is empty. In the stream object coming from the web service call, it looks like this:

{
    "Contact Information Service": {
        "response": {
            "phone_numbers": [

]
        }
    }
}

This causes ParseJSONValue to return a nil value.

However, if I change the empty phone_numbers array to this in my test txt file:

{
    "Contact Information Service": {
        "response": {
            "phone_numbers": []
        }
    }
}

it works fine (i.e. returns a TJSONObject). The difference being the whitespace in the empty array. For some reason the first JSON response with whitespace in the empty array causes ParseJSONValue to return nil. It works fine with no whitespace between the square braces.

What am I doing wrong with my JSON parsing? Is there some sort of pre-parsing I need to do before calling ParseJSONValue?

Answer 1

This issue is not exclusive of the Delphi JSON implementation (DBXJSON), I worked with some JSON PHP parsers with the same limitation.

Now because all the blanks outside a double quoted strings literals are (and must be) ignored by the JSON parsers, you can remove these white spaces safely, So A possible workaround is Minify your Json string, before to parse it.

Try this sample, which uses regular expressions to remove the extra whitespaces from a string.

{$APPTYPE CONSOLE}

{$R *.res}


uses
  System.RegularExpressions,
  System.Classes,
  System.SysUtils,
  Data.DBXJSON;

const
JsonString=
'{'+
'    "Contact Information Service": {'+
'        "response": {'+
'            "phone_numbers": [        ]'+
'        }'+
'    }'+
'}';

function JsonMinify(const S: string): string;
begin
 Result:=TRegEx.Replace(S,'("(?:[^"\\]|\\.)*")|\s+', '$1');
end;

procedure TestJSon;
var
  s : string;
  SL : TStringStream;
  LJsonObj : TJSONObject;
begin
  SL := TStringStream.Create;
  try
    s:=JsonMinify(JsonString);
    SL.WriteString(s);
    LJsonObj := TJSONObject.ParseJSONValue(TEncoding.ASCII.GetBytes(SL.DataString), 0) as TJSONObject;
    Writeln(LJsonObj.Size);
  finally
    SL.Free;
  end;
end;

begin
 try
    TestJSon;
 except
    on E:Exception do
        Writeln(E.Classname, ':', E.Message);
 end;
 Writeln('Press Enter to exit');
 Readln;
end.

Answer 2

Have a look at TJsonObject.ParseArray. You'll find this:

while ValueExpected or (Br.PeekByte <> Ord(']')) do
begin
  ConsumeWhitespaces(Br);
  Pos := ParseValue(Br, JsonArray);
  if Pos <= 0 then
    Exit(Pos);

So at the top of the array (immediately after it reads the open bracket), if the next character is not a close bracket, eat whitespace and then try to read a valid JSON value. A close bracket is not a valid JSON value, so it bails out at this point.

This does appear to be valid JSON, (I can get my browser to accept it as a valid JavaScript object), so this ought to be considered a bug in the DBXJSON library. You might need to pre-parse this, use a different JSON library (there are a handful for Delphi) or find a way to ensure that the information being sent to you does not contain this pattern.

Either way, you ought to report this to QC as a bug.