Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I completely sort arbitrary JSON using jq?

Tags:

json

sorting

jq

I want to diff two JSON text files. Unfortunately they're constructed in arbitrary order, so I get diffs when they're semantically identical. I'd like to use jq (or whatever) to sort them in any kind of full order, to eliminate differences due only to element ordering.

--sort-keys solves half the problem, but it doesn't sort arrays.

I'm pretty ignorant of jq and don't know how to write a jq recursive filter that preserves all data; any help would be appreciated.

I realize that line-by-line 'diff' output isn't necessarily the best way to compare two complex objects, but in this case I know the two files are very similar (nearly identical) and line-by-line diffs are fine for my purposes.

Using jq or alternative command line tools to diff JSON files answers a very similar question, but doesn't print the differences. Also, I want to save the sorted results, so what I really want is just a filter program to sort JSON.

like image 654
Jeff Learman Avatar asked Jul 08 '16 01:07

Jeff Learman


People also ask

How do I sort a list of JSON objects?

Using json. dumps() function is one way to sort the JSON object. It is used to convert the array of JSON objects into a sorted JSON object. The value of the sort_keys argument of the dumps() function will require to set True to generate the sorted JSON objects from the array of JSON objects.

Can JSON be sorted?

You cannot sort a JSON string.. JSON is an Object Notation for data transport - ie, a string.

Can we sort JSON array?

A JSONArray can parse text from a String to produce a vector-like object and supports java. util. List interface. We can sort a JSONArray in the below example.

Can jq write JSON?

The safest way to create JSON on the command line is through using a tool that constructs it for you as jq does.


1 Answers

Here is a solution using a generic function sorted_walk/1 (so named for the reason described in the postscript below).

normalize.jq:

# Apply f to composite entities recursively using keys[], and to atoms
def sorted_walk(f):
  . as $in
  | if type == "object" then
      reduce keys[] as $key
        ( {}; . + { ($key):  ($in[$key] | sorted_walk(f)) } ) | f
  elif type == "array" then map( sorted_walk(f) ) | f
  else f
  end;

def normalize: sorted_walk(if type == "array" then sort else . end);

normalize

Example using bash:

diff <(jq -S -f normalize.jq FILE1) <(jq -S -f normalize.jq FILE2)

POSTSCRIPT: The builtin definition of walk/1 was revised after this response was first posted: it now uses keys_unsorted rather than keys.

like image 93
peak Avatar answered Sep 17 '22 21:09

peak