Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I collect unique elements of an array-valued field across multiple objects in jq?

Tags:

json

arrays

jq

I have json data that looks like:

{"foo":"one", "bar":2, "baz":[0]}
{"foo":"two", "bar":3, "baz":[1]}
{"foo":"one", "bar":3, "baz":[2,3]}
{"foo":"one", "bar":2, "baz":[2,4]}

I want to group everything with the same "foo" and collect the unique values of bar, and then collect the unique values inside of the "baz" arrays:

[
  {"foo":"one", "bar":[2, 3], "baz":[0,2,3,4]},
  {"foo":"two", "bar":[3], "baz":[1]}
]

(I don't care if the result is in an array or just a raw sequence of whitespace separated JSON objects, and I don't care about the order of the items in the "baz" array)

I've got jq version 1.4 installed from source. I can properly group by "foo" and collect unique values of "bar" with:

jq -s 'group_by(.foo) | map({foo: .[0].foo, bar: map(.bar) | unique})'

yielding:

[
  {"foo":"one","bar":[2,3]},
  {"foo":"two","bar":[3]}
]

but I can't figure out how to do the collection of unique values of "baz".

What am I missing?

like image 848
bskaggs Avatar asked Sep 30 '14 19:09

bskaggs


People also ask

How to find unique elements from an array in Java?

In Java, there is more than one way to find unique elements from an array which are as follows: By storing all the elements to the hashmap's key. By using nested loop.

How to filter unique items from an array in JavaScript?

We will see a different approach to filter the unique items from the simple, nested arrays as well as an array of objects. filter () method takes a callback function in which we can provide the condition to filter the unique items.

How to get unique values in JavaScript arrays of primitive values?

There are two common ways (in ES5 and ES6, respectively) of getting unique values in JavaScript arrays of primitive values. Basically, in ES5, you first define a distinct callback to check if a value first occurs in the original array, then filter the original array so that only first-occurred elements are kept. In ES6, the code is much simpler.

How to get the unique properties of an array of objects?

We can use the spread … along with map and Set to get the unique properties from the array of objects. map returns a new array with the added items. We are passing the new array of person.name or person.age to the Set which will have the unique items and then spreading it as individual item using spread … operator.


1 Answers

Edit: new "flatten" function not needed (thanks @JeffMercado)

I can run

jq -s 'group_by(.foo) | map({foo: .[0].foo, bar: map(.bar) | unique, baz: map(.baz) | add | unique})

which produces:

[
  {"foo":"one","bar":[2,3],"baz":[0,2,3,4]},
  {"foo":"two","bar":[3],"baz":[1]}
]
like image 70
bskaggs Avatar answered Oct 20 '22 23:10

bskaggs