Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

remove duplicates in JSON values using jq

Tags:

json

jq

I have the following JSON:

[
  {
    "function": "ping",
    "name": "start",
    "servers": [
      {
        "load": 581.6875,
        "last_heard": 2.379324197769165,
        "version": "1.0",
        "hidden": false,
        "pid": "19735"
      },
      {
        "load": 444.0625,
        "last_heard": 1.3227169513702393,
        "version": "1.0",
        "hidden": false,
        "pid": "12092"
      }
    ]
  },
  {
    "function": "pong",
    "name": "middle",
    "servers": [
      {
        "load": 581.6875,
        "last_heard": 2.379324197769165,
        "version": "2.0",
        "hidden": false,
        "pid": "19735"
      },
      {
        "load": 444.0625,
        "last_heard": 1.3227169513702393,
        "version": "3.0",
        "hidden": false,
        "pid": "12092"
      },
      {
        "load": 444.0625,
        "last_heard": 1.3227169513702393,
        "version": "3.0",
        "hidden": false,
        "pid": "12093"
      }
    ]
  },
  {
    "function": "pang",
    "name": "end",
    "servers": [
      {
        "load": 581.6875,
        "last_heard": 2.379324197769165,
        "version": "2.0",
        "hidden": false,
        "pid": "19735"
      },
      {
        "load": 444.0625,
        "last_heard": 1.3227169513702393,
        "version": "2.0",
        "hidden": false,
        "pid": "12092"
      }
    ]
  }
]

(it's just a sample, it's hundreds of entries)

What I need is to get

[{"name": "start", "version": ["1.0"]},
{"name": "middle", "version": ["2.0", "3.0"]},
{"name": "end", "version": ["2.0"]}]

So I need to remove useless data and then get a list of names with their unique values for "version".

I can get until the point where I have

{
  "name": "ping",
  "version": [
    "1.0",
    "1.0"
  ]
}
{
  "name": "pong",
  "version": [
    "2.0",
    "3.0",
    "3.0"
  ]
}
{
  "name": "pang",
  "version": [
    "2.0",
    "2.0"
  ]
}

using

jq '.[] | {name: .function, version: [.servers[].version]}'

But I need to get rid of the duplicated values. Is this possible using jq?

like image 702
Daniel Avatar asked Nov 17 '15 20:11

Daniel


People also ask

Does JSON Stringify remove duplicates?

We can use the JSON. stringify method to convert a plain JavaScript object into a string. This lets us check all the properties at once instead of hard coding the check as we did in the previous example. to remove the duplicate item by using JSON.

Can jq write JSON?

jq is an amazing little command line utility for working with JSON data. We've written before about how you can use jq to parse JSON on the command line, but in this post I want to talk about using jq to create JSON data from scratch or make changes to existing data.

How do you remove duplicates from JSON in Java?

How to remove the duplicates? You will need to convert the JSON to Java Objects and then perform the duplicate removal operation. Added code snippet for each of the steps. Hope this helps!


1 Answers

You were almost there. Just pipe the version array to the unique function:

jq '[.[]|{name, "version": [.servers[].version]|unique}]' input
like image 111
hek2mgl Avatar answered Sep 19 '22 11:09

hek2mgl