Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Remove duplicates from json data

Tags:

python

Is there an efficient way to remove duplicates 'person_id' fields from this data with python? In this case just keep the first occurrence.

{
  {obj_id: 123,
    location: {
      x: 123,
      y: 323,
  },
  {obj_id: 13,
    location: {
      x: 23,
      y: 333,
  },
 {obj_id: 123,
    location: {
      x: 122,
      y: 133,
  },
}

Should become:

{
  {obj_id: 123,
    location: {
      x: 123,
      y: 323,
  },
  {obj_id: 13,
    location: {
      x: 23,
      y: 333,
  },
}
like image 429
9-bits Avatar asked Feb 17 '23 06:02

9-bits


1 Answers

Presuming your JSON is valid syntax and you are indeed requesting help for Python you will need to do something like this

import json
ds = json.loads(json_data_string) #this contains the json
unique_stuff = { each['obj_id'] : each for each in ds }.values()

if you want to always retain the first occurrence, you will need to do something like this

all_ids = [ each['obj_id'] for each in ds ] # get 'ds' from above snippet
unique_stuff = [ ds[ all_ids.index(id) ] for id in set(ids) ]
like image 192
Kalyan02 Avatar answered Feb 28 '23 18:02

Kalyan02