Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

jq error "object is not valid in a csv row"

Tags:

json

bash

csv

jq

I'm using jq to convert a nested .json file to .csv, however, there seems to be an issue when I have a particular key, value within the JSON. The script I'm using comes from this answer, which works if I eliminate one line from the input file.

ERROR:

jq: error (at example.json:0): object ({"favicon":...) is not valid in a csv row

The error message isn't very descriptive, since the object it mentions doesn't seem to be the culprit.

json2csv.jq

paths as $path
| {path: $path, value: getpath($path)}
| select(.value|type == "object" )
| select( [.value[]][0] | type != "object")
| .path + ([.value[]])
| @csv

example.json (which validates with https://jsonlint.com):

{
  "items": [
    {
      "id": 1234,
      "title": "Title of Item",
      "internet": {
        "favicon": "https://example.com/icon.png",
        "domain": "https://example.com"
      },
      "image": {
        "gif": null,
        "main": "https://example.com/image_large.png"
      },
      "description": {
        "origin": null,
        "resource": null
      },
      "referral": {
        "owner": null,
        "name": null
      },
      "url": "https://example.com/image.png?version=1"
    }
  ],
  "other": {
    "data": 0,
    "notes": 1,
    "total": 1,
    "misc": 1
  }
}

Removing the following from the JSON doesn't throw error from jq:

, "url": "https://example.com/image.png?version=1"

Invocation:

$ jq -er -f json2csv.jq example.json

I can't really change the .json file, because it needs the problematic line, so I'm looking for a solution from jq.

like image 662
l'L'l Avatar asked Jan 25 '23 09:01

l'L'l


1 Answers

Both @tsv and @csv require their inputs to be flat arrays (i.e., arrays of atomic values). There are many (infinitely many?) ways to "flatten" objects and arrays, so without knowing something about what you expect, I'd just be guessing wildly.

If you want a wild guess, replace .path + [.value[]] by

.path + [.value[]|tostring]

At least that will always work (assuming the input is valid JSON) while producing easy-to-read results.

like image 108
peak Avatar answered Jan 31 '23 09:01

peak