How to fix "Error: spawn ENAMETOOLONG" (probably cause by jq) in javascript?

Question

I'm on windows 10, and I work with node.js, and I work with [email protected] (the lastest version at this moment)

I want to retrieve some information from a JSON with jq. So I use jq.run function with 'json' as option parameter. But it throw me an error, I know my queries are good, and all my code work without the line which start with "let AllKOMoves = ...".

My JSON are well formatted and I try this program in JMES Path and it's work well, so there no problem with JSON. I did some researches and some people said that my OS cannot support a too long request. I want to precise that on of my JSON weighs 6Mo.

I did some researches on internet and some of them talk about "Not specifying a .catch(…)" on this website: here and this one talk about the too long query: here

Here's my code I put the const jq = require('node-jq'); out of the async function.

(async ()=>{
    let name = "abomasnow"; 
    const fileContent = fs.readFileSync(`./Pokemon with move CSV/Queries/nonStatusMove/${name}nonStatusMove.csv`);
    let pokemonSubject = JSON.parse(await jq.run(`. | map(select(.name == "${name}"))[0]`, './nameBSTypeAbilityWeight.json'));
    var allMoves = moveToJSON(fileContent);
    let AllKOMoves = await jq.run(`. | map(select(.Moves | map(select(.Max == 100)) | length > 1)) | map({Nature: .Nature, EV: .EV, Moves: [.Moves | map(select(.Max == 100))[] | .Move], MovesCount: .Moves | length})`,getAllInflictedDamage(pokemonSubject, allMoves, pokemonJSON[0]),{ input: 'json' })
    console.log(AllKOMoves);
    })();

Here's a sample of the JSON generate by this portion of code getAllInflictedDamage(pokemonSubject, allMoves, pokemonJSON[0]):

[  
   {  
      "Opponent":"abomasnow",
      "Move":"wood-hammer",
      "Min":16.15,
      "Max":19.25,
      "Crit":28.88
   },
   {  
      "Opponent":"abomasnow",
      "Move":"round",
      "Min":15.84,
      "Max":18.94,
      "Crit":28.26
   }
]

Here's is a sample of the JSON contains in the variable allMoves:

[  
   {  
      "name":"bulldoze",
      "accuracy":"100",
      "category":"physical",
      "power":"60",
      "priority":"0",
      "target":"all-other-pokemon",
      "type":"ground"
   },
   {  
      "name":"frost-breath",
      "accuracy":"90",
      "category":"special",
      "power":"60",
      "priority":"0",
      "target":"selected-pokemon",
      "type":"ice"
   }
]

I expected the content of my JSON, but it throw this error message:

(node:21792) UnhandledPromiseRejectionWarning: Error: spawn ENAMETOOLONG
    at ChildProcess.spawn (internal/child_process.js:366:11)
    at Object.spawn (child_process.js:538:9)
    at C:\Users\adela\node_modules\node-jq\lib\exec.js:24:43
    at new Promise (<anonymous>)
    at exec (C:\Users\adela\node_modules\node-jq\lib\exec.js:20:10)
    at C:\Users\adela\node_modules\node-jq\lib\jq.js:24:24
    at new Promise (<anonymous>)
    at Object.run (C:\Users\adela\node_modules\node-jq\lib\jq.js:19:10)
    at C:\Users\adela\node_code\calcDamage1forAll.js:757:28
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:21792) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:21792) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Normally my code should retrieve information like this:

[  
   {  
      "Nature":"brave",
      "EV":[  
         4,
         0,
         0,
         0,
         252,
         252
      ],
      "Moves":[  
         "focus-punch",
         "natural-gift-fire90",
         "natural-gift-fire100",
         "natural-gift-fire80"
      ],
      "MovesCount":4
   },
   {  
      "Nature":"brave",
      "EV":[  
         0,
         252,
         252,
         4,
         0,
         0
      ],
      "Moves":[  
         "natural-gift-fire90",
         "natural-gift-fire100",
         "natural-gift-fire80"
      ],
      "MovesCount":3
   }
]

EDIT: If I only do this:

let AllKOMoves = await jq.run(`.`,getAllInflictedDamage(pokemonSubject, allMoves, pokemonJSON[0]),{ input: 'json' })

it also give me the same error !

EDIT2: If I do this:

let AllKOMoves = await jq.run(`.`,{"foo":"bar"},{ input: 'json' })

it work perfectly ! With some researches I think it's just about the size of json because my json size ~12Mo, I try with a 300 Ko json it doesn't work too so I think just my json are too big.

My question now is to know if it exists other solution that create a temporary files like it proposed in the first answer...

Answer 1

EDIT:

This has now been re-examined and addressed in node-jq 1.10.0 (pull #191) thanks to this question, so making sure the version of node-jq is up to date in your project's dependencies should prevent this error.

Background for node-jq pre-1.10.0 (or people looking into ENAMETOOLARGE problems)

The trouble with an input: besides 'file' is that it gets translated by node-jq versions before 1.10.0 into a command line like jq --null-input '[json] | [filter]' so your JSON and filter combined must be smaller than your OS's largest acceptable single argument and/or total arguments limit, on windows this may be as small as 8192 bytes. (Most UNIX-clones should fail with an E2BIG error instead around for arguments around 128-256k.)

To pass a large amount of data to jq through earlier versions of node-jq without it going directly through a command line argument you would need to write to a temp file with a name ending in .json to pass your JSON using {input:file} if it is sizeable.

While in theory, you could provide an array of [] for files, making jq wait for data over stdin, node-jq wrapped the subprocess, so it would be ugly to get a handle on its stdin. Censequently it needed to be fixed or one would need to write an alternate wrapper to pass the data to jq over stdin, like:

child_process.spawn... 
process.stdin.write('myJsonAsAstring')
process.stdin.end()

Further discussion is available in the original node-jq issue.