marklogic mlcp custom transform split aggregate document to multiple files

Question

I have a JSON "aggregate" file that I want to split up and ingest as multiple documents into MarkLogic using mlcp.

I want to transform the content during ingestion using javascript.

My JSON file looks something like this:

{
  "type": "FeatureCollection",
  "features": [
    {blobA}, {blobB}, {blobC} ......
    ]
 }

...and I want to run this file through MLCP so that each document contains an item in the array.

i.e. One document will contain {blobA}, another will contain {blobB}, and another will contain {blobC}....and so forth.

How do I write my custom .sjs transform module?

Answer 1

Check out the example here: http://docs.marklogic.com/guide/mlcp/import#id_26044

The original input document is expected to be of the following form:

{ uri: string,
  value: node
}

That is also the expected output form for each document. You'll also want your return to be of type document-node, since you want mlcp to split it up and ingest it as JSON documents.

So, your .sjs custom transform module will look something like this....

function splitFeatures(doc) {
  const features = doc.value.toObject().features;
  return xdmp.arrayValues(
    features.map(function(feature) {
      return {
        uri: '/path/itemhere-' + xdmp.random() + '.json',
        value: xdmp.toJSON(feature)
      }
    })
  );
}

exports.transform = splitFeatures;

As an aside, this is a useful resource when working with JSON in MarkLogic.