Antora playbook execution crashes when having multiple source repositories

Question

I use Antora 2.3 since sometime now and I had to introduce more source repositories into my playbook.yaml file as the documentation-site grew.

Currently, I have 50-60 Git repos, each of size around 15 MB, mentioned in the playbook. Afterwards, my site generation began to crash while parallelly cloning the repositories mentioned in my content.sources[] of playbook.yaml (i.e. exits from the Antora command in Linux shell without any errors).

I tried allocating more memory and received same above issue

node --max-old-space-size=16384 `which antora` --cache-dir=./.cache/antora --generator custom-generate-site playbook.yaml --stacktrace 

Answer 1

As the solution we have overridden the playbook build (generate-site.js module). We have split the main playbook into small set of playbooks which are easy to swallow. We call them as canary playbooks.

async function buildCanaryPlaybooks(playbook, sourcesPerPb = 3) {
  let canaryPlaybooks = {};
  if(playbook.content.sources.length > sourcesPerPb){
    let sources = playbook.content.sources;
    for(let i=0,j=sources.length; i < j; i+=sourcesPerPb){
        let chunk = sources.slice(i, i+ sourcesPerPb)
        let canaryPlaybook = getNewPlaybook(playbook,chunk)
        canaryPlaybooks[i] = canaryPlaybook
    }
  }
  return canaryPlaybooks;
}
function getNewPlaybook(playbook, sources) {
  let nc = Object.assign({}, playbook.content, sources:sources })
  return Object.assign({}, playbook, { content: nc })
}