How to speedup java maven build on Google Cloud Build (100s of dependencies)

Question

I am using Google Cloud Build to build a java project which has 100s of dependencies. By default the local maven repository cache will be empty and it downloads all dependencies each time there is a build.

The google documentation only suggests "Caching directories with Google Cloud Storage" https://cloud.google.com/cloud-build/docs/speeding-up-builds but it takes a long time to sync 7000 files (which means the build is slower)

just one dependency is 5 files

repository/org/mockito
repository/org/mockito/mockito-core
repository/org/mockito/mockito-core/2.15.0
repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.jar
repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.jar.sha1
repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.pom
repository/org/mockito/mockito-core/2.15.0/mockito-core-2.15.0.pom.sha1
repository/org/mockito/mockito-core/2.15.0/_remote.repositories

An example cloudbuild.yaml file

steps:
- name: gcr.io/cloud-builders/gsutil
  args: ['rsync', '-r', 'gs://my-mavencache-bucket/repository', '.']

- name: 'gcr.io/$PROJECT_ID/mvn'
  args: ['package']
...

I would like to mount gs://my-mavencache-bucket at as a volume - but I dont see an option to do that

Answer 1

After much experimentation, this solution seems to work quite well. google-storage-wagon. This maven plugin provides a mechanism to read and publish maven artifacts from a google data bucket

Maven pom.xml contains

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
...
    <repositories>
        <repository>
            <id>my-repo-bucket-release</id>
            <url>gs://bucket-ave-build-artifact/external</url>
            <releases>
                <enabled>true</enabled>
                <!-- TODO figure out why checksums do not match when artifact pulled from GCP -->
                <checksumPolicy>ignore</checksumPolicy>
            </releases>
        </repository>
    </repositories>

    <distributionManagement>
        <snapshotRepository>
            <id>my-repo-bucket-snapshot</id>
            <url>gs://my-build-artifact-bucket/snapshot</url>
        </snapshotRepository>
        <repository>
            <id>my-repo-bucket-release</id>
            <url>gs://my-build-artifact-bucket/release</url>
        </repository>
    </distributionManagement>
...
        <extensions>
            <extension>
                <groupId>com.gkatzioura.maven.cloud</groupId>
                <artifactId>google-storage-wagon</artifactId>
                <!-- version 1.8 seems to produce exception, ticket logged -->
                <version>1.7</version>
            </extension>
        </extensions>

    </build>

and cloudbuild.yaml is simply

steps:
 - name: 'gcr.io/cloud-builders/mvn'
  # -X here simply for verbose maven debugging
  args: ['deploy', '-X']

this will:

1. maven publish artifacts to a data bucket gs://my-build-artifact-bucket/release
2. download external dependencies from gs://my-build-artifact-bucket/external (if they exist in this directory)