Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Maven is trying to install every jar from a private repository

Tags:

java

maven

amazon-s3

I have a private repository in S3 that I use to release a library that then I use in another project.

To release, I have this setup:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.7.0</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>

    <extensions>
        <extension>
            <groupId>com.allogy.maven.wagon</groupId>
            <artifactId>maven-s3-wagon</artifactId>
            <version>1.1.0</version>
        </extension>
    </extensions>
</build>

<distributionManagement>
    <repository>
        <id>projectx-aws-release</id>
        <name>ProjectX AWS Release Repository</name>
        <url>s3://projectx-support/maven2</url>
    </repository>
    <snapshotRepository>
        <id>projectx-aws-snapshots</id>
        <name>Projectx AWS Snapshot Repository</name>
        <url>s3://projectx-support/maven2</url>
    </snapshotRepository>
</distributionManagement>

and then, on the application that's using that library:

<build>
    ...

    <extensions>
        <extension>
            <groupId>com.allogy.maven.wagon</groupId>
            <artifactId>maven-s3-wagon</artifactId>
            <version>1.1.0</version>
        </extension>
    </extensions>
</build>

<repositories>
    <repository>
        <id>projectX-aws</id>
        <url>s3://projectX-support/maven2</url>
        <releases><enabled>true</enabled></releases>
        <snapshots><enabled>true</enabled></snapshots>
    </repository>
</repositories>

with .circleci/settings.xml having the credentials for the S3 repo:

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <servers>
        <server>
            <id>projectx-aws</id>
            <username>....</username>
            <password>....</password>
        </server>
    </servers>
</settings>

When running mvn --batch-mode --settings .circleci/settings.xml dependency:go-offline something odd happens. It quickly downloads some dependencies:

mvn --batch-mode --settings .circleci/settings.xml dependency:go-offline
[INFO] Scanning for projects...
Downloading: http://repo.maven.apache.org/maven2/com/allogy/maven/wagon/maven-s3-wagon/1.1.0/maven-s3-wagon-1.1.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/allogy/maven/wagon/maven-s3-wagon/1.1.0/maven-s3-wagon-1.1.0.pom (7 KB at 47.0 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/com/allogy/allogy-maven-pom-oss/2.0/allogy-maven-pom-oss-2.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/allogy/allogy-maven-pom-oss/2.0/allogy-maven-pom-oss-2.0.pom (10 KB at 808.4 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/com/google/guava/guava/15.0/guava-15.0.pom
Downloaded: http://repo.maven.apache.org/maven2/com/google/guava/guava/15.0/guava-15.0.pom (7 KB at 508.8 KB/sec)
.
.
.
Downloading: http://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/apache/16/apache-16.pom (16 KB at 1879.5 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.10/maven-dependency-plugin-2.10.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.10/maven-dependency-plugin-2.10.jar (157 KB at 12014.5 KB/sec)

then it shows this:

[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building projectx 1.0.0-beta.2
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] >>> maven-dependency-plugin:2.10:go-offline (default-cli) @ projectx >>>
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:resolve-plugins (resolve-plugins) @ projectx ---

and downloads some more deps:

Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/2.2.1/maven-artifact-2.2.1.pom (2 KB at 171.5 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven/2.2.1/maven-2.2.1.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven/2.2.1/maven-2.2.1.pom (22 KB at 2431.5 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/11/maven-parent-11.pom
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/11/maven-parent-11.pom (32 KB at 3516.6 KB/sec)
.
.
.
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.6/maven-resources-plugin-2.6.jar (29 KB at 4118.2 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2/maven-jar-plugin-2.2.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/2.2/maven-jar-plugin-2.2.jar (27 KB at 3252.7 KB/sec)
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.3/maven-install-plugin-2.3.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.3/maven-install-plugin-2.3.jar (23 KB at 2775.0 KB/sec)

but then it attempts to download jars that should be and are in Maven Central from the S3 repository:

[INFO] Plugin Resolved: copy-rename-maven-plugin-1.0.1.jar
Downloading: s3://projectx-support/maven2/com/coderplus/maven/plugins/copy-rename-maven-plugin/1.0.1/copy-rename-maven-plugin-1.0.1.pom
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/com/coderplus/maven/plugins/copy-rename-maven-plugin/1.0.1/copy-rename-maven-plugin-1.0.1.pom
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 27ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: http://repo.maven.apache.org/maven2/com/coderplus/maven/plugins/copy-rename-maven-plugin/1.0.1/copy-rename-maven-plugin-1.0.1.pom
Downloaded: http://repo.maven.apache.org/maven2/com/coderplus/maven/plugins/copy-rename-maven-plugin/1.0.1/copy-rename-maven-plugin-1.0.1.pom (5 KB at 418.0 KB/sec)
Downloading: s3://projectx-support/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 11ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: http://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
Downloaded: http://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom (7 KB at 712.8 KB/sec)
Downloading: s3://projectx-support/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 12ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar (86 KB at 9475.0 KB/sec)
Downloading: s3://projectx-support/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
[INFO] File permissions: Private
[INFO] Logged in - projectx-support
[INFO] Downloading: s3://projectx-support/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
[INFO] Logged off - projectx-support
[INFO] Transfers: 1 Time: 12ms Amount: 0 bytes Throughput: 0.000 bytes/s
Downloading: http://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar
Downloaded: http://repo.maven.apache.org/maven2/org/apache/maven/maven-project/2.0.9/maven-project-2.0.9.jar (119 KB at 11892.6 KB/sec)
Downloading: s3://projectx-support/maven2/org/apache/maven/maven-core/2.0.9/maven-core-2.0.9.jar

After a while of that, AWS starts to throw Connection refused errors (some protection system maybe?) and the dependency that is only present in the S3 repo is not downloaded:

Downloading: s3://projectx-support/maven2/tech/projectx/projectxcommon/1.0.0-SNAPSHOT/maven-metadata.xml
[WARN] s3://projectx-support/maven2 - Connection refused
[INFO] Logged off - projectx-support
[WARNING] Could not transfer metadata tech.projectx:projectxcommon:1.0.0-SNAPSHOT/maven-metadata.xml from/to projectx-aws (s3://projectx-support/maven2): Could not connect to repository
[WARNING] Failure to transfer tech.projectx:projectxcommon:1.0.0-SNAPSHOT/maven-metadata.xml from s3://projectx-support/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of projectx-aws has elapsed or updates are forced. Original error: Could not transfer metadata tech.projectx:projectxcommon:1.0.0-SNAPSHOT/maven-metadata.xml from/to projectx-aws (s3://projectx-support/maven2): Could not connect to repository
Downloading: s3://projectx-support/maven2/tech/projectx/projectxcommon/1.0.0-SNAPSHOT/projectxcommon-1.0.0-SNAPSHOT.pom
[WARN] s3://projectx-support/maven2 - Connection refused
[INFO] Logged off - projectx-support

which causes the compilation to fail:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17.929s
[INFO] Finished at: Wed Apr 18 16:03:19 UTC 2018
[INFO] Final Memory: 32M/1067M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project projectx: Could not resolve dependencies for project tech.projectx:projectx:jar:1.0.0-beta.2: Failed to collect dependencies for [tech.projectx:projectxcommon:jar:1.0.0-SNAPSHOT (compile), org.projectlombok:lombok:jar:1.16.20 (compile), org.springframework:spring-web:jar:5.0.5.RELEASE (compile), org.springframework.retry:spring-retry:jar:1.2.2.RELEASE (compile), com.fasterxml.jackson.core:jackson-databind:jar:2.9.5 (compile), org.kordamp.ikonli:ikonli-javafx:jar:2.1.1 (compile), org.kordamp.ikonli:ikonli-fontawesome5-pack:jar:2.1.1 (compile), net.harawata:appdirs:jar:1.0.1 (compile), io.sentry:sentry:jar:1.7.3 (compile), org.javassist:javassist:jar:3.22.0-CR2 (compile), junit:junit:jar:4.12 (test)]: Failed to read artifact descriptor for tech.projectx:projectxcommon:jar:1.0.0-SNAPSHOT: Could not transfer artifact tech.projectx:projectxcommon:pom:1.0.0-SNAPSHOT from/to projectx-aws (s3://projectx-support/maven2): Could not connect to repository: Couldn't initialize a SAX driver to create an XMLReader: SAX2 driver class org.apache.xerces.parsers.SAXParser not found -> [Help 1]

Any ideas what's going on? This seems to be non-deterministic. It comes and goes with changes that shouldn't have any effect on it, although it seems to be quite consistent in that re-building doesn't seem to change the outcome. I'm sure the credentials work because they are used in two projects and one is working and they are the same that were used in this project when it was working.

This has been happening since I switched from Gradle to Maven, but since that's a completely different system, it's not a very useful comparison I believe.

When I run mvn --batch-mode --settings .circleci/settings.xml dependency:go-offline install -e -X on my own computer, after deleting ~/.m2/repository, the output looks like this: https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-maven-local-machine-log and when I do the same in CircleCI, it looks like this: https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-maven-circleci-log

When listing the repositories present, following the answer at How can I ask Maven for a list of the default repositories?, I see this:

<repositories>
    <repository>
        <id>projectx-aws</id>
        <url>s3://projectx-support/maven2</url>
        <layout>default</layout>
        <locations class="linked-hash-map">
            <entry>
                <string></string>
                <inputLocation>
                    <lineNumber>461</lineNumber>
                    <columnNumber>21</columnNumber>
                    <source>
                        <modelId>tech.projectx:projectx:1.0.0-beta.2</modelId>
                        <location>C:\Users\pupeno\Documents\projectx\code\projectx\pom.xml</location>
                    </source>
                </inputLocation>
            </entry>
            <entry>
                <string>id</string>
                <inputLocation>
                    <lineNumber>462</lineNumber>
                    <columnNumber>17</columnNumber>
                    <source reference="../../../entry/inputLocation/source"/>
                </inputLocation>
            </entry>
            <entry>
                <string>url</string>
                <inputLocation>
                    <lineNumber>463</lineNumber>
                    <columnNumber>18</columnNumber>
                    <source reference="../../../entry/inputLocation/source"/>
                </inputLocation>
            </entry>
        </locations>
        <releases>
            <enabled>true</enabled>
            <locations class="linked-hash-map">
                <entry>
                    <string></string>
                    <inputLocation>
                        <lineNumber>464</lineNumber>
                        <columnNumber>23</columnNumber>
                        <source reference="../../../../../locations/entry/inputLocation/source"/>
                    </inputLocation>
                </entry>
                <entry>
                    <string>enabled</string>
                    <inputLocation>
                        <lineNumber>465</lineNumber>
                        <columnNumber>26</columnNumber>
                        <source reference="../../../../../locations/entry/inputLocation/source"/>
                    </inputLocation>
                </entry>
            </locations>
        </releases>
        <snapshots>
            <enabled>true</enabled>
            <locations class="linked-hash-map">
                <entry>
                    <string></string>
                    <inputLocation>
                        <lineNumber>467</lineNumber>
                        <columnNumber>24</columnNumber>
                        <source reference="../../../../../locations/entry/inputLocation/source"/>
                    </inputLocation>
                </entry>
                <entry>
                    <string>enabled</string>
                    <inputLocation>
                        <lineNumber>468</lineNumber>
                        <columnNumber>26</columnNumber>
                        <source reference="../../../../../locations/entry/inputLocation/source"/>
                    </inputLocation>
                </entry>
            </locations>
        </snapshots>
    </repository>
    <repository>
        <id>central</id>
        <name>Central Repository</name>
        <url>https://repo.maven.apache.org/maven2</url>
        <layout>default</layout>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
</repositories>

s3://projectx-support/maven2/tech/projectx/projectxcommon/1.0.0-SNAPSHOT/maven-metadata.xml is present and contains the following:

<?xml version="1.0" encoding="UTF-8"?>
<metadata modelVersion="1.1.0">
  <groupId>tech.projectx</groupId>
  <artifactId>projectxcommon</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <versioning>
    <snapshot>
      <timestamp>20180418.113836</timestamp>
      <buildNumber>5</buildNumber>
    </snapshot>
    <lastUpdated>20180418113836</lastUpdated>
    <snapshotVersions>
      <snapshotVersion>
        <extension>jar</extension>
        <value>1.0.0-20180418.113836-5</value>
        <updated>20180418113836</updated>
      </snapshotVersion>
      <snapshotVersion>
        <extension>pom</extension>
        <value>1.0.0-20180418.113836-5</value>
        <updated>20180418113836</updated>
      </snapshotVersion>
    </snapshotVersions>
  </versioning>
</metadata>

The dependency tree of my app looks like this:

> mvn dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------< tech.projectx:projectx >------------------------
[INFO] Building projectx 1.0.0-beta.2
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ projectx ---
[INFO] tech.projectx:projectx:jar:1.0.0-beta.2
[INFO] +- tech.projectx:projectxcommon:jar:1.0.0-SNAPSHOT:compile
[INFO] |  +- org.bouncycastle:bcprov-jdk15on:jar:1.59:compile
[INFO] |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.9.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.glassfish:javax.el:jar:3.0.1-b10:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.5.RELEASE:compile
[INFO] |  |  +- org.springframework:spring-aop:jar:5.0.5.RELEASE:compile
[INFO] |  |  \- org.springframework:spring-expression:jar:5.0.5.RELEASE:compile
[INFO] |  +- com.pubnub:pubnub-gson:jar:4.19.0:compile
[INFO] |  |  +- com.squareup.retrofit2:retrofit:jar:2.3.0:compile
[INFO] |  |  +- com.squareup.okhttp3:okhttp:jar:3.9.1:compile
[INFO] |  |  |  \- com.squareup.okio:okio:jar:1.13.0:compile
[INFO] |  |  +- com.squareup.okhttp3:logging-interceptor:jar:3.9.1:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.8.2:compile
[INFO] |  |  \- com.squareup.retrofit2:converter-gson:jar:2.3.0:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.5:compile
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.5:compile
[INFO] |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.5:compile
[INFO] +- org.projectlombok:lombok:jar:1.16.20:compile
[INFO] +- org.springframework:spring-web:jar:5.0.5.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.5.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.5.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.5.RELEASE:compile
[INFO] +- org.springframework.retry:spring-retry:jar:1.2.2.RELEASE:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.5:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.5:compile
[INFO] +- org.kordamp.ikonli:ikonli-javafx:jar:2.1.1:compile
[INFO] |  \- org.kordamp.ikonli:ikonli-core:jar:2.1.1:compile
[INFO] +- org.kordamp.ikonli:ikonli-fontawesome5-pack:jar:2.1.1:compile
[INFO] +- net.harawata:appdirs:jar:1.0.1:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] |  \- net.java.dev.jna:jna-platform:jar:4.5.0:compile
[INFO] |     \- net.java.dev.jna:jna:jar:4.5.0:compile
[INFO] +- io.sentry:sentry:jar:1.7.3:compile
[INFO] +- org.javassist:javassist:jar:3.22.0-CR2:compile
[INFO] \- junit:junit:jar:4.12:test
[INFO]    \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.188 s
[INFO] Finished at: 2018-04-23T20:38:39+01:00
[INFO] ------------------------------------------------------------------------

My .cicleci\config.yml file looks like this:

# Java Gradle CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-java/ for more details
#
version: 2
jobs:
  build:
    docker:
      # specify the version you desire here
      - image: circleci/jdk8:0.1.1

      # Specify service dependencies here if necessary
      # CircleCI maintains a library of pre-built images
      # documented at https://circleci.com/docs/2.0/circleci-images/
      # - image: circleci/postgres:9.4

    working_directory: ~/repo

    environment:
      # Customize the JVM maximum heap limit
      JVM_OPTS: -Xmx3200m
      TERM: dumb

    steps:
       - checkout

       - run: apt-get update

       - run: apt-get install --assume-yes maven

       - restore_cache:
           key: projectx-{{ checksum "pom.xml" }}

       - run: mvn --batch-mode --settings .circleci/settings.xml dependency:go-offline install -e -X

       - save_cache:
           paths:
             - ~/.m2
           key: projectx-{{ checksum "pom.xml" }}

       - run: mvn --batch-mode --settings .circleci/settings.xml test

I have logging enabled for this bucket and one entry looks like this (with the expected 404):

https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-2018-04-23-21-10-39-7467973073c586b9

These are the last 3 accesses to maven-metadata.xml:

  • https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-2018-04-20-15-59-22-60727568fac6b5db
  • https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-2018-04-21-07-43-41-4fec5248b2163d87
  • https://gist.github.com/pupeno/da4f53a6e9673ff611909021306191a4#file-2018-04-23-15-56-21-0b070f7c98f2e22e
like image 262
pupeno Avatar asked Apr 10 '18 03:04

pupeno

People also ask

How do I download dependencies from private repository in Maven?

If the dependencies are found in 'R' then "~/. m2" will get it from 'R'. If the dependencies are not found in 'R' then these dependencies are downloaded from the central repositories and keep the dependencies in 'R' for further uses. After that '~/.

What command should one use to install jar files to the local repository?

Using the mvn Command You can install the JAR file to the local Maven repository, which is specific to the current user on the current machine.

Video Answer

2 Answers

Probably main problem was with too many requests to Amazon S3 repository.

By analysing logs we can see over 120 request of downloading files from Amazon S3 in short time 1-3 seconds:

  • Maven is downloading artefacts that don't exist in Amazon S3, then download it from Maven Central, those requests to Amazon S3 are useless
  • Every download needs a login and logout request, that's strange!
  • Last 40 request are connection refused
  • Last 40 request don't have [INFO] Logged in - projectx-support statement for them, so maybe some limits of log in in short time?

To solve the problem, there are atleast 2 solutions:

  1. Force maven to download artefact FIRSTLY from Maven Central, then rest from Amazon S3, to decrease requests to Amazon S3. This is a good solutions, but remember if you will have more artefacts to download only from Amazon S3 you will get same problem!

    By default Maven Central comes LAST in repository, to set highest priority and download first from Maven Center, we have to add it manually in FIRST position in our repositories, because the order of the repositories will also DECIDE the order of the repository access. (Source):

    <repositories>
    <repository>
        <id>central</id>
        <name>Central Repository</name>
        <url>https://repo.maven.apache.org/maven2</url>
        <layout>default</layout>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>projectX-aws</id>
        <url>s3://projectX-support/maven2</url>
        <releases><enabled>true</enabled></releases>
        <snapshots><enabled>true</enabled></snapshots>
    </repository>
</repositories>
  2. Find a way to change limits of requests by IP on Amazon S3 repository. This solution will always works but your Amazon S3 will get many useless requests.

So the best way to solve the problem is to mix this two solutions. :)

like image 184
Maciej Pulikowski Avatar answered Oct 16 '22 11:10

Maciej Pulikowski

The problem is that Maven Central as defined by Maven itself, in the super POM, always comes last in the repository order, so, projectx's S3 repo is being hit by many requests that fail and eventually AWS S3 temporarily blocks the client, before the projectxcommon jar can be downloaded.

The only way to have Maven Central before the S3 repository is manually defining it in your pom.xml before the projectx-aws repo:

<repositories>
    <repository>
        <id>central</id>
        <name>Central Repository</name>
        <url>http://repo.maven.apache.org/maven2</url>
        <layout>default</layout>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
    <repository>
        <id>projectX-aws</id>
        <url>s3://projectX-support/maven2</url>
        <releases><enabled>true</enabled></releases>
        <snapshots><enabled>true</enabled></snapshots>
    </repository>
</repositories>

More information about setting multiple repositories and their order can be found at:

  • Setting up Multiple Repositories
  • Repository search order
  • https://stackoverflow.com/a/5325685/4704510

Some answers in Stack Overflow mention alphabetical order. At least in Maven 3 that is not true and only the order in which they are defined matters.

like image 1
UsamaAmjad Avatar answered Oct 16 '22 11:10

UsamaAmjad
Sign in to Comment

Related questions

Application crashes only in release version
How to make option menu appear on bottom of the screen?
Task com.google.firebase.a.v rejected from java.util.concurrent.ThreadPoolExecutor
Spring RedisConnectionFactory with transaction not returning connection to Pool and then blocks when exhausted
Java Swing app looks tiny on high-DPI screen when it should be scaled to normal size
What are the differences between javadoc.jar, sources.jar and .jar?
How to fix "Failed to notify build listener" error?
Is it dead code in LinkedHashMap in JDK11？
Does log4j security violation vulnerability affect log4net?
Java Portals and Portlets
Spring MVC Controller Design
Write-Behind Cache for JPA
Java CSV parser with string separator (multi-character)
Deploying Maven artifact to multiple repositories with different settings
Why hasn't EventListenerList been replaced? (Or: what are the pitfalls in replacing it?)
Couldn't Expand RemoteViews - Bad notification
Eclipse build loop caused by protobuf generated code (related to Maven Project Builder)
Java 8 functional interface ambiguous reference (is this a bug?)
Live video streaming in java [duplicate]
Ubuntu, JavaFX, A fatal error has been detected by the Java Runtime Environment

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!