“500 Internal Server Error” with job artifacts on minio

Question

I'm running gitlab-ce on-prem with min.io as a local S3 service. CI/CD caching is working, and basic connectivity with the S3-compatible minio is good. (Versions: gitlab-ce:13.9.2-ce.0, gitlab-runner:v13.9.0, and minio/minio:latest currently c253244b6fb0.)

Is there additional configuration to differentiate between job-artifacts and pipeline-artifacts and storing them in on-prem S3-compatible object storage?

In my test repo, the "build" stage builds a sparse R package. When I was using local in-gitlab job artifacts, it succeeds and moves on to the "test" and "deploy" stages, no problems. (And that works with S3-stored cache, though that configuration is solely within gitlab-runner.) Now that I've configured minio as a local S3-compatible object storage for artifacts, though, it fails.

...
Created cache
Uploading artifacts for successful job
Uploading artifacts...
/builds/git/mygroup/citest/ci/build/*.tar.gz: found 1 matching files and directories 
/builds/git/mygroup/citest/ci/lib: found 67 matching files and directories 
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=397 responseStatus=500 Internal Server Error status=500 token=q42snHs9
FATAL: invalid argument                            
Cleaning up file based variables
ERROR: Job failed: exit code 1

The only fail is in the artifact-storage attempt. (With in-gitlab artifacts, this all succeeded and worked.)

I can find no reference of "invalid argument" within the minio logs (which say nothing during this time). I'm not certain that it is a minio-problem.

The relevant portion of the gitlab config:

gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'host' => "minio.mydomain.com",
  # 'region' => '',
  'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
  'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>',
  'path_style' => true
}
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts-storage'
gitlab_rails['object_store']['objects']['external_diffs']['enabled'] = false
gitlab_rails['object_store']['objects']['lfs']['enabled'] = false
gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads-storage'
gitlab_rails['object_store']['objects']['packages']['enabled'] = false
gitlab_rails['object_store']['objects']['dependency_proxy']['enabled'] = false
gitlab_rails['object_store']['objects']['terraform_state']['enabled'] = false
gitlab_rails['object_store']['objects']['pages']['enabled'] = false

That configuration is adapted from https://docs.gitlab.com/ee/administration/object_storage.html, deselecting storage of components I don't think I need. I added 'path_style' => true because without it, the default of bucket.minio.mydomain.com was not resolved correctly (so this is mostly-S3-compatible, not perfectly AWS). I have also tried this with 'proxy_download' of true, no change.

I'm logged into the minio console, and the gitlab-artifacts-storage bucket exists. In fact, after these failed "build" tests, I'm seeing newly created job artifacts stored in this bucket (as job.log), so I know that basic connectivity (i.e., access key and secret) works:

In my research, I've seen similar errors linked to nginx rev-proxy issues, which suggests that something in the traefik configuration (or just its presence) might be an issue. It is merely passing traffic, it is doing no path-translation/stripping. Regardless, all of the rev-proxy discussion was on nginx, often attempting path-munging of some sort, and was resolved with some seemingly-unrelated change to the nginx configuration. I haven't found any that map into the traefik domain.

traefik access.log

{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44742","ClientHost":"172.19.0.1","ClientPort":"44742","ClientUsername":"-","DownstreamContentSize":329,"DownstreamStatus":200,"DownstreamStatusLine":"200 OK","Duration":3472517,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":329,"OriginDuration":3422850,"OriginStatus":200,"OriginStatusLine":"200 OK","Overhead":49667,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131470,"RequestHost":"minio.mydomain.com","RequestLine":"POST /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?uploads HTTP/1.1","RequestMethod":"POST","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?uploads","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:53.890788923Z","StartUTC":"2021-03-18T20:37:53.890788923Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"329","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:53 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45D8DDC4B3","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"329","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:53 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45D8DDC4B3","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203753Z","time":"2021-03-18T20:37:53Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44748","ClientHost":"172.19.0.1","ClientPort":"44748","ClientUsername":"-","DownstreamContentSize":467,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1571234,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":467,"OriginDuration":1525159,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":46075,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131472,"RequestHost":"minio.mydomain.com","RequestLine":"GET /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203753Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature HTTP/1.1","RequestMethod":"GET","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203753Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.090024071Z","StartUTC":"2021-03-18T20:37:54.090024071Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"467","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45E4BD6115","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"467","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45E4BD6115","origin_X-Xss-Protection":"1; mode=block","request_User-Agent":"Go-http-client/1.1","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44754","ClientHost":"172.19.0.1","ClientPort":"44754","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1205368,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1145885,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":59483,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131473,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.162829022Z","StartUTC":"2021-03-18T20:37:54.162829022Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45E9114D33","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45E9114D33","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44758","ClientHost":"172.19.0.1","ClientPort":"44758","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1087332,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1031618,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":55714,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131474,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.206750001Z","StartUTC":"2021-03-18T20:37:54.206750001Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EBAE7A4E","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EBAE7A4E","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44762","ClientHost":"172.19.0.1","ClientPort":"44762","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1126408,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1068170,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":58238,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131475,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.243629562Z","StartUTC":"2021-03-18T20:37:54.243629562Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EDE0062E","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EDE0062E","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44766","ClientHost":"172.19.0.1","ClientPort":"44766","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1279861,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":1227773,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":52088,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131476,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6 HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099873-1316-0002-0683-2dfe06e7a451447ff0b4a5518c8e19c6","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:54.277572935Z","StartUTC":"2021-03-18T20:37:54.277572935Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:54 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A45EFE57F90","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:54 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A45EFE57F90","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203754Z","time":"2021-03-18T20:37:54Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44774","ClientHost":"172.19.0.1","ClientPort":"44774","ClientUsername":"-","DownstreamContentSize":329,"DownstreamStatus":200,"DownstreamStatusLine":"200 OK","Duration":4317260,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":329,"OriginDuration":4261951,"OriginStatus":200,"OriginStatusLine":"200 OK","Overhead":55309,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131480,"RequestHost":"minio.mydomain.com","RequestLine":"POST /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?uploads HTTP/1.1","RequestMethod":"POST","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?uploads","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:55.503920248Z","StartUTC":"2021-03-18T20:37:55.503920248Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"329","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:55 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A4639004631","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"329","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:55 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A4639004631","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203755Z","time":"2021-03-18T20:37:55Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44778","ClientHost":"172.19.0.1","ClientPort":"44778","ClientUsername":"-","DownstreamContentSize":467,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":1879630,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":467,"OriginDuration":1830988,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":48642,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131482,"RequestHost":"minio.mydomain.com","RequestLine":"GET /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203755Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature HTTP/1.1","RequestMethod":"GET","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a?X-Amz-Expires=15300\u0026X-Amz-Date=20210318T203755Z\u0026X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=MyS3AccessKey%2F20210318%2F%2Fs3%2Faws4_request\u0026X-Amz-SignedHeaders=host\u0026X-Amz-Signature=SomeSignature","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:55.703174362Z","StartUTC":"2021-03-18T20:37:55.703174362Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"467","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Content-Type":"application/xml","downstream_Date":"Thu, 18 Mar 2021 20:37:55 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A4644E482E7","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"467","origin_Content-Security-Policy":"block-all-mixed-content","origin_Content-Type":"application/xml","origin_Date":"Thu, 18 Mar 2021 20:37:55 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A4644E482E7","origin_X-Xss-Protection":"1; mode=block","request_User-Agent":"Go-http-client/1.1","time":"2021-03-18T20:37:55Z"}
{"BackendAddr":"172.19.0.2:9000","BackendName":"backend-minio-myswarm","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.19.0.2:9000","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":""},"ClientAddr":"172.19.0.1:44782","ClientHost":"172.19.0.1","ClientPort":"44782","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":404,"DownstreamStatusLine":"404 Not Found","Duration":2076600,"FrontendName":"Host-minio-mydomain-com-2","OriginContentSize":0,"OriginDuration":2009920,"OriginStatus":404,"OriginStatusLine":"404 Not Found","Overhead":66680,"RequestAddr":"minio.mydomain.com","RequestContentSize":0,"RequestCount":131484,"RequestHost":"minio.mydomain.com","RequestLine":"HEAD /gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a HTTP/1.1","RequestMethod":"HEAD","RequestPath":"/gitlab-artifacts-storage/tmp/uploads/1616099875-1422-0002-1921-4427532294f444d4582d12ec3b75ba3a","RequestPort":"-","RequestProtocol":"HTTP/1.1","RetryAttempts":0,"StartLocal":"2021-03-18T20:37:56.652353505Z","StartUTC":"2021-03-18T20:37:56.652353505Z","downstream_Accept-Ranges":"bytes","downstream_Content-Length":"0","downstream_Content-Security-Policy":"block-all-mixed-content","downstream_Date":"Thu, 18 Mar 2021 20:37:56 GMT","downstream_Referrer-Policy":"same-origin","downstream_Server":"MinIO","downstream_Strict-Transport-Security":"max-age=315360000","downstream_Vary":"Origin","downstream_X-Amz-Request-Id":"166D8A467D7A9332","downstream_X-Xss-Protection":"1; mode=block","level":"info","msg":"","origin_Accept-Ranges":"bytes","origin_Content-Length":"0","origin_Content-Security-Policy":"block-all-mixed-content","origin_Date":"Thu, 18 Mar 2021 20:37:56 GMT","origin_Referrer-Policy":"same-origin","origin_Server":"MinIO","origin_Strict-Transport-Security":"max-age=315360000","origin_Vary":"Origin","origin_X-Amz-Request-Id":"166D8A467D7A9332","origin_X-Xss-Protection":"1; mode=block","request_Authorization":"AWS4-HMAC-SHA256 Credential=MyS3AccessKey/20210318//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=SomeSignature","request_Content-Length":"0","request_User-Agent":"fog-core/2.1.0","request_X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","request_X-Amz-Date":"20210318T203756Z","time":"2021-03-18T20:37:56Z"}

I understand there is a difference between job artifacts and pipeline artifacts, so I think what I'm missing is a configuration distinction between "job" and "pipeline".

.gitlab-ci.yml snippet

variables:
  GIT_DEPTH: 10
  R_LIBS_USER: "$CI_PROJECT_DIR/ci/lib"
  BUILD_DIR: "$CI_PROJECT_DIR/ci/build"
  CHECK_DIR: "$CI_PROJECT_DIR/ci/logs"
  BUILD_LOGS_DIR: "$CI_PROJECT_DIR/ci/logs/$CI_PROJECT_NAME.Rcheck"

default:
  image: rocker/shiny-verse:4.0.3
  interruptible: true

build-package:
  stage: build
  script:
    - mkdir -p "$R_LIBS_USER" "$BUILD_DIR"
    - R -e '
        devtools::install_deps(dependencies = TRUE, lib = Sys.getenv("R_LIBS_USER")) ;
        devtools::build(path = Sys.getenv("BUILD_DIR")) ;'
  artifacts:
    paths:
      - $BUILD_DIR/*.tar.gz
      - $R_LIBS_USER
  cache:
    key: "${CI_COMMIT_REF_SLUG}__cilib"
    paths:
      - $R_LIBS_USER

(Yes, it is both cached and an artifact. This is an instance of me testing the CI subsystem, not something I intend to maintain.)

This is in a docker-swarm behind a traefik reverse-proxy (and SSL terminator).

Answer 1

The answer is to bypass the empty-string test; the underlying protocol does not support region-less configuration, nor is there a configuration option to support it.

The trick is able to work because the use of 'endpoint' causes the 'region' to be ignored. With that, setting the region to something and forcing the endpoint allows it to work:

gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
  'host' => "minio.mydomain.com",
  'region' => 'us-east-1',                      # this must be non-empty, but is ignored ...
  'endpoint' => 'https://minio.mydomain.com',   # ... because of 'endpoint'
  'aws_access_key_id' => '<AWS_ACCESS_KEY_ID>',
  'aws_secret_access_key' => '<AWS_SECRET_ACCESS_KEY>',
  'path_style' => true
}

(I owe discovery of this to Florian, in gitlab-org/gitlab#297227.)