Not able to configure Elasticsearch snapshot repository using OCI Amazon S3 Compatibility API

Question

My Elasticsearch7.8.0 is running in OCI OKE (Kubernetes running in Oracle Cloud). I want to setup Elasticsearch backup snapshot with OCI Object store using OCI Amazon S3 Compatibility API. Added repository-s3 plugin and configured ACCESS_KEY and SECRET_KEY in the PODs. While repository, I am getting "s_s_l_peer_unverified_exception"

PUT /_snapshot/s3-repository

{
  "type": "s3",
  "settings": {
    "client": "default",
    "region": "OCI_REGION",
    "endpoint": "OCI_TENANCY.compat.objectstorage.OCI_REGION.oraclecloud.com",
     "bucket": "es-backup"    
  }
}

Respose :

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[s3-repository] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[s3-repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object [tests-0J3NChNRT9WIQJknHAssKg/master.dat] using a single upload",
      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "Unable to execute HTTP request: Certificate for <es-backup.OCI_TENANCY.compat.objectstorage.OCI_REGION.oraclecloud.com> doesn't match any of the subject alternative names: [swiftobjectstorage.us-ashburn-1.oraclecloud.com]",
        "caused_by" : {
          "type" : "s_s_l_peer_unverified_exception",
          "reason" : "Certificate for <es-backup.OCI_TENANCY.compat.objectstorage.OCI_REGION.oraclecloud.com> doesn't match any of the subject alternative names: [swiftobjectstorage.us-ashburn-1.oraclecloud.com]"
        }
      }
    }
  },
  "status" : 500
}

Answer 1

Well you shouldn't be talking to es-backup.OCI_TENANCY.compat.objectstorage.OCI_REGION.oraclecloud.com where your bucket name is part of the domain. You can try it in your browser and you'll get a similar security warning about certs.

If you look at https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm#usingAPI you'll see a mention of:

The application must use path -based access. Virtual host-style access (accessing a bucket as bucketname.namespace.compat.objectstorage.region.oraclecloud.com) is not supported.

AWS is migrating from path based to sub-domain based URLs for S3 (https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/) so the ES S3 plugin is probably defaulting to doing things the new AWS way.

Does it make a difference if you use an https:// URL for the endpoint value? Looking at my 6.8 config I have something like:

{
  "s3-repository": {
    "type": "s3",
    "settings": {
      "bucket": "es-backup",
      "client": "default",
      "endpoint": "https://{namespace}.compat.objectstorage.us-ashburn-1.oraclecloud.com/",
      "region": "us-ashburn-1"
    }
  }
}

What I'm guessing is that having a full URL for the endpoint probably sets the protocol and path_style_access or 6.8 didn't require you to set path_style_access to true but 7.8 might. Either way, try a full URL or setting path_style_access to true. Relevant docs at https://www.elastic.co/guide/en/elasticsearch/plugins/master/repository-s3-client.html