GO - Docker ask certificate on K8S container

Question

I use the following code with this lib

provider, err := oidc.NewProvider(ctx, providerURI)
if err != nil {
    log.Println(err)
}

While running it locally with same providerURI it works, I was able to get the provider successfully!

I deployed it to K8S with the exact same provider url (as env variable ) and debug it using port-forwarding , However, in k8S I got error and dont get the provider.

The error is:

2020/08/14 16:42:22 Get "https://ace.svar.com/.well-known/openid-configuration": x509: certificate signed by unknown authority

I've added the certificate to the image and verify it, I exec into k8s container after deploy and I see the server.crt file under /usr/local/share/ca-certificates/ path.

And still got the same error, any idea if I miss here something else ... Not sure if it really related to the OIDC lib or something more general..

FROM golang:1.14.7 AS builder
RUN go get github.com/go-delve/delve/cmd/dlv
ADD server.crt /usr/local/share/ca-certificates/server.crt
RUN chmod 644 /usr/local/share/ca-certificates/server.crt && update-ca-certificates
RUN mkdir /app

ADD . /app
WORKDIR /app
RUN CGO_ENABLED=0 GOOS=linux go build -gcflags="all=-N -l" -o main ./...

FROM debian:buster AS production
COPY --from=builder /app .
COPY --from=builder /go/bin/dlv /
COPY --from=builder /usr/local/share/ca-certificates/ /usr/local/share/ca-certificates/
EXPOSE 8000 40000
ENV SSL_CERT_DIR=/usr/local/share/ca-certificates/
ENV PORT=8000
CMD ["/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "./main"]

I got replay from the author of the GO-OIDC repository to try to use

https://godoc.org/github.com/coreos/go-oidc#ClientContext

Not sure how, any idea?

Answer 1

From oidc.ClientContext docs it shows how to pass in a custom http.Client:

myClient := &http.Client{}
ctx := oidc.ClientContext(parentContext, myClient)

// This will use the custom client
provider, err := oidc.NewProvider(ctx, "https://accounts.example.com")

providing a custom http.Client allows for custom TLS handling.

---

TLS Config

To create a http.Client with a specific CA-trust file, I employ these helper functions:

func tlsConf(trustfile string) (t *tls.Config, err error) {
    if trustfile == "" {
        // DON'T USE IN PRODUCTION (but handy for testing)
        t = &tls.Config{InsecureSkipVerify: true}
        return
    }

    pembody, err := ioutil.ReadFile(trustfile)
    if err != nil {
        err = fmt.Errorf("failed to load trust file %q: %w", trustfile, err)
        return
    }

    rootCAs := x509.NewCertPool()
    if ok := rootCAs.AppendCertsFromPEM(pembody); !ok {
        err = fmt.Errorf("invalid PEM file %q", trustfile)
        return

    }

    t = &tls.Config{RootCAs: rootCAs}
    return
}

and:

func httpCli(trustfile string) (hc *http.Client, err error) {
    tc, err := tlsConf(trustfile)
    if err != nil {
        return
    }
    hc = &http.Client{Transport: &http.Transport{TLSClientConfig: tc}}
    return
}

So to use the above with the OIDC package for a quick test:

hc, err := httpCli("") // DON'T USE IN PRODUCTION - will trust any TLS cert

ctx := oidc.ClientContext(parentContext, hc)
provider, err := oidc.NewProvider(ctx, "https://accounts.example.com")

If this works, then add the correct trust file to your app:

hc, err := httpCli("/usr/local/share/ca-certificates/server.crt"))

---

CA Trust

If your server.crt trust file is not working, you may have the wrong subject/issuer listed.

To know for sure, you can grab the trust cert (and optional signing chain) from any remote server (port 443 is the default https port):

echo | openssl s_client -connect ace.svar.com:443 -showcerts 2> /dev/null > ace.txt

Since I don't know what your infrastructure looks like, I'll use the example output from google.com:443:

---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=*.google.com
   i:/C=US/O=Google Trust Services/CN=GTS CA 1O1
-----BEGIN CERTIFICATE-----
MIIKIzCCCQugAwIBAgIQF9rkH7fB/M4IAAAAAE2d0TANBgkqhkiG9w0BAQsFADBC
MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMRMw
...
-----END CERTIFICATE-----

The s: indicates the subject of a cert - and you can see the server name is identified by the wildcard CN=*.google.com. If you see something similar within ace.txt - your server.crt should include these lines (starting at BEGIN CERTIFICATE and ending with END CERTIFICATE).

You may also note the i: line indicates the issuer cert name. If this is the same name as s: - then it is self-signed cert and you are done.

In the google.com:443 example the subject (s:) differs from the issuer (i:). So instead of trusting the subject cert - one can trust the issuer cert instead - allowing potentially multiple servers to be trust. Since the subject cert is signed by that issuer - the chain-of-trust is complete.

Answer 2

Golang standard ssl library is looking for certificates in the following directories: https://github.com/golang/go/blob/master/src/crypto/x509/root_unix.go#L18-L37 && https://github.com/golang/go/blob/master/src/crypto/x509/root_linux.go#L8-L15, if you want to look it up in the new location, you can use environment variables: SSL_CERT_FILE or SSL_CERT_DIR and set location of your certificate. So in your case it would be:

export SSL_CERT_DIR=/usr/local/share/ca-certificates/

and then run your application.