Using client-go to `kubectl apply` against the Kubernetes API directly with multiple types in a single YAML file

Question

I'm using https://github.com/kubernetes/client-go and all works well.

I have a manifest (YAML) for the official Kubernetes Dashboard: https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml

I want to mimic kubectl apply of this manifest in Go code, using client-go.

I understand that I need to do some (un)marshalling of the YAML bytes into the correct API types defined in package: https://github.com/kubernetes/api

I have successfully Createed single API types to my cluster, but how do I do this for a manifest that contains a list of types that are not the same? Is there a resource kind: List* that supports these different types?

My current workaround is to split the YAML file using csplit with --- as the delimiter

csplit /path/to/recommended.yaml /---/ '{*}' --prefix='dashboard.' --suffix-format='%03d.yaml'

Next, I loop over the new (14) parts that were created, read their bytes, switch on the type of the object returned by the UniversalDeserializer's decoder and call the correct API methods using my k8s clientset.

I would like to do this to programmatically to make updates to any new versions of the dashboard into my cluster. I will also need to do this for the Metrics Server and many other resources. The alternative (maybe simpler) method is to ship my code with kubectl installed to the container image and directly call kubectl apply -f -; but that means I also need to write the kube config to disk or maybe pass it inline so that kubectl can use it.

I found this issue to be helpful: https://github.com/kubernetes/client-go/issues/193 The decoder lives here: https://github.com/kubernetes/apimachinery/tree/master/pkg/runtime/serializer

It's exposed in client-go here: https://github.com/kubernetes/client-go/blob/master/kubernetes/scheme/register.go#L69

I've also taken a look at the RunConvert method that is used by kubectl: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/convert/convert.go#L139 and assume that I can provide my own genericclioptions.IOStreams to get the output?

It looks like RunConvert is on a deprecation path

I've also looked at other questions tagged [client-go] but most use old examples or use a YAML file with a single kind defined, and the API has changed since.

Edit: Because I need to do this for more than one cluster and am creating clusters programmatically (AWS EKS API + CloudFormation/eksctl), I would like to minimize the overhead of creating ServiceAccounts across many cluster contexts, across many AWS accounts. Ideally, the only authentication step involved in creating my clientset is using aws-iam-authenticator to get a token using cluster data (name, region, CA cert, etc). There hasn't been a release of aws-iam-authenticator for a while, but the contents of master allow for the use of a third-party role cross-account role and external ID to be passed. IMO, this is cleaner than using a ServiceAccount (and IRSA) because there are other AWS services the application (the backend API which creates and applies add-ons to these clusters) needs to interact with.

Edit: I have recently found https://github.com/ericchiang/k8s. It's definitely simpler to use than client-go, at a high-level, but doesn't support this behavior.

Answer 1

It sounds like you've figured out how to deserialize YAML files into Kubernetes runtime.Objects, but the problem is dynamically deploying a runtime.Object without writing special code for each Kind.

kubectl achieves this by interacting with the REST API directly. Specifically, via resource.Helper.

In my code, I have something like:

import (
    meta "k8s.io/apimachinery/pkg/api/meta"
    "k8s.io/cli-runtime/pkg/resource"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/restmapper"
    "k8s.io/apimachinery/pkg/runtime"
)

func createObject(kubeClientset kubernetes.Interface, restConfig rest.Config, obj runtime.Object) error {
    // Create a REST mapper that tracks information about the available resources in the cluster.
    groupResources, err := restmapper.GetAPIGroupResources(kubeClientset.Discovery())
    if err != nil {
        return err
    }
    rm := restmapper.NewDiscoveryRESTMapper(groupResources)

    // Get some metadata needed to make the REST request.
    gvk := obj.GetObjectKind().GroupVersionKind()
    gk := schema.GroupKind{Group: gvk.Group, Kind: gvk.Kind}
    mapping, err := rm.RESTMapping(gk, gvk.Version)
    if err != nil {
        return err
    }

    name, err := meta.NewAccessor().Name(obj)
    if err != nil {
        return err
    }

    // Create a client specifically for creating the object.
    restClient, err := newRestClient(restConfig, mapping.GroupVersionKind.GroupVersion())
    if err != nil {
        return err
    }

    // Use the REST helper to create the object in the "default" namespace.
    restHelper := resource.NewHelper(restClient, mapping)
    return restHelper.Create("default", false, obj, &metav1.CreateOptions{})
}

func newRestClient(restConfig rest.Config, gv schema.GroupVersion) (rest.Interface, error) {
    restConfig.ContentConfig = resource.UnstructuredPlusDefaultContentConfig()
    restConfig.GroupVersion = &gv
    if len(gv.Group) == 0 {
        restConfig.APIPath = "/api"
    } else {
        restConfig.APIPath = "/apis"
    }

    return rest.RESTClientFor(&restConfig)
}

Answer 2

I was able to get this working in one of my projects. I had to use much of the source code from kubectl's apply command to get it working correctly.

https://github.com/billiford/go-clouddriver/blob/master/pkg/kubernetes/client.go#L63