Preface

Having self service capabilities for cluster provisioning is a great thing for many organizations, but deploying a cluster is not enough.

We need the ability to have certain software installed in every cluster, before our end user begins working with it.

This could be infra services like an Ingress Controller, External DNS or Cert Manager. It could also be monitoring tools like Prometheus or Grafana, logging tools like FluentBit or Loki, and many times it could be security tooling like Aqua or Prisma.

Many organizations due to the complexity of this setup, decide to have some sort of pipeline that provisions clusters, installs software imperatively, and then sends a notification to the requesting user with the details for connecting to this cluster.

In this post, we will see how this can be achieved in a declarative approach, without custom automation, using TMC.

TMC Continuous Delivery – Initial Release

TMC back in July 2022 added a new feature "Continuous Delivery".
This feature is in essence a wrapper implementation of FluxCD, which is one the 2 most common GitOps controllers alongside ArgoCD.

TMC in the initial release of this feature supported the ability to define raw YAML manifests or Kustomize configurations to be deployed to a cluster from a Git repository.

The configuration of what to deploy was done at the cluster level. We also needed to first enable the service on a cluster manually, as it is not deployed by default when a cluster is created or attached to TMC.

While this was a great feature, it simply did not solve many issues we still had, one of which is the title and subject of this post.

TMC Continuous Delivery – January 31st 2023

On January 31st 2023, VMware released a major update to TMC which included 2 key features that are relevant to this post:

1. Added support for installing a Helm chart from a Git repository
2. Added support for continuous delivery to cluster groups

Lets take a look into what each of these features includes.

Helm Chart Support

FluxCD supports mainly 2 deployment mechanisms, Kustomize and Helm. Kustomize is a very common tool, and the integration in FluxCD allows us to deploy either raw YAML definitions or Kustomize configurations to our clusters. While kustomize is a great solution in some cases, most software we consume from vendors or OSS projects today are packaged as helm charts.

FluxCD supports the deployment of helm charts that are sourced from Git repositories, and in this new release of TMC, we now have the ability to configure Helm Releases as well, just like we would Kustomizations.

This opens up additional options of what we can easily integrate into deployment and management flows using TMC.

While adding in support for Helm is great, just like Kustomize in the past, it requires us to enable the service on a cluster by cluster basis, and the configuration is also at the cluster level only.

Cluster Group Continuous Delivery

The main feature in my mind that will give huge value to customers from the January 31st release, is this feature.

In TMC we have always had the concept of a Cluster Group, which is a grouping of clusters under a single logical entity, that we can apply policies to such as security policies, RBAC and much more.

In this release, we now have the ability to enable Continuous Delivery at the Cluster Group level. This means we can now say that any cluster that gets added to a specific cluster group, will automatically have the CD service enabled.

We also have the ability to define Kustomization configurations at the Cluster Group level, which will be automatically applied to all clusters in the relevant Cluster Group!

This feature opens up some really awesome capabilities.

Installing Carvel Packages on all clusters

TMC beyond providing FluxCD also supports my favorite package management solution Carvel.

Not only does TMC install and provide the APIs and UI for installing Carvel packages, it also configures the "Tanzu Standard" package repository which includes common tooling needed for kubernetes clusters such as Contour, Cert Manager, Harbor, Prometheus, Grafana, FluentBit, External DNS and more.

The great thing is, is that these packages are supported by VMware!

The difficulty is that just like with the CD service, the package management capabilities are cluster scoped.

While this is true, in the end, Carvel package management is all kubernees resources in the end.

Recently I have begun playing around with deploying carvel package installations declaratively instead of doing it imperatively via the Tanzu CLI or the TMC GUI, and doing so actually offers us more advanced capabilities and fine grained control, as we can utilize advanced fields in the resource specs, that we cant configure via the Tanzu CLI or via the TMC GUI.

What gets create when we install a package via TMC

When we create a package installation, 4 or 5 resources are created for us:

1. Service Account – A service account that is used to perform the installation.
2. Cluster Role – A cluster role with Cluster Admin credentials used to allow the installation of the packages resources.
3. Cluster Role Binding – A Cluster Role Binding which binds our new service account, with the new Cluster Role from above.
4. Package Install – The actual package install resource.
5. (optional) Secret – A secret containing the values file we want to use for the installation.

How do we do this declaratively

The first step, is to create the relevant manifests. In this example we will configure the contour package from the Tanzu Standard Repository. While not needed, in the example bellow i have added annotations to the resources, to match what the Tanzu CLI does, as well as named the resources in the same manner. This just seems like a beneficial thing to do in my mind, but is not a requirement.

• First is the Service Account YAML:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: contour-tkg-packages
  name: contour-tkg-packages-sa
  namespace: tkg-system

• Next is the Cluster Role YAML

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: contour-tkg-system
  name: contour-tkg-system-cluster-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'

• Next is the Cluster Role Binding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: contour-tkg-system
  name: contour-tkg-system-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: contour-tkg-system-cluster-role
subjects:
- kind: ServiceAccount
  name: contour-tkg-system-sa
  namespace: tkg-system

• Next is the Secret with the data values for contour on vSphere with NSX ALB

apiVersion: v1
kind: Secret
metadata:
  name: contour-tkg-system-values
  namespace: tkg-system
type: Opaque
stringData:
  contour-data-values.yaml: |
    infrastructure_provider: vsphere
    contour:
      configFileContents: {}
      useProxyProtocol: false
      replicas: 2
      pspNames: "vmware-system-restricted"
      logLevel: info
    envoy:
      service:
        type: LoadBalancer
        annotations: {}
        nodePorts:
          http: null
          https: null
        externalTrafficPolicy: Cluster
        disableWait: false
      hostPorts:
        enable: false
      hostNetwork: false
      terminationGracePeriodSeconds: 300
      logLevel: info
      pspNames: null
    certificates:
      duration: 8760h
      renewBefore: 360h

• Finally is the Package Install itself

apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package-ClusterRole: contour-tkg-system-cluster-role
    tkg.tanzu.vmware.com/tanzu-package-ClusterRoleBinding: contour-tkg-system-cluster-rolebinding
    tkg.tanzu.vmware.com/tanzu-package-Secret: contour-tkg-system-values
    tkg.tanzu.vmware.com/tanzu-package-ServiceAccount: contour-tkg-system-sa
  name: contour
  namespace: tkg-system
spec:
  packageRef:
    refName: contour.tanzu.vmware.com
    versionSelection:
      constraints: 1.20.2+vmware.1-tkg.1
      prereleases: {}
  serviceAccountName: contour-tkg-system-sa
  values:
  - secretRef:
      name: contour-tkg-system-values

But why Cluster Admin???????

Indeed, all Tanzu Package installation performed via TMC or Tanzu CLI, use Cluster Admin permissions. This is because they don’t know what permissions any specific package may or may not need, in order to be installed. While that may be acceptable for some, I wanted to perform least privilege concepts, and that is where I found a great tool called audit2rbac.

What is Audit 2 RBAC

Audit 2 RBAC is an amazing tool put together by Jordan Liggitt. The tool can auto generate RBAC resource YAMLs for a specific user or service account based on a kubernetes Audit log. Luckily, TKG which is the distribution I am using, enables us to have audit logging enabled on our clusters. Once you have an audit log file, you simply pass the file to audit2rbac and tell the CLI tool, what user or service account you are interested in, and it will create the YAML definitions for the relevant RBAC needs.

Using audit2rbac for the Contour Package

After installing the contour package in one of my clusters I copied the audit.log file from the Control Plane node in my cluster to my local machine where I have audit2rbac installed.

The audit log in TKG is configured to be placed at the path /var/log/kubernetes/audit.log on the Control Plane nodes.

The next step was to simply run the following command:

audit2rbac -f audit.log --serviceaccount tkg-system:contour-tkg-system-sa \
  --generate-annotations tkg.tanzu.vmware.com/tanzu-package=contour-tkg-system \
  --generate-labels="" \
  --generate-name contour-tkg-system > contour-pkgi-rbac.yaml

If we take a look at what we receive as output we can look at the file we just created and see that it created 4 resources:

• Cluster Role

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: contour-tkg-system
  name: contour-tkg-system
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - create
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  - nodes
  - pods
  - serviceaccounts
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resourceNames:
  - tanzu-system-ingress
  resources:
  - namespaces
  verbs:
  - create
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - create
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - apps
  resources:
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - cert-manager.io
  resources:
  - certificates
  - issuers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  - clusterroles
  verbs:
  - create
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  - roles
  verbs:
  - get
  - list
  - watch

• Cluster Role Binding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: contour-tkg-system
  name: contour-tkg-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: contour-tkg-system
subjects:
- kind: ServiceAccount
  name: contour-tkg-system-sa
  namespace: tkg-system

• Role

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: contour-tkg-system
  name: contour-tkg-system
  namespace: tanzu-system-ingress
rules:
- apiGroups:
  - ""
  resourceNames:
  - tanzu-system-ingress
  resources:
  - namespaces
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  - services
  verbs:
  - create
  - get
  - patch
  - update
- apiGroups:
  - apps
  resourceNames:
  - envoy
  resources:
  - daemonsets
  verbs:
  - create
  - get
  - patch
  - update
- apiGroups:
  - apps
  resourceNames:
  - contour
  resources:
  - deployments
  verbs:
  - create
  - get
  - patch
  - update
- apiGroups:
  - cert-manager.io
  resources:
  - certificates
  - issuers
  verbs:
  - create
  - get
  - patch
  - update
- apiGroups:
  - rbac.authorization.k8s.io
  resourceNames:
  - contour-rolebinding
  resources:
  - rolebindings
  verbs:
  - create
  - get
  - patch
  - update
- apiGroups:
  - rbac.authorization.k8s.io
  resourceNames:
  - contour
  resources:
  - roles
  verbs:
  - create
  - get
  - patch
  - update

• Role Binding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: contour-tkg-system
  name: contour-tkg-system
  namespace: tanzu-system-ingress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: contour-tkg-system
subjects:
- kind: ServiceAccount
  name: contour-tkg-system-sa
  namespace: tkg-system

The issue

While this output is much better then giving cluster admin credentials, this configuration, will require you to pre create the tanzu-system-ingress namespace. as well as it will require some additional permissions.

The reason additional permissions are needed in many cases, is that in kubernetes, you cant create a cluster role or role that give higher permissions then what you yourself have. If the package includes RBAC resources, you must add the permissions from them, into your package service accounts RBAC, otheriwse Kapp Controller will fail to reconcile the package installation.

There is also a bug in the audit2rbac tool where it sometimes can make incorrect over zealous configurations, which simply cant work. The main change one must make in the manifests is that if we take this snippet for example:

- apiGroups:
  - rbac.authorization.k8s.io
  resourceNames:
  - contour-rolebinding
  resources:
  - rolebindings
  verbs:
  - create
  - get
  - patch
  - update

The issue is that the "create" verb does not work in kubernetes RBAC in conjunction with the resourceNames section. the solution to this with the highest level of security is:

- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  verbs:
  - create
- apiGroups:
  - rbac.authorization.k8s.io
  resourceNames:
  - contour-rolebinding
  resources:
  - rolebindings
  verbs:
  - get
  - patch
  - update
  - delete

Here we are creating 2 sets of permissions, 1 for creating the resource which cant be scoped down to a specific resources name, and then the rest of the verbs in a seperate rule, where we do scope the permissions to the specific resource names.

The reason we need to pre-create the namespace is that as this is a least privilege approach, it has split the RBAC into both cluster wide and namespace specific permissions, and as the installation targets the tanzu-system-ingress namespace, that is where the role and role binding are targeted at.

Solving the missing permissions issue

The easiest way to solve this requires some scripting and a few CLI tools. the CLI tools I use for this process are:

• kubectl
• kubectl split-yaml plugin
• imgpkg
• ytt
• jq

The general flow is:

1. set a few environment variables

export PKG_NAME=contour.tanzu.vmware.com
export PKG_VERSION=1.22.3+vmware.1-tkg.1
export PKG_NS=tkg-system

2. pull down the imgpkg bundle of this package

export BUNDLE_URI=$(kubectl get pkg $PKG_NAME.$PKG_VERSION -n $PKG_NS -ojson | \
  jq -r .spec.template.spec.fetch[0].imgpkgBundle.image)
imgpkg pull -b $BUNDLE_URI -o ./contour-package-bundle
cd contour-package-bundle

3. create a file with the values you would supply when applying a package

cat <<EOF > my-custom-values.yaml
infrastructure_provider: vsphere
contour:
  configFileContents: {}
  useProxyProtocol: false
  replicas: 2
  pspNames: "vmware-system-restricted"
  logLevel: info
envoy:
  service:
    type: LoadBalancer
    annotations: {}
    nodePorts:
      http: null
      https: null
    externalTrafficPolicy: Cluster
    disableWait: false
  hostPorts:
    enable: false
  hostNetwork: false
  terminationGracePeriodSeconds: 300
  logLevel: info
  pspNames: null
certificates:
  duration: 8760h
  renewBefore: 360h

3. template out the package and retrieve all RBAC relevant resources

ytt -f my-custom-values.yaml -f config/ | kubectl split-yaml -f - 
cd split-yaml
if [ -d "rbac.authorization.k8s.io_v1--ClusterRole" ] 
then
  mkdir ../additional-cluster-roles/
  mv rbac.authorization.k8s.io_v1--ClusterRole/* ../additional-cluster-roles/
fi
if [ -d "rbac.authorization.k8s.io_v1--Role" ] 
then
  mkdir ../additional-roles/
  mv rbac.authorization.k8s.io_v1--Role/* ../additional-roles/
fi
cd ..
rm -rf split-yaml

4. now you can either write a custom script or manually take the values from the roles and clusterroles in the new folders and update the baseline recieved from the audit2rbac tool.
5. create overlay to remove resourceNames field when verb is create

cat <<EOF > remove-resource-name-config-when-create.yaml
#@ load("@ytt:overlay","overlay")

#@ def bad_rule():
verbs:
- create
#@ end

#@overlay/match by=overlay.all
---
rules:
#@overlay/match by=overlay.subset(bad_rule()), expects="1+"
- 
  #@overlay/remove
  resourceNames: []
EOF

6. run the overlay against all rbac files and then add those updated files to your git repo.

Solving the Role vs Cluster Role issue

Solution 1 – Pre Create The Namespace

The easiest solution is to pre create the namespace. while this requires a step before installation, you could also simply add the namespace configuration to the FluxCD Kustomization repo as well.

Solution 2 – Move everything to a Cluster Role

You could also simply get rid of the seperation made between Cluster Role and Role permissions and just merge the 2 sets of permissions.

While this is less resources to maintain, and aligns to the resources created by Tanzu CLI and the TMC GUI, it veers away from the least privilege model, and is a suboptimal choice.

None the less, this is still a much better choice from a security perspective, over the default Cluster Admin approach!

What would a file manifest look like

In the end after configuring RBAC correctly and everything else, you can see an example of a working configuration in the following Git repo.

Summary

While there is definitely a learning curve with these things, the ability to configure flux at the cluster group level is huge. I think that overtime we will see the carvel packaging ecosystem evolve to more easily support fine grained RBAC as we have tried to do here.