One of the greatest features in TAP 1.2 in my opinion is something that many have overlooked, including myself in my What’s new in TAP 1.2 blog post.
A big challenge that exists with platforms such as TAP, is how do i go about customizing the platform when the defaults or configuration nobs provided by the vendor simply are not enough.
This could be for many different use cases and scenarios, and we will cover in this blog post one key feature that I think can be very valuable in many scenarios, which is Auto Rendering of Tech Docs.
Back in May 2022 with TAP 1.1, I wrote a blog post on how I configured this which can be viewed here for more details.
While that approach mentioned in the original post is still possible, in TAP 1.2 we have a much better solution.
TAP is configured utilizing the Carvel toolset and is bundled into packages that we deploy to our cluster.
Carvel packages are a really awesome solution for managing applications on Kubernetes and you can see some of my blog posts in the Carvel topic on this site, that go deeper into the mechanism and why it is so useful.
TAP has also chosen to utilize a method referred to by many now as a Meta-Package which is basically a single package that installs many other packages.
This is how you can install TAP via a single YAML manifest and a single command, but still get over 30 different applications and components installed and managed for you that build up the platform.
While the idea of a package is that the author will build it to accept all the inputs they believe should be configurable by the end user, sometimes the end user may disagree with this decision.
In Carvel packages, because we have the access to utilize YTT under the hood, an escape hatch mechanism was made which allows you to save a YTT Overlay in a secret, and then annotate the PackageInstall CR in your cluster with the secret name. By doing this, those YTT Overlays you save will be taken into account when rendering the YAML manifests during the packages reconciliation.
Now this has always existed, so why am I talking about this in the context of something amazing in TAP 1.2?
Well the reason is, is that when you use a mechanism like TAP does of a Meta-Package which installs many other packages, the mechanism to overlay things becomes very difficult.
In the pre 1.2 days of TAP, you would have to either pause the reconciliation of the Meta Package like I showed as an example in the previous blog post on Tech Docs, or you could write an overlay for the meta package, that would overlay the package installation of the sub package, and that is a pretty messy overlay to deal with and not a great UX. It also would typically happen as a Day2 step, as most installations of TAP will happen via the Tanzu CLI as documented in the Official Documentation.
So with that background lets dive into the new feature in 1.2!
Package Overlays!!!
In TAP 1.2, a new field was added as a value option in the TAP Meta Package which is “package_overlays”. This field accepts an array of objects, where we can define the secret name and the package we want the secret/s applied to.
Let’s take a look at the Techdocs scenario as an example.
The first step is to Create a secret with my overlay for adding the docker socket container and volume mounts to the TAP GUI deployment.
cat << EOF > overlay.yaml
#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({"kind": "Deployment", "metadata":{"name":"server","namespace":"tap-gui"}})
---
spec:
template:
spec:
containers:
#@overlay/match by=overlay.subset({"name":"backstage"})
- name: backstage
#@overlay/match missing_ok=True
env:
- name: DOCKER_HOST
value: tcp://localhost:2375
volumeMounts:
- mountPath: /tmp
name: tmp
- mountPath: /output
name: output
#@overlay/append
- command:
- dockerd
- --host
- tcp://127.0.0.1:2375
image: harbor.vrabbi.cloud/tap/docker:dind-rootless
imagePullPolicy: IfNotPresent
name: dind-daemon
resources: {}
securityContext:
privileged: true
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp
name: tmp
- mountPath: /output
name: output
#@overlay/match missing_ok=True
volumes:
- emptyDir: {}
name: tmp
- emptyDir: {}
name: output
EOF
This overlay is basically the same as the patch file created in the previous post, just using some YTT annotations to tell YTT how to overlay this config onto the manifests in the package.
Now lets create the secret with this file as its content. To do this, we need to create the secret in the same namespace the TAP Meta Package is installed in which unless you didn’t follow the official documentation is called tap-install.
kubectl create secret generic -n tap-install tap-gui-techdocs-overlay --from-file=overlay.yaml
This will create the secret that we need for overlaying the TAP GUI package.
Now the final step before installing or updating our TAP installation, is to add a few lines to our TAP values file which would look in this example like:
package_overlays: - name: tap-gui secrets: - name: tap-gui-techdocs-overlay
Once we have that in our values file, we can proceed as regular with deploying or updating TAP and this overlay will be applied to the relevant package automatically.
So How does it work?
In TAP, the carvel packages are all saved as imgpkg bundles in a Container/OCI registry.
Lets find the imgpkg bundle URL from the relevant package CR after we have imported the TAP Package repo into our cluster:
IMAGE_URL=`kubectl get pkg -n tap-install tap.tanzu.vmware.com.1.2.0 \
-o jsonpath="{.spec.template.spec.fetch[0].imgpkgBundle.image}"`
This command will give us the URL of the package in our OCI registry if we relocated the packages as is suggested or in the official registry if you are installing TAP in a POC mode.
In any event, we can now pull down the package to our machine to see the manifests it includes by using the imgpkg CLI tool:
imgpkg pull -b $IMAGE_URL -o ./tap-12-meta-package
Once the command finishes, we will get a folder generated with all of the manifests that were in the meta package.
Lets take a look at the file which is relevant for us:
cat ./tap-12-meta-package/config/package-overlays.yaml
And that will return the following YAML:
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
#@ def build_package_overlay(package):
kind: PackageInstall
metadata:
name: #@ package
#@ end
#@ for package in data.values.package_overlays:
#@overlay/match by=overlay.subset(build_package_overlay(package.name))
---
metadata:
#@overlay/match missing_ok=True
annotations:
#@ i=0
#@ for secret in package.secrets:
#@overlay/match missing_ok=True
#@yaml/text-templated-strings
ext.packaging.carvel.dev/ytt-paths-from-secret-name.(@= str(i) @): #@ secret.name
#@ i=i+1
#@ end
#@ end
Lets break this up into a few pieces to better understand whats going on
1. High level Annotations
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
In these 2 lines, we are loading the overlay and data libraries of YTT into our files execution.
2. Defining a template to use for matching resources in the package
#@ def build_package_overlay(package): kind: PackageInstall metadata: name: #@ package #@ end
As can be seen, we are creating the base structure of a PackageInstall CR which contains the kind, and name of the object.
we are defining this as a function and it has an input called package which we use to fill in the package installs name when calling the function.
3. Iterating over the package overlays supplied in the values file
#@ for package in data.values.package_overlays:
This is a simple for loop which we close in the final line of the file via the “#@ end” annotation.
4. Defining the object the overlay should apply to
#@overlay/match by=overlay.subset(build_package_overlay(package.name)) ---
In this overlay annotation we are telling YTT to overlay on objects based on them matching the output of the function we defined above called build_package_overlay.
This is a very elegant way to specify, I want to update an object in an array of many objects, that meets specific criteria. while there is no need for the function necessarily, and we could just inline its definition into the overlay annotation, doing it this way makes it much easier to understand, and update if needed.
5. Defining the overlay patch itself
metadata: #@overlay/match missing_ok=True annotations:
In these lines we are defining where the overlay should be applied meaning, we need the addition of an annotation to a package install CR.
The annotation we see here, is a great feature of YTT. YTT is a very safe tool, and it is precautious and doesn’t want to break things for you, or allow you to break things by mistake as much as possible.
The way YTT works with overlays, is that if a field doesn’t exist for example metadata.annotations at all, we can’t simply add metadata.annotations.bla to our manifests.
The reason for this is that we want to be certain that the added field is not simply lets say a typo and that you wrote metdata instead of metadata. this would make it so that the outputted YAML, while it does follow what you inputted, does not follow what you intended.
This YTT annotation is the way to say, I know that this field is defined correctly, so whether it exists or not please proceed and add the underlying values.
6. Adding the annotations/s to the CR
#@ i=0
#@ for secret in package.secrets:
#@overlay/match missing_ok=True
#@yaml/text-templated-strings
ext.packaging.carvel.dev/ytt-paths-from-secret-name.(@= str(i) @): #@ secret.name
#@ i=i+1
#@ end
#@ end
This section is using the benefit of YTT exposing Starlark, a pythonic language to us via YAML annotations, and is basically setting an index and then iterating over each secrets i have said should be used as an overlay for this package, and then it constructs the kubernetes annotation using the index count as needed in order to support multiple overlays for a single package.
It finally increases the counter index by one for each iteration, and then it closes out the 2 for loops we are in.
Summary
It is pretty amazing the impact 21 simple lines of YTT can have on the UX of a project, but it really does make the product so much more customizable in an elegant, and simple way.
I really look forward to trying out this mechanism more and more over time as different use cases come up which need some tweaking of the platform.
While most people won’t need to use this mechanism necessarily, it is great that it exists in the event that you do need it. whether that is now, or in the future as your implementation of TAP evolves.