• Deploying TAP on TKGi

    Deploying TAP on TKGi

    Recently i was working on a deployment of TAP for a customer on top of a few TKGi clusters.

    While TAP works on any conformant kubernetes cluster, as I have said many times, Kubernetes is not 100% cloud agnostic, and every distribution can have some weird quirks.

    When deploying to TKGi a few of these quirks came up, and in this post we will discuss the quirks, and how to solve them.

    Docker issue

    One thing that is important to validate is that you have performed the migration to Containerd from Docker on your TKGi clusters ahead of installing TAP.

    In most cases this should happen automatically when moving to a recent version of TKGi but if for some reason you are still on Docker, be warned that the installation will not succeed. TAP does not work on Docker based container runtimes in kubernetes and requires the use of Containerd in order to function properly.

    Contour Issue

    When we deployed TAP onto the clusters, we had an issue that the envoy pods of Contour simply would not start.

    After investigating this issue together with GSS, the issue was found to be that the Contour package provided in TAP, does not work out of the box with clusters that have their nodes configured to only support IPv4 networking and have disabled at the node level, IPv6 networking.

    This was a change made in the Tanzu packaging of Contour, where as of TAP 1.3, the behavior of Contour was changed and it’s defaulting to IPv6 with IPv4 compatibility now.

    When debugging the issue, in the envoy pod logs we found the following:

    [2023-03-16 12:05:57.334][1][info][upstream] [source/common/upstream/cds_api_helper.cc:35] cds: add 10 cluster(s), remove 2 cluster(s)
    [2023-03-16 12:05:57.334][1][info][upstream] [source/common/upstream/cds_api_helper.cc:72] cds: added/updated 0 cluster(s), skipped 10 unmodified cluster(s)
    [2023-03-16 12:15:09.584][1][warning][config] [source/common/config/grpc_subscription_impl.cc:126] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) ingress_http: malformed IP address: ::
    ingress_https: malformed IP address: ::
    stats-health: malformed IP address: ::

    To solve this, we need to add a simple overlay which will change the flags passed to the Contour deployment, and switch it to use IPv4 instead of IPv6.

    The first step is to create a secret with the overlay like bellow:

    apiVersion: v1
    kind: Secret
      name: ipv4-overlay
      namespace: tap-install
      ipv4-overlay.yaml: |
        #@ load("@ytt:overlay", "overlay")
        #@overlay/match by=overlay.subset({"metadata":{"name":"contour"}, "kind": "Deployment"})
                #@overlay/match by="name"
                - name: contour
                  - serve
                  - --incluster
                  - '--xds-address='
                  - --xds-port=8001
                  - '--stats-address='
                  - '--http-address='
                  - '--envoy-service-http-address='
                  - '--envoy-service-https-address='
                  - '--health-address='
                  - --contour-cafile=/certs/ca.crt
                  - --contour-cert-file=/certs/tls.crt
                  - --contour-key-file=/certs/tls.key
                  - --config-path=/config/contour.yaml

    Next we need to update our TAP values file to instruct TAP to use this overlay and apply it to the contour package.

    This can easily be done by using the package_overlays section in our TAP values and adding a snippet like bellow:

    - name: contour
      - name: ipv4-overlay

    This will solve the issue, and once applied to the cluster, the envoy pods will enter into a running state and contour will successfully deploy as expected.

    Source Testing Issue

    On TKGi when using NCP as the CNI, there are some quirks one can encounter. One of these quirks is related to the fact that NCP syncs labels from pods into NSX tags.

    The issue seems to be with labels that have a value of "true" such as:

    apps.tanzu.vmware.com/auto-configure-actuators: "true"
    apps.tanzu.vmware.com/has-tests: "true"

    With these labels, NCP seems to not be able to create the tag and as such does not finish networking config of the pod, causing the images to get stuck in an initialization phase indefinitely.

    This issue is odd, but can easily be fixed by adding a simple overlay to remove these labels from the pods before they are created.

    The first step is to create a secret as follows:

    apiVersion: v1
    kind: Secret
      name: testing-template-labels-overlay
      namespace: tap-install
    type: Opaque
      testing-template-labels-overlay.yaml: |
        #@ load("@ytt:overlay","overlay")
        #@ def testing_template_matcher():
        apiVersion: carto.run/v1alpha1
        kind: ClusterSourceTemplate
          name: testing-pipeline
        #@ end
        #@overlay/match by=overlay.subset(testing_template_matcher())
          ytt: |
            #@ load("@ytt:data", "data")
            #@ load("@ytt:overlay", "overlay")
            #@ def merge_labels(fixed_values):
            #@   labels = {}
            #@   if hasattr(data.values.workload.metadata, "labels"):
            #@     labels.update(data.values.workload.metadata.labels)
            #@   end
            #@   labels.update(fixed_values)
            #@   return labels
            #@ end
            #@ def bad_labels():
            #@   if/end hasattr(data.values.workload.metadata.labels, "apps.tanzu.vmware.com/has-tests"):
            apps.tanzu.vmware.com/has-tests: "true"
            #@   if/end hasattr(data.values.workload.metadata.labels, "apps.tanzu.vmware.com/auto-configure-actuators"):
            #@overlay/remove missing_ok=True
            apps.tanzu.vmware.com/auto-configure-actuators: "true"
            #@ end
            #@ def merged_tekton_params():
            #@   params = []
            #@   if hasattr(data.values, "params") and hasattr(data.values.params, "testing_pipeline_params"):
            #@     for param in data.values.params["testing_pipeline_params"]:
            #@       params.append({ "name": param, "value": data.values.params["testing_pipeline_params"][param] })
            #@     end
            #@   end
            #@   params.append({ "name": "source-url", "value": data.values.source.url })
            #@   params.append({ "name": "source-revision", "value": data.values.source.revision })
            #@   return params
            #@ end
            apiVersion: carto.run/v1alpha1
            kind: Runnable
              name: #@ data.values.workload.metadata.name
              labels: #@ overlay.apply(merge_labels({ "app.kubernetes.io/component": "test" }),bad_labels())
              #@ if/end hasattr(data.values.workload.spec, "serviceAccountName"):
              serviceAccountName: #@ data.values.workload.spec.serviceAccountName
                name: tekton-source-pipelinerun
                kind: ClusterRunTemplate
                  apiVersion: tekton.dev/v1beta1
                  kind: Pipeline
                #@ not hasattr(data.values, "testing_pipeline_matching_labels") or fail("testing_pipeline_matching_labels param is required")
                matchingLabels: #@ data.values.params["testing_pipeline_matching_labels"] or fail("testing_pipeline_matching_labels param cannot be empty")
                tekton-params: #@ merged_tekton_params()

    Once you apply this secret we simply need to use the package_overlays section in the TAP values file to instruct TAP to apply this change to the OOTB Templates package:

    - name: ootb-templates
      - name: testing-template-labels-overlay

    Once this is applied and TAP reconciles, your testing pods will work as expected.

    Prisma Scanner issue

    For this customer we are using the Prisma scanner and there too we encountered the Label issue, as well as another issue regarding security context configuration.

    These issues are easily fixable with 2 steps.

    As the prisma package is not part of the TAP installation it will be fixed in one step, and the label issue will be fixed in a different step.

    To fix the labels issue, we can create the following overlay secret:

    apiVersion: v1
    kind: Secret
      name: scan-stamping-labels-overlay
      namespace: tap-install
    type: Opaque
      scan-stamping-labels-overlay.yaml: |
        #@ load("@ytt:overlay","overlay")
        #@ def scan_template_matcher():
        apiVersion: carto.run/v1alpha1
        kind: ClusterSourceTemplate
          name: source-scanner-template
        #@ end
        #@overlay/match by=overlay.subset(scan_template_matcher())
          ytt: |
            #@ load("@ytt:data", "data")
            #@ load("@ytt:overlay", "overlay")
            #@ def merge_labels(fixed_values):
            #@   labels = {}
            #@   if hasattr(data.values.workload.metadata, "labels"):
            #@     labels.update(data.values.workload.metadata.labels)
            #@   end
            #@   labels.update(fixed_values)
            #@   return labels
            #@ end
            #@ def bad_labels():
            #@ if/end hasattr(data.values.workload.metadata.labels, "apps.tanzu.vmware.com/has-tests"):
            apps.tanzu.vmware.com/has-tests: "true"
            #@ if/end hasattr(data.values.workload.metadata.labels, "apps.tanzu.vmware.com/auto-configure-actuators"):
            #@overlay/remove missing_ok=True
            apps.tanzu.vmware.com/auto-configure-actuators: "true"
            #@ end
            apiVersion: scanning.apps.tanzu.vmware.com/v1beta1
            kind: SourceScan
              name: #@ data.values.workload.metadata.name
              labels: #@ overlay.apply(merge_labels({ "app.kubernetes.io/component": "source-scan" }),bad_labels())
                url: #@ data.values.source.url
                revision: #@ data.values.source.revision
              scanTemplate: #@ data.values.params.scanning_source_template
              #@ if data.values.params.scanning_source_policy != None and len(data.values.params.scanning_source_policy) > 0:
              scanPolicy: #@ data.values.params.scanning_source_policy
              #@ end

    We can now use the package_overlays section in the TAP values file to apply these changes:

    - name: ootb-templates
      - name: scan-stamping-labels-overlay

    As mentioned above, we also need to update the Prisma package installation.

    In this case, as prisma is installed not as part of the TAP installation itself we need to apply the overlay ourselves on the package installation.

    First we need to create the overlay secret:

    apiVersion: v1
    kind: Secret
      name: prisma-sec-context-overlay
      namespace: tap-install
    type: Opaque
      prisma-sec-context-overlay.yaml: |
        #@ load("@ytt:overlay","overlay")
        #@ def st_matcher():
        apiVersion: scanning.apps.tanzu.vmware.com/v1beta1
        kind: ScanTemplate
        #@ end
        #@overlay/match by=overlay.subset(st_matcher()), expects="1+"
            #@overlay/match missing_ok=True
              runAsNonRoot: true 

    We can now apply this overlay to our prisma package installation with the following command:

    kubectl annotate pkgi -n tap-install prisma ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=prisma-sec-context-overlay


    While there were indeed some issues encountered with TAP on TKGi, overall with just a few overlays, we can get this working end to end pretty easily. Understanding the mechanisms and intricacies of YTT overlays and Carvel packaging is indeed a steep learning curve, but once you get a hold of it, it is extremely powerful and an amazing toolset to have at your fingertips.

  • TAP 1.5 – IDE Plugin Enhancements

    TAP 1.5 – IDE Plugin Enhancements

    TAP IDE Plugins

    In TAP one of the key elements and interaction points with the platform, are the IDE plugins which currently exist for 3 IDES:

    1. Visual Studio Code
    2. IntelliJ
    3. Visual Studio

    These plugins are the key interaction point that developers utilize on a day to day basis, in order to develop their applications with TAP enabled kubernetes clusters.

    In every release of TAP, many advances are made in these plugins, making them more and more capable, feature rich, and user friendly.

    Let’s take a look at the key new features in TAP 1.5 for the IDE plugins.

    Visual Studio Code Extensions

    In visual studio code, we have 2 extensions available for TAP:

    1. Developer Tools
    2. Application Accelerator

    The Developer tools, is the extension which allows us to utilize the key features for inner loop development which includes live update functionality, and remote debugging of Java applications.

    The application accelerator extension, brings the app accelerator catalog from TAP GUI, directly into our IDE, to allow developers to bootstrap new applications, directly from their IDE!

    In TAP 1.5, the major changes were made to the Developer Tools plugin, so lets dig into it and see what the changes are!

    Developer Tools Enhancements

    This extension got a bunch of really awesome new features in this release.

    1. A new Tanzu Activity panel was added
    2. Multi namespace visibility
    3. Tanzu quick actions are available in the workload panel

    Lets see what these enhancements provide.

    Tanzu Activity Panel

    TAP is an amazing platform, and the user interface of a workload YAML, is truly awesome. One of the key selling points and benefits of TAP, is that the developer really does not need to be a kubernetes expert in order to use the platform, as TAP deals with all the underlying steps, and manifest generation etc.

    While this is true in most cases, we all know that errors can happen, and getting an understanding of what is happening under the hoods, is also a really important thing.

    Previously, if a developer wanted to understand what resources were created, what failed, what the logs of the failed step are, what the error is, etc., would need to use the terminal and start using Tanzu and kubectl commands.

    With this release of the extension, we now have a new panel in the IDE called the activity panel, which shows us the entire resource hierarchy of our workloads, in a clear an easily understandable way. It also allows us to see where the errors if there are any are to be found, and allows us to get the logs, describe a resource, and receive the error message itself from any of the relevant resources.

    Lets see what this looks like:

    You can receive a list of all the running workloads: Alt

    You can then drill down and see the 3 sections the resources are broken up into:

    Running Application:


    Supply Chain:




    On any of these resources, we can also click and get the action menu:


    This is a huge improvement, and makes debugging when needed a breeze!

    Multi Namespace Visibility

    Till TAP 1.5, workloads that were shown in the Workloads panel, and had errors visible in the Tanzu Debug panel, all were visualized from a single namespace, which is the namespace you were targeting via your kubeconfig.

    While this in it of itself was already very nice, in TAP 1.5, we now can visualize workloads across multiple namespaces!

    Lets see what this looks like.

    In the Tanzu Workload panel, you can click the three dots at the tops and select the option to select namespaces: Alt

    This will open up an action bar at the top of the window where you can select a set of namespaces you want to view: Alt

    From this point on, workloads in any of the selected namespaces, are visualized for you automatically when you open up VSCode!

    This brings a much wider perspective to the developer directly in their IDE, again enhancing the Developer experience.

    Quick Actions From The Workloads Panel

    Previously, actions such as starting live update, deleting a workload, starting remote debugging and applying a workload, all were done in the file viewer under the relevant workspace. With TAP 1.5, we can now run any of these actions also from the Tanzu Workloads panel, again just making it easier for the developer to run what they need when then need it, without jumping around to find exactly where to click!


    IntelliJ Extension Enhancements

    In IntelliJ, the major changes that were made are:

    1. A new Application Accelerator Plugin
    2. Multi Namespace Visibility

    New Application Accelerator Plugin

    Regarding the new Application Accelerator plugin, I have written a dedicated blog, regarding the new functionality in the application accelerator component of TAP with details of the new plugin and the UX, but here I will just mention, that we now have this plugin available to us, bridging the Gap between the VSCode experience and the IntelliJ experience, and bringing a truly amazing developer experience directly to the IDE, from the initial phase of project generation, all the way through deployment to kubernetes, and remote debugging of our apps on kubernetes.

    Multi Namespace Visibility

    This addition, similar to the enhancement in the VSCode extension, allows us to visualize workloads across multiple namespaces at the same time. This is a great enhancement, making the UX for developers much smoother.

    From the current Tanzu Activity Panel, one ca now click on the settings icon and select the new "Select Namespaces" option: Alt

    We then will be presented with a popup list, where you can select the namespaces you want to see workloads from:


    As can be seen, the UX is very intuitive, and this small but powerful feature makes the developer experience much smoother in real world scenarios.


    As you can see, the enhancements in the IDE plugins, makes the end user interface with TAP, much smoother, and user friendly, helping in making TAP the ultimate DevEx platform!

    I’m truly excited to see where the extensions move forwards in the upcoming releases!

  • TAP 1.5 – Application Configuration Service

    TAP 1.5 – Application Configuration Service

    What Is ACS

    ACS or Application Configuration Service, is a new optional component added in TAP 1.5. ACS, is a Kubernetes native replacement for the Spring Cloud Config Server, which was an essential part of a Spring Cloud based microservices based architecture, heavily used by Spring based workloads in the Cloud Foundry and Azure Spring Apps world.

    Spring Cloud Config Server helped by enabling setting runtime config for spring apps, in Git repositories. These configs could be stored on different branches and in directories that could be used to generate runtime configuration properties for applications running in multiple environments.

    ACS is compatible with the existing Git repository configuration management approach, and offers a Kubernetes native integration layer, making the configuration fit much more seamlessly, with application when deployed to TAP. ACS filters runtime configuration for any application by using slices that produce secrets, which can the be bound using Service Claims, like all other external configurations or service binding in a TAP environment.

    When ACS Should be used

    ACS is a great replacement for any usage you may currently have of Spring Cloud Config Server, and with minimal changes can replace it, without any major changes.

    ACS is extremely beneficial if you are migrating to TAP for PCF/TAS environments, where SCCS is very commonly used, and having a native integration for the same features via ACS, allows for a much easier migration path for your applications.

    What does the UX look like

    The first thing one would do, is configure a CR of type "Configuration Source", pointing at the Git repository where your SCCS config resides. For example:

    apiVersion: "config.apps.tanzu.vmware.com/v1alpha4"
    kind: ConfigurationSource
      name: greeter-config-source
      namespace: my-apps
        - type: git
          uri: https://github.com/your-org/your-config-repo

    Once we have our source configured, we then can define a slice, where points at a specific config in that source repo. for example:

    apiVersion: config.apps.tanzu.vmware.com/v1alpha4
    kind: ConfigurationSlice
      name: greeter-config
      namespace: my-apps
      configurationSource: greeter-config-source
      - greeter/dev
      configMapStrategy: applicationProperties
      interval: 10m

    As you can see, this is a very simple and clear UX. the spec of these resources can be much more complex, and these are just examples, but the idea is always the same.

    You can find the full documentation on this service here.

    Once we have configured the ACS resources, we can simply create a resource claim which will allow us to bind the config to our workload as such:

    apiVersion: services.apps.tanzu.vmware.com/v1alpha1
    kind: ResourceClaim
      name: greeter-config-claim
      namespace: my-apps
        apiVersion: config.apps.tanzu.vmware.com/v1alpha4
        kind: ConfigurationSlice
        name: greeter-config

    And now we can reference this resource claim in our workload as such:

    apiVersion: carto.run/v1alpha1
    kind: Workload
      name: greeter-messages
      namespace: my-apps
        apps.tanzu.vmware.com/workload-type: web
        app.kubernetes.io/part-of: greeter
        - name: BP_JVM_VERSION
          value: "17"
        - name: BP_GRADLE_BUILT_MODULE
          value: "greeter-messages"
        value: "${SERVICE_BINDING_ROOT}/spring-properties/"
      - name: spring-properties
          apiVersion: services.apps.tanzu.vmware.com/v1alpha1
          kind: ResourceClaim
          name: greeter-config-claim
          url: https://github.com/spring-cloud-services-samples/greeting
            branch: main


    While this feature may not be a game changing feature for many, the ability it brings to easily migrate apps from TAS/PCF/ASA environments into TAP, is a huge accomplishment, and is a huge step in the direction of TAP becoming the de-facto standard for running Spring based apps at scale in containerized environments!

    I am truly excited to see these types of features being added in to TAP, making the migration story into TAP, that much smoother for brownfield environments.

  • TAP 1.5 – App SSO Enhancements

    TAP 1.5 – App SSO Enhancements

    App SSO Overview

    App SSO is one of the great features offered in TAP, which has received a bunch of love in TAP 1.5 App SSO provides the needed APIs for curating and consuming a "Single Sign-On as a service" offering on Tanzu Application Platform.

    Through App SSO, one can easily integrate TAP based workloads with SSO, in a secure, simple, and straightforward manner.

    What’s new in TAP 1.5?

    In TAP 1.5, App SSO has been enhanced with a few key new sets of functionality. The ones I am extremely excited about are:

    1. New AuthServer CORS API
    2. Role claim mapping from External IDP group membership
    3. New default auth scopes for users

    Let’s take a look at each of these features and what they bring to the table.

    Auth Server CORS API

    When working with SPAs and mobile apps with SSO, we need to configure CORS which is never a fun thing to deal with. Now, in TAP 1.5, we have a very simple and clean UX for defining CORS for Public Clients as part of the Auth Server CR.

    With this new API, we can simply enable web apps, that utilize the PKCE authentication flow.

    While TAP does support allowing all origins for CORS, it is not recommended to do so from a security perspective, and this should be done with great caution.

    Let’s see what the UX looks like. The first step one would do is define the Auth Server as they would previously, just now you would add the CORS configuration:

    kind: AuthServer
    # ...
        - "https://vrabbi.cloud"
        - "https://*.vrabbi.cloud"

    As can be seen, both exact matches, and wildcards are supported in the allowed origins array.

    You could as mentioned allow all origins if needed as follows:

    kind: AuthServer
        sso.apps.tanzu.vmware.com/allow-unsafe-cors: ""
        allowAllOrigins: true

    As seen, we need to specify both in the config that we want all sources to be allowed, as well as with the allow-unsafe-cors annotation, as TAP does not want to prevent you from creating such a client, but wishes to deter you from doing so unless needed, as this is not a secure configuration and she be avoided when possible.

    Once you have defined a Public Client, and an Auth Server with CORS enabled, you must set the authentication mode to none when creating the client registration as can be seen bellow:

    kind: ClientRegistration
      clientAuthenticationMethod: none

    This is needed as the Public Client flow is without authentication, and instead the PKCE flow will be utilized.

    Role claim mapping from External IDP group membership

    This new feature is another great enhancement, making App SSO much more user friendly.

    This feature allows us to easily map and filter groups a user is a part of that are returned from the upstream IDP at login, into a set of roles, under the roles claim in the provided JWT token to your apps, which is provided by App SSO, and the relevant Auth Server.

    This new feature supports 2 types of filters, for how to find the relevant groups in order to add the relevant roles. the 2 types are exactMatch and regex.

    These 2 methods, enable just enough flexibility, while still keeping the API simple in my mind, and allow for us to map credentials from our upstream IDP, into our downstream apps, in a simple manner.

    This feature is configured at the Auth Server level, per IDP. For example:

        - name: my-ldap
                - exactMatch: "admin-users"
                - regex: "^users-*"
        - name: my-oidc
                - exactMatch: "admin-users"
                - regex: "^users-*"
        - name: my-saml
                - exactMatch: "admin-users"
                - regex: "^users-*"

    As can be seen, this is supported for OIDC, LDAP and SAML IDPs, making the configuration really simple and easy to integrate, no matter your setup.

    Default Auth Scopes

    This is a long requested feature, that is truly awesome to see being added. With this feature, you can now define authorization scopes that are automatically granted to all users from a specific IDP, regardless of their user role.

    For example, given an AuthServer with an OIDC IDP configured, with defined authorization scope defaults:

    kind: AuthServer
        - name: my-oidc
                    - "user.read"
                    - "user.write"

    With the above config, a client registration can be created, requesting the scopes:

    kind: ClientRegistration
        - name: "roles"
        - name: "user.read"

    Now that the client registration is added, when a Workload is registered by using the ClientRegistration, that workload, on behalf of the user, can request and be granted with the scope user.read automatically within the issued access token, without relation to the group memberships of that user.

    This allows for some pretty powerful use cases, and i am excited to see where people take this.

    Spring Cloud Gateway Integration

    Another great feature, which is not new in TAP 1.5, but worth noting, is the simple integration one can do between Spring Cloud Gateway (SCG) and App SSO. The reason this is worth mentioning, is that in TAP 1.5, SCG is now included as an optional package, making the possibility of utilizing SCG much greater for a wider set of customers.

    This is an area i expect to see grow over the next few releases, and it will be interesting to see how customers end up implementing these integrations and configurations in real world scenarios.


    As you can tell, VMware have put a lot of effort into App SSO in this release, adding in key features which simplify the UX, and make the amount of customization needed in your own app to include SSO that much smaller, and easier to cope with, bringing us all a step closer to a more secure landscape, and better protected applications, which is always a huge benefit, especially when provided by a platform, with almost no overhead to manage!

  • TAP 1.5 – Spring Cloud Gateway

    TAP 1.5 – Spring Cloud Gateway

    In TAP 1.5, an amazing addition was made, with the inclusion of an additional package "Spring Cloud Gateway For Kubernetes"! This is a VMware product which was initially released in February 2021 as a standalone product, and is now included as part of TAP!

    Spring Cloud Gateway for Kubernetes is based on the open source Spring Cloud Gateway project.

    What Is Spring Cloud Gateway For Kubernetes

    Spring Cloud Gateway for Kubernetes is based on the open source Spring Cloud Gateway project, and is targetted to be the API gateway solution that application developers love, and IT Operators are so happy that it exists! Spring Cloud Gateway for Kubernetes handles cross-cutting concerns on behalf of development teams, such as:

    • Single Sign-On (SSO)
    • access control
    • rate limiting
    • resiliency
    • security SCG helps to accelerate API delivery using modern cloud native patterns, using any programming language you choose, and integration with your existing CI/CD pipeline strategy.

    What Does SCG For Kubernetes Add On Top Of The Open Source Project

    Beyond the upstream SCG capabilities, the SCG for Kubernetes offering, enhances the offering by integrating with other Spring ecosystem projects such as Spring Security and Spring Session.

    Beyond these key additions, we also get additional features, only available in the commercial offering which include:

    • Kubernetes native integration, using a set of CRDs, managed by the SCG for Kubernetes operator
    • Dynamic API route configuration
    • Support for API updates through existing CI/CD pipelines
    • Simple SSO configuration
    • Commercial API route filters to enable authentication and access control
    • OpenAPI v3 auto-generated documentation
    • Horizontal and Vertical scaling configuration, to reach High Availability and meet performance requirements

    What Does the UX look like

    The first step, after installing SCG from the TAP package repository, is to create a gateway instance, which can be as simple as:

    apiVersion: tanzu.vmware.com/v1
    kind: SpringCloudGateway
      name: my-api-gateway
      count: 1

    While this resource seems very simple, we can actually add a lot of logic here as well at the gateway level. a more advance setup may look more like this:

    apiVersion: "tanzu.vmware.com/v1"
    kind: SpringCloudGateway
      name: my-advanced-api-gateway
      namespace: platform-ops-system
        serverUrl: https://my-advanced-api-gateway.example.com
        title: my advanced api gateway
        description: Micro Gateway to control internal APIs of my app
        version: 0.1.0
            - api-portal.example.com
      count: 3
        secret: sso-secret
          - name: vault-jwt-keys
              roleName: scg-role
            enabled: true
            secretsProviderName: vault-jwt-keys   
        - name: spring.cloud.gateway.httpclient.connect-timeout
          value: "90s"
        enabled: true
        interval: 30s
            enabled: true
            enabled: true
          secret: wavefront-secret
          source: my-advanced-api-gateway
          application: my-app
          service: my-advanced-api-gateway
            enabled: true
              enabled: true
                release: my-advanced-api-prometheus"

    As you can see, we can add a lot of logic in a gateway, and by abstracting it away from our applications, this makes everyone’s lives, so much easier!

    Once we have a Gateway, we now need to define our routes. This is done in a separate CR called "SpringCloudGatewayRouteConfig". Lets see an example of what this may look like:

    apiVersion: "tanzu.vmware.com/v1"
    kind: SpringCloudGatewayRouteConfig
      name: suppliers-routes-config
        name: suppliers-api
        - predicates:
            - Path=/list-outgoing-payments/
            - Method=GET
            - RateLimit=2,10s
            - CircuitBreaker="suppliersCircuitBreaker,forward:/alternative-payments-service"
        - predicates:
            - Path=/process-payment/*/supplier/**
            - Method=POST,PUT,DELETE
          ssoEnabled: true

    As you can see, we can easily define, based on many characteristics of a request, how it should be handled.

    The upside of splitting the route configuration from the gateway is huge, because the configuration at the gateway level, are the platform and IT teams concerns, while the routes themselves, are what developers own. by splitting these concerns into separate APIs, we give a much cleaner boundry and seperation of concerns between the relevant teams in the organization.

    The final resource we have is the glue that stiches this all together which is the 3rd and final CR added as part of SCG which is called "SpringCloudGatewayMapping".

    This resource simply binds a route configuration to a specific gateway.

    This would for example look like:

    apiVersion: "tanzu.vmware.com/v1"
    kind: SpringCloudGatewayMapping
      name: suppliers-routes-mapping
        name: my-advanced-api-gateway
        namespace: platform-ops-system
        name: suppliers-routes-config

    The power we get with SCG for kubernetes is amazing, and having it in a kubernetes native implementation, allows us to manage our API Gateway configuration using industry standards such as GitOps. We also, because this is being defined as a kubernetes resource, can apply any policy tooling to enforce certain organizational requirements. for Example we could create OPA based policies using Tanzu Mission control, that dont allow any routes without SSO enabled, or we could require cirtuit breaker configs to exist for every route.


    Having SCG for Kubernetes now at our fingertips when using TAP unlocks a wide range of advanced use cases, making the possibilities endless.

    SCG also integrate with App Live View, allowing us to easily visualize the status of our metrics directly in TAP GUI alongside our applications themselves!

    I am truly excited to see how organizations adopt this technology in their own paths to production, and make everyone’s lives easier, while at the same time, increasing security, application resilience, and visibility!

  • TAP 1.5 – Azure DevOps Is Now Supported

    TAP 1.5 – Azure DevOps Is Now Supported


    Back in December 2022, While working on designing a TAP implementation for a customer of mine, I came to learn that they use Azure DevOps as their Git server.

    Immediately i decided to take a look at how it would work with TAP, and ran into a bunch of issues.

    Azure DevOps is not a standard Git server, and many limitations and differences are present that make the integration difficult.

    I ended up diving deep into the weeds, and was able to make the integration work, through some custom overlays.

    This solved my issue, but was not an officially supported configuration, and as such I shared with people on the TAP team my findings, and my implementation of a POC on building in such an integration. I also wrote a blog post detailing the steps needed to get this working.

    TAP 1.5 – Official Support for Azure DevOps

    Now in TAP 1.5, VMware have built in support for Azure DevOps out of the box which is amazing!

    Many enterprise customers use Azure DevOps, and integrating with it out of the box, makes the barrier of entry to TAP much lower for many customers.

    What you need to configure to work with Azure DevOps

    As mentioned above, Azure DevOps is not a standard Git implementation and has some hard restrictions as well as some quirks.

    The main 2 differences are:

    1. Azure DevOps requires Multi Ack authentication from the git client
    2. Azure DevOps has non standard paths for repositories as compared to other git providers

    In order to handle these cases, we need to make a few changes in our TAP Values, as well as some changes in our Workload and Deliverable YAML files.

    TAP Values level changes

    We have 2 main integrations with Git that need configuration in our supply chain.

    The first type of integration, is how the platform pulls down source code from git, which is handled by our supply chains.

    The second type of integration, is the GitOps flow, in which we push the generated kubernetes manifests into a git repository either via a direct commit to a specified branch or via opening a pull request.

    Configuring the supply chain to pull from Azure DevOps

    In our TAP values file we need to set the git_implementation key to libgit2 instead of the default which is go-git. This is because of the requirment for multi ack support, which is not provided by go-git.

    This key is a parameter of the supply chain specific settings in your TAP values, so depending on the out of the box supply chain you choose to use, it will like like one of the following:

      git_implementation: libgit2
      git_implementation: libgit2
      git_implementation: libgit2

    While doing this setting globally is the better UX, and will prevent many issues, this could also be set on a workload by workload basis. This may be beneficial if you have multiple teams using the platform and some use Azure DevOps, and others use a different Git provider such as Github. This can be easily configured on a workload by adding the parameter "gitImplementation" like bellow:

    apiVersion: carto.run/v1alpha1
    kind: Workload
        - name: gitImplementation
          value: libgit2

    Configuring the supply chain to push to an Azure DevOps GitOps repository

    When we configure the GitOps flow, we add under our top level key of the relevant supply chain a gitops section with the needed values which differ depending on if you want to use the direct commit approach or the PR approach, however in either case, the Azure DevOps related changes are the same.

    under the gitops key, we first have 3 important keys:

    1. server_address – this is the URL to your Git server
    2. repository_owner – this is typically the github/gitlab organization or user
    3. repository_name – the name of the repo we want to push the config to.

    While the hierarchy of most Git Providers is <ORG NAME>/<REPONAME>, in Azure DevOps, we have a middle level object called a project.

    This means that when a typical Gtihub URL to a repo would look like:


    In Azure DevOps it will look a bit different. A project in Azure DevOps does not only contain a git repo but also many other functionalities and as such the URLs are built in the following format:


    For example:


    If you notice in the example above, the project name and repo name are the same. This is typically the case, however you can actually have multiple repositories under a single project.

    Due to this structure of the URL, we need to fill out the values mentioned above in a specific manner:

    1. server_address – this is the same as other providers (eg https://dev.azure.com)
    2. repository_owner – this must be <ORG>/<PROJECT> (eg vrabbi/vrabbi-gitops)
    3. repository_name – this is the same as other providers (eg tap-gitops)

    The final setting we must configure, whether using the PR flow or the direct commit flow is, gitops.pull_request.server_kind which must be set to the value "azure".

    In the end a sample configuration with the commit flow would look like:

        server_address: https://dev.azure.com
        repository_owner: vrabbi/tap-gitops
        repository_name: tap-gitops
          server-kind: azure

    And a sample with the PR flow would look like:

        server_address: https://dev.azure.com
        repository_owner: vrabbi/tap-gitops
        repository_name: tap-gitops
        commit_strategy: pull_request
        branch: main
          server-kind: azure
          commit_branch: ""
          pull_request_title: ready for review
          pull_request_body: generated by supply chain

    Changes to our workload and deliverable manifests

    The only real change here, is to pay attention to the Git URL and make sure you use the correct Azure DevOps format which is as mentioned above:


    While this seems trivial, it is important to not add the ".git" suffix to the name of the repo, as this is unsupported by Azure DevOps, and will cause your supply chains to fail to pull the source code.


    It is great to see that VMware have now added official support for Azure DevOps to TAP. The integration is extremely easy to setup, and the changes mentioned above should be quite well known for those that deal with Azure DevOps.

    The ability to support so many different customer setups is an amazing part of TAP, due to its extreme flexibility, and the ease of customization of the platform, and having another key Git providers implementation officially supported is a great proof of this.

  • TAP 1.5 – Application Accelerator Updates

    TAP 1.5 – Application Accelerator Updates

    Application Accelerator Overview

    App Accelerator helps you bootstrap developing your applications and deploying them in a discoverable and repeatable way.

    You can think of app accelerator as a way to publish a software catalog to your end users, which will allow them to fill out a simple and customizable form, that will then generate a templated set of files which they can start iterating on for there new project.

    Enterprise Architects author and publish accelerator projects that provide developers and operators in their organization ready-made, enterprise-conformant code and configurations.

    App Accelerator, is based on the Backstage Software Template plugin, and enriches it with additional capabilities and features to make it extremely easy to use.

    What Is New In TAP 1.5

    With every release of TAP, we get some new and exciting features in App Accelerator, and this is very much the case in TAP 1.5!

    There are 3 key new features in App Accelerator:

    1. IntelliJ Plugin for App Accelerator
    2. Accelerator Generated Projects Provenance
    3. Global activation or deactivation of Git repo creation flow

    Lets take a look at each of these features and what they bring to the table.

    IntelliJ Plugin for App Accelerator

    Just like we have had for a few releases in VSCode, we can now generate projects from accelerators, directly from Intellij!!!!

    The flow is really great, as they have embedded the accelerator plugin in a way that makes it a seamless experience, just like creating a project using any other plugin in IntelliJ.

    Let’s see what the flow looks like:

    First, the developer opens IntelliJ and would select to create a new project: Alt

    Next, the developer select Application Accelerator on the left hand panel, and would select the relevant accelerator they want to use: Alt

    Next they would fill out the form of that accelerator: Alt

    Next they would review the inputs, and generate the project: Alt

    Finally, they would press create, and the new project will be opened for them automatically in IntelliJ! Alt

    As you can see, the flow is extremely intuitive, and flows very well with the current developer flow that they have, with all the additional benefits that application accelerators bring to the table included!

    Accelerator Generated Projects Provenance

    Provenance and attestation, are the new buzz words, and for good reason. With supply chain attacks happening day after day, we need a way of stating, where something was generated, how it was generated, by whom at is what generated, with what configuration settings etc.

    Now in TAP 1.5, we are getting a new feature in Application Accelerators, which is enabled in all of the OOTB accelerators, and is easily implementable in all of your own custom accelerators, which will provide provenance for a project generated from an accelerator.

    When you generate a project using an accelerator a new file accelerator-info.yaml is generated as part of the generated zip. If you choose to provision a git repo then this file is pushed along with other files in the generated zip.

    Lets see an example file that is generated, and what it contains:

    id: a572d705-1bb4-49ca-9164-2f8918548d01
    timestamp: 2023-03-28T15:03:04Z
    username: ScottRosenberg
    source: VSCODE
      name: tanzu-java-web-app
        image: harbor.vrabbi.cloud/tap/tap-packages@sha256:28e34f0cbf0de2c6f317ea870caf21b79aa52a5359c28666a6ceec90184eb409
      - name: build-wrapper-maven
          image: harbor.vrabbi.cloud/tap/tap-packages@sha256:195a3ca6585fa91c41292584a19c2807c72ecdf986ce860a7147451e89d467d4
      - name: java-version
          image: harbor.vrabbi.cloud/tap/tap-packages@sha256:fa976ccf1609cb69e74a0162f0f49581fd0d393003e2fbe5d54d12eae62b4ff9
      - name: tap-workload
          image: harbor.vrabbi.cloud/tap/tap-packages@sha256:dbf0dedb6848ad8a7704c1c19465a1ddae9039b0e63c1dd0df3e2ed9cbda6093
      includeBuildToolWrapper: true
      javaVersion: 11
      projectName: tanzu-java-web-app
      repositoryPrefix: harbor.vrabbi.cloud/library/sample
      updateBoot3: false

    Here we can see, that the project was generated via the VSCode plugin, by Scott Rosenberg, o0n March 28th 2023, using the Tanzu Java Web App accelerator which is coming from an imgpkg bundle with the exact SHA noted, which also invoked specific fragments which all are also defined in this case as imgpkg bundles, again with the exact SHA mentioned, and we also have the exact inputs I entered, available at the bottom of the file.

    This type of provenance is extremely powerful, and I am really excited to see where this goes in future releases!

    Global activation or deactivation of Git repo creation flow

    One of the great features of Application Accelerator, is the ability to auto create a Git repository when i generate a new project.

    While this is a great feature, some organizations do not want to expose this option. In cases where this functionality is not wanted, there is now an easy configuration that one can add to there TAP Values, under the tap_gui section which will disable this feature globally. This means it will disable the feature from both the IDE plugins as well as from TAP GUI itself.

    To deactivate this feature, one simply needs to add the following YAML snippet to their TAP values file:

      # Existing configuration .......
        # Existing configuration .......
          # Existing configuration .......
              gitRepoCreation: false

    This will disable this feature globally, and align the platform to your environments needs, in a simple and easy way!


    As you can see, VMware are putting a lot of effort into improving the Developer experience, and making the platform teams happy as well. These updates of Application Accelerator, are truly awesome to see, and I am even more excited to see what will become of these features in the future!

  • TAP 1.5 – New Trivy / Aqua Integration

    TAP 1.5 – New Trivy / Aqua Integration


    TAP has always included a pluggable scanning mechanism allowing for the platform to perform source code and image scanning as part of your supply chain.

    By default TAP utilizes Grype as the scanner, but has added support for additional scanner over time including:

    1. Carbon Black Container Security – for image scanning
    2. Snyk – for image scanning
    3. Prisma Cloud (alpha) – for source code and image scanning

    As I have many customers that use Aqua, as well as many customers that utilize Aquas open source scanner trivy, I was asked many times if TAP could integrate with these tools.

    Back in November 2022, I started working on a trivy integration in TAP, and put together a POC implementation of trivy as a scanner for TAP.

    This process was relatively very simple, and the results were pretty good, but i did not get around to building it for Aqua simply out of lack of time.

    I ended up also writing a blog post describing what was needed in order to build the solution, with detailed instructions.

    I also brought this up with the TAP team, as they are always extremely happy to get customer feedback, and they truly do listen to this feedback and take it seriously.

    Trivy Scanner In TAP 1.5

    Now in TAP 1.5, VMware have added a new Alpha scanner for Trivy which supports both the Opensource trivy scanner, as well as Aqua Enterprise, which is implemented using the Aqua plugin for trivy, making it a seamless change and experience between the OSS and Enterprise solutions.

    Like the Prisma scanner which was added in TAP 1.4, The trivy scanner is released as an alpha feature, and is located in a seperate package repository.

    While the second repository does require a few extra steps in terms of configuration, the benefit is huge, as these packages have lifecycles of there own, and as these are alpha integrations, bugs can occur. I found some issues for example in the Prisma scanner about a month ago, and let the relevant people know at VMware, and within 4 days, a new version was released, with updated documentation, and ready for consumption!

    The scanner is extremely nice, in that we can not only scan with Aqua, but we also can have a scan policy resource, that instead of defining locally in each namespace what are policy around vulnerabilities is within a rego file, we can simply define a policy that checks if our scan results pass the Aqua scan policy defined globally within our Aqua console. This means that we can manage our scanning policies, and allow list and deny list of vulnerabilities, at the organizational level, without regard to where how and by whom an image was built.


    It is great to see the new integrations of scanning solutions in TAP with every release, and Aqua + Trivy are truly an amazing addition to the already quite impressive list of integrations.

    As these integrations mature, it will be great to see them fully integrated as well, as first class packages in the TAP package repository itself!

  • TAP 1.5 – GitOps Installation

    TAP 1.5 – GitOps Installation

    TAP GitOps Reference Implementation Overview

    In TAP 1.5 we now have a new and very exciting installation model, based on the GitOps methodology.

    The new installation model offers a secure and simple way to manage a TAP installation, no matter what type of configuration you are targeting.

    What Problems Is This Trying To Solve

    One of the issues with any system we manage that must be dealt with, is where is my source of truth, and where do i store my configuration.

    TAP requires credentials to be provided at installation time to multiple different types of systems such as Git Repositories, Image Registries, Authentication Providers for TAP GUI, Kubernetes credentials (in multi cluster setups), etc.

    The challenge this brings is that when we have such needs, storing the config file simply in git as is would not be a secure solution. With that being said, the industry has standardized on Git being our source of truth, and keeping config locally on your machine, is not a sustainable solution either.

    Another issue with the current installation flow, is the imperative nature of the process. While the installation can be greatly streamlined with some experience, the real ideal is to take the kubernetes approach of being declarative and not imperative.

    We also want to solve the issue of drift detection and remediation. In the current state of TAP installation, drift is very common, and very often what is configured in the TAP Values file, is not what is actually running on the cluster. This occurs due to people adding overlays manually, or tweaking configurations inline like pausing package reconciliation, and other things that simply happen, whether we like it or not.

    Another issue is that while the installation of TAP can be very simple, TAP is not all we need to do typically. We usually want to install the TBS Full dependencies, which is another package repository and package installation. We may want to install Kubernetes operators like the RabbitMQ operator, or the VMware SQL operators, to allow for easy provisioning of data services for our workloads to bind to. Or another common example is that we want to install an optional package that is part of TAP but not in a profile such as External Secrets Operator (ESO), Spring Cloud Gateway (SCG), Application Configuration Service (ACS) etc.

    The challenge this brings, is that while in our minds, this is all one solution, the installation process is a bunch of bespoke imperative commands, with different configuration files, that all need to be managed in combination with one another, but nothing ties them together and provides the well needed glue.

    What Is The Traditional Installation Model

    The traditional TAP installation in a simple situation is in general built out of a few different steps:

    1. Accept The EULAs
    2. Install Tanzu CLI
    3. Relocate Tanzu Cluster Essentials images to your own registry
    4. Relocate TAP Package Repository to your own registry
    5. Relocate TBS Full Dependencies Package Repository to your own registry
    6. Install Tanzu Cluster Essentials
    7. Add the TAP package repository to your cluster
    8. Add the TBS Full Dependencies repo to your cluster
    9. Generate your TAP values file
    10. Install TAP via a single Tanzu CLI command
    11. Install the TBS full dependencies package to your cluster

    After this, there are many additional steps one may take such as those mentioned above in the previous section.

    What Does The GitOps Flow Look Like

    In the GitOps flow the steps start out the same:

    1. Accept The EULAs
    2. Install Tanzu CLI
    3. Relocate Tanzu Cluster Essentials images to your own registry
    4. Relocate TAP Package Repository to your own registry
    5. Relocate TBS Full Dependencies Package Repository to your own registry
    6. Install Tanzu Cluster Essentials

    At this point though things change.

    The following steps in the GitOps flow greatly depend on which of the 2 solutions provided you use in order to manage your secrets in the TAP configuration.

    Choosing A Secret Management Solution

    As mentioned above, one of the things we must take care of when dealing with storing things in Git, is how to protect and secure our credentials that are needed as part of the configuration.

    In TAP 1.5, the GitOps installation process, offers support for 2 different options, SOPS and ESO.

    Let’s quickly try and understand what each solution provides, and why one may want to use it.

    SOPS – Mozilla Secret OPerationS

    With this approach, which is the easier approach to configure, we store all of our sensitive data in an encrypted format within our Git repository.

    SOPS is a tool from Mozilla that allows us to easily encrypt and decrypt files.

    SOPS supports YAML, JSON, ENV, INI and BINARY formatted files and can encrypt the needed data with different mechanisms such as:

    • AWS KMS
    • GCP KMS
    • Azure Key Vault
    • age
    • PGP

    This approach allows for the easiest way to get started, and allows all of our configuration to be stored in Git, as well as the fact that it has no dependencies on the existence of other tools in your environment.

    What are the remaining steps if we choose SOPS

    While SOPS supports multiple different solutions for managing the keys used for encrypting and decrypting our data, currently AGE is the supported mechanism. While you could with some customization make GPG work as well, the supported mechanism is AGE so that is what we will use.

    The remaining steps if we go down this path are:

    1. Install SOPS and AGE CLIs to your machine
    2. Skaffold a Git repository with the needed configuration based on the GitOps download bundle included in the TAP 1.5 downloads
    3. Create an AGE key which will be used to encrypt our sensitive data
    4. Generate a TAP Values file excluding any sections that are sensitive
    5. Generate another YAML file with all of the sensitive fields you left out of the TAP values file from the previous step
    6. encrypt the sensitive file using the SOPS CLI with your new AGE key generated above
    7. Push the configuration to a git repository
    8. run the provided bash script to configure the GitOps flow. This will create a secret with your new AGE key in order to be able to decrypt the sensitive values, install a carvel application CR which will pull down the git configuration and deploy TAP for you automatically, as well as keep the clusters config always in sync with your git repositories configuration.

    Getting started with this approach is extremely easy, and the documentation is also very good at walking you through all of the needed steps to get it up and running in no time.

    ESO – External Secrets Operator

    While SOPS is the easiest solution to implement, many organizations have a central Key management system such as Hashicorp Vault, AWS KMS, Azure Key Vault or many other tools that exist in this area.

    External Secrets Operator is a kubernetes operator that reads information from a third-party service like those mentioned above as well as many more, and automatically injects the values into Kubernetes Secrets.

    This approach allows us to store our sensitive values needed for TAP installation in a key management tool, and have ESO pull those down into our cluster as part of the installation process.

    This approach is more involved in terms of setup requirements, as well as it splits our source of truth between 2 different systems, as our sensitive data is not saved in Git, but it aligns with the general secret management strategy many companies are adopting in the market, of storing all sensitive data in a dedicated system built exactly for this purpose.

    What are the remaining steps if we choose ESO

    In TAP 1.5, the ESO integration is curated for AWS Secret Manager and EKS clusters specifically, but with some changes to the scripts and configuration files, one could make it work with any supported provider in ESO, however EKS + AWS Secrets Manager is the only tested and verified solution as of now.

    The remaining steps if we go down this path are:

    1. install and configure AWS CLI
    2. Skaffold a Git repository with the needed configuration based on the GitOps download bundle included in the TAP 1.5 downloads
    3. Generate a TAP Values file excluding any sections that are sensitive
    4. Generate another YAML file with all of the sensitive fields you left out of the TAP values file from the previous step
    5. create 2 IAM policies to allow the GitOps flow as well as TAP installation itself to pull the secrets from the AWS secret store
    6. Create 2 IRSA pairs for your cluster, binding the policies from above to the needed 2 service accounts in your cluster for the GitOps flow and TAP installation itself.
    7. create the needed secrets in AWS secrets manager using the AWS CLI
    8. Create a configuration file which will be stored in git pointing to the relevant secrets you have stored the TAP sensitive values in
    9. Push the configuration to a git repository
    10. run the provided bash script to configure the GitOps flow. This will create secrets to access your container registry, install a carvel application CR which will pull down the git configuration and deploy TAP for you automatically, as well as install ESO and generate the external secret CRs needed to pull down the sensitive values from your AWS secret manager and keep them in sync. Just as with SOPS, this flow will keep the clusters config always in sync with your git repositories configuration.

    As you can tell, this is a bit more work to configure, but also here the documentation is very well written, with good examples, and clearly guides you along the path to setting this up.

    Adding Custom Configurations To The Mix

    While this approach has huge value already for just installing TAP itself, it becomes even more powerful when we add our own additional configurations in the mix, to make it a much more holistic and end to end solution, for our cluster configuration.

    Lets see a few examples of where this could be extremely valuable:

    TBS Full dependencies

    TAP comes pre baked with the lite dependencies of TBS, which while fine for a POC, should really be replaced with the full dependencies package in a production setup.

    The TBS dependencies installation is similar to TAP in that we need to install a dedicated package repository and then install a package from that repository into our cluster.

    We can easily add the manifests for the repository and package installation in our GitOps repo, and let the system install it for us automatically, directly alongside our TAP installation itself.

    You can see an example of such a configuration in my Sample TAP GitOps Repository.

    Configuring TAP Overlays

    While TAP exposes many knobs for us to configure things as we need, there are situations where we may need to use YTT overlays, to overcome either an issue / bug, or simply to change a specific decision made by the TAP engineering team in a component that does not meet the needs of our specific environments.

    The need for such a mechanism is understood well by the TAP team, and they have provided us with a great mechanism for allowing such overlays directly via secrets references in our TAP values file.

    In the traditional non GitOps installation mode of TAP, we would simply create the tap-install namespace, create these secrets containing our overlays, and add references to the secrets in our TAP values before installing TAP.

    As the tap-install namespace, and TAP itself are now all part of the GitOps installation flow, we need another way to create these secrets, and optimally they should be managed side by side with our TAP config itself.

    We can simply store our secret manifests alongside our TAP configuration in the same repo, and we can have the GitOps process manage these secrets for us as well!

    You can see some examples of these types of overlays and the setup in my Sample TAP GitOps Repository.

    Installing Operators

    Another common thing many people will want in a TAP cluster, is additional operators to be installed such as the VMware Data Services operators for MySQL, PostgreSQL and RabbitMQ, or other operators you want to utilize in this cluster alongside TAP.

    This is also not limited to operators, and can actually be any software you want installed in your cluster.

    Typically these operators would also be installed imperatively into our cluster, and we would have the same management challenges as we mentioned above for TAP.

    Because this solution is easily customizable, we can add the installation of these operators into our repo alongside the TAP installation config, and manage the entire setup together.

    You can see some examples of these types of additions in my Sample TAP GitOps Repository.

    Creating TAP Resources

    Another common use case may be to install your own custom supply chains, or Cluster instance classes, or any other TAP related custom resources.

    TAP is truly amazing in the amount of capabilities it offers, and as it is a kubernetes native solution, where almost everything is defined in a dedicated CR, we have many extension points, and configurations we can setup, in order to make our clusters perfectly fit to our needs.

    By adding these custom supply chains, or cluster instance classes, or really any other resources that are TAP related, you can truly manage YOUR TAP configuration in a unified manner, following industry standard approaches such as GitOps.

    You can see an example of some custom resources i have put together in my Sample TAP GitOps Repository.


    The new GitOps installation model is extremely beneficial, and makes the management of TAP environments a much cleaner and seamless experience.

    It is great to see GitOps being added as an official way to manage TAP environments, and I’m sure we will see more and more innovation in this area as the product keeps maturing and growing.

    You can tell that the engineers behind this feature, truly thought deeply of how this flow should look, and have built the solution in a manner that makes lifecycle management easy, will make upgrades a much more seamless process, and at the same time have left enough room for us to be able to add in our own customizations, to fit the solution to our needs, in a simple and secure manner!

  • TAP 1.5 – Dynamic Service Provisioning With Crossplane

    TAP 1.5 – Dynamic Service Provisioning With Crossplane

    What Is Crossplane

    Crossplane is an open source, CNCF project built on the foundation of Kubernetes to orchestrate anything.

    Its basically the kubernetes native approach to solve similar issues and challenges that tools like Terraform, Pulumi and IDEM are trying to solve.

    Tanzu Application Platform makes use of Crossplane to power a number of capabilities, such as dynamic provisioning of services instances with Services Toolkit as well as the out of the box Bitnami Services.

    What Is Dynamic Service Provisioning

    One of the strong capabilities in TAP, is the ability to easily bind a workload to backing services like databases or message brokers.

    The issue that has existed till now in TAP, is that the provisioning of these services was a manual process, where there was no self service element in play.

    This required the Service Operators such as DBAs or dedicated experts in specific services, to be involved in the process of deploying a service, before the application operator could then claim that service, and then, the developer could utilize that claim in their own workload.

    While the APIs have been there for the developer to easily bind to a service, all of the manual steps needed to get to that point, was a major stumbling block for many, causing delays in development, and made it so our developers were not truly self sufficient.

    Now it is important to note, this is not a trivial issue to solve, and nearly all platforms on the market have this the of issue, and now as of TAP 1.5, VMware have added in some amazing features, which allow us to easily and securely offer self service capabilities to our end users, to provision and lifecycle manage their applications backing services, without needing all of the manual toil.

    The idea with dynamic service provisioning, is that a service operator, can define via a Crossplane XRD and composition (just a bunch of YAML), a service they want to provide. they can define what fields a user can change, and which ones they can’t change. they can set validations on values, and really any constraints that they want, which are defined in a simple OpenAPIv3 Schema within the service definition.

    Once the service operator has defined these services they want to offer in the cluster to the end users, a developer can simply claim a new instance of that service, and provide the parameters he wants to set (which the service operator defined for him), and the platform will automatically provision a new instance of that service, and create the needed configurations for the TAP workloads to seamlessly bind to the services, just like they would till TAP 1.5, using the Service Bindings for Kubernetes Spec.

    Bitnami Services

    Out of the box, TAP 1.5 comes with a few very common services packaged as Crossplane compositions and XRDs, for Bitnami helm charts.

    The included services are:

    • RabbitMQ
    • PostgreSQL
    • MySQL
    • Redis

    While this in it of itself is great, VMware have taken it a step further, and via a few values in your TAP values file, you can configure these compositions to instead of deploying the OSS Bitnami helm charts, deploy VMware Application Catalog based helm charts, as long as you have a VAC subscription and that you provide the needed credentials to make it work.

    These services, are probably the most common backing services today in the kubernetes ecosystem, and as such it is great to see them being included out of the box in TAP!

    Building Your Own Services

    Building your own services based on Crossplane can seem quite overwhelming at the beginning, but once you get a hold of it, its truly easy to do.

    Similar to all tools in this area, such as Helm, YTT, Terraform etc., the learning curve is indeed a hurdle one must cross, but once you do, the options are endless, and the value is huge!

    Another great feature in TAP 1.5 is the GitOps installation flow, and you can see in my Sample GitOps Repo where I defined my entire TAP installation, I have put together a few examples of custom services including:

    • RabbitMQ Operator based cluster
    • VMware PostgreSQL Operator based clusters
    • VMware MySQL Operator based clusters
    • MongoDB using the Bitnami OSS Helm chart
    • Kafka using the Bitnami OSS Helm chart
    • Microsoft SQL Server using raw kubernetes YAML and the official SQL Server image from Microsoft.

    What does the new experience look like for a developer

    The first step they would do is see what classes of services are available to them:

    $ tanzu services classes list

    This will return an output similar to:

      NAME                  DESCRIPTION
      kafka-unmanaged       Kafka by Bitnami
      mongodb-unmanaged     MongoDB by Bitnami
      mysql                 MySQL
      mysql-unmanaged       MySQL by Bitnami
      postgresql            PostgreSQL
      postgresql-unmanaged  PostgreSQL by Bitnami
      rabbitmq              RabbitMQ
      rabbitmq-unmanaged    RabbitMQ by Bitnami
      redis-unmanaged       Redis by Bitnami
      sqlserver-unmanaged   SQL Server by TeraSky

    Then the developer can see what parameters were exposed to him for a specific service for example:

    $ tanzu services classes get postgresql-unmanaged

    Which will return output like the following:

    NAME:           postgresql-unmanaged
    DESCRIPTION:    PostgreSQL by Bitnami
    READY:          true
      KEY        DESCRIPTION                               TYPE     DEFAULT  REQUIRED
      storageGB  The desired storage capacity in GB.       integer  1        false

    As we can see, one parameter has been exposed to the developer, where they can set the amount of storage they want allocated to the database.

    If the developer wants to now deploy a database with 20 GB of storage for example he simply runs a command like the following:

    $ tanzu services class-claim create my-db --class postgresql-unmanaged \
        --parameter storageGB=20

    which will return output similar to the following:

    Creating claim 'my-db' in namespace 'test-01'.
    Please run `tanzu services class-claims get my-db --namespace test-01` to see the progress of create.

    And now if the developer runs the command suggested in the output they can see the status of the provisioning, which will look similar to:

    Name: my-db
    Namespace: test-01
    Claim Reference: services.apps.tanzu.vmware.com/v1alpha1:ClassClaim:my-db
    Class Reference:
      Name: postgresql-unmanaged
      Ready: True
      Claimed Resource:
        Name: 98ec8a57-7c2a-405e-82fc-9102dc6c0717
        Namespace: test-01
        Version: v1
        Kind: Secret

    This shows them all the needed info in order to simply now bind this service to there workload either via the CLI using the –service-ref flag or by adding in a serviceClaims section to their workload YAML pointing at the class claim resource that was created. An example of the imperative command mechanism would look like:

    $ tanzu apps workload create -f config/workload.yaml --service-ref db=services.apps.tanzu.vmware.com/v1alpha1:ClassClaim:my-db

    And the declarative YAML configuration would look like:

      - name: db
          apiVersion: services.apps.tanzu.vmware.com/v1alpha1
          kind: ClassClaim
          name: my-db

    This simple user experience, is extremely powerful, and provides an unprecedented UX for developers, while finding the right balance between developer autonomy and platform and operational needs and concerns.


    The new dynamic provisioning functionality is a true game changer in terms of Developer experience, and helps platform teams truly meet their goal of being enablers for the business and not being stumbling blocks in the way to production.

    The integration with crossplane is a great choice in my opinion, and I am truly excited to see how this integration deepens and grows over future releases.