In TAP 1.4, VMware added an additional optional package to TAP for the External Secrets Operator, and in TAP 1.5 they have enhanced the integration.

What Is External Secrets Operator (ESO)

The External Secrets Operator is a Kubernetes operator that integrates with external secret management systems, for example, AWS Secrets Manager, Google Secrets Manager and Hashicorp Vault. It reads information from external APIs and automatically injects the values into Kubernetes secrets.

ESO In TAP

ESO is available as a package in the TAP package repository, but is not included in any of the profiles.

ESO can be utilized in TAP for many use cases, including for TAP installation itself, using the new GitOps installation method!

You can also utilize ESO for simplifying the management of secrets which are needed for Service Bindings, Git credentials or registry credentials just to mention a few common cases. It can also be very helpful when working with a scanner other than vault within TAP such as Snyk, Carbon Black, Prisma or Aqua where we need to provide credentials for running the scans in every developer namespace.

ESO is not integral to TAP at this point, and there is no dependency on it, but any environment that already has a secret management system can gain huge value from this integration.

What’s New In TAP 1.5

In TAP 1.5, we now have 3 key update to ESO:

1. ESO is now a Beta component (previously alpha) in TAP
2. Integration with TAP Installation flow using a GitOps methodology
3. External Secrets plugin for Tanzu CLI

Let’s see what these updates add to the picture

ESO is now a Beta component (previously alpha) in TAP

This is a great sign, showing that the direction towards deeper integration, and full on GA status is moving in the right direction. ESO can bring a lot to the table for a platform like TAP, and it is great to see VMware integrating with ESO more and more with each release.

Integration with TAP Installation flow using a GitOps methodology

I have written a dedicated post regarding the GitOps installation option, and what that entails, so for here what I will state in short is that the new installation method allows us to split our TAP Values into 2 separate files, one with non sensitive values, and the other, with sensitive values.

The sensitive values are then stored as a secret in your external secrets management system, and at deployment time, The ESO package will be deployed and configured and the values will be populated into a kubernetes secret, which is in turn used, alongside the non sensitive values file which is pulled down from Git, as the values for the TAP installation.

This method offers a great UX, and a simple lifecycle management story for TAP, utilizing industry standard approaches like GitOps, while seamlessly integrating security at the forefront, utilizing ESO, as an agnostic integration layer, with nearly all common secret management solutions on the market today.

External Secrets plugin for Tanzu CLI

One of the main things VMware have done with Tanzu, is to build a full end to end cloud native portfolio, and not just a single tool. As such, a unified entrypoint is needed, and for Tanzu this can be found in the Tanzu CLI.

While not all Tanzu products are integrated yet into the Tanzu CLI, much work is happening to add more and more of the products into this model, where we can have a single CLI entry point to the entire Tanzu ecosystem, and for each tool, a set of dedicated or shared plugins will be available in order to support the relevant use cases.

This is already the case with TKG and TAP, and TMC have already announced that soon they will be moving to a Tanzu CLI based interface instead of the standalone TMC CLI which currently exists.

As part of this, TAP has since the beginning had multiple Tanzu CLI plugins which are dedicated for TAP, as well as TAP utilizing some of the core shared Tanzu CLI plugins for its needs.

In TAP 1.5, a new CLI plugin was added for ESO which can be accessed simple via the Tanzu CLI sub-command "external-secrets" or for those like me that hate typing long words 2 aliases have been added for the command "es" and "eso".

With the current CLI we can interact with 3 resource types, which are part of ESO installation:

1. ClusterStores
2. Stores
3. ExternalSecrets

Let’s see what we can do with this new CLI plugin.

Creating resources

We can create resources using the Tanzu CLI very easily.

For example, to create a secret store targeting vault, using the kubernetes authentication method, one could run a command like:

cat <<EOF | tanzu external-secrets stores create -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-secret-store
spec:
  provider:
    vault:
      server: "https://vault.vrabbi.cloud"
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: demo-vault-role
          serviceAccountRef:
            name: my-sa
EOF

And to create an external secret we can follow the same approach:

cat <<EOF | tanzu external-secrets secret create -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: vault-example
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-secret-store
    kind: SecretStore
  target:
    name: example-sync
  data:
  - secretKey: foobar
    remoteRef:
      key: secret/foo
      property: my-value
EOF

We can also list all the relevant ESO resources and see their status for example:

Getting all Secret Stores:

$ tanzu external-secrets store list -A

NAMESPACE  NAME                PROVIDER         STATUS
default    vault-secret-store  Hashicorp Vault  Valid
demo-app   vault-secret-store  Hashicorp Vault  Valid
scott-dev  vault-secret-store  Hashicorp Vault  Valid

We can all list all external secrets:

$ tanzu external-secrets secret list -A

NAMESPACE  NAME                  SECRET NAME      STORE               REFRESH INTERVAL  STATUS             LAST UPDATED  LAST REFRESH  
default    git-creds             registry-secret  vault-secret-store  15m               SecretSynced       18s           10m
default    registry-creds        registry-secret  vault-secret-store  15m               SecretSynced       18s           10m
demo-app   git-creds             registry-secret  vault-secret-store  15m               SecretSynced       7s
demo-app   registry-creds        registry-secret  vault-secret-store  15m               SecretSynced       7s            10m
scott-dev  aqua-creds            registry-secret  vault-secret-store  15m               SecretSynced       35s           10m
scott-dev  git-creds             registry-secret  vault-secret-store  15m               SecretSynced       2s            10m
scott-dev  registry-creds        registry-secret  vault-secret-store  15m               SecretSynced       63s           10m

This is all very nice, but what i think is probably the most useful command so far, is the ability to trigger a sync of a secret manually. While ESO has a sync interval at which it will check if there is an updated value in the configured secret store, and if there is, will update the generated secret, sometimes we want to trigger a sync on demand.

To trigger the sync of a secret manually, one can simply run for example:

$ tanzu external-secrets secret sync aqua-creds -n scott-dev

Summary

ESO is a great open source project, and it is great to see that once again, the VMware team, are leveraging the tremendous ecosystem out there in the kubernetes space, and enhancing the UX around these tools by seamlessly integrating them into TAP, without falling down the common pitfall of forking the project. In this case, like nearly all other key components in TAP, VMware wrap the OSS solutions into a cohesive end to end solution, abstracting away the complexities and intricacies of the individual tools for end users, but still enabling advanced users to utilize the full capabilities of these capabilities and to so called "break glass" when needed.

I am truly excited to see the deeper integration with ESO in TAP 1.5, and am excited to see what comes next!