Integrating Active Directory CA (ADCS) with TAP

Recently a colleague of mine wrote 2 great blog posts (blog-1, blog-2) regarding configuration of TAP to issue certificates signed by ADCS. The solution he documented utilizes Hashicorp Vault as an intermediate CA between cert-manager and ADCS.

This approach is a very scalable and simple approach which is very well suited for production setups.

The reason for using Hashicorp Vault was that ADCS is not supported by the default cert-manager ClusterIssuer and Issuer resources, whereas Vault is supported OOTB making it a great way to integrate these systems.

A while back i remebered seeing that Nokia had built an external Cert Manager issuer for ADCS which also seemed like an interesting direction to investigate, however they have stopped maintaining the provider, and till recently this seemed like a lost cause.

Recently when taking a look at this again i found that there is an active fork of this provider which has since been updated and seems to be well maintained now making it an interesting project to explore.

In this post I will show how we can integrate the ADCS Cert Manager Provider into TAP directly.

The first step is to install the Provider using the Helm chart using a custom values file with some specific configurations. Lets create that values file:

cat <<EOF > adcs-values.yaml
crd:
  install: true
controllerManager:
  manager:
    image:
      repository: ghcr.io/vrabbi/adcs-issuer
      tag: 2.0.8
    resources:
      limits:
        cpu: 100m
        memory: 500Mi
      requests:
        cpu: 100m
        memory: 100Mi
  rbac:
    enabled: true
    serviceAccountName: adcs-issuer
    certManagerNamespace: cert-manager
    certManagerServiceAccountName: cert-manager
  replicas: 1
  environment:
    KUBERNETES_CLUSTER_DOMAIN: cluster.local
    ENABLE_WEBHOOKS: "false"
    ENABLE_DEBUG: "false"
  arguments:
    enable-leader-election: "true"
    cluster-resource-namespace: "adcs-issuer"
    zap-log-level: 5
    disable-approved-check: "false"
  securityContext:
    runAsUser: 1000
  enabledWebHooks: false
  enabledCaCerts: false
  caCertsSecretName: ca-certificates
metricsService:
  enabled: true
  ports:
  - name: https
    port: 8443
    targetPort: https
  type: ClusterIP
webhookService:
  ports:
  - port: 443
    targetPort: 9443
  type: ClusterIP
EOF

Now we can add the helm repository and install the chart:

helm repo add djkormo-adcs-issuer https://djkormo.github.io/adcs-issuer/
helm install adcs-issuer  djkormo-adcs-issuer/adcs-issuer \
 --version 2.0.8 \
 --namespace adcs-issuer \
 --values adcs-values.yaml \
 --create-namespace

With the ADCS provider installed we now need to create our ClusterAdcsIssuer which is the equivalent of the default ClusterIssuer but for the ADCS provider.

To do this we need to create a secret with NTLM credentials for ADCS:

cat <<EOF | kubectl apply -n adcs-issuer -f - 
apiVersion: v1
stringData:
  password: MySecretPassword
  username: MyAwesomeUsername
kind: Secret
metadata:
  name: adcs-issuer-credentials
  namespace: adcs-issuer
type: Opaque
EOF

And now we can create our ClusterAdcsIssuer. In order to do this we need 3 pieces of information:

• The Base64 encoded CA cert of the ADCS serve
• The FQDN of the ADCS Web Enrollment URL
• The Certificate Template we want to use to generate the certificates

With that data we can create our issuer:

cat <<EOF | kubectl apply -f -
apiVersion: adcs.certmanager.csf.nokia.com/v1
kind: ClusterAdcsIssuer
metadata:
  name: adcs
spec:
  caBundle: <BASE64_ENCODED_CA_DATA>
  credentialsRef:
    name: adcs-issuer-credentials
  statusCheckInterval: 1m
  retryInterval: 1m
  url: https://<ADCS_WEB_ENROLLMENT_FQDN>/certsrv/
  templateName: "<TEMPLATE NAME>"
EOF

Now that we have our setup of the ADCS provider configured we can test out the generation of a certificates by using the following example:

cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: adcs-cert-tap-test
spec:
  dnsNames:
  - test.vrabbi.cloud
  issuerRef:
    group: adcs.certmanager.csf.nokia.com
    kind: ClusterAdcsIssuer
    name: adcs
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - vRabbi
  secretName: adcs-cert-tap-test
EOF

If the setup works as expected you should have a secret called adcs-cert-tap-test with 3 keys including tls.crt, tls.key and ca.crt.

kubectl get secret adcs-cert-tap-test -o yaml

With that now working, we now need to integrate this into TAP which requires a few overlays and depends on the profile in which you are setting this up.

I will describe bellow the configuration for a full profile cluster, but for any other profile, you simply can exclude the overlays for the irrelevant packages.

When creating a certificate using an external Cert Manager provider, as we see in the example above we need to specify the issuerRef a bit differently as we need the API Group of the custom issuer and we need the kind to be the specific kind of the custom issuer vs. the default used in TAP which is the standard OOTB ClusterIssuer kind.

In order to make these changes we first need to understand which parts of TAP need to be manipulated. This splits into 2 key areas:

• Knative – All web workloads get auto generated TLS certificates and we need to configure Knative to use our custom issuer
• Other Packages with Exposed HTTPProxy resources – some packages (TAP GUI, Metadata Store, API Portal, API Auto Registration) expose services outside the cluster and define certificates and HTTPProxy resources.

For the Knative configuration, if we search through the docs, we can see that there is a configmap in the knative-serving namespace called config-certmanager where we can define the issuer to use.

When trying to see how this is handled in TAP I used the following process:

• Find the imgpkg bundle URI of the cnrs package

kubectl get package -n tap-install cnrs.tanzu.vmware.com.2.4.1 -o json | jq -r .spec.template.spec.fetch[0].imgpkgBundle.image

• Pull down the bundle to our local machine

imgpkg pull -b harbor.vrabbi.cloud/tap/tap-packages@sha256:57569892d8371ed52c5ebd6177930d3de490e1a1095b39873dc9c12d717cd16a -o cnrs

• Explore the package files. Here we can find the relevant files at the following path:

packages/serving/bundle/config/overlays/

• The overlay used in TAP for configuring this configmap is called overlay-knative-config-certmanager.yaml The contents of the file are:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:yaml", "yaml")
#@ load("@ytt:assert", "assert")
#@ load("@ytt:data", "data")
#@ load("values.star", "issuer_ref")

#@ if data.values.ingress_issuer:
#@ if data.values.domain_name or data.values.domain_config:
#@overlay/match by=overlay.subset({"metadata":{"name":"config-certmanager","namespace":"knative-serving"}})
---
data:
  #@overlay/match missing_ok=True
  issuerRef: #@ yaml.encode(issuer_ref())
#@ else:
#@ assert.fail("cannot set an ingress_issuer without configuring a custom domain")
#@ end
#@ end

• Create a custom overlay which similarly updates the issuerRef but with our specific needs
• Apply the overlay via TAP values

With the idea of how we can do this now understood we can write our overlay which needs to be stored in a secret in the tap-install namespace so it can be used as an overlay on the cnrs package.

The final overlay for KNative is as follows:

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"metadata":{"name":"config-certmanager","namespace":"knative-serving"}})
---
data:
  #@overlay/match missing_ok=True
  issuerRef: |
    kind: ClusterAdcsIssuer
    name: adcs
    group: adcs.certmanager.csf.nokia.com

We can now take this and create secret with the content:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: adcs-knative-overlay
  namespace: tap-install
stringData:
  zzz-adcs-package-overlay.yml: |
    #@ load("@ytt:overlay", "overlay")

    #@overlay/match by=overlay.subset({"metadata":{"name":"config-certmanager","namespace":"knative-serving"}})
    ---
    data:
      #@overlay/match missing_ok=True
      issuerRef: |
        kind: ClusterAdcsIssuer
        name: adcs
        group: adcs.certmanager.csf.nokia.com
EOF

We can now move on to all of thew other packages where the general exploration phase is the same but what we notice in the relevant packages is that TAP uses Certificate resources directly stamped out as part of the packages which actually works great for us, as we can create one overlay that can apply to all of these packages!

The final overlay for the other packages is:

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind": "Certificate"}), expects="1+"
---
spec:
  #@overlay/match missing_ok=True
  issuerRef:
    #@overlay/match missing_ok=True
    name: adcs
    #@overlay/match missing_ok=True
    kind: ClusterAdcsIssuer
    #@overlay/match missing_ok=True
    group: adcs.certmanager.csf.nokia.com

And we can just like with the Knative overlay create a secret with this overlay as well:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: adcs-certificate-overlay
  namespace: tap-install
stringData:
  zzz-adcs-package-overlay.yml: |
    #@ load("@ytt:overlay", "overlay")

    #@overlay/match by=overlay.subset({"kind": "Certificate"}), expects="1+"
    ---
    spec:
      #@overlay/match missing_ok=True
      issuerRef:
        #@overlay/match missing_ok=True
        name: adcs
        #@overlay/match missing_ok=True
        kind: ClusterAdcsIssuer
        #@overlay/match missing_ok=True
        group: adcs.certmanager.csf.nokia.com
EOF

With those secrets created the final step is to tell TAP to use these overlays and apply them to the relevant package installations which can easily be done via the following stanza in your TAP Values file:

package_overlays:
- name: tap-gui
  secrets:
  - name: adcs-certificate-overlay
- name: api-auto-registration
  secrets:
  - name: adcs-certificate-overlay
- name: api-portal
  secrets:
  - name: adcs-certificate-overlay
- name: metadata-store
  secrets:
  - name: adcs-certificate-overlay
- name: cnrs
  secrets:
  - name: adcs-knative-overlay

With this behind us we can now apply the new values using the tanzu CLI as usual and we should see all certificates being generated with our custom ADCS issuer!