Posted on by vrabbi

TAP PR Flow with Azure DevOps

TAP has an awesome GitOps PR flow, but OOTB it does not work with Azure DevOps. In this post we will see how we can make it work with Azure DevOps as well.

Why Azure DevOps Is Difficult

While Azure DevOps does have git repositories, There are 3 main issues with the Azure DevOps Git implementation.

1. They have not actually implemented the Git v1 API and require the use of the “multi ack” protocol which is not implemented in go-git which is the main git implementation library in go which is used in nearly all gitops tools.

2. The URL of a git repository does not follow the standard path format like is implemented in github, gitlab, gitea, etc. which makes templating difficult.

3. Azure DevOps does not support cloning repositories with the stanmdard “.git” suffix added to the URL, which in many platforms and solutions is added automatically to the URL and with Azure DevOps it simply does not work.

Beyond these specific low level issues in Azure DevOps, but very likely due to these issues, the tool which is used in TAP to create PRs in the GitOps flow which is jx-scm from the JenkinsX project does not support Azure DevOps.

Time to find a solution

How to create a PR

When searching how to create a PR via the command line in Azure DevOps, I found that the official method is to use the Azure CLI, and then adding the Azure DevOps extension. The issue i found is that this would increase the size of the container image used to create the PRs by 1.2GB!!!!!!!!!!

With this being immediately disregarded as a possible solution for me, I decided that using the REST API via simple curl commands is probably the way to go.

While Azure DevOps is a clunky Git implementation, the REST API is actually very well documented, so finding the right API was actually really easy.

Once i found the right API, I took a look at the OOTB Tekton ClusterTask which is used for pushing commits and opening a PR, and updated the script to support Azure DevOps as well.

The change that was needed is in the final step, which is called “open-pr” and the new script value needs to basically, check if the git provider type is azure devops, and if so use our custom logic and otherwise, simply use the default as it comes.

The final script looks like this:

#!/usr/bin/env bash

set -o errexit
set -o pipefail

head_branch=$(cat /workspaces/ws/commit_branch | tr -d '\n')
token=$(cat $(credentials.path)/.git-credentials | sed -e 's/https:.*://' | sed -e 's/@.*//')
if [[ "$(params.git_server_kind)" == "azure" ]]; then
  JSON_FMT='{"sourceRefName":"refs/heads/%s","targetRefName":"refs/heads/%s","title":"%s","description":"%s"}\n'
  bodyJson=`printf "$JSON_FMT" "$head_branch" "$(params.base_branch)" "$(params.pull_request_title)" "$(params.pull_request_body)"`
  repo_api_path=`echo $(params.repository_name) | sed 's|/_git/|/_apis/git/repositories/|g' -`
  uri="$(params.git_server_address)/$(params.repository_owner)/$repo_api_path/pullRequests?api-version=3.0"

  echo $bodyJson
  echo $uri

  base64AuthInfo=$(echo -n "test:$token" | base64)
  result=$(curl -X POST -u "test:$token" -H "Content-Type: application/json" -d "$bodyJson" "$uri")
  
  echo $result | jq -r '.repository.webUrl + "/pullrequest/" + (.pullRequestId|tostring)' > $(results.pr-url.path)
else
  jx-scm pull-request create \
  --kind "$(params.git_server_kind)" \
  --server "$(params.git_server_address)" \
  --token "$token" \
  --owner "$(params.repository_owner)" \
  --name "$(params.repository_name)" \
  --head "$head_branch" \
  --title "$(params.pull_request_title)" \
  --body "$(params.pull_request_body)" \
  --base "$(params.base_branch)" \
  --allow-update 2>&1 |
  tee stdoutAndSterr.txt
  cat stdoutAndSterr.txt | sed -n -e 's/^.*\. url: //p' > $(results.pr-url.path)
fi

As you can see, we are simply checking the parameter which is supplied in the tap-values file called git_server_kind and if it is set to “azure” we use our own custom logic, and otherwise we use the logic provided by VMware using the jx-scm CLI.

Solving the Path issue

This actually doesn’t need to be solved, but what it requires is that when filling in the TAP Values, the repository_name variable under the gitops section for the supply chain config, must be in the format <PROJECT NAME>/_git/<REPO NAME>.

While this could be solved and most likely hidden from the end user in multiple different ways, It would require changes in too many places, it simply was not worth the effort as this is a global setting and is not a value that is needed to be configured by end users on a day 2 day basis.

Solving the lack of Git v1 API support

As mentioned above, Azure DevOps does not work with go-git which is the default git implementation used in TAP specifically but also is the standard use in nearly all go based solutions that integrate with git.

Luckily TAP does support configuring the git implementation to use, with the default being go-git, but also supporting libgit2.

libgit2 does include support for the multi ack protocol which is what Azure DevOps supports, so that issue can be solved by adding the variable “git_implementation: libgit2” under the ootb_delivery_basic top level key as well as under the configured supply chains top level key.

libgit2 does work, however it should not be used unless no other option is available, as it is much less stable then go-git. but in this situation, we do not have a real choice.

Solving the “.git” suffix issue

This issue unfortunately requires changes to be made in multiple different locations, as the OOTB templates, add the “.git” suffix to repo URLs when templating the resources out.

The templates we need to change are the ones that create the gitrepository CRs and the ones that create the delivery CRs.

This means we have 2 ClusterSourceTemplates to update:

  • source-template
  • delivery-source-template

And we also need to update the ClusterTemplate that creates the delivery resource:

  • deliverable-template

The change in all of them is the same. We have 2 YTT functions in all 3 of these templates called “git_repository” and “mono_repository”.

In these function we simply need to remove the addition of the “.git” at the end of the functions.

Once you have completed these changes, you are ready to use the GitOps flow and PR flow with Azure DevOps.

Example GitOps section in TAP Values files for Azure DevOps

An example of the relevant sections in the TAP Values that you can use once the above steps are completed would be

ootb_delivery_basic:
  git_implementation: libgit2
ootb_supply_chain_testing_scanning:
  git_implementation: libgit2
  gitops:
    server_address: https://dev.azure.com
    repository_owner: vrabbi
    repository_name: test-az-devops-tap/_git/test-az-devops-tap
    branch: main
    commit_strategy: pull_request
    pull_request:
      server_kind: azure
      commit_branch: ""
      pull_request_title: ready for review
      pull_request_body: generated by supply chain
    ssh_secret: git-creds

Summary

While TAP OOTB currently doesn’t support Azure DevOps in the GitOps PR flow, as I hope you can see from this post, the extensible nature of TAP, makes such integrations relatively very easy to do.

Building this solution was a matter of about 1.5 hours of investigation and trial and error, which is not bad considering the intricacies in solving this particular solution which I hope again shows just how flexible the platform really is, and how easy it is to customize it to your own needs!

Leave a Reply

%d bloggers like this: