In the platform engineering space, two of the most impactful and interesting tools are Backstage and Crossplane. In this post, we will discuss and demonstrate how the two tools can be integrated to provide a great baseline for the ultimate developer platform.
What is Backstage
Backstage is an open platform for building developer portals. It provides a unified interface for managing software, infrastructure, and resources. By leveraging Backstage’s plugin architecture, organizations can customize and extend its capabilities to suit their needs, offering developers a centralized view of their tools and workflows.
What is Crossplane
Crossplane is an open-source tool that enables organizations to provision and manage cloud infrastructure and services using Kubernetes declaratively. It extends Kubernetes APIs with custom resource definitions (CRDs) to manage both infrastructure and application workloads in a unified way.
What is Possible Out of the Box
Out of the box, Backstage offers plugins and templates to manage software catalogs, scaffolding, CI/CD integrations, and more. Similarly, Crossplane provides a rich ecosystem for declaratively managing infrastructure resources. However, the integration of the two offers a unique capability to visualize and manage infrastructure as part of the developer workflow.
What is Missing Currently
While Backstage and Crossplane are powerful tools individually, some gaps exist in their integration:
- Seamless visualization of Crossplane resources in Backstage.
- Auto-ingestion of Kubernetes resources into Backstage catalogs.
- Enhanced scaffolding capabilities tailored for Crossplane resources.
- Efficient management and cleaning of catalog entities.
How This Can Be Solved
This is where custom plugins come into play. The plugins crossplane-resources, kubernetes-ingestor, and scaffolder-backend-module-terasky-utils bridge these gaps by providing enhanced integration capabilities.
The source code of these plugins, as well as installation instructions for each of the plugins discussed bellow and more can be found on Github in the following repository.
Building a Catalog Processor
The kubernetes-ingestor plugin acts as a catalog processor, ingesting Kubernetes workloads and Crossplane claims directly into Backstage. It supports annotations to customize the ingestion process.
For each workload that is registered based on a standard Kubernetes resource we will recieve the following type of UX OOTB without any annotations or custom configurations:
And for Crossplane Claims it will look like this:
Example Configuration:
kubernetesIngestor:
components:
enabled: true
taskRunner:
frequency: 10
timeout: 600
crossplane:
claims:
ingestAllClaims: true
This configuration enables automatic ingestion of claims and other Kubernetes resources into the catalog without needing to create and manage catalog-info.yaml files in git for each object. Via annotations we can manipulate how that backstage component will be created by providing the data directly on the kubernetes resources. While this is very beneficial for Crossplane, it has a much wider usage opportunity for the entire Kubernetes ecosystem.
An example of the usage of these annotation on a standard Kubernetes Deployment resource could look like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: yelb-appserver
labels:
app: yelb-appserver
tier: middletier
annotations:
terasky.backstage.io/add-to-catalog: 'true'
terasky.backstage.io/owner: dev-team
terasky.backstage.io/component-type: service
terasky.backstage.io/lifecycle: experimental
terasky.backstage.io/dependsOn: 'Component:yelb-db,Component:yelb-redis'
terasky.backstage.io/kubernetes-label-selector: 'app=yelb-appserver,tier=middletier'
terasky.backstage.io/source-code-repo-url: "https://github.com/dambor/yelb-catalog"
terasky.backstage.io/source-branch: "main"
terasky.backstage.io/techdocs-path: "components/yelb-appserver"
spec:
replicas: 1
selector:
matchLabels:
app: yelb-appserver
tier: middletier
template:
metadata:
labels:
app: yelb-appserver
tier: middletier
spec:
containers:
- name: yelb-appserver
image: mreferre/yelb-appserver:0.7
ports:
- containerPort: 4567
This will end up with a resource that has much more information attached and is much more accurately documented in Backstage:
The potential of this, and the simplification this brings to management is endless, and with small customizations in the code, you could add any other annotations or logic as needed per your organizations needs.
Building a Frontend Plugin
The crossplane-resources plugin provides a frontend interface for visualizing Crossplane resources. Developers can view resource details, YAML manifests, and relationships through a graph view, directly on the auto-ingested components based on any crossplane claim in your attached clusters!
Example Usage in Entity Pages
<EntityLayout>
<EntityLayout.Route path="/crossplane-resources" title="Crossplane Resources">
<CrossplaneAllResourcesTable />
</EntityLayout.Route>
<EntityLayout.Route path="/crossplane-graph" title="Crossplane Graph">
<CrossplaneResourceGraph />
</EntityLayout.Route>
</EntityLayout>
This adds Crossplane-specific views to Backstage’s entity pages.
There are 2 main options for the visualization.
Table View
This also supports viewing the YAML of a resource:
And an events viewer:
Graph View
This also supports viewing the YAML and events of a resource:
Generating Software Templates
The kubernetes-ingestor plugin beyond ingesting resources as components into backstage, also has the ability to generate backstage software templates for Crossplane claims based on the corresponding XRDs. These templates are automatically registered in the Backstage catalog and provide an auto updating, constantly evolving self service portal for generating manifests for your custom APIs defined in Crossplane, without needing to double the work of defining the schema in backstage in order to expose a user friendly UI above the Crossplane abstractions.
The flow in these templates is very simple:
Provide the basic metadata
This is where you supply the name and namespace of the resource you want to create:
Resource Schema
The next page is the spec fields you have defined in your XRD:
This will enforce required fields, enums, field validations and more, just like you defined in your XRD!
Crossplane Settings
This is where you can set the crossplane specific settings like the connection secret name to create lifecycle related fields and composition selection method which by default is set to runtime which means the composition will be chosen at runtime and not specified in the manifest:
You can also select to have it with a direct reference which when this option is selected a dropdown of all the discovered compositions is provided to select from:
Or you can select to use the label based selection:
Creation Settings
These templates can be configured via the app-config.yaml to publish the generated manifests to git via creating a PR for you, or can simply generate a YAML file which you can download. The recommended approach would be to auto push the manifests to git, and have a GitOps tool like FluxCD, Carvel Kapp Controller, or ArgoCD auto deploy these manifests to your cluster, for a seamless end to end experience. When choosing GitOps you can either hide the repo selection and have it defined globally in the app-config.yaml or it can support repo selection in the template itself as shown bellow. This page also enables you to select how the manifest should be placed in the GitOps repo. The default option is cluster-scoped which when selected will allow you to select from a list of the connected clusters where this XRD has been found, and the template will push a copy of the manifest into cluster dedicated folders in the GitOps repo:
You could also chose to have it be namespace scoped in cases where you deploy the same configurations to all clusters and don’t maintain separate folder structures for each cluster:
And if needed there is also a custom option where you can supply the path in the repo you want the file created under:
There also may be some cases where while the environment is setup to support GitOps based management that you don’t want to push the manifest to git. For this use case, you can uncheck the box at the top of the page, and no manifest will be pushed to git, but it will be available for download directly.
Review and Submit
After reviewing the inputs and submitting the form you will receive the results:
In this example we get the link to the generated PR, as well as a link to download the generated YAML manifest directly. Because we selected 2 clusters “kubetopus” and “spectro-shared” and our SQLServerInstance is set to be deployed in the demo namespace, when we look at the PR we will see 2 files being created:
The file paths are in the format <CLUSTER NAME>/<NAMESPACE>/<RESOURCE KIND>/<RESOURCE NAME>.yaml This enables easy gitops configuration, but can easily be changed with just a few lines of code in the kubernetes-ingestor plugin if needed, or you can use one of the other methods mentioned above if they meet your needs.
Building Custom Scaffolder Actions
The scaffolder-backend-module-terasky-utils plugin introduces custom actions like terasky:claim-template for generating YAML manifests for Crossplane claims based off of the inputted parameters by the user, and terasky:catalog-info-cleaner for cleaning catalog entities, and allowing the auto generated component manifests from the Kubernetes Ingestor plugin to be pushed to Git, if and when you decide that you want to manage your catalog in the more traditional manner rather then via Kubernetes annotations.
Example Action: Claim Template
steps:
- id: generate-claim-manifest
name: Generate Kubernetes Resource Manifest
action: terasky:claim-template
input:
parameters: ${{ parameters }}
apiVersion: vrabbi.cloud/v1alpha1
kind: MyCustomCrossplaneAPI
clusters: ${{ parameters.clusters }}
This generates a YAML manifest for a claim and saves it to the filesystem. This can then be further processed and pushed to a git repo for example using the github, gitlab, bitbucket or azure devops actions, or exposed as a yaml file for downloading directly.
Implementing the Permission Framework
The crossplane-resources frontend plugin also includes configuration options to enable permission checks for different exposed elements, allowing you to tailor the view to different personas in the organization. This requires also installing the crossplane-permissions-backend plugin in order to add the permissions to the permission framework within backstage.
In the repository you can see that I have integrated also the community RBAC plugin which allows you to easily manage via CSV files or from the UI creation of roles, and assigning roles to users and groups within your backstage instance, to provide a much more streamlined and simple process, rather then needing to implement permission policies in Typescript embedded within your backstage code base. The permissions available in the plugin are granular by design to allow for you to customize roles based on the different personas and requirements in your organization. When a user does not have any crossplane related permissions they will see for example:
But with the permissions to list claims and composite resources, and view the yaml of claims only it would look like this:
In order to enable the frontend to use the permission framework, you can simply add in your app-config.yaml
crossplane:
enablePermissions: true
Lessons Learned
- Clear integration points exist between Backstage and Crossplane, but they require thoughtful customization.
- Annotation-driven approaches simplify resource ingestion but need proper documentation for end-users.
- Combining frontend, backend, and scaffolding plugins creates a holistic developer experience.
- Building Backstage plugins is complex at the beginning but becomes much easier over time.
Next Steps
- Explore additional integrations and possible optimizations of the cluster querying to speed up the frontend plugin rendering, and require less permissions from a Kubernetes perspective to run the plugins.
- Extend the scaffolder actions for more advanced workflows.
- Optimize the UI for larger-scale Crossplane resources, enabling a better UX for complex XRDs, including collapsible sections, and more UX tweaking.
Summary
By combining Backstage and Crossplane with custom plugins, organizations can offer developers a seamless platform experience. These plugins address key gaps in visibility, automation, and management, unlocking the potential for enhanced developer productivity.