November 22, 2022

Reading Time:


Dynamically Launch Infrastructure With ArgoCD, Crossplane, and Sealed Secrets


At OpsVerse, we provision infrastructure. A ton of it.

For the uninitiated, infrastructure is any underlying system that your core app relies upon to function properly. This can be:

    • A Kubernetes cluster where your packaged app is deployed to and runs
    • A Kafka topic your app produces to for other applications to consume
    • A PostgresQL database that your app uses to store critical and stateful information

A Simple Use Case

Say you have an application that simply creates a file every minute and uploads it to an AWS S3 bucket

One prerequisite (for this script to run as intended) is that an S3 bucket must be created.

What Does It Mean to Shift Left

By dynamically provisioning our infrastructure earlier in the development lifecycle, we’re “shifting left”.  That is simply a fancy buzzword (phrase?) we’ve seen thrown around lately describing the self-serviceability in engineering teams. In other words: can we as developers bring up some infrastructure without opening up a ticket down the line to involve someone else?

Let’s look at the above use case as an example.

How it was before

In order to create the my-cool-s3-bucket bucket, we would still follow the Infrastructure-as-Code (IaC) principles – where all infrastructure changes are in Git. In this case, we’d use the Terraform AWS provider to create an S3 resource with a commit like the following:

Then a team member (or a CI Runner) who has access to the cloud environment can run something like terraform apply to reconcile the desired state.
This is an absolutely okay way to manage and provision cloud infrastructure. But, as you can see, it is somewhat out-of-band.

How we do it now

Since our apps run on Kubernetes (more on this in a later post), we’re able to make sure all our Kubernetes clusters have ArgoCD, Bitnami Sealed Secrets, and Crossplane controllers installed on them.

Therefore, because the app is already defined as a raw Kubernetes manifest (or Helm chart), this allows the developer to simply add another manifest to the deployment (or chart), that would look like this:

Just like how the app is deployed, Kubernetes will treat the management of the S3 bucket like a first-class citizen alongside the app.

Setup – How to achieve this

  • Install Argo CD
    • Where: This can essentially run anywhere, as long as it can reach your K8s control plan (shameless plug: if you’re an OpsVerse user, you can launch a DeployNow instance to quickly get a managed Argo CD)
    • Why: This will enable pure GitOps-based deployments going forward
  • Install Bitnami’s Sealed Secrets
    • Where: On your target cluster (where your apps will run)
    • Why: So you don’t have to worry about checking in plain-text secrets into Git. This will allow you to commit kind: SealedSecrets (your K8s Secret in its encrypted form) into Git… which the SealedSecrets controller in your target cluster will convert to a standard Secret once it sees the resource.

At this point, you can use GitOps-based deployments to get up-and-running with Crossplane (so that cloud resources, like the S3 bucket described above, can be provisioned as a first-class Kubernetes resource alongside the app).

To get Crossplane setup in your target cluster via Argo, we use the App of Apps pattern (where we define a “root” Argo CD Application that points to a directory with other child Argo Applications).1

For example, you Argo App manifest may look something like this:

… then in path/to/argo/apps you’ll want 3 Argo apps for Crossplane:

Let’s go over which specific Kubernetes resources each of these Argo Apps will deploy.


This is the actual Crossplane controller (with the applicable ClusterRoles, ServiceAccounts, and Bindings). It’s needed first to reconcile any Crossplane custom resources that we’ll eventually create on the cluster. So, the path in crossplane-app.yaml should be pointing to what is generated from the helm repo at


Now that the Crossplane controller is in place, we want to make sure Argo syncs the specific Crossplane provider (e.g., the cloud provider you care for, like AWS, GCP, Azure, etc). In order for this to sync after the controller is synced, we utilize Argo sync waves. So, in our crossplane-provider-app.yaml, we’ll have an annotation like this (arbitrarily picking a value of 5):

The resources inside path/to/crossplane-provider/resources/ are basically a Crossplane Provider CR and a SealedSecret CR which has the creds for the provider.

For example, if wanting an AWS provider, you may have this in that directory:

And also the SealedSecret, which you can auto-generate by making sure your kube context is on the target cluster where the sealed-secrets controller was installed, so you can use kubeseal to convert Secret to SealedSecret (which will be fitting to commit to Git). The secret.yaml can be created by the steps provided by Crossplane’s documentation, which we then further convert to a SealedSecret:

When these resources are synced to your cluster, all that’s needed is for us to tell Crossplane which configuration to use for the provider.


With the controller synced, followed by the Crossplane AWS Provider (along with a SealedSecret of the creds, which gets reconciled to a Secret), we can at last configure the credential for the AWS provider so that it is ready to use.

Using Argo sync waves, we ensure this is synced after the provider resource is there, to avoid any “provider not found” type errors. So, in our crossplane-provider-cfg-app.yaml, we’ll have an annotation like this (arbitrarily picking a value greater than 5):

Then, in path/to/crossplane-provider-cfg/resources/, we simpy tie the credentials secret to the provider by creating a ProviderConfig CR:

Once synced, we can begin provisioning cloud resources natively!

Now, if a developer had to update his/her deployment.yaml to his/her latest container image (where this version of the code also requires an S3 bucket), (s)he can simply add a file like bucket.yaml:

End Result

… and bucket.yaml will be a native Bucket resource, or any other
AWS managed resources people need 2 :

The benefits are immediate. This allows us to do many things automatically:

  • This provisioning stays in-band with the app for which it’s intended
  • New developers save time on learning where to add, say, a comparable Terraform module (and how to run it)
  • Options exist to cleanup on resource deletion. So lingering resources are auto-deleted too, saving cost on both infra and time.

We can deep dive further into any of these topics, including further best practices, hardening, and other patterns we use. This is intended as a high-level, hands-on “Getting Started” post by putting Argo CD, Crossplane, and SealedSecrets together for seamless shifting of infrastructure provisioning leftward.

What’s Next

We’ll have future blog posts that cover some of the topics that arose from this:

  • We’ll extend on this example to see how the app can access the bucket using Service Accounts and IAM roles
  • Why Kubernetes is King; allowing you to shift left, stay cloud-agnostic, run anywhere, and avoid any vendor lock-in


1 With the newer releases of Argo, the “App Of App” pattern may be giving way to the ApplicationSet controller. Check it out!

2 Technically, these are composite resource claims (XRC) that bind to composite resources (XR). More details here.


Written by Satbir Chahal

Subscribe to the OpsVerse blog

New posts straight to your inbox