Dynamically Launch Infrastructure With ArgoCD, Crossplane, and Sealed Secrets – new Satbir Chahal November 22, 2022

Dynamically Launch Infrastructure With ArgoCD, Crossplane, and Sealed Secrets

At OpsVerse, we provision infrastructure. A ton of it.

For the uninitiated, infrastructure is any underlying system that your core app relies upon to function properly. This can be:

    • A Kubernetes cluster where your packaged app is deployed to and runs
    • A Kafka topic your app produces to for other applications to consume
    • A PostgresQL database that your app uses to store critical and stateful information

A Simple Use Case

Say you have an application that simply creates a file every minute and uploads it to an AWS S3 bucket

"""
This script will :
  (1) Wait for a minute
  (2) Create a file:
        Named .txt
        A random number as its contents
  (3) Upload the file to an S3 bucket
"""
import logging
import random
import time
import boto3
from botocore.exceptions import ClientError

BUCKET_NAME = 'my-cool-s3-bucket'


while True:
    time.sleep(3)

    file_name = time.strftime('%Y%m%d%H%M%S') + '.txt'

    f = open(file_name, "w")
    n = f.write(str(random.randint(0, 100)))
    f.close()

    print("Uploading {} to {}...".format(file_name, BUCKET_NAME))
    s3_client = boto3.client('s3')
    try:
        key = 'test/dropoff/' + file_name
        response = s3_client.upload_file(file_name, BUCKET_NAME, key)
    except ClientError as exc:
        logging.error(exc)

One prerequisite (for this script to run as intended) is that an S3 bucket must be created.

What Does It Mean to Shift Left

By dynamically provisioning our infrastructure earlier in the development lifecycle, we’re “shifting left”.  That is simply a fancy buzzword (phrase?) we’ve seen thrown around lately describing the self-serviceability in engineering teams. In other words: can we as developers bring up some infrastructure without opening up a ticket down the line to involve someone else?

Let’s look at the above use case as an example.

How it was before

In order to create the my-cool-s3-bucket bucket, we would still follow the Infrastructure-as-Code (IaC) principles – where all infrastructure changes are in Git. In this case, we’d use the Terraform AWS provider to create an S3 resource with a commit like the following:

resource "aws_s3_bucket" "cool_bucket" {
  bucket = "my-cool-s3-bucket"

  tags = {
    Environment = "Dev"
  }
}
Then a team member (or a CI Runner) who has access to the cloud environment can run something like terraform apply to reconcile the desired state. This is an absolutely okay way to manage and provision cloud infrastructure. But, as you can see, it is somewhat out-of-band.

How we do it now

Since our apps run on Kubernetes (more on this in a later post), we’re able to make sure all our Kubernetes clusters have ArgoCD, Bitnami Sealed Secrets, and Crossplane controllers installed on them.

Therefore, because the app is already defined as a raw Kubernetes manifest (or Helm chart), this allows the developer to simply add another manifest to the deployment (or chart), that would look like this:

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: my-cool-s3-bucket
  namespace: my-cool-ns
spec:
  deletionPolicy: Delete
  providerConfigRef:
    name: my-dev-aws
  forProvider:
    acl: private
    publicAccessBlockConfiguration:
      blockPublicAcls: true
      blockPublicPolicy: true
      ignorePublicAcls: true
      restrictPublicBuckets: true
    locationConstraint: us-east-1

Just like how the app is deployed, Kubernetes will treat the management of the S3 bucket like a first-class citizen alongside the app.

Setup – How to achieve this

  • Install Argo CD
    • Where: This can essentially run anywhere, as long as it can reach your K8s control plan (shameless plug: if you’re an OpsVerse user, you can launch a DeployNow instance to quickly get a managed Argo CD)
    • Why: This will enable pure GitOps-based deployments going forward
  • Install Bitnami’s Sealed Secrets
    • Where: On your target cluster (where your apps will run)
    • Why: So you don’t have to worry about checking in plain-text secrets into Git. This will allow you to commit kind: SealedSecrets (your K8s Secret in its encrypted form) into Git… which the SealedSecrets controller in your target cluster will convert to a standard Secret once it sees the resource.

At this point, you can use GitOps-based deployments to get up-and-running with Crossplane (so that cloud resources, like the S3 bucket described above, can be provisioned as a first-class Kubernetes resource alongside the app).

To get Crossplane setup in your target cluster via Argo, we use the App of Apps pattern (where we define a “root” Argo CD Application that points to a directory with other child Argo Applications).1

For example, you Argo App manifest may look something like this:

project: default
source:
  repoURL: 'https://.git'
  path: path/to/argo/apps
  targetRevision: HEAD
  directory:
    recurse: true
destination:
  server: 'https://kubernetes.default.svc'
  namespace: default

… then in path/to/argo/apps you’ll want 3 Argo apps for Crossplane:

path/to/argo
└── apps
    ├── crossplane-controller-app.yaml
    ├── crossplane-provider-app.yaml
    ├── crossplane-provider-cfg-app.yaml

Let’s go over which specific Kubernetes resources each of these Argo Apps will deploy.

crossplane-controller-app.yaml

This is the actual Crossplane controller (with the applicable ClusterRoles, ServiceAccounts, and Bindings). It’s needed first to reconcile any Crossplane custom resources that we’ll eventually create on the cluster. So, the path in crossplane-app.yaml should be pointing to what is generated from the helm repo at https://charts.crossplane.io/stable

crossplane-provider-app.yaml

Now that the Crossplane controller is in place, we want to make sure Argo syncs the specific Crossplane provider (e.g., the cloud provider you care for, like AWS, GCP, Azure, etc). In order for this to sync after the controller is synced, we utilize Argo sync waves. So, in our crossplane-provider-app.yaml, we’ll have an annotation like this (arbitrarily picking a value of 5):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    # This must be synced after the crossplane-controller but
    # before crossplane-provider-config is run
    argocd.argoproj.io/sync-wave: "5"
...
spec:
  ...
  source:
    ...
    path: path/to/crossplane-provider/resources/

The resources inside path/to/crossplane-provider/resources/ are basically a Crossplane Provider CR and a SealedSecret CR which has the creds for the provider.

For example, if wanting an AWS provider, you may have this in that directory:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: "crossplane/provider-aws:v0.24.1"
And also the SealedSecret, which you can auto-generate by making sure your kube context is on the target cluster where the sealed-secrets controller was installed, so you can use kubeseal to convert Secret to SealedSecret (which will be fitting to commit to Git). The secret.yaml can be created by the steps provided by Crossplane’s documentation, which we then further convert to a SealedSecret:
# Generated via taking a "kind: Secret" (creds key = the toml base64 encoded) and running
#   cat secret.yaml | kubeseal --controller-namespace \
#   --controller-name -sealedsecrets --format yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: aws-creds
spec:
  encryptedData:
    creds: AgB6brqgnLqNRSV
  template:
    metadata:
      name: aws-creds
    type: Opaque

When these resources are synced to your cluster, all that’s needed is for us to tell Crossplane which configuration to use for the provider.

crossplane-provider-app.yaml

With the controller synced, followed by the Crossplane AWS Provider (along with a SealedSecret of the creds, which gets reconciled to a Secret), we can at last configure the credential for the AWS provider so that it is ready to use.

Using Argo sync waves, we ensure this is synced after the provider resource is there, to avoid any “provider not found” type errors. So, in our crossplane-provider-cfg-app.yaml, we’ll have an annotation like this (arbitrarily picking a value greater than 5):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    # This must be synced after both the crossplane-controller
    # and crossplane-provider
    argocd.argoproj.io/sync-wave: "10"
...
spec:
  ...
  source:
    ...
    path: path/to/crossplane-provider-cfg/resources/
Then, in path/to/crossplane-provider-cfg/resources/, we simpy tie the credentials secret to the provider by creating a ProviderConfig CR:
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  # ProviderConfigs are cluster-scoped, so no namespace needed
  name: my-provider-config
spec:
  credentials:
    source: Secret
    secretRef:
      name: aws-creds
      key: creds

Once synced, we can begin provisioning cloud resources natively!

Now, if a developer had to update his/her deployment.yaml to his/her latest container image (where this version of the code also requires an S3 bucket), (s)he can simply add a file like bucket.yaml:

End Result

app
└── deploy
    ├── deployment.yaml
    ├── bucket.yaml
apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  name: my-cool-s3-bucket
  namespace: my-cool-ns
spec:
  deletionPolicy: Delete
  providerConfigRef:
    name: my-dev-aws
  forProvider:
    acl: private
    publicAccessBlockConfiguration:
      blockPublicAcls: true
      blockPublicPolicy: true
      ignorePublicAcls: true
      restrictPublicBuckets: true
    locationConstraint: us-east-1

The benefits are immediate. This allows us to do many things automatically:

  • This provisioning stays in-band with the app for which it’s intended
  • New developers save time on learning where to add, say, a comparable Terraform module (and how to run it)
  • Options exist to cleanup on resource deletion. So lingering resources are auto-deleted too, saving cost on both infra and time.

We can deep dive further into any of these topics, including further best practices, hardening, and other patterns we use. This is intended as a high-level, hands-on “Getting Started” post by putting Argo CD, Crossplane, and SealedSecrets together for seamless shifting of infrastructure provisioning leftward.

What’s Next

We’ll have future blog posts that cover some of the topics that arose from this:

  • We’ll extend on this example to see how the app can access the bucket using Service Accounts and IAM roles
  • Why Kubernetes is King; allowing you to shift left, stay cloud-agnostic, run anywhere, and avoid any vendor lock-in

Footnotes

1  With the newer releases of Argo, the “App Of App” pattern may be giving way to the ApplicationSet controller. Check it out!

2 Technically, these are composite resource claims (XRC) that bind to composite resources (XR). More details here.