At OpsVerse, we provision infrastructure. A ton of it.
For the uninitiated, infrastructure is any underlying system that your core app relies upon to function properly. This can be:
-
- A Kubernetes cluster where your packaged app is deployed to and runs
- A Kafka topic your app produces to for other applications to consume
- A PostgresQL database that your app uses to store critical and stateful information
A Simple Use Case
Say you have an application that simply creates a file every minute and uploads it to an AWS S3 bucket
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
""" This script will : (1) Wait for a minute (2) Create a file: Named .txt A random number as its contents (3) Upload the file to an S3 bucket """ import logging import random import time import boto3 from botocore.exceptions import ClientError BUCKET_NAME = 'my-cool-s3-bucket' while True: time.sleep(3) file_name = time.strftime('%Y%m%d%H%M%S') + '.txt' f = open(file_name, "w") n = f.write(str(random.randint(0, 100))) f.close() print("Uploading {} to {}...".format(file_name, BUCKET_NAME)) s3_client = boto3.client('s3') try: key = 'test/dropoff/' + file_name response = s3_client.upload_file(file_name, BUCKET_NAME, key) except ClientError as exc: logging.error(exc) |
One prerequisite (for this script to run as intended) is that an S3 bucket must be created.
What Does It Mean to Shift Left
By dynamically provisioning our infrastructure earlier in the development lifecycle, we’re “shifting left”. That is simply a fancy buzzword (phrase?) we’ve seen thrown around lately describing the self-serviceability in engineering teams. In other words: can we as developers bring up some infrastructure without opening up a ticket down the line to involve someone else?
Let’s look at the above use case as an example.
How it was before
In order to create the my-cool-s3-bucket bucket, we would still follow the Infrastructure-as-Code (IaC) principles – where all infrastructure changes are in Git. In this case, we’d use the Terraform AWS provider to create an S3 resource with a commit like the following:
|
1 2 3 4 5 6 |
resource "aws_s3_bucket" "cool_bucket" { bucket = "my-cool-s3-bucket" tags = { Environment = "Dev" } } |
Then a team member (or a CI Runner) who has access to the cloud environment can run something like terraform apply to reconcile the desired state.
This is an absolutely okay way to manage and provision cloud infrastructure. But, as you can see, it is somewhat out-of-band.
How we do it now
Since our apps run on Kubernetes (more on this in a later post), we’re able to make sure all our Kubernetes clusters have ArgoCD, Bitnami Sealed Secrets, and Crossplane controllers installed on them.
Therefore, because the app is already defined as a raw Kubernetes manifest (or Helm chart), this allows the developer to simply add another manifest to the deployment (or chart), that would look like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
apiVersion: s3.aws.crossplane.io/v1beta1 kind: Bucket metadata: name: my-cool-s3-bucket namespace: my-cool-ns spec: deletionPolicy: Delete providerConfigRef: name: my-dev-aws forProvider: acl: private publicAccessBlockConfiguration: blockPublicAcls: true blockPublicPolicy: true ignorePublicAcls: true restrictPublicBuckets: true locationConstraint: us-east-1 |
Just like how the app is deployed, Kubernetes will treat the management of the S3 bucket like a first-class citizen alongside the app.
Setup – How to achieve this
- Install Argo CD
- Where: This can essentially run anywhere, as long as it can reach your K8s control plan (shameless plug: if you’re an OpsVerse user, you can launch a DeployNow instance to quickly get a managed Argo CD)
- Why: This will enable pure GitOps-based deployments going forward
- Install Bitnami’s Sealed Secrets
- Where: On your target cluster (where your apps will run)
- Why: So you don’t have to worry about checking in plain-text secrets into Git. This will allow you to commit kind: SealedSecrets (your K8s Secret in its encrypted form) into Git… which the SealedSecrets controller in your target cluster will convert to a standard Secret once it sees the resource.
At this point, you can use GitOps-based deployments to get up-and-running with Crossplane (so that cloud resources, like the S3 bucket described above, can be provisioned as a first-class Kubernetes resource alongside the app).
To get Crossplane setup in your target cluster via Argo, we use the App of Apps pattern (where we define a “root” Argo CD Application that points to a directory with other child Argo Applications).1
For example, you Argo App manifest may look something like this:
|
1 2 3 4 5 6 7 8 9 10 |
project: default source: repoURL: 'https://.git' path: path/to/argo/apps targetRevision: HEAD directory: recurse: true destination: server: 'https://kubernetes.default.svc' namespace: default |
… then in path/to/argo/apps you’ll want 3 Argo apps for Crossplane:
|
1 2 3 4 5 |
path/to/argo └── apps ├── crossplane-controller-app.yaml ├── crossplane-provider-app.yaml ├── crossplane-provider-cfg-app.yaml |
Let’s go over which specific Kubernetes resources each of these Argo Apps will deploy.
crossplane-controller-app.yaml
This is the actual Crossplane controller (with the applicable ClusterRoles, ServiceAccounts, and Bindings). It’s needed first to reconcile any Crossplane custom resources that we’ll eventually create on the cluster. So, the path in crossplane-app.yaml should be pointing to what is generated from the helm repo at https://charts.crossplane.io/stable
crossplane-provider-app.yaml
Now that the Crossplane controller is in place, we want to make sure Argo syncs the specific Crossplane provider (e.g., the cloud provider you care for, like AWS, GCP, Azure, etc). In order for this to sync after the controller is synced, we utilize Argo sync waves. So, in our crossplane-provider-app.yaml, we’ll have an annotation like this (arbitrarily picking a value of 5):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: annotations: # This must be synced after the crossplane-controller but # before crossplane-provider-config is run argocd.argoproj.io/sync-wave: "5" ... spec: ... source: ... path: path/to/crossplane-provider/resources/ |
The resources inside path/to/crossplane-provider/resources/ are basically a Crossplane Provider CR and a SealedSecret CR which has the creds for the provider.
For example, if wanting an AWS provider, you may have this in that directory:
|
1 2 3 4 5 6 |
apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-aws spec: package: "crossplane/provider-aws:v0.24.1" |
And also the SealedSecret, which you can auto-generate by making sure your kube context is on the target cluster where the sealed-secrets controller was installed, so you can use kubeseal to convert Secret to SealedSecret (which will be fitting to commit to Git). The secret.yaml can be created by the steps provided by Crossplane’s documentation, which we then further convert to a SealedSecret:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# Generated via taking a "kind: Secret" (creds key = the toml base64 encoded) and running # cat secret.yaml | kubeseal --controller-namespace # --controller-name -sealedsecrets --format yaml apiVersion: bitnami.com/v1alpha1 kind: SealedSecret metadata: name: aws-creds spec: encryptedData: creds: AgB6brqgnLqNRSV template: metadata: name: aws-creds type: Opaque |
When these resources are synced to your cluster, all that’s needed is for us to tell Crossplane which configuration to use for the provider.
crossplane-provider-app.yaml
With the controller synced, followed by the Crossplane AWS Provider (along with a SealedSecret of the creds, which gets reconciled to a Secret), we can at last configure the credential for the AWS provider so that it is ready to use.
Using Argo sync waves, we ensure this is synced after the provider resource is there, to avoid any “provider not found” type errors. So, in our crossplane-provider-cfg-app.yaml, we’ll have an annotation like this (arbitrarily picking a value greater than 5):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: annotations: # This must be synced after both the crossplane-controller # and crossplane-provider argocd.argoproj.io/sync-wave: "10" ... spec: ... source: ... path: path/to/crossplane-provider-cfg/resources/ |
Then, in path/to/crossplane-provider-cfg/resources/, we simpy tie the credentials secret to the provider by creating a ProviderConfig CR:
|
1 2 3 4 5 6 7 8 9 10 11 |
apiVersion: aws.crossplane.io/v1beta1 kind: ProviderConfig metadata: # ProviderConfigs are cluster-scoped, so no namespace needed name: my-provider-config spec: credentials: source: Secret secretRef: name: aws-creds key: creds |
Once synced, we can begin provisioning cloud resources natively!
Now, if a developer had to update his/her deployment.yaml to his/her latest container image (where this version of the code also requires an S3 bucket), (s)he can simply add a file like bucket.yaml:
End Result
|
1 2 3 4 |
app └── deploy ├── deployment.yaml ├── bucket.yaml |
… and bucket.yaml will be a native Bucket resource, or any other AWS managed resources people need 2 :
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
apiVersion: s3.aws.crossplane.io/v1beta1 kind: Bucket metadata: name: my-cool-s3-bucket namespace: my-cool-ns spec: deletionPolicy: Delete providerConfigRef: name: my-dev-aws forProvider: acl: private publicAccessBlockConfiguration: blockPublicAcls: true blockPublicPolicy: true ignorePublicAcls: true restrictPublicBuckets: true locationConstraint: us-east-1 |
The benefits are immediate. This allows us to do many things automatically:
- This provisioning stays in-band with the app for which it’s intended
- New developers save time on learning where to add, say, a comparable Terraform module (and how to run it)
- Options exist to cleanup on resource deletion. So lingering resources are auto-deleted too, saving cost on both infra and time.
We can deep dive further into any of these topics, including further best practices, hardening, and other patterns we use. This is intended as a high-level, hands-on “Getting Started” post by putting Argo CD, Crossplane, and SealedSecrets together for seamless shifting of infrastructure provisioning leftward.
What’s Next
We’ll have future blog posts that cover some of the topics that arose from this:
- We’ll extend on this example to see how the app can access the bucket using Service Accounts and IAM roles
- Why Kubernetes is King; allowing you to shift left, stay cloud-agnostic, run anywhere, and avoid any vendor lock-in
Footnotes
1 With the newer releases of Argo, the “App Of App” pattern may be giving way to the ApplicationSet controller. Check it out!
2 Technically, these are composite resource claims (XRC) that bind to composite resources (XR). More details here.
