Fernando Souza

Senior DE / DataOps / DevOps engineer

A k8s Deployment is a very useful resource, where you can control how your app rollouts when pushing new versions.

You can choose between 2 strategies to work with a Deployment: RollingUpdate and Replace.

Both cover the basic mechanism on how your new version is rolled out, one replaces pod by pod, the other just replaces everything at once.

We can include probes as well, Readiness and Liveness, which influence pod health and let the rollout progress.

So what if you need to do more analysis and add stronger guarantees during rollouts?

Let’s say that you have a very strict SLO for the performance of your service — how would you implement this check with the deployments above?

In some cases you have health services starting, but then begin reporting errors and defects only detected when calling the API.

Those requirements above may become harder to implement when using plain k8s deployments.

You still can though, but it may end up being something outside your k8s strategy for rolling your app, something customized.

On searching recently for something to cover the scenarios above, I’ve bumped into ArgoRollouts.

It’s a standalone solution from the ArgoCD team that performs the analysis of your rollout and lets it progress, or not, based on the results.

It’s very versatile, allowing steps in your rollout, analysis templates that can serve multiple rollouts, and two strategies: Canary and Blue-Green, with several options.

ArgoRollouts can as well be integrated with ArgoCD.

Let’s first just review the concepts of both deployment strategies, and then I’ll describe how ArgoRollouts fits into them.

Canary

You gradually release a new version of your app, routing traffic to it while measuring its metrics, then either progressing or rolling back based on the analysis. Progress is analyzed at certain stages or steps during the rollout, and rollback happens if checks fail.

flowchart LR U[Users] --> LB["Load balancer / Service"] LB -- "90%" --> S["Stable v1
9 pods"] LB -- "10%" --> C["Canary v2
1 pod"]

Progression flow:

flowchart LR A["Step 1
v1: 100%
v2: 0%"] --> B["Step 2
v1: 90%
v2: 10%"] B --> C["Step 3
v1: 50%
v2: 50%"] C --> D["Step 4
v1: 0%
v2: 100%"] B -. "metrics bad" .-> R["Roll back to v1"] C -. "metrics bad" .-> R

Ps: Load-test or real traffic is pointing to LB / Service serving both rollouts. I don’t recommend this strategy to check the correctness of your code or business requirements, this has to be mainly for checking and comparing for example if your new app is still performing under your SLO with real traffic. Business requirements and if the your service is working properly should be done on unit-tests and/or integrations tests before reaching the deployment / release phase.

Blue-Green

You have two full environments, but you keep user traffic only to Blue. Blue is live / stable, and you deploy the new version to Green alongside it and test it with no user traffic. When it’s ready, you switch all traffic over at once, and scale down Blue.

flowchart LR U[Users] --> SVC["Service / Router"] SVC == "live traffic" ==> B["Blue v1
active"] SVC -. "preview only" .-> G["Green v2
idle"]

Progression flow:

flowchart TD A["Blue v1 serves 100%"] --> B["Deploy Green v2 alongside"] B --> C["Test Green via preview endpoint"] C --> D{"Healthy?"} D -- "yes" --> E["Switch all traffic to Green"] D -- "no" --> F["Delete Green, Blue untouched"] E --> G["Keep Blue for instant rollback"] G --> H["Scale down Blue"]

Ps: Direct or create a special load-test pointing to Green, for generating metrics in order to analyse the health. Again this strategy should not be used as a test strategy.

ArgoRollouts

What resources can we use with ArgoRollouts to implement deployment strategies:

  • Rollout : It replaces the Deployment k8s resource and adds both strategies above for the rollout.
    • Define steps for the rollout, at which steps does the analysis start and more.
  • AnalysisTemplate : It’s a ArgoRollouts CRD that allows you to create generic analysis based on inputs from prometheus metrics and others.
    • Defines the query for the analysis, the result for comparison, interval and more.
    • Can receive arguments in order to run the analysis, like labels to filter metrics from prometheus about your app.
    • Has to be configured to collect the analysis, failure limits, total count of runs and more.
  • AnalysisRun : It’s another ArgoRollouts CRD that can be triggered manually to do the same analysis as the AnalysisTemplate.
    • Can be used for analysing for example the first revision, or to run the analysis even out of the rollout cycle.
    • Can be created from an AnalysisTemplate, pass arguments to create it making use of the template definition.
    • Can be put to run indefinitely if there’s no count.

In order to use ArgoRollouts we have to deploy a controller to k8s, and the controller will be monitoring namespaces and taking care of the whole Rollout flow.

graph LR subgraph k8s direction LR subgraph argorollouts["ArgoRollouts namespace"] direction LR C["Controller"] end subgraph app["App namespace"] direction LR R["Rollout"] AT["AnalysisTemplate"] AR["AnalysisRun"] end end C --->|rollout revision| R C --->|analysis| AT C -.->|manual analysis trigger| AR AT -.- R AR -.- R

ArgoRollouts has many other features like:

  • Notifications for all the events, and push notifications to slack for example.
  • Intelligent traffic management with several gatewars like ambassador, NGINX and more.
  • A built-in UI / dashboard to monitor Rollouts and take manual actions if needed.
  • Experiment CRD in order to run ephemeral replicasets of your app, and apply analysis on it.
Demo

I’ve built a Canary strategy demo for ArgoRollouts:

It’s a full demo with all resource config, and a service in go.

I’ve also added a LoadTester to the project using oha, and it’s deployed to k8s via a CronJob.

The demo shows how Rollout revisions are allowed to progress or not based on AnalysisTemplates; therefore I have 3 versions of the same app. Check the beginning of the repo with the podman build statements to generate all 3 versions, as all 3 will have different behavior for the analysis.

I’m exploring most of the topics presented above, but topics like Experiments, Notifications and gateway integration are not part of the demo.

The single point of access to the app is a Service, with a selector that points to both stable and new deployments.

Ps: It’s for now only including the Canary scenario and resources, but will soon be adding the Blue-Green scenario.

References

Updated:

You May Also Enjoy