In this Cloud Native Short Take, instructor Chris Hanson covers pipeline optimization, discusses decoupling task files from pipeline manifests, and demonstrates pipeline instancing. RX-M, a cloud-native training, consulting, and advisory firm, provides vital strategies, insights, and tactical approaches in its Cloud Native Short Takes series. Subscribe to the RX-M YouTube Channel to be notified of each release of the popular Cloud Native Short Take Series.
Video Transcript
Welcome to another Cloud Native Short Take from RX-M! My name is Chris Hanson. Today we are going to be covering the Concourse module: Pipeline Patterns. In this module we cover topics such as pipeline optimization using configurations like parallel git steps; we also talk about decoupling task files from pipeline manifests; as well as techniques like pipeline instancing. That is what today’s session is going to cover; we are going to look at dynamically creating pipelines from git branches so that you can reuse a pipeline template that builds and tests your code on different branches without having to either create the branched pipelines manually or clean them up afterwards.
We use an empty Concourse to keep things clean for our demo. I use a couple of different pipelines: the first is the template which will be auto deployed by the second, the tracker. The tracker pipeline will invoke the template and then we will have the same build and test process for each branch that we have in our git repo. Looking at the tracker pipeline we can see that we are using the git branches resource type. It will let us track git branches and we will use that for creating the branch-based pipelines.
We use a local copy of the RX-M concourse examples repo being served by git daemon so we can push back to that copy–git daemon is actually a pod running on my Kubernetes cluster So I do not have worry about having to push to an external dependency (of course it would still work if you use something like gitlab or github).
The lone job is the update branch pipelines job. It uses the “load var” step to load variables based on our branches and the “set pipeline” step which creates a group called dev for “development” and uses the file within the git repo for our template. Remember that we are creating pipelines based on our branches which deploy to kubernetes namespaces also named for each branch so we create a branch based namespace with “kubectl create” prior to triggering the pipeline(s). This step is necessary because the Kubernetes Deployment resource type is currently not capable of creating the namespace itself (or we would do that in the pipeline too!).
With the namespace created, we can create the pipeline itself using fly. We call it “tracker” and have saved the pipeline in a file named “tracker.yaml”, which we submit with “fly”. Concourse is going to confirm our changes before accepting them and by confirming (with a simple “y”). The tracker pipeline starts paused but we can easily use the the Concourse UI to unpause our pipeline. When we drill in we can see the examples and repro branches resources that feed the update branch pipelines job. Because the pipeline is unpaused the triggers have started the first build. Once that job is successful a dev pipeline group is created dynamically by the tracker pipeline.
The main branch in the dev pipeline group automatically triggers using the copy of the examples repo and runs the test from the “go-test.yaml” file. Drilling into “test” we see that the commit is pulled from our local repo.
Once the “test” job has completed, that means we have passed the test and can build the image using the “build” job. When the build completes it pushes the image to the registry which will allow the final job, “deploy” to trigger to deploy the new image onto the Kubernetes cluster. Confirming is as easy as using “kubectl” in the “main” namespace. Since we deployed a K8s Service in our “deploy” job, from the host shell we can use the ClusterIP with the port that the app listens on, port 8080, to get the “Hello World!” message!
For the branching part of the demo, we use git to create a branch called “issue” where we will commit a message in the main.go code that says “Hello Issue!”. Once we commit and push the code change, the “dev” group should have the new “issue” branch pipeline which immediately triggers. The “issue” pipeline will require a K8s namespace of the same name, which we create with “kubectl” for the demo.
Once the “issue” pipeline completes, we use the ClusterIP once again to curl the app and this time receive the “Hello Issue!” message! That’s just one of the things you’ll learn in our Concourse Foundation course!