CI/CD through Gitlab

Review of CI/CD as a concept

DevOps_CICD
Figure 1. DevOps and CI/CD (1/3)
DevOps_CICD
Figure 2. DevOps and CI/CD (2/3)
DevOps_CICD
Figure 3. DevOps and CI/CD (3/3)
CI (Continuous Integration)

  • Is your code good?

  • earlier you run tests, sooner you find problems

  • makes sure that any file edits made integrate well with the stable code base

    • functional tests

    • security scans

    • code quality scans

    • performance tests

    • license scanning

    • fuzz testing

  • gives everyone on the team an understanding of the state of thesoftware

CD (Continuous Delivery / Continous Deployment)

  • more ambiguous

  • finding out where your code should go

    • and putting it there

  • make releases "boring"

  • different environments

    • testing/development environments (run against feature branches etc)

    • staging/pre-prodcution (run against main)

    • production envs (run against tags)

  • continuous delivery#

    • automatic deploy to test and dev envs

    • not automatically deployed to production without manual approval

  • continuous deployment

    • deploy to production automatically

  • building/packaging

    • preparing the code into deployable form

      • ruby gem

      • java war/ear

      • c code → docker image

      • tarball

      • no packaging/building

  • sending/delivering/deploying

    • pushing docker image to repository

    • using a cli tool to deploy to AWS environment

    • last (sometimes only) task of CD

CI/CD Pipeline can be tool agnostic

  • pipelines ultimately boil down to a series of shell commands

    • that run automatically

    • i.e. series of tasks run by a robot that relate to

      • building

      • verifying

      • deploying

Overview of Gitlab CI

Gitlab_CI
Figure 4. GitLab CI Diagram
The three major parts of a Gitlab CI/CD pipeline

  • the Gitlab instance

  • The Gitlab CI Workflow file

  • Gitlab Runner(s)

The Gitlab CI Workflow file

  • is a series of steps performed on your files

  • when ever you commit and push source code changes

  • can be thought of as a blueprint/recipe

Steps

  • tasks that are performed on your files

    • tests

    • security scans

    • build / packaging

      • zip file?

      • linux package?

      • docker container?

    • deploy / set up

      • test env

      • prod env

  • basically, steps are any manipulation of your repository files

Runners

  • Runners are robots/daemons/programs that

    • take these steps from somewhere (e.g. Github/Gitlab instance)

    • and execute them in an environment

    • where the rubber meets the road

      • CI/CD code to actual tasks

  • GitLab Runners are open source programs

    • written in Go

    • gitlab.com/gitlab-org/gitlab-runner

    • can be self-hosted (installed on your own infrastructure)

    • or used through 3rd party installations (e.g. Gitlab.com runners)

    • Each Runner Agent can spawn multiple runner processes.

  • Runners can be private or shared

    • specifically registered to a single repository

    • Or shared with a project/organisation

    • Or belong to an entire GitLab instance

  • Runners have different "executors"

    • This allows them to perform or execute your steps

    • In different environments.

      • Shell

      • Docker

      • Virtual Machines

      • Kubernetes

      • Custom

Workflow_File
Figure 5. Anatomy of a Pipeline Workflow
Parts of a pipeline

  • Stages

    • Each pipeline is made up of one or more "Stages"

    • help readibility/maintainibility of pipeline

    • collection of pipeline tasks that are related

      • build

        • compile

      • package?

      • test

        • linting

        • unit tests

        • scans

      • deploy

        • package

        • deliver

      • each stage is executed sequentially by default

        • one stage after another

        • can be changed using the needs keyword

  • Jobs

    • equivilant to steps/tasks

    • Each stage contains one more more jobs

    • best practice to have each job do one task

    • all jobs within a stage are executed in parallel by default

      • can be changed using the needs keyword

  • Commands

    • each job contains one or more commands

    • job is the robot/runner take types commands in to a bash shell

      • javac *.java

      • docker build --tag my_app:1.2

      • mvn test

    • in the end, running all of the commands of a pipeline manually

    • is the same thing as running the pipeline through a runner

Running Pipelines

  • automatically

    • commit push

    • merge request

      • source code of branch of merge request

    • merged result

      • pipeline on a temporary merge

    • merge train

      • seperate, concurrent merged result piepeline

      • temp merge of source branches from every merge request that’s ahead of the current merge request in the queue.

  • manually

    • branch

    • tag

    • skipping pipelines

  • certain "features" of the single pipeline can be turned on or off,

    • depending on which features make sesnse for the type of changes.

    • unit tests on code changes

    • spelling/editing tests on documentation changes

Karlina and IT4I’s CI/CD setup

IT4I_CICD
Figure 6. IT4I’s CI/CD setup
Documentation
Gitlab Instance

  • code.it4i.cz

    • If you have been granted access to Karolina or other IT4I systems, you should have an account here.

Gitlab Runners

  • 7 System Runners

    • 4 runners (of this type) running in the login nodes of Karolina,

    • 2 runners (of this type) running in the login nodes of Barbora,

    • 1 runner (of this type) running in the login node of Complementary systems.

  • 5 Docker Runners

  • We are not able to choose exactly which of the GitLab runners will be picked.

    • We use tags to select which cluster we want the CI job to be submitted to as a SLURM job.

    • One of the GitLab runners that match our CI job’s tags(running in the login node of the specified cluster) will automatically pick our job.

Jacamar CI

A comparison of Gitlab CI and Github Actions

Similarities

  • both systems base their workflow language on YAML

  • both systems support multiple stages, and work directly on your files in the associated repo.

Differences

  • Gitlab CI does not have a comparable marketplace like Github Actions.

    • new feature/component under development, available: "Gitlab Steps"

      • docs.gitlab.com/17.9/ci/steps/

      • official components developed by Gitlab (safe but not so many)

      • support now added to use Github Actions in Gitlab CI. Weird!

  • Gitlab has support for custom executers

    • JacamarCI

    • a native Kubernetes executer.

  • only one pipeline workflow file per project

  • .gitlab-ci.yml

What is a Github Action?

  • A Github action can be thought of as a simple program, or more like a Function As A Service.

  • It consists of

    • an action.yml that describes inputs and what program the action calls.

    • Either a docker file to make a container based action. This can run anything from shell scripts to python code.

  • Or a Nodejs application, usually calling index.html as the starting point.

Gitlab YAML syntax

documentation
key differences
GitHub GitLab

"on"

no keyword

"run"

no keyword

"jobs"

"stages"

"env"

"variables"

"runs-on"

"tags"

"steps"

"script"

"uses"

"include"

"container"

"image"

  • no "on" keyword

  • no "run" keyword

  • upload-artifact and download-artifact replaced with "artifcats"

  • secrets management

    • github has integrated secrets management

    • gitlab encourages 3rd party tools (like Vault)

    • or have to use masked and protected ci/cd variables