.. _templates: CI Templates ============ The CI templates are a set of Gitlab CI job *templates* that can be integrated into your ``.gitlab-ci.yml`` file. These templates simplify building and re-using container images for the project's registry. To make use of the templates, you need to - :ref:`include the template ` and - :ref:`define a job extending the template ` The CI templates provide a set of templates per distribution and you need to :ref:`include ` the one for the distribution(s) you want to use. The examples below use the Fedora templates. .. _templates_why: Why use the CI templates? ------------------------- With the CI templates you can easily build a **persistent container image**. Images are identified by a **unique tag** that allows for them to be used in the actual CI jobs. So you can say *"I want a Fedora 31 container with packages X, Y, Z installed"*, tag this with *"2020-02-03.0"* for the current date (best practice but not required) and that image can be used from all tests. To build a new container or rebuild the existing container with updated packages, simply change the tag in your ``.gitlab-ci.yml``. Merge request re-use the same container image for CI. The CI templates will take care of the rest. This gives you **reproducible test results** as all merge requests will use the same container images. Where a merge request changes the tag, a new container is built **for that merge request** and this merge request runs CI with the new container image. All other merge requests are unaffected. Only once the merge request has been merged into the upstream project will future merge requests use the new container image. All this is available with minimal boilerplate - you specify the distribution, version and the packages to install and the templates do the rest. .. _templates_including: Including the CI templates -------------------------- There are two ways of including a template in your project, depending on whether your project is :ref:`hosted on gitlab.freedesktop.org ` or :ref:`hosted elsewhere `. In both cases, you should use a specific git sha of the files you want to include - the CI templates do not use version numbers. .. warning:: You can use ``master`` as git ref but we do not recommend this. Using a specific git sha protects **you** from unplanned CI failures caused by changes in the CI templates. .. _templates_including_fdo_projects: Projects hosted on gitlab.freedesktop.org .................................................... If your project is hosted on ``gitlab.freedesktop.org`` you can include it as follows: .. code-block:: yaml .templates_sha: &templates_sha 123456deadbeef include: - project: 'freedesktop/ci-templates' # the project to include from ref: *templates_sha # git ref of that project file: '/templates/fedora.yml' # the actual file to include - project: 'freedesktop/ci-templates' ref: *templates_sha file: '/templates/alpine.yml' # rest of your .gitlab-ci.yml goes here The above snippet first defines the git sha of the CI templates we want to include as a YAML anchor. It then includes the Fedora and Alpine templates from the CI templates repository of that specific sha. Using a YAML anchor is recommended to avoid duplication. You can specify different shas for different files though it is not something we recommend. You can specify any valid git ref (e.g. ``master``) though we recommend that you use a specific sha. For more information on the ``include:`` statement, see the `GitLab documentation `__ .. _templates_including_outside_projects: Projects not hosted on gitlab.freedesktop.org ............................................. If youre project is not hosted on ``gitlab.freedesktop.org`` you can use the CI templates as follows. .. code-block:: yaml include: - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/123456deadbeef/templates/fedora.yml' - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/123456deadbeef/templates/alpine.yml' # rest of your .gitlab-ci.yml goes here The above snippets links to the files directly using the git sha. You can specify different shas for different files though it is not something we recommend. You can specify any valid git ref (e.g. ``master``) though we recommend that you use a specific sha. For more information on the ``include:`` statement, see the `GitLab documentation `__ .. _templates_extends: Extending the template jobs --------------------------- All jobs provided by the CI templates use a naming scheme in the form ``.fdo.@``. They start with a dot (``.``) and thus do not get invoked by the GitLab CI runners. To make use of them, use ``extends:``. .. code-block:: yaml # include statements go here myjob: extends: .fdo.container-build@fedora stage: somestage variables: FDO_DISTRIBUTION_VERSION: 31 FDO_DISTRIBUTION_PACKAGES: 'curl wget valgrind' In the (incomplete) example above a job called ``myjob`` is defined to be invoked in the ``somestage`` stage (user's choice). It extends the ``.fdo.container-build@fedora`` template (see :ref:`templates_building_containers`). The ``variables`` will be used by the CI template to generate the correct container image. For more information on the ``extends:`` statement, see the `GitLab documentation `__ .. _templates_building_containers: Building container images ------------------------- The CI templates provide two ways of building a container image: ``container-build`` for a normal container and ``qemu-build`` for building an image that can be run through QEMU on a KVM-enabled host. For projects without specific hardware interactions the ``container-build`` is sufficient. As the name implies, this builds a container image if it does not already exist. At runtime, it will check your image repository (i.e. the GitLab ``username/project``) for the container image and then the upstream repository. If the image does not exist in your repository but it does exist upstream, the image is copied to your repository. Thus, even if upstream changes the image later, you have a copy of that image. If the image does not exist upstream, it is built in **your** repository. Once your merge request is merged, **the image will be rebuilt for the upstream repository**. .. note:: If you want to **force** a container build, set the variable ``FDO_FORCE_REBUILD``. Below is an example on how to build a container. To avoid repetition of boilerplate code, we define a template for the distribution we want to build on and re-use that template to fill in the variables for us where required. .. code-block:: yaml include: - project: 'freedesktop/ci-templates' ref: 12345deadbeef file: '/templates/fedora.yml' # Let's define some stages to get the correct order of jobs. stages: - prep - test variables: # The upstream repository path on gitlab.freedesktop.org to check # for existing container images. FDO_UPSTREAM_REPO: some/path # A simple template so we only have one place where we need to # define the Fedora version and the image tag # Using a date as tag is best practice but it can be any string # allowed by the GitLab registry. .myproject.fedora:30: variables: FDO_DISTRIBUTION_VERSION: 31 FDO_DISTRIBUTION_TAG: '2020-03-10.0' # A job to build a Fedora 31 container with valgrind and gcc # installed. # # You must not define script: in this job, it is used by the # container-build template. build-fedora-container: extends: - .fdo.container-build@fedora # the CI template - .myproject.fedora:30 # our template job above stage: prep variables: # Packages to install on the container FDO_DISTRIBUTION_PACKAGES: "valgrind gcc" # The test job for your project. Extending from the CI Templates # .fdo.distribution-image makes it use the same image we built above. mytest: extends: - .fdo.distribution-image@fedora - .myproject.fedora:30 stage: test script: # FDO_DISTRIBUTION_NAME is set by the distribution-image job. # It should be considered read-only - echo "Hello world in $FDO_DISTRIBUTION_NAME" .. warning:: The ``.fdo.qemu-build`` template uses an arch-specific suffix. See :ref:`templates_multiarch` for important details. .. _templates_building_qemu_images: Building QEMU-capable container images ...................................... Some tests need to run in a virtual machine instead of a container. For those, qemu-capable images can be prepared with the CI templates. The templates are identical to the ones shown above but use the term ``qemu`` instead of ``container``. The above example thus becomes: .. code-block:: yaml # A job to build a Fedora 31 qemu image with valgrind and gcc # installed. # # You must not define script: in this job, it is used by the # qemu-build template. build-fedora-vm: extends: - .fdo.qemu-build@fedora@x86_64 # the CI template - .myproject.fedora:30 # our template job above stage: prep variables: # Packages to install on the vm FDO_DISTRIBUTION_PACKAGES: "valgrind gcc" .. warning:: The ``.fdo.qemu-build`` template uses an arch-specific suffix. See :ref:`templates_multiarch` for important details. Once built, the container image provides the script ``/app/vmctl start`` to start the virtual machine. The VM is configured to accept ssh connection and aliased as host ``vm``. Commands can be run on the virtual machine with the ``/app/vmctl exec`` helper. .. code-block:: yaml # The test job for your project. Extending from the CI Templates # .fdo.distribution-image makes it use the same image we built above. mytest: extends: - .fdo.distribution-image@fedora - .myproject.fedora:30 stage: test script: # start the VM. This also sets up ssh/scp to connect to "vm" # correctly. - /app/vmctl start # copy our workspace to the VM # The quotes are required to stop the ':' from parsing as yaml - scp -r $PWD "vm:" # We don't want any failed commands to exit our script until VM # cleanup has been completed. - set +e # run test-command on the VM and create the .success file if it # succeeds - /app/vmctl exec "cd $CI_PROJECT_NAME ; test-command" && touch .success # copy any test results from the VM to our container so we can # save them as artifacts - scp -r vm:$CI_PROJECT_NAME/test-results.xml . # shut down the VM - /app/vmctl stop # VM cleanup is complete, any command failures now should result in # a CI failed job - set -e # our CI script exit code should match the test command exit status - test -e .success || exit 1 Noteworthy in the above is that the actual test command and subsequent VM cleanup are wrapped by a ``set +e`` and ``set -e`` call. This ensures that any failed call does not immediately terminate the CI job but allows for the proper cleanup of the VM. The ``.success`` file is what determines the actual success status of the CI job. .. templates_container_labels: Image labels ------------ Images built with the CI templates have a number of labels set that can be used by jobs or the GitLab setup itself. The example below shows how to access the labels from a CI job using `skopeo `__ and `jq `__: .. code-block:: yaml check label: extends: - .myproject.fedora:30 # our template job above - .fdo.distribution_image@fedora # We don't want/need to use the actual image we built earlier, we can use # any image that has skopeo and jq, or install those as part of # script: image: any-image-with-skopeo-and-jq script: # FDO_DISTRIBUTION_IMAGE still has indirections - DISTRO_IMAGE=$(eval echo ${FDO_DISTRIBUTION_IMAGE}) # retrieve the infos from the registry (once) - JSON_IMAGE=$(skopeo inspect docker://$DISTRO_IMAGE) # Parse the the pipeline_id label - IMAGE_PIPELINE_ID=$(echo $JSON_IMAGE | jq -r '.Labels["fdo.pipeline_id"]') # If the image was built as part of this pipeline, the image's pipeline # ID is the same as the current pipeline ID. # This can be used to poke other projects to rebuild dependent images. - if [[ x"$IMAGE_PIPELINE_ID" == x"$CI_PIPELINE_ID" ]]; then echo "Image was built in this pipeline" fi .. note:: Extending from ``.fdo.distribution_image@fedora`` provides ``FDO_DISTRIBUTION_IMAGE``. We do not actually use the image itself, any image with ``skopeo`` and ``jq`` will work here. The labels currently set by the CI templates are as follows: .. list-table:: * - Image Label - Value * - ``fdo.pipeline_id`` - ``$CI_PIPELINE_ID`` * - ``fdo.job_id`` - ``$CI_JOB_ID`` * - ``fdo.commit`` - ``$CI_COMMIT_SHA`` * - ``fdo.project`` - ``$CI_PROJECT_PATH`` * - ``fdo.upstream-repo`` - ``$FDO_UPSTREAM_REPO`` (optional) * - ``fdo.expires-after`` - ``$FDO_EXPIRES_AFTER`` (optional) For the values starting with ``CI_``, see the `GitLab environment variables documentation `__ As in the example above, the image's ``fdo.pipeline_id`` can be used to check if an image was built as part of the current pipeline. .. _templates_deleting_containers: Deleting container images ------------------------- Unfortunately, deleting container images is nontrivial with templates and it requires extra authentication tokens. Use the :ref:`ci-fairy tool ` for this task: .. code-block:: yaml delete-containers: extends: - .fdo.distribution-image@fedora - .myproject.fedora:30 stage: cleanup image: golang:alpine before_script: - apk add python3 git - pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates script: # Go to your Profile, Settings, Access Tokens # Create a personal token with 'api' scope, copy the value. # Go to CI/CD, Schedules, schedule a new monthly job (or edit the existing one) # Define a variable of type File named AUTHFILE. Content is that token # value. # # This example assumes that you want to delete all but the current tag - ci-fairy -v --authfile $AUTHFILE delete-image --repository $FDO_DISTRIBUTION_NAME/$FDO_DISTRIBUTION_VERSION --exclude-tag $FDO_DISTRIBUTION_TAG only: - schedules This is a job to run container cleanup on a `schedule job `__. We get ``$FDO_DISTRIBUTION_NAME`` by extending ``.fdo.distribution-image@fedora`` but since we only need to run a simple python tool, we can just run off a ``golang:alpine`` image. All other variables are courtesy of ``.myproject.fedora:30`` (see :ref:`templates_extends`) Because of restrictions in GitLab, this can only be run with an API token, ``CI_JOB_TOKEN`` does not have permissions to delete images. The ``ci-fairy`` command as run here will delete all images in the ``fedora/30`` image repository, excluding the one with the tag ``2020-03-10.0``. .. _templates_multiarch: Handling multi-arch images -------------------------- The ``.fdo.container-build`` and ``.fdo.qemu-build`` templates use an arch-specific suffix (``@x86_64`` or ``@aarch64``) to build for the appropriate architecture. This arch-specific suffix is not encoded in the image name, and a potential pitfall when building identical images for multiple architectures is that those images overwrite each other. This example illustrates the problem: .. code-block:: yaml # DO NOT USE THIS SNIPPET. This example illustrates a bug .fedora: variables: FDO_DISTRIBUTION_VERSION: 32 FDO_DISTRIBUTION_TAG: '2020-11-12.0' # uses distribution name, version and tag to store the image in the # registry build-x86: stage: prep extends: - .fdo.container-build@fedora - .fedora # uses the same distribution name, version and tag to store the image # in the registry, potentially overwriting the other image. # if this job runs after the build-x86 job completed, it does nothing # because an image with the given tag is already in the registry build-arm: stage: prep extends: - .fdo.container-build@fedora - .fedora tags: - aarch64 # this job runs on whichever image got stored in the registry run-image: stage: build extends: - .fdo.distribution-image@fedora - .fedora script: - echo "I don't know which image I'm running on" Both ``build-`` jobs produce the same image tag and which image ends up in the registry depends on the (non-deterministic) order the jobs are run and/or completed. Where multi-arch jobs are required, the arch must be encoded in the image name using ``FDO_DISTRIBUTION_TAG`` (or ``FDO_REPO_SUFFIX``): .. code-block:: yaml .fedora: variables: FDO_DISTRIBUTION_VERSION: 32 BASE_TAG: "2020-11-12.0" .fedora-x86: extends: - .fedora variables: FDO_DISTRIBUTION_TAG: "x86_64-$BASE_TAG" .fedora-arm: extends: - .fedora tags: - aarch64 variables: FDO_DISTRIBUTION_TAG: "aarch64-$BASE_TAG" build-x86: stage: prep extends: - .fdo.container-build@fedora - .fedora-x86 build-arm: stage: prep extends: - .fdo.container-build@fedora - .fedora-arm .run-image: stage: build extends: - .fdo.distribution-image@fedora script: - echo "I'm running on $FDO_DISTRIBUTION_TAG" run-x86: extends: - .run-image - .fedora-x86 run-arm: extends: - .run-image - .fedora-arm :ref:`Templating the .gitlab-ci.yml ` can ease the maintenance of such pipelines.