.. _templates_contributing:

Hacking on the CI Templates
===========================

This is an outline of how everything fits together in this repository.

This repository contains both the source for the templates **and** the
generated templates that projects use. In addition, the GitLab CI pipeline
of the CI Templates produces several public container images that are used
when projects employ the CI Templates to build images.

.. _templates_ci_pipeline:

Explanation of the image build process
--------------------------------------

When the CI Templates CI pipeline runs, the first stage is ``bootstrap``.
This stage builds several base images that are used by the
``.fdo.container-build`` and ``.fdo.qemu-build`` templates.
The other stages are to **test** the distribution-specific CI templates but do
not otherwise alter any global state.

To reduce the bandwidth costs, the base images are pushed to
https://quay.io and their names are hardcoded into the ``.fdo.container-build``
and ``.fdo.qemu-build`` templates.

.. graphviz::

   digraph ci {
   compound=true;
   rankdir="TB";
   node [shape="box"];

   subgraph cluster_quay {
       label="quay.io";
       style="dashed";
       q_run_base [ label="qemu run base image"];
       q_build_base [ label="qemu build base image"];
       c_build_base [ label="container build base image"];
   }

   bootstrap_ci [ label="bootstrap CI stage", shape="component" ];

   bootstrap_ci -> c_build_base;
   bootstrap_ci -> q_build_base;
   bootstrap_ci -> q_run_base;

   }


The bootstrapping stage builds three images:

- a container build base image. This image is used exclusively by the
  ``.fdo.container-build`` templates and is capable of **building** a normal
  container image.
- a qemu run base image. This image is capable of **starting** a
  QEMU-compatible raw image file.
- a qemu build base image. This image is used exclusively by the ``.fdo.qemu-build``
  templates and is capable of **building** a QEMU-compatible raw image file.
  The ``.fdo.qemu-build`` job then places that raw image file inside the
  qemu run image and publishes that as the image to be run by the
  project-specific pipelines.

The actual templates then use those https://quay.io images to compose
the project-specific images.

In the graphs below, an arrow **to** an image indicates that image is built,
an error **from** an image indicates that image is **used**.

.. graphviz::

   digraph ci {
   compound=true;
   rankdir="TB";
   node [shape="box"];

   subgraph cluster_quay {
       label="quay.io";
       c_build_base [ label="container build base image"];
       q_build_base [ label="qemu build base image"];
       q_run_base [ label="qemu run base image"];
   }

   subgraph cluster_registry {
       label="project registry";
       style="dashed";
       project_img [ label="project image" ];

       subgraph cluster_qemuimg {
           style="solid";
           node [ shape="ellipse"; ]
           label="project qemu image";
           imagefile [label="image.qcow2", shape="folder"];
       }
   }

   container_build [ label=".fdo.container-build", shape="component" ];
   qemu_build [ label=".fdo.qemu-build", shape="component" ];
   distribution_img [ label=".fdo.distribution_image", shape="component" ];
   distribution_img_q [ label=".fdo.distribution_image (qemu)", shape="component" ];

   c_build_base -> container_build [style="dashed", label="runs on"];
   container_build -> project_img;
   project_img -> distribution_img [style="dashed", label="runs on"];


   q_build_base -> qemu_build [style="dashed", label="runs on"];
   q_run_base -> qemu_build [label="uses"];
   q_run_base -> imagefile[
        lhead=cluster_qemuimg,
        arrowhead="none",
        label="copy of",
        style="dotted"
        ];
   qemu_build -> imagefile;
   qemu_build -> imagefile [lhead=cluster_qemuimg];
   imagefile -> distribution_img_q[
       ltail=cluster_qemuimg,
       style="dashed",
       label="runs on"
       ];
   }

In summary, the ``.fdo.*-build`` templates use the https://quay.io images to
compose project-specific images and store those in the project's image
registry. The ``.fdo.distribution-image`` and ``.fdo.suffixed-image``
templates then use those images to run the CI jobs.

.. note:: The images stored in the project registry are independent of
   the quay.io images.

CI Templates directory layout
-----------------------------

The directory structure is as follows:

- ``bootstrap``: Scripts used by the bootstrap images. The primary scripts
  of interest are `cbuild <https://gitlab.freedesktop.org/freedesktop/ci-templates/-/blob/master/bootstrap/cbuild>`__,
  the script that ends up building the various containers,
  and `vmctl <https://gitlab.freedesktop.org/freedesktop/ci-templates/-/blob/master/bootstrap/vmctl>`__
  a helper script to control the QEMU virtual machines generated with the
  ``.fdo.qemu-build`` templates.
- ``src``: The source template files and per-distribution configuration.
  Templates ending in ``-ci.tmpl`` are those used by **our** CI run to build
  the base images and test the templates themselves.
- ``templates``: The **generated** templates. Do not edit.
- ``test``: Scripts used by the CI to test images.
- ``tools``: The :ref:`ci-fairy <ci-fairy>` tool.

When templates change, ``make templates`` generates
both the resulting ``templates/`` **and** the ``.gitlab-ci.yml`` file that
generates the consumable images and the various CI pipelines to test these
images and the templates.

The consumable images are built in the ``bootstrap`` phase of the CI
pipeline, see the ``.gitlab-ci/bootstrap-ci.yml`` file and
:ref:`templates_ci_pipeline` for details.

Modifying templates
-------------------

A typical sequence of steps to modify the distribution templates is:

- edit the ``src/*.tmpl`` files
- run ``make templates``
- commit and file MR

Where the ``bootstrap`` phase or the ``cbuild`` and ``vmctl`` scripts are
modified, new images for https://quay.io must be built:

- edit the scripts
- bump the respective tag(s) in ``src/config/ci-globals.yml``
- run ``make templates``
- commit and file MR

Use a pending MR in a different project
---------------------------------------

When implementing a new feature, it is best to also provide a usage
of the currently opened MR in a separate project.

Thanks to gitlab's magic, users can actually reference any sha in any opened
MR even if they are not merged, yet.

However, the ``cbuild`` script will not be pushed to the canonical repo,
meaning that hackers need to overwrite the URL to have a working use case.

To achieve that, please have a look at any job that makes use of ``cbuild``
in the ``ci-templates`` local fork.

At the beginning of the logs (``alpine:latest@container-build@x86_64``
from the ``alpine`` child pipeline for example), you'll see

.. code-block:: bash

  $ echo $FDO_CBUILD
  https://gitlab.freedesktop.org/api/v4/projects/5764/packages/generic/cbuild/sha256-3606152dfd34af076f6c09d24bb8cdf128fce83d07d53d91745c101ec06c7209/cbuild

This URL is the current ``cbuild`` that is used by this version of the MR.

.. tip::

  The format is the following:

  ``https://gitlab.../api/v4/projects/{fork_project_id}/packages/generic/cbuild/sha256-{sha256 of cbuild}/cbuild``

Then, in your example of usage, in the job that extends
``.fdo.container-build@distribution``, override the ``FDO_CBUILD`` variable
with the URL above.

A more complete example is the following:

.. code-block:: yaml

  include:
  - project: 'freedesktop/ci-templates'
      ref: 62b11fa3f746b38ae413fc936cc6ae96960dd448
      file:
        - '/templates/freebsd.yml'

  stages:
   - prep
   - use

  freebsd:13.0@qemu-build@x86_64:
    extends:
    - .fdo.qemu-build@freebsd@x86_64
    stage: prep
    variables:
      FDO_CBUILD: https://gitlab.freedesktop.org/api/v4/projects/5764/packages/generic/cbuild/sha256-3606152dfd34af076f6c09d24bb8cdf128fce83d07d53d91745c101ec06c7209/cbuild
      FDO_DISTRIBUTION_PACKAGES: 'wget curl'
      FDO_DISTRIBUTION_VERSION: '13.0'

  freebsd:13.0@qemu-check@x86_64:
    extends:
    - .fdo.distribution-image@freebsd
    stage: use
    script:
      - ...