.. _ci-fairy: ci-fairy - a CLI tool ===================== ``ci-fairy`` is a commandline helper tool for some commonly performed tasks, primarily those executed by the CI. ``ci-fairy`` is optimized to be run from a CI script where its arguments will be hardcoded and change very little. It does not do a lot of error handling, expect to see exceptions where things go wrong. Use within gitlab-ci.yml ------------------------ The ``ci-fairy`` tool is available by extending the ``.fdo.ci-fairy`` template: .. code-block:: yaml include: - project: 'freedesktop/ci-templates' file: '/templates/ci-fairy.yml' check-mr: extends: - .fdo.ci-fairy script: - ci-fairy check-merge-request --require-allow-collaboration The ``.fdo.ci-fairy`` template guarantees the following preinstalled packages: ``bash``, ``git``, ``python`` and ``pip``. It is recommended that CI jobs that want to ``pip install`` use a `Python venv `__ to avoid version conflicts. The distribution the template is based on is subject to change without notice. Installation ------------ As the ``ci-fairy`` is likely used within a CI job, installation via ``pip3`` directly from git is recommended: :: pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates@12345deadbeef The ``@12345deadbeef`` suffix specifies the git sha to check out. The git sha should match the sha of the :ref:`included templates `. Or where a local checkout is available: :: pip3 install . Alternatively, you may directly invoke the ``ci-fairy`` tool from within the git tree without prior installation. Note that ``pip3`` will take care of all dependencies, for local invocations you must install those manually. Use of the GitLab CI Environment -------------------------------- ``ci-fairy`` will make use of the `predefined environment variables set by GitLab CI `__ where possible. This includes but is not limited to ``CI_PROJECT_PATH``, ``CI_SERVER_URL``, and ``CI_JOB_TOKEN``. In most cases ``ci-fairy`` does not need arguments beyond what is specific to the interaction with the project at the time. Authentication -------------- Authentication is handled automatically through the ``$CI_JOB_TOKEN`` environment variable. Where the ``ci-fairy`` tool is called outside a CI job, use the ``--authfile`` argument to provide the path to a file containing the value of a `GitLab private token `__ with 'api' access. For example, if your private token has the value ``abcd1234XYZ``, authentication can be performed like this: :: $ echo "abcd1234XYZ" > /path/to/authfile $ ci-fairy --authfile /path/to/authfile <....> Where ``--authfile`` is used within a CI job, specificy the token as a `GitLab predefined environment variable `__ of type "File". The "Key" is the file name given to ``--authfile``, the value is the token value of your private token. .. _ci-fairy-linting: Linting ------- GitLab has an online linter but it requires copy/pasting the ``.gitlab-ci.yml`` file into the web form. ``ci-fairy`` makes this simpler: :: $ cd /path/to/project.git $ ci-fairy lint ``ci-fairy`` will find the ``.gitlab-ci.yml`` file and **upload it to the CI linter** and print any errors. .. _ci-fairy-pre-commit: pre-commit ---------- You can incorporate some ci-fairy functions check in your `pre-commit `_ setup. Add the following snippet to your ``.pre-commit-config.yaml`` .. code-block:: yaml - repo: https://gitlab.freedesktop.org/freedesktop/ci-templates.git rev: master hooks: # The hooks below are pre-push by default - id: lint - id: check-commits - id: generate-template Currently all ci-fairy pre-commit hooks default to ``stages: ["push"]``, i.e. they run pre-push, not pre-commit. Please see the `pre-commit Documentation `_ for further details. .. note:: By default ``pre-commit`` installs the pre-commit hook only. Run ``pre-commit install --hook-type pre-push`` to enable the ci-fairy hooks. .. _ci-fairy-deleting-images: Deleting registry images ------------------------ .. note:: The ``CI_JOB_TOKEN`` does not have sufficient permissions to delete images, ``--authfile`` is required for this task. To delete an image from your container registry, use the ``delete-image`` subcommand. This subcommands provides three modes - deleting a specific image tag, deleting all but an image tag or deleting all images. :: ci-fairy delete-image --repository fedora/30 --all ci-fairy delete-image --repository fedora/30 --exclude-tag 2020-03.11.0 ci-fairy delete-image --tag "2020-03-*" During testing, use the ``--dry-run`` argument to ensure that no tags are actually deleted. .. _ci-fairy-templating: Templating .gitlab-ci.yml ------------------------- The ``.gitlab-ci.yml`` file can become repetitive where multiple jobs use the same setups (e.g. such as testing a build across different distributions). ``ci-fairy`` provides a command to generate a ``.gitlab-ci.yml`` (or any file, really) from a `Jinja2 template `__, sourcing the data from a YAML file. For example, let's assume the YAML file ``.gitlab-ci/config.yml`` with a list of distributions: .. code-block:: yaml distributions: fedora: [32, 31] ubuntu: [19.04] packages: fedora: needed: ['gcc', 'valgrind'] use_qemu: false ubuntu: needed: ['curl', 'wget', 'valgrind'] use_qemu: true debian: needed: ['curl', 'wget'] And the template ``.gitlab-ci/ci.template`` that uses those to generate jobs (see the `Jinja2 documentation `__ for details): .. code-block:: {% for distro in distributions %} {% for version in distributions[distro] %} .job.{{distro}}.{{version}}: extends: .fdo.container-build@{{distro}}@x86_64 variables: FDO_DISTRIBUTION_VERSION: "{{version}}" FDO_DISTRIBUTION_PACKAGES: "{{" ".join(packages[distro].needed)}}" {% endfor %} {% endfor %} Now let's generate the ``.gitlab-ci.yml`` file: :: $ ci-fairy generate-template and our file will look like this: .. code-block:: yaml .job.fedora.32: extends: .fdo.container-build@fedora variables: FDO_DISTRIBUTION_VERSION: "32" FDO_DISTRIBUTION_PACKAGES: "gcc valgrind" .job.fedora.31: extends: .fdo.container-build@fedora variables: FDO_DISTRIBUTION_VERSION: "31" FDO_DISTRIBUTION_PACKAGES: "gcc valgrind" .job.ubuntu.19.04: extends: .fdo.container-build@ubuntu@x86_64 variables: FDO_DISTRIBUTION_VERSION: "19.04" FDO_DISTRIBUTION_PACKAGES: "curl wget valgrind" .. note:: Without arguments, ``ci-fairy generate-templates`` always uses the files ``.gitlab-ci/config.yml`` and ``.gitlab-ci/ci.template`` and (over)write the file ``.gitlab-ci.yml``. This command does not need a connection to the GitLab instance. Templating any file ................... While the defaults generate a ``.gitlab-ci.yml`` file, it's possible to invoke ``ci-fairy`` with any YAML or template file: :: $ ci-fairy generate-template --config somefile.yml mytemplate.tmpl ``ci-fairy`` doesn't care about the template file type, you can use any text file as template source. To allow using the same YAML file for multiple templates, ``ci-fairy`` allows for the selection of the root node. For example: :: $ cat template.tpl qemu needed? {{use_qemu}} $ ci-fairy generate-template --root=/packages/fedora --config distributions.yml template.tpl qemu needed? true $ ci-fairy generate-template --root=/packages/ubuntu --config distributions.yml template.tpl qemu needed? false ``ci-fairy`` also provides helper functions that can be used from Jinja2 templates: * ``ci_fairy.hashfiles()``: takes a list of one or more filenames and produces a 64 character hexdigest string. It hashes both the files' content and the filenames themselves. The used hash algorithm is unspecified but stable. This can be used to generate checksums from files. Example: :: variables: FEDORA_TAG: {{ (ci_fairy.hashfiles('.gitlab-ci/config.yml', '.gitlab-ci/ci.template', '.gitlab-ci/fedora-install.sh'))[0:12] }}' * ``ci_fairy.sha256sum(path, prefix=True)``: takes a path to a file and produces a string in the form of ``sha256-`` if ``prefix=True`` and in the form of ```` if ``prefix=False``. Unlike ``hashfiles()`` it hashes only the file contents. This can be used to generate checksums from files. Example: :: variables: CHECKSUM: {{ci_fairy.sha256sum('tarball.tar.xz')}} * ``ci_fairy.import_module(module_name)``: takes a string which is a python module name and exports that module as a variable. This is useful to import a module not exposed by ``ci_fairy`` itself. Example: :: {% set time = ci_fairy.import_module( 'time' ) %} {{ time.time() }} * ``ci_fairy.nodes``: gives the full config nodes as a dictionary. This allows to iterate over all root nodes. Example: :: {% for key in ci_fairy.nodes %} - root node in config: { key } {% endfor %} .. note:: Root nodes in the config files must not be named ``ci_fairy`` to avoid namespace clashes. For convenience, users are encouraged to use the unicode character ``🧚`` as an alias for ``ci_fairy``. This makes the template clearer that this object is *not* part of the config file. Example: .. code-block:: variables: FEDORA_TAG: {{ (🧚.hashfiles('.gitlab-ci/config.yml', '.gitlab-ci/ci.template', '.gitlab-ci/fedora-install.sh'))[0:12] }}' Extensions to YAML syntax: .......................... In addition to the ``ci_fairy`` or ``🧚`` keyword, some keywords have been added to the YAML language to allow easier configuration. extends: ^^^^^^^^ ``extends`` aims at replicating `Gitlab CI/CD extends keyword `_. There are a few differences however, and ``ci-fairy`` only supports the following: Supported in: top-level dictionaries The ``extends:`` keyword makes the current dictionary inherit all members of the extended dictionary, according to the following rules: - where the value is a non-empty dict, the base and new dicts are merged - where the value is a non-empty list, the base and new list are concatinated - where the value is an empty dict or empty list, the new value is the empty dict/list. - otherwise, the new value overwrites the base value. Example: .. code-block:: yaml foo: bar: [1, 2] baz: a: 'a' b: 'b' bat: 'foobar' subfoo: extends: bar bar: [3, 4] baz: c: 'c' bat: 'subfoobar' Results in the effective values for subfoo: .. code-block:: yaml subfoo: bar: [1, 2, 3, 4] baz: {a: 'a', b: 'b', c: 'c'} bat: 'subfoobar' foo: bar: 1 include: ^^^^^^^^ ``include`` aims at replicating the local include from `Gitlab CI/CD include keyword `_. Supported in: top-level only The ``include:`` keyword includes the specified file at the place. The path to the included file is relative to the source file. Example: .. code-block:: yaml # content of firstfile foo: bar: [1, 2] #content of secondfile bar: baz: [3, 4] include: firstfile foobar: extends: foo .. note:: the included file will work with the normal YAML rules, specifically: where the included file has a key with the same name this section will be overwritten with the later-defined. The position of the ``include`` statement thus matters a lot. version: ^^^^^^^^ The YAML file version (optional). Handled as attribute on this object and does not show up in the dictionary otherwise. This version must be a top-level entry in the YAML file in the form "version: 1" or whatever integer number and is exposed as the "version" attribute. Where other files are included, the version of that file must be identical to the first version found in any file. .. _ci-fairy-commit-checks: Checking commits ---------------- Git commit messages have a few standard requirements that apply across projects. ``ci-fairy`` can verify the commits to be merged for those standards. :: $ ci-fairy check-commits master..HEAD By default, this checks for any leftover ``fixup!``/``squash!`` commits and some cosmetic requirements. Other options to check for is whether a ``Signed-off-by:`` line is present (``--signed-off-by``) or ensure that such a line is **not** included (``--no-signed-off-by``). See the ``--help`` output for a list fo checks available. It is possible to provide custom rule definitions in YAML format, either in ``.gitlab-ci/commit-rules.yml`` if it exists, or by specifying a file with the ``--rules-file`` option. An example rules files: .. code-block:: yaml patterns: require: - regex: ($CI_MERGE_REQUEST_PROJECT_URL/(-/)?(issues|merge_requests)/[0-9]+) message: Commit message must contain a link to an issue or merge request deny: - regex: '^\S*\.[ch]:' message: Commit message subject prefix should not include .c, .h etc. where: subject ``ci-fairy`` uses the CI environment where available and will do the right thing without requiring options. It does need the ``$FDO_UPSTREAM_REPO`` variable to be set to the upstream project's full name. An example ``.gitlab-ci.yml`` section: .. code-block:: yaml variables: FDO_UPSTREAM_REPO: libinput/libinput commit message check: image: something_that_has_pip before_script: - pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates script: - ci-fairy check-commits --signed-off-by ``ci-fairy`` uses ``$CI_MERGE_REQUEST_TARGET_BRANCH_NAME`` if available, otherwise it defaults to the ``main`` or ``master`` branch of the ``$FDO_UPSTREAM_REPO``, whichever exists. For the above snippet, the commit range checked will thus be ``upstream/master..HEAD``. .. _ci-fairy-mr-checks: Checking merge requests ----------------------- ``ci-fairy`` can be used to check merge requests for common requirements: :: $ ci-fairy check-merge-request --require-allow-collaboration The ``--require-allow-collaboration`` flag checks for the merge request to `allow edits from maintainers `__. This can be used in the CI to fail a merge request pipeline if that checkbox is not set. There are two ways to invoke this through the CI: either through a `detached merge pipeline `__ or by searching for a merge request with the same commit sha as our branch. The detached merge pipeline has side-effects you should familiarize yourself with. Checking merge requests within a pipeline ......................................... Here is an example for utilising ``FDO_UPSTREAM_REPO``: .. code-block:: yaml variables: FDO_UPSTREAM_REPO: 'project/name' check-merge-request: image: golang:alpine before_script: - apk add python3 git - pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates script: - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml artifacts: when: on_failure reports: junit: check-merge-request.xml When pushing to a branch, this snippet will search ``project/name`` for an **open** merge requests that has the same git sha as this pipeline. That merge request is the one the checks are performed on. If no suitable merge request is found, ``ci-fairy``'s exits code is 2. Otherwise, ``ci-fairy`` exits with status 0 on success or 1 in case of failure. The advantag of this approach is that the job is run as part of the normal CI pipeline and no detached merge pipeline needs to be run. The disadvantages of this approach is that the pipeline may complete before a merge request is filed and the job may fail despite the (future) merge request having the checkbox set. Checking merge requests within a detached pipeline .................................................. Here is an example for utilizing detached merge pipelines: .. code-block:: yaml # work around the detached merge request pipeline issues by defining # rules for all jobs workflow: rules: - when: always check-merge-request: image: golang:alpine before_script: - apk add python3 git - pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates script: - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml rules: - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' when: always - when: never # override the workflow's always rule artifacts: when: on_failure reports: junit: check-merge-request.xml Note that defining ``rules:`` for a single job has the side-effect of running `detached merge pipelines `__. This is almost never what you really want. To work around this, the above example defines global rules in ``workflow``, effectively providing a ``rules`` definition for all jobs. Ensure that this is compatible with your project's CI. .. _ci-fairy-wait-for-pipeline: Waiting for a pipeline ---------------------- ``ci-fairy`` can be used to wait for the completion of a pipeline. :: $ ci-fairy wait-for-pipeline In the default invocation with no arguments ``ci-fairy`` will guess the gitlab project path based on the ``$GITLAB_USER_ID`` if set, or ``$USER`` otherwise, combined with the basename of the current directory. The pipline to wait for defaults to the pipeline matching the current git sha:: $ whoami bob $ cd myproject $ git push -q gitlab mybranch $ ci-fairy wait-for-pipeline Pipeline https://gitlab.freedesktop.org/bob/myproject/-/pipelines/12345 status: success | 90/90 | created: 0 | pending: 0 | running: 0 | failed: 1 | success: 89 Please see the ``--help`` output for more details on controlling which pipeline to wait for.