Local CI ======== Run the GitLab CI pipeline locally against the same ``.gitlab-ci.yml`` your runners use. Useful for: * Verifying a change won't break CI before pushing. * Producing and pushing a Docker image when the real CI is down. * Iterating on the pipeline itself without burning shared CI minutes. No host installation of ``gitlab-ci-local``, Node, or any Python tooling is required. Everything runs in throwaway containers on your local Docker daemon. How it works ------------ ``./scripts/ci_local.sh`` launches ``node:22-alpine``, installs ``git``, ``bash``, ``rsync``, and ``docker-cli``, then runs the `gitlab-ci-local `_ npm package via ``npx``. The host Docker socket is bind-mounted into the container so each CI job runs as a *sibling* container on your host daemon (Docker-out-of-Docker, not nested). The repo is mounted at its real host path inside the container so sibling jobs see the same paths as the parent. Sensitive variables that real CI provides automatically (``CI_REGISTRY_USER``, ``CI_REGISTRY_PASSWORD``, ``DOCKER_IO_USER``, ``DOCKER_IO_PASSWORD``) are read from ``.gitlab_ci_local_variables.yml`` in the repo root. That file is gitignored -- never commit credentials. Prerequisites ------------- * Docker installed and running on the host. * For the **build** / **build_branch** jobs (which push to the GitLab Container Registry): a project Deploy Token (see below). * For the **build_docker_hub** job: Docker Hub credentials. * No prerequisites for the **code** stage (``vulture``, ``ruff``, ``typos``) or the **test** stage (``tests``, ``tests:network``). One-time setup -------------- 1. Create a GitLab Deploy Token ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Deploy Tokens authenticate against the project's Container Registry without exposing your personal credentials. 1. Open ``_. 2. Expand **Deploy tokens**. 3. **Add token**: * **Name**: anything memorable, e.g. ``local-registry-push``. * **Expiration**: 6-12 months is reasonable. * **Scopes**: tick ``read_registry`` **and** ``write_registry``. Leave ``read_repository`` / ``write_repository`` unchecked unless you need git access too. 4. Click **Create deploy token**. Copy **both** the username (typically ``gitlab+deploy-token-``) and the token value **immediately** -- the token is shown only once. 2. Create a Docker Hub Personal Access Token ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Only needed if you plan to run the ``build_docker_hub`` job locally. 1. Open ``_. 2. **New Access Token**. Tick **Read, Write, Delete** scope. 3. Copy the token value immediately. 3. Fill in the variables file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The variables file is created by the initial repo checkout with ``REPLACE_ME`` placeholders. Open it: .. code-block:: text .gitlab_ci_local_variables.yml Paste your values: .. code-block:: yaml CI_REGISTRY: registry.gitlab.com CI_REGISTRY_USER: gitlab+deploy-token- CI_REGISTRY_PASSWORD: DOCKER_IO_USER: DOCKER_IO_PASSWORD: The file is in ``.gitignore``. Run ``git status`` after editing -- it should not appear. Running ------- The script accepts the same arguments as ``gitlab-ci-local``. Run every job in the pipeline ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: bash $ ./scripts/ci_local.sh Run a single job ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: bash $ ./scripts/ci_local.sh tests $ ./scripts/ci_local.sh tests:network $ ./scripts/ci_local.sh ruff $ ./scripts/ci_local.sh build List available jobs without running ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: bash $ ./scripts/ci_local.sh --list Run a build job from a non-master branch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``build`` and ``build_docker_hub`` jobs in ``.gitlab-ci.yml`` only run on ``master``, tag ``stable``, or version tags (``[0-9.]+``). To exercise them locally from a feature branch, override the ref name: .. code-block:: bash $ ./scripts/ci_local.sh --variable CI_COMMIT_REF_NAME=master build Same trick works for ``build_docker_hub``. ``build_branch`` is gated on merge-request pipelines: .. code-block:: bash $ ./scripts/ci_local.sh --variable CI_MERGE_REQUEST_ID=1 build_branch What runs where --------------- .. list-table:: :header-rows: 1 :widths: 18 28 18 36 * - Stage / job - Image used - Needs creds - Notes * - ``vulture``, ``ruff``, ``typos`` - Alpine images - No - Static checks. * - ``tests`` - ``docker:25.0.2`` + ``docker:25.0.2-dind`` service - No - Builds the dev image, runs Django tests with ``--exclude-tag=network``. * - ``tests:network`` - Same as ``tests`` - No - Runs network-tagged tests. ``allow_failure: true`` in real CI; local runs surface the failure normally. * - ``build`` - ``docker:25.0.2`` + dind - **Yes** (GitLab Deploy Token) - Builds and pushes to ``registry.gitlab.com/imn1/richy``. * - ``build_branch`` - Same as ``build`` - **Yes** (GitLab Deploy Token) - Branch image variant. Gated by MR pipeline. * - ``build_docker_hub`` - ``docker:25.0.2`` + dind - **Yes** (Docker Hub PAT) - Mirrors the image to ``docker.io/n1cz/richy``. Differences from real CI ------------------------ * **Token type.** Real CI uses an ephemeral ``CI_JOB_TOKEN``; locally we substitute a Deploy Token. ``.gitlab-ci.yml`` reads ``CI_REGISTRY_USER`` / ``CI_REGISTRY_PASSWORD`` (which GitLab sets automatically in CI), so the same script works in both contexts. * **Network egress IP.** Real CI runs from data-center IPs that Yahoo Finance and other upstream APIs frequently throttle or tarpit. Locally your residential / office IP is rarely affected -- network tests that flake in CI will usually pass cleanly here. * **Helper utility image pulled lazily.** ``docker.io/firecow/gitlab-ci-local-util`` is pulled the first time any test job runs -- ``gitlab-ci-local`` uses it internally as a helper image. Cached after the first pull. Performance notes ----------------- .. list-table:: :header-rows: 1 :widths: 40 25 35 * - Operation - First run - Cached run * - ``apk add git bash rsync docker-cli`` - ~10s - ~5s (no apk cache between runs) * - ``npx -y gitlab-ci-local`` - ~10s - instant (cached in named volume ``ci_local_npm_cache``) * - Pulling upstream CI base images - ~1.5 GB once - instant (host Docker daemon image cache) Total cold-start overhead before the first job actually runs: ~25 seconds. Subsequent runs add ~10 seconds of overhead before the first job kicks off. Files involved -------------- .. list-table:: :header-rows: 1 :widths: 35 65 * - File - Purpose * - ``.gitlab-ci.yml`` - The actual CI pipeline definition; same file the GitLab runner uses. * - ``compose.ci.yaml`` - Compose stack the ``tests`` job brings up (richy app, redis, postgres). * - ``scripts/ci_local.sh`` - Entry-point wrapper that launches ``gitlab-ci-local`` in a container. * - ``.gitlab_ci_local_variables.yml`` - Local credentials. Gitignored. * - ``.gitlab-ci-local/`` - Working directory created by ``gitlab-ci-local`` (artifacts, service logs, etc.). Gitignored. See also -------- * :doc:`ci` -- the real pipeline this mirrors. * :doc:`/architecture/testing` -- what the ``tests`` and ``tests:network`` jobs actually run.