# CI/CD Pipelines Hearthforge runs pipelines in Docker or Podman containers. A pipeline is described by a `.hearthforge-ci.toml` file at the root of the repository. The **Pipelines** tab of a repository shows the run history, a short reference, and a link to the full [template](web/static/assets/hearthforge-ci-template.toml). ## How a run works 1. A push, a tag, or the **Run pipeline** button creates a run. Branches and tags changed in the web UI count as pushes. The config is read from the pushed commit, so every branch can carry its own pipeline. The branch picker next to the button selects the branch a manual run builds. The form fields come from that branch's config. 2. Hearthforge pulls `image`, creates the cache volumes, and starts one container named `hearthforge-ci-` with `work_dir` as its working directory. Files listed under `[[copy]]` are taken from their source images first. 3. The commit is exported with `git archive` on the host and uploaded into `clone_project_to`. The image needs no git. No `.git` directory reaches the container. Directories are created by the upload itself, so the image needs no `mkdir` either. 4. Steps run in file order, in that one container, so files and installed packages persist from step to step. Each step is one `docker exec` of `shell` with `shell_setup` prepended. 5. Published files are streamed out of the container into `$DATA_DIR/ci/artifacts//`. Archives are built on the host, so the image needs no tar, gzip, zstd or zip. The container is removed. Cache volumes stay. Everything up to the first step is shown as the step `pipeline setup`. It fails after 30 minutes. The image must provide a POSIX shell at `/bin/sh` (or whatever `shell` names), `sleep` for the container's main process, and `rm` plus `mkdir` if a step uses `clear`. Any image with busybox or coreutils qualifies. Nothing else is required. ## Setup Hearthforge talks to the container engine over its Unix socket. It checks, in order: 1. `CI_DOCKER_SOCKET`, when set 2. `/var/run/docker.sock` 3. `/run/podman/podman.sock` 4. `/run/user//podman/podman.sock` Without a socket the Pipelines tab still works, and every run ends in one skipped `pipeline setup` step that names the problem. ### Host install Docker needs no extra setup. Rootless Podman needs the API socket enabled: ```bash systemctl --user enable --now podman.socket ``` ### Hearthforge in a container The socket must be mounted into the Hearthforge container. `compose.yml` carries commented-out bindings for both engines. Mount the socket at `/var/run/docker.sock` and the auto-detection finds it, or mount it elsewhere and set `CI_DOCKER_SOCKET`. Mounting the socket gives Hearthforge full control over the container engine. That is equivalent to root on the host for Docker, and to the socket owner's account for rootless Podman. Cache volumes, image pulls, and the CI containers live on the engine side. Nothing about them refers to paths inside the Hearthforge container, so a containerised Hearthforge works the same as a host install. ### Server settings | Variable | Default | Description | |----------------------|-------------------|------------------------------------------------------| | `CI_DOCKER_SOCKET` | _(auto-detected)_ | Path to the Docker/Podman socket | | `CI_MAX_CONCURRENT` | `2` | Runs executed at once. Further runs wait in a queue. | | `CI_MAX_HISTORY` | `50` | Runs kept per repository, artifacts included | | `CI_DEFAULT_TIMEOUT` | `3600` | Step timeout in seconds when the config sets none | | `CI_NETWORK` | _(engine default)_ | Network the CI container joins | | `CI_VM_IMAGE` | _(built from `vm/`)_ | Image of the build VM container, pulled when missing | | `CI_VM_CPUS` | `2` | vCPUs of a build VM | | `CI_VM_MEMORY_MB` | `2048` | RAM of a build VM, at least 1024 | | `CI_VM_DISK_MB` | `20480` | Scratch disk of a build VM, for layers and pulls | | `CI_MAX_IMAGE_BYTES` | `4294967296` | Largest image a build may push (4 GiB) | ## Example ```toml image = "docker.io/debian:stable" work_dir = "/ci/build" clone_project_to = "/ci/build/project" # must be absolute shell_setup = "set -euo pipefail" timeout = 3600 # per-step default, in seconds memory_limit = "2g" cache = [ { path = "/root/.cache/pip", max_size = "1g" }, ] [on] push = ["main", "release/*"] # branch globs, or ["*"]. `*` does not cross a `/` tag = true [variables] [variables.PYTHON] default = "3.12" description = "Interpreter version, overridable from the Run pipeline form" [[steps]] name = "setup" run_sh = "apt-get update -qq && apt-get install -y --no-install-recommends python3 python3-pip" timeout = 300 [[steps]] name = "test" run_sh = "cd project && python3 -m pytest" [[steps]] name = "package" run_if = 'test -n "${CI_COMMIT_TAG}"' run_sh = "cd project && python3 -m build" publish_gzip = ["/ci/build/project/dist/"] [[steps]] name = "report" always = true run_sh = "cat /ci/build/project/.pytest_cache/v/cache/lastfailed || true" ``` The template in the Pipelines tab lists every key with a comment. ### Step options | Key | Effect | |----------------|------------------------------------------------------------------------------| | `run_sh` | The command. Runs through `shell`, after `shell_setup`. | | `timeout` | Seconds. Falls back to the top-level `timeout`, then `CI_DEFAULT_TIMEOUT`. Also bounds `run_if` and `clear`. | | `run_if` | Shell expression. A non-zero exit skips the step. A timeout fails it. | | `always` | Run even after an earlier step failed. | | `warn_on_fail` | A failure marks the step `warning` and the run continues. | | `clear` | Delete `clone_project_to` and extract a fresh checkout before the step. | | `build_image` | Build a Containerfile in a VM after `run_sh`, see below. | | `publish_*` | `publish_file`, `publish_tar`, `publish_gzip`, `publish_zstd`, `publish_zip` | ### Environment Every step sees `CI=true` plus: | Variable | Value | |-----------------------|--------------------------------| | `CI_PIPELINE_ID` | numeric run ID | | `CI_REPO_NAME` | repository name | | `CI_SERVER_URL` | Hearthforge `BASE_URL` | | `CI_TRIGGER_SOURCE` | `push`, `tag`, or `manual` | | `CI_COMMIT_SHA` | full commit hash | | `CI_COMMIT_SHORT_SHA` | first 8 characters | | `CI_COMMIT_BRANCH` | branch name, empty for tags | | `CI_COMMIT_TAG` | tag name, empty for branches | | `CI_COMMIT_REF_NAME` | branch or tag name | | `CI_REGISTRY` | `/` with the repo name lowercased, the image name prefix for the built-in registry | `[variables]` defaults come next, then overrides from the **Run pipeline** form, then secrets. A later source wins on a name clash. ### Secrets Admins add secrets under **Repository settings → CI Secrets**. They are injected as environment variables. Their values are replaced with `[MASKED]` in step logs. The masking is a plain text match: a secret printed in encoded or split form is not caught. ### Status badge ``` ![CI](https://your-forge.example.com/REPO_NAME/ci/badge.svg) ``` ## Container networking CI containers join the engine's default network. `CI_NETWORK` names a different one. Hearthforge never creates it; the network must exist before a run starts. The common reason to set it is IPv6. A rootful Docker or Podman bridge is IPv4-only, so a step cannot reach a host that publishes only an AAAA record. Create a network once and name it: ```bash podman network create --ipv6 hearthforge-ci # or: docker network create --ipv6 ... ``` `CI_NETWORK=host` puts the container on the host's network stack instead, which also gives it the host's IPv6. A step then sees every port the host listens on, so use it only where that is acceptable. Rootless Podman needs none of this. Its containers already share the host's addresses. ## Building container images A step with `build_image` builds a Containerfile and pushes the image to the repository's namespace in the built-in registry. It runs after the step's `run_sh`, so earlier steps can prepare the context. A step may have `build_image` alone. ```toml [[steps]] name = "image" [steps.build_image] tags = ["$CI_COMMIT_SHORT_SHA", "latest"] ``` | Key | Default | Meaning | |-----------|--------------------------------------|--------------------------------------------------------------| | `context` | `clone_project_to` | Absolute directory in the CI container | | `file` | `Containerfile`, then `Dockerfile` | Relative to `context` | | `target` | last stage | Stage of a multi-stage build | | `args` | none | Build args. `$VARS` are expanded | | `secrets` | none | CI secret names, for `RUN --mount=type=secret,id=NAME` | | `image` | none | Path below the repository: `//` | | `tags` | `["$CI_COMMIT_SHORT_SHA"]` | `$VARS` are expanded. A tag that expands to nothing is dropped | `tags` and `args` expand the predefined variables and `[variables]`, but no secrets. Tags are public, and an arg used in `RUN` is stored in the image history. Pass a secret through `secrets` instead. The step log lists every pushed reference and the manifest digest. Pull the image with the admin password, see the registry section of the README. ### How a build runs 1. Hearthforge reads `context` out of the CI container. 2. It starts a build VM container from `CI_VM_IMAGE`. The container gets `/dev/kvm`, its own memory and CPU limits, and `CI_NETWORK`. It gets no bind mount, no engine socket and no privileges. 3. The container boots a QEMU microVM with KVM. The VM runs buildah, which builds the context like `docker build`. Any valid Containerfile works: multi-stage builds, heredocs, `RUN --mount` of type cache, secret and bind, `HEALTHCHECK`, `ONBUILD`. The image is stored in Docker format with one layer per instruction. 4. The VM writes the image to the container. Hearthforge checks every blob against its digest and stores the image in the registry. Each build starts cold. The VM keeps no layer cache and pulls its base images again every time. ### Setup The binary carries `vm/`. The first `build_image` step builds the image `localhost/hearthforge-buildvm:` from it through the engine. That takes a minute or two and needs internet access. The step log shows the build output. Later runs reuse the image. A Hearthforge version that changes `vm/` builds a new image under a new tag. Old tags stay until `podman image prune` or `docker image prune -a` removes them. `CI_VM_IMAGE` replaces the built image with your own, for example a pinned image from a registry. Hearthforge pulls it when the engine lacks it. The engine host needs `/dev/kvm`. A cloud VM needs nested virtualization for that. There is no fallback without KVM, because QEMU does not isolate a guest without it. This works the same when Hearthforge itself runs in a container. The build container is a sibling started through the engine socket. The Hearthforge container needs no device and no extra mount. Rootless Podman passes `/dev/kvm` when the user can open it. Hearthforge asks crun to keep the user's supplementary groups, so membership in the `kvm` group is enough. The build container always gets a CPU limit, so rootless Podman needs the `cpu` cgroup controller delegated to the user. ### Isolation | Layer | Stops | |-------|-------| | buildah's `RUN` container in the VM | normal build code | | KVM | everything in the guest, its root and its kernel included | | QEMU `-sandbox on` (seccomp) | most exploits of the emulated devices | | The build container | a QEMU escape. The container holds no data and no socket | | Disk sizes and `CI_MAX_IMAGE_BYTES` | filling the engine's disk | The guest reaches the network through QEMU user networking inside the build container. Its address for the host, `10.0.2.2`, is the container's own loopback, where nothing listens. The guest still reaches everything the container reaches, including host services on the bridge gateway. Block those with a host firewall rule, as for any CI container. ## Caches Each `cache` path becomes a named volume, `hearthforge-ci-cache-`, labelled `com.hearthforge.repo=`. The volume is mounted at the same path in every run of that repository. - A cache without `max_size` grows without bound. - A cache with `max_size` is deleted after the run that takes it over the limit. The run detail page shows a `cache` step naming the dropped volume. The next run starts cold. Size the cap at about twice the working set. - A volume the current config no longer lists is deleted after a run on the default branch. Other branches keep it. - **Purge caches** on the Pipelines tab deletes all volumes of the repository. Deleting or renaming a repository does the same. - Everything else needs the engine's own tools: ```bash docker volume ls --filter label=com.hearthforge.repo=REPO_NAME docker volume rm $(docker volume ls -q --filter label=com.hearthforge.repo) # all Hearthforge caches ``` The size check reads the daemon's reported usage. A daemon that does not report volume sizes never triggers the cap. ## Gotchas - **`clone_project_to` must be absolute**, and no `cache` path may lie inside it or above it. The checkout is extracted over that directory and a `clear` step deletes it. The config is rejected otherwise. - **`publish_zip` drops symlinks.** The zip format has no symlink entry, so the archive keeps regular files and directories only. Use `publish_tar` or `publish_gzip` when links matter. - **Steps share one container.** A file left behind by one step is visible to the next. Use `clear` on a step that must start from a clean tree. Do not `rm -rf` `work_dir` itself: the container's working directory is gone and every later step fails to start. - **Logs are cut at 2 MB per step.** Redirect verbose output to a file and publish the file instead. - **A Hearthforge restart cancels running runs.** Every CI container carries the label `com.hearthforge.ci`, and all of them are force-removed on startup. The runs show as `cancelled` and can be retried. - **Deleting a repository cancels its runs** and deletes their artifacts and cache volumes. - **The container is on the engine's default network.** It reaches the internet and any other container on that network. Set `cpu_limit` and `memory_limit`; there are no defaults. - **Old runs are deleted.** Only the `CI_MAX_HISTORY` most recently finished runs per repository are kept, and their artifacts go with them. Attach anything permanent to a release.