# CI/CD Pipelines Hearthforge runs pipelines in Docker or Podman containers. A pipeline is described by a `.hearthforge-ci.toml` file at the root of the repository. The **Pipelines** tab of a repository shows the run history, a short reference, and a link to the full [template](public/assets/hearthforge-ci-template.toml). ## How a run works 1. A push, a tag, or the **Run pipeline** button creates a run. The config is read from the pushed commit, so every branch can carry its own pipeline. The branch picker next to the button selects the branch a manual run builds. The form fields come from that branch's config. 2. Hearthforge pulls `image`, creates the cache volumes, and starts one container named `hearthforge-ci-` with `work_dir` as its working directory. Files listed under `[[copy]]` are taken from their source images first. 3. The commit is exported with `git archive` on the host and uploaded into `clone_project_to`. The image needs no git. No `.git` directory reaches the container. Directories are created by the upload itself, so the image needs no `mkdir` either. 4. Steps run in file order, in that one container, so files and installed packages persist from step to step. Each step is one `docker exec` of `shell` with `shell_setup` prepended. 5. Published files are streamed out of the container into `$DATA_DIR/ci/artifacts//`. Archives are built on the host, so the image needs no tar, gzip, zstd or zip. The container is removed. Cache volumes stay. Everything up to the first step is shown as the step `pipeline setup`. It fails after 30 minutes. The image must provide a POSIX shell at `/bin/sh` (or whatever `shell` names), `sleep` for the container's main process, and `rm` plus `mkdir` if a step uses `clear`. Any image with busybox or coreutils qualifies. Nothing else is required. ## Setup Hearthforge talks to the container engine over its Unix socket. It checks, in order: 1. `CI_DOCKER_SOCKET`, when set 2. `/var/run/docker.sock` 3. `/run/podman/podman.sock` 4. `/run/user//podman/podman.sock` Without a socket the Pipelines tab still works, and every run ends in one skipped `pipeline setup` step that names the problem. ### Host install Docker needs no extra setup. Rootless Podman needs the API socket enabled: ```bash systemctl --user enable --now podman.socket ``` ### Hearthforge in a container The socket must be mounted into the Hearthforge container. `compose.yml` carries commented-out bindings for both engines. Mount the socket at `/var/run/docker.sock` and the auto-detection finds it, or mount it elsewhere and set `CI_DOCKER_SOCKET`. Mounting the socket gives Hearthforge full control over the container engine. That is equivalent to root on the host for Docker, and to the socket owner's account for rootless Podman. Cache volumes, image pulls, and the CI containers live on the engine side. Nothing about them refers to paths inside the Hearthforge container, so a containerised Hearthforge works the same as a host install. ### Server settings | Variable | Default | Description | |----------------------|-------------------|------------------------------------------------------| | `CI_DOCKER_SOCKET` | _(auto-detected)_ | Path to the Docker/Podman socket | | `CI_MAX_CONCURRENT` | `2` | Runs executed at once. Further runs wait in a queue. | | `CI_MAX_HISTORY` | `50` | Runs kept per repository, artifacts included | | `CI_DEFAULT_TIMEOUT` | `3600` | Step timeout in seconds when the config sets none | | `CI_ENGINE_SOCKET` | `0` | Allow `engine_socket = true` steps | | `CI_NETWORK` | _(engine default)_ | Network the CI container joins | ## Example ```toml image = "docker.io/debian:stable" work_dir = "/ci/build" clone_project_to = "/ci/build/project" # must be absolute shell_setup = "set -euo pipefail" timeout = 3600 # per-step default, in seconds memory_limit = "2g" cache = [ { path = "/root/.cache/pip", max_size = "1g" }, ] [on] push = ["main", "release/*"] # branch globs, or ["*"]. `*` does not cross a `/` tag = true [variables] [variables.PYTHON] default = "3.12" description = "Interpreter version, overridable from the Run pipeline form" [[steps]] name = "setup" run_sh = "apt-get update -qq && apt-get install -y --no-install-recommends python3 python3-pip" timeout = 300 [[steps]] name = "test" run_sh = "cd project && python3 -m pytest" [[steps]] name = "package" run_if = 'test -n "${CI_COMMIT_TAG}"' run_sh = "cd project && python3 -m build" publish_gzip = ["/ci/build/project/dist/"] [[steps]] name = "report" always = true run_sh = "cat /ci/build/project/.pytest_cache/v/cache/lastfailed || true" ``` The template in the Pipelines tab lists every key with a comment. ### Step options | Key | Effect | |----------------|------------------------------------------------------------------------------| | `run_sh` | The command. Runs through `shell`, after `shell_setup`. | | `timeout` | Seconds. Falls back to the top-level `timeout`, then `CI_DEFAULT_TIMEOUT`. Also bounds `run_if` and `clear`. | | `run_if` | Shell expression. A non-zero exit skips the step. A timeout fails it. | | `always` | Run even after an earlier step failed. | | `warn_on_fail` | A failure marks the step `warning` and the run continues. | | `clear` | Delete `clone_project_to` and extract a fresh checkout before the step. | | `engine_socket`| Give the step the Docker/Podman socket. Needs `CI_ENGINE_SOCKET=1`. | | `publish_*` | `publish_file`, `publish_tar`, `publish_gzip`, `publish_zstd`, `publish_zip` | ### Environment Every step sees `CI=true` plus: | Variable | Value | |-----------------------|--------------------------------| | `CI_PIPELINE_ID` | numeric run ID | | `CI_REPO_NAME` | repository name | | `CI_SERVER_URL` | Hearthforge `BASE_URL` | | `CI_TRIGGER_SOURCE` | `push`, `tag`, or `manual` | | `CI_COMMIT_SHA` | full commit hash | | `CI_COMMIT_SHORT_SHA` | first 8 characters | | `CI_COMMIT_BRANCH` | branch name, empty for tags | | `CI_COMMIT_TAG` | tag name, empty for branches | | `CI_COMMIT_REF_NAME` | branch or tag name | | `CI_REGISTRY` | `/` with the repo name lowercased, the image name prefix for the built-in registry | `[variables]` defaults come next, then overrides from the **Run pipeline** form, then secrets. A later source wins on a name clash. ### Secrets Admins add secrets under **Repository settings → CI Secrets**. They are injected as environment variables. Their values are replaced with `[MASKED]` in step logs. The masking is a plain text match: a secret printed in encoded or split form is not caught. ### Status badge ``` ![CI](https://your-forge.example.com/REPO_NAME/ci/badge.svg) ``` ## Container networking CI containers join the engine's default network. `CI_NETWORK` names a different one. Hearthforge never creates it; the network must exist before a run starts. The common reason to set it is IPv6. A rootful Docker or Podman bridge is IPv4-only, so a step cannot reach a host that publishes only an AAAA record. Create a network once and name it: ```bash podman network create --ipv6 hearthforge-ci # or: docker network create --ipv6 ... ``` `CI_NETWORK=host` puts the container on the host's network stack instead, which also gives it the host's IPv6. A step then sees every port the host listens on, so use it only where that is acceptable. Rootless Podman needs none of this. Its containers already share the host's addresses. ## Building container images Steps run in a plain container without an engine of their own. A step with `engine_socket = true` gets the host engine's socket mounted at `/run/hearthforge/engine.sock`, and `DOCKER_HOST` and `CONTAINER_HOST` point at it. `docker build` and `podman build` then run on the host engine with the full Dockerfile feature set and the engine's own layer cache. ```toml [[steps]] name = "image" engine_socket = true run_sh = """ apt-get install -y -qq docker.io > /dev/null echo "$REGISTRY_PASSWORD" | docker login "${CI_REGISTRY%%/*}" -u admin --password-stdin docker build -t "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA" project docker push "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA" """ ``` `REGISTRY_PASSWORD` is a CI secret holding the admin password. The server must allow it with `CI_ENGINE_SOCKET=1`. Without that, a step with `engine_socket = true` fails and the run stops. The bind source is the socket path as Hearthforge sees it. When Hearthforge runs in a container, mount the socket at the same path it has on the host, for example `/run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock`. The engine resolves the source on the host, so a different path inside the container makes the step fail with a missing socket. > **Warning:** a step with the socket controls the whole engine. It can start > privileged containers on the host. That is the same access Hearthforge > itself has, and only admins push CI configs, so the trust level does not > change. The socket is mounted for the whole run once any step asks for it. > Other steps do not get `DOCKER_HOST`, but they can still reach the socket > file. ## Caches Each `cache` path becomes a named volume, `hearthforge-ci-cache-`, labelled `com.hearthforge.repo=`. The volume is mounted at the same path in every run of that repository. - A cache without `max_size` grows without bound. - A cache with `max_size` is deleted after the run that takes it over the limit. The run detail page shows a `cache` step naming the dropped volume. The next run starts cold. Size the cap at about twice the working set. - A volume the current config no longer lists is deleted after a run on the default branch. Other branches keep it. - **Purge caches** on the Pipelines tab deletes all volumes of the repository. Deleting or renaming a repository does the same. - Everything else needs the engine's own tools: ```bash docker volume ls --filter label=com.hearthforge.repo=REPO_NAME docker volume rm $(docker volume ls -q --filter label=com.hearthforge.repo) # all Hearthforge caches ``` The size check reads the daemon's reported usage. A daemon that does not report volume sizes never triggers the cap. ## Gotchas - **`clone_project_to` must be absolute**, and no `cache` path may lie inside it or above it. The checkout is extracted over that directory and a `clear` step deletes it. The config is rejected otherwise. - **`publish_zip` drops symlinks.** The zip format has no symlink entry, so the archive keeps regular files and directories only. Use `publish_tar` or `publish_gzip` when links matter. - **Steps share one container.** A file left behind by one step is visible to the next. Use `clear` on a step that must start from a clean tree. Do not `rm -rf` `work_dir` itself: the container's working directory is gone and every later step fails to start. - **Logs are cut at 2 MB per step.** Redirect verbose output to a file and publish the file instead. - **A Hearthforge restart cancels running runs.** Every CI container carries the label `com.hearthforge.ci`, and all of them are force-removed on startup. The runs show as `cancelled` and can be retried. - **Deleting a repository cancels its runs** and deletes their artifacts and cache volumes. - **The container is on the engine's default network.** It reaches the internet and any other container on that network. Set `cpu_limit` and `memory_limit`; there are no defaults. - **Old runs are deleted.** Only the `CI_MAX_HISTORY` most recently finished runs per repository are kept, and their artifacts go with them. Attach anything permanent to a release.