CI/CD Pipelines

Hearthforge runs pipelines in Docker or Podman containers. A pipeline is described by a .hearthforge-ci.toml file at the root of the repository. The Pipelines tab of a repository shows the run history, a short reference, and a link to the full template.

How a run works

A push, a tag, or the Run pipeline button creates a run. The config is read from the pushed commit, so every branch can carry its own pipeline.
Hearthforge pulls image, creates the cache volumes, and starts one container named hearthforge-ci-<run id> with work_dir as its working directory. Files listed under [[copy]] are taken from their source images first.
The commit is exported with git archive on the host and uploaded into clone_project_to. The image needs no git. No .git directory reaches the container. Directories are created by the upload itself, so the image needs no mkdir either.
Steps run in file order, in that one container, so files and installed packages persist from step to step. Each step is one docker exec of shell with shell_setup prepended.
Published files are streamed out of the container into $DATA_DIR/ci/artifacts/<run id>/. Archives are built on the host, so the image needs no tar, gzip, zstd or zip. The container is removed. Cache volumes stay.

Everything up to the first step is shown as the step pipeline setup.

The image must provide a POSIX shell at /bin/sh (or whatever shell names), sleep for the container's main process, and rm plus mkdir if a step uses clear. Any image with busybox or coreutils qualifies. Nothing else is required.

Setup

Hearthforge talks to the container engine over its Unix socket. It checks, in order:

CI_DOCKER_SOCKET, when set
/var/run/docker.sock
/run/podman/podman.sock
/run/user/<uid>/podman/podman.sock

Without a socket the Pipelines tab still works, and every run ends in one skipped pipeline setup step that names the problem.

Host install

Docker needs no extra setup. Rootless Podman needs the API socket enabled:

systemctl --user enable --now podman.socket

Hearthforge in a container

The socket must be mounted into the Hearthforge container. compose.yml carries commented-out bindings for both engines. Mount the socket at /var/run/docker.sock and the auto-detection finds it, or mount it elsewhere and set CI_DOCKER_SOCKET.

Mounting the socket gives Hearthforge full control over the container engine. That is equivalent to root on the host for Docker, and to the socket owner's account for rootless Podman.

Cache volumes, image pulls, and the CI containers live on the engine side. Nothing about them refers to paths inside the Hearthforge container, so a containerised Hearthforge works the same as a host install.

Server settings

Variable	Default	Description
`CI_DOCKER_SOCKET`	(auto-detected)	Path to the Docker/Podman socket
`CI_MAX_CONCURRENT`	`2`	Runs executed at once. Further runs wait in a queue.
`CI_MAX_HISTORY`	`50`	Runs kept per repository, artifacts included
`CI_DEFAULT_TIMEOUT`	`3600`	Step timeout in seconds when the config sets none
`CI_ENGINE_SOCKET`	`0`	Allow `engine_socket = true` steps
`CI_NETWORK`	(engine default)	Network the CI container joins

Example

image = "docker.io/debian:stable"
work_dir = "/ci/build"
clone_project_to = "/ci/build/project"   # must be absolute
shell_setup = "set -euo pipefail"
timeout = 3600                           # per-step default, in seconds
memory_limit = "2g"

cache = [
  { path = "/root/.cache/pip", max_size = "1g" },
]

[on]
push = ["main", "release/*"]   # branch globs, or ["*"]. `*` does not cross a `/`
tag = true

[variables]
  [variables.PYTHON]
  default = "3.12"
  description = "Interpreter version, overridable from the Run pipeline form"

[[steps]]
name = "setup"
run_sh = "apt-get update -qq && apt-get install -y --no-install-recommends python3 python3-pip"
timeout = 300

[[steps]]
name = "test"
run_sh = "cd project && python3 -m pytest"

[[steps]]
name = "package"
run_if = 'test -n "${CI_COMMIT_TAG}"'
run_sh = "cd project && python3 -m build"
publish_gzip = ["/ci/build/project/dist/"]

[[steps]]
name = "report"
always = true
run_sh = "cat /ci/build/project/.pytest_cache/v/cache/lastfailed || true"

The template in the Pipelines tab lists every key with a comment.

Step options

Key	Effect
`run_sh`	The command. Runs through `shell`, after `shell_setup`.
`timeout`	Seconds. Falls back to the top-level `timeout`, then `CI_DEFAULT_TIMEOUT`.
`run_if`	Shell expression. A non-zero exit skips the step.
`always`	Run even after an earlier step failed.
`warn_on_fail`	A failure marks the step `warning` and the run continues.
`clear`	Delete `clone_project_to` and extract a fresh checkout before the step.
`engine_socket`	Give the step the Docker/Podman socket. Needs `CI_ENGINE_SOCKET=1`.
`publish_*`	`publish_file`, `publish_tar`, `publish_gzip`, `publish_zstd`, `publish_zip`

Environment

Every step sees CI=true plus:

Variable	Value
`CI_PIPELINE_ID`	numeric run ID
`CI_REPO_NAME`	repository name
`CI_SERVER_URL`	Hearthforge `BASE_URL`
`CI_TRIGGER_SOURCE`	`push`, `tag`, or `manual`
`CI_COMMIT_SHA`	full commit hash
`CI_COMMIT_SHORT_SHA`	first 8 characters
`CI_COMMIT_BRANCH`	branch name, empty for tags
`CI_COMMIT_TAG`	tag name, empty for branches
`CI_COMMIT_REF_NAME`	branch or tag name
`CI_REGISTRY`	`<BASE_URL host>/<repo>`, the image name prefix for the built-in registry

[variables] defaults come next, then overrides from the Run pipeline form, then secrets. A later source wins on a name clash.

Secrets

Admins add secrets under Repository settings → CI Secrets. They are injected as environment variables. Their values are replaced with [MASKED] in step logs. The masking is a plain text match: a secret printed in encoded or split form is not caught.

Status badge

![CI](https://your-forge.example.com/REPO_NAME/ci/badge.svg)

Container networking

CI containers join the engine's default network. CI_NETWORK names a different one. Hearthforge never creates it; the network must exist before a run starts.

The common reason to set it is IPv6. A rootful Docker or Podman bridge is IPv4-only, so a step cannot reach a host that publishes only an AAAA record. Create a network once and name it:

podman network create --ipv6 hearthforge-ci     # or: docker network create --ipv6 ...

CI_NETWORK=host puts the container on the host's network stack instead, which also gives it the host's IPv6. A step then sees every port the host listens on, so use it only where that is acceptable.

Rootless Podman needs none of this. Its containers already share the host's addresses.

Building container images

Steps run in a plain container without an engine of their own. A step with engine_socket = true gets the host engine's socket mounted at /run/hearthforge/engine.sock, and DOCKER_HOST and CONTAINER_HOST point at it. docker build and podman build then run on the host engine with the full Dockerfile feature set and the engine's own layer cache.

[[steps]]
name = "image"
engine_socket = true
run_sh = """
apt-get install -y -qq docker.io > /dev/null
echo "$REGISTRY_PASSWORD" | docker login "${CI_REGISTRY%%/*}" -u admin --password-stdin
docker build -t "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA" project
docker push "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA"
"""

REGISTRY_PASSWORD is a CI secret holding the admin password.

The server must allow it with CI_ENGINE_SOCKET=1. Without that, a step with engine_socket = true fails and the run stops.

The bind source is the socket path as Hearthforge sees it. When Hearthforge runs in a container, mount the socket at the same path it has on the host, for example /run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock. The engine resolves the source on the host, so a different path inside the container makes the step fail with a missing socket.

Warning: a step with the socket controls the whole engine. It can start privileged containers on the host. That is the same access Hearthforge itself has, and only admins push CI configs, so the trust level does not change. The socket is mounted for the whole run once any step asks for it. Other steps do not get DOCKER_HOST, but they can still reach the socket file.

Caches

Each cache path becomes a named volume, hearthforge-ci-cache-<hash>, labelled com.hearthforge.repo=<repository>. The volume is mounted at the same path in every run of that repository.

A cache without max_size grows without bound.
A cache with max_size is deleted after the run that takes it over the limit. The run detail page shows a cache step naming the dropped volume. The next run starts cold. Size the cap at about twice the working set.
A volume the current config no longer lists is deleted after a run on the default branch. Other branches keep it.
Purge caches on the Pipelines tab deletes all volumes of the repository. Deleting or renaming a repository does the same.
Everything else needs the engine's own tools:

docker volume ls --filter label=com.hearthforge.repo=REPO_NAME
docker volume rm $(docker volume ls -q --filter label=com.hearthforge.repo)   # all Hearthforge caches

The size check reads the daemon's reported usage. A daemon that does not report volume sizes never triggers the cap.

Gotchas

clone_project_to must be absolute, and no cache path may lie inside it or above it. The checkout is extracted over that directory and a clear step deletes it. The config is rejected otherwise.
publish_zip drops symlinks. The zip format has no symlink entry, so the archive keeps regular files and directories only. Use publish_tar or publish_gzip when links matter.
Steps share one container. A file left behind by one step is visible to the next. Use clear on a step that must start from a clean tree. Do not rm -rf work_dir itself: the container's working directory is gone and every later step fails to start.
Logs are cut at 2 MB per step. Redirect verbose output to a file and publish the file instead.
A Hearthforge restart cancels running runs. Their containers are force-removed on startup. The runs show as cancelled and can be retried.
The container is on the engine's default network. It reaches the internet and any other container on that network. Set cpu_limit and memory_limit; there are no defaults.
Old runs are deleted. Only the newest CI_MAX_HISTORY runs per repository are kept, and their artifacts go with them. Attach anything permanent to a release.