add ci doc
ACI.md
@@ -0,0 +1,219 @@
# CI/CD Pipelines
Hearthforge runs pipelines in Docker or Podman containers. A pipeline is
described by a `.hearthforge-ci.toml` file at the root of the repository.
The **Pipelines** tab of a repository shows the run history, a short
reference, and a link to the full [template](public/assets/hearthforge-ci-template.toml).
## How a run works
1. A push, a tag, or the **Run pipeline** button creates a run. The config is
read from the pushed commit, so every branch can carry its own pipeline.
2. Hearthforge pulls `image`, creates the cache volumes, and starts one
container named `hearthforge-ci-<run id>` with `work_dir` as its working
directory. Files listed under `[[copy]]` are taken from their source images
first.
3. The commit is exported with `git archive` on the host and uploaded into
`clone_project_to`. The image needs no git. No `.git` directory reaches the
container.
4. Steps run in file order, in that one container, so files and installed
packages persist from step to step. Each step is one `docker exec` of
`shell` with `shell_setup` prepended.
5. Published files are copied out of the container into
`$DATA_DIR/ci/artifacts/<run id>/`. The container is removed. Cache volumes
stay.
Everything up to the first step is shown as the step `pipeline setup`.
## Setup
Hearthforge talks to the container engine over its Unix socket. It checks,
in order:
1. `CI_DOCKER_SOCKET`, when set
2. `/var/run/docker.sock`
3. `/run/podman/podman.sock`
4. `/run/user/<uid>/podman/podman.sock`
Without a socket the Pipelines tab still works, and every run ends in one
skipped `pipeline setup` step that names the problem.
### Host install
Docker needs no extra setup. Rootless Podman needs the API socket enabled:
```bash
systemctl --user enable --now podman.socket
```
### Hearthforge in a container
The socket must be mounted into the Hearthforge container. `compose.yml`
carries commented-out bindings for both engines. Mount the socket at
`/var/run/docker.sock` and the auto-detection finds it, or mount it elsewhere
and set `CI_DOCKER_SOCKET`.
Mounting the socket gives Hearthforge full control over the container engine.
That is equivalent to root on the host for Docker, and to the socket owner's
account for rootless Podman.
Cache volumes, image pulls, and the CI containers live on the engine side.
Nothing about them refers to paths inside the Hearthforge container, so a
containerised Hearthforge works the same as a host install.
### Server settings
| Variable | Default | Description |
|----------------------|-------------------|------------------------------------------------------|
| `CI_DOCKER_SOCKET` | _(auto-detected)_ | Path to the Docker/Podman socket |
| `CI_MAX_CONCURRENT` | `2` | Runs executed at once. Further runs wait in a queue. |
| `CI_MAX_HISTORY` | `50` | Runs kept per repository, artifacts included |
| `CI_DEFAULT_TIMEOUT` | `3600` | Step timeout in seconds when the config sets none |
## Example
```toml
image = "docker.io/debian:stable"
work_dir = "/ci/build"
clone_project_to = "/ci/build/project" # must be absolute
shell_setup = "set -euo pipefail"
timeout = 3600 # per-step default, in seconds
memory_limit = "2g"
cache = [
{ path = "/root/.cache/pip", max_size = "1g" },
]
[on]
push = ["main", "release/*"] # branch globs, or ["*"]
tag = true
[variables]
[variables.PYTHON]
default = "3.12"
description = "Interpreter version, overridable from the Run pipeline form"
[[steps]]
name = "setup"
run_sh = "apt-get update -qq && apt-get install -y --no-install-recommends python3 python3-pip"
timeout = 300
[[steps]]
name = "test"
run_sh = "cd project && python3 -m pytest"
[[steps]]
name = "package"
run_if = 'test -n "${CI_COMMIT_TAG}"'
run_sh = "cd project && python3 -m build"
publish_gzip = ["/ci/build/project/dist/"]
[[steps]]
name = "report"
always = true
run_sh = "cat /ci/build/project/.pytest_cache/v/cache/lastfailed || true"
```
The template in the Pipelines tab lists every key with a comment.
### Step options
| Key | Effect |
|----------------|------------------------------------------------------------------------------|
| `run_sh` | The command. Runs through `shell`, after `shell_setup`. |
| `timeout` | Seconds. Falls back to the top-level `timeout`, then `CI_DEFAULT_TIMEOUT`. |
| `run_if` | Shell expression. A non-zero exit skips the step. |
| `always` | Run even after an earlier step failed. |
| `warn_on_fail` | A failure marks the step `warning` and the run continues. |
| `clear` | Delete `clone_project_to` and extract a fresh checkout before the step. |
| `publish_*` | `publish_file`, `publish_tar`, `publish_gzip`, `publish_zstd`, `publish_zip` |
### Environment
Every step sees `CI=true` plus:
| Variable | Value |
|-----------------------|--------------------------------|
| `CI_PIPELINE_ID` | numeric run ID |
| `CI_REPO_NAME` | repository name |
| `CI_SERVER_URL` | Hearthforge `BASE_URL` |
| `CI_TRIGGER_SOURCE` | `push`, `tag`, or `manual` |
| `CI_COMMIT_SHA` | full commit hash |
| `CI_COMMIT_SHORT_SHA` | first 8 characters |
| `CI_COMMIT_BRANCH` | branch name, empty for tags |
| `CI_COMMIT_TAG` | tag name, empty for branches |
| `CI_COMMIT_REF_NAME` | branch or tag name |
`[variables]` defaults come next, then overrides from the **Run pipeline**
form, then secrets. A later source wins on a name clash.
### Secrets
Admins add secrets under **Repository settings → CI Secrets**. They are
injected as environment variables. Their values are replaced with `[MASKED]`
in step logs. The masking is a plain text match: a secret printed in encoded
or split form is not caught.
### Status badge
```
```
## Caches
Each `cache` path becomes a named volume, `hearthforge-ci-cache-<hash>`,
labelled `com.hearthforge.repo=<repository>`. The volume is mounted at the
same path in every run of that repository.
- A cache without `max_size` grows without bound.
- A cache with `max_size` is deleted after the run that takes it over the
limit. The run detail page shows a `cache` step naming the dropped volume.
The next run starts cold. Size the cap at about twice the working set.
- A volume the current config no longer lists is deleted after a run on the
default branch. Other branches keep it.
- **Purge caches** on the Pipelines tab deletes all volumes of the
repository. Deleting or renaming a repository does the same.
- Everything else needs the engine's own tools:
```bash
docker volume ls --filter label=com.hearthforge.repo=REPO_NAME
docker volume rm $(docker volume ls -q --filter label=com.hearthforge.repo) # all Hearthforge caches
```
The size check reads the daemon's reported usage. A daemon that does not
report volume sizes never triggers the cap.
## Gotchas
- **`clone_project_to` must be absolute**, and no `cache` path may lie inside
it or above it. The checkout is extracted over that directory and a `clear`
step deletes it. The config is rejected otherwise.
- **The image needs no git, and the checkout has no `.git`.** `git describe`
or a version from the log needs a step that installs git and fetches the
history itself, for example over the forge's HTTP clone URL. Pushing back
from a pipeline needs the admin's credentials, kept as a secret.
- **The image is pulled on every run.** Even with a local copy, the daemon
contacts the registry. There is no registry authentication, so private
images must already be present and pullable without login. Pin a tag; a
moving tag changes the toolchain under the pipeline.
- **Match the libc for `[[copy]]`.** A binary from an `-alpine` image is
musl and fails on a glibc image with an error that looks like a missing
file.
- **`publish_*` uses tools inside the image.** `publish_gzip`, `publish_tar`,
and `publish_zstd` need `tar` (plus `zstd`). `publish_zip` needs `zip`.
`publish_file` needs nothing.
- **Steps share one container.** A file left behind by one step is visible to
the next. Use `clear` on a step that must start from a clean tree. Do not
`rm -rf` `work_dir` itself: the container's working directory is gone and
every later step fails to start.
- **Logs are cut at 2 MB per step.** Redirect verbose output to a file and
publish the file instead.
- **A Hearthforge restart cancels running runs.** Their containers are
force-removed on startup. The runs show as `cancelled` and can be retried.
- **The container is on the engine's default network.** It reaches the
internet and any other container on that network. Set `cpu_limit` and
`memory_limit`; there are no defaults.
- **Old runs are deleted.** Only the newest `CI_MAX_HISTORY` runs per
repository are kept, and their artifacts go with them. Attach anything
permanent to a release.
MREADME.md
@@ -119,7 +119,7 @@ A `BASE_URL` with the `https://` scheme is what activates HTTPS hardening:
Hearthforge includes a built-in CI/CD system that runs pipelines in Docker or Podman containers,
configured via a `.hearthforge-ci.toml` file at the root of your repository. The Pipelines tab contains a small tutorial and
an example file.
an example file. Setup, the run model, caches, and known pitfalls are documented in [CI.md](CI.md).
## Development
Mcompose.yml
@@ -7,6 +7,10 @@ services:
- "2222:2222"
volumes:
- ./data:/data
# CI needs the container engine's socket, see CI.md. Mount one of:
# - /var/run/docker.sock:/var/run/docker.sock
# - ${XDG_RUNTIME_DIR}/podman/podman.sock:/var/run/docker.sock # rootless podman
# - /run/podman/podman.sock:/var/run/docker.sock # rootful podman
environment:
DATA_DIR: /data
PORT: 3000
@@ -14,9 +18,9 @@ services:
OWNER_DISPLAY_NAME: Admin
BASE_URL: "http://localhost:3000"
# ADMIN_PASSWORD: change-me
# CI_DOCKER_SOCKET: /var/run/docker.sock # only when mounted at another path
# SSH_DISABLED: 0
# SCANNED_REPO_PRIVATE: 1
# ARCHIVE_ZST_ENABLED: 0
# TRUSTED_PROXY: 1
# REGISTRATION_TYPE: enabled
# REGISTER_QUESTION: ""
Mpublic/assets/hearthforge-ci-template.toml
@@ -9,7 +9,7 @@ work_dir = "/ci/build"
clone_project_to = "/ci/build/project"
# shell = ["/bin/sh", "-c"]
shell_setup = "set -euo pipefail"
# timeout = 3600 # overall run timeout in seconds
# timeout = 3600 # default per-step timeout in seconds
# cpu_limit = 2.0 # CPU cores limit
# memory_limit = "2g" # memory limit (k/m/g suffix)
# Caches persist between runs. An entry is a bare path, or a table with a