Build container images in a KVM microVM instead of the engine socket
Remove the engine_socket step option and CI_ENGINE_SOCKET. A step no longer gets access to the host's container engine. Add the build_image step key. It copies the build context out of the CI container into a sibling "build VM container". That container gets /dev/kvm and nothing else: no bind mount, no engine socket, no privileges. It boots a QEMU microVM, and buildah builds the Containerfile inside it. Any valid Containerfile works, including multi-stage builds, heredocs and RUN --mount of type cache, secret and bind. The VM writes the image to a virtio-serial port. Hearthforge imports it into the built-in registry. It links only the blobs the manifest names, and it checks each one against its digest. - vm/ holds the build VM image. The binary embeds it and builds the image through the engine on first use, tagged by a hash of vm/. The step log shows the build output. CI_VM_IMAGE replaces it with a pulled image. - New settings: CI_VM_IMAGE, CI_VM_CPUS, CI_VM_MEMORY_MB, CI_VM_DISK_MB, CI_MAX_IMAGE_BYTES. - build_image keys: context, file, target, args, secrets, image, tags. Tags and args expand CI variables but never secrets. Secrets reach the build as RUN --mount=type=secret. - Registry: blob commit and manifest storage are split out of the push handlers, so a push and an import run the same checks. - Image path and tag validation moves to util, shared by the registry and the CI config.
MCI.md
@@ -79,8 +79,12 @@ containerised Hearthforge works the same as a host install.
| `CI_MAX_CONCURRENT` | `2` | Runs executed at once. Further runs wait in a queue. |
| `CI_MAX_HISTORY` | `50` | Runs kept per repository, artifacts included |
| `CI_DEFAULT_TIMEOUT` | `3600` | Step timeout in seconds when the config sets none |
| `CI_ENGINE_SOCKET` | `0` | Allow `engine_socket = true` steps |
| `CI_NETWORK` | _(engine default)_ | Network the CI container joins |
| `CI_VM_IMAGE` | _(built from `vm/`)_ | Image of the build VM container, pulled when missing |
| `CI_VM_CPUS` | `2` | vCPUs of a build VM |
| `CI_VM_MEMORY_MB` | `2048` | RAM of a build VM, at least 1024 |
| `CI_VM_DISK_MB` | `20480` | Scratch disk of a build VM, for layers and pulls |
| `CI_MAX_IMAGE_BYTES` | `4294967296` | Largest image a build may push (4 GiB) |
## Example
@@ -138,7 +142,7 @@ The template in the Pipelines tab lists every key with a comment.
| `always` | Run even after an earlier step failed. |
| `warn_on_fail` | A failure marks the step `warning` and the run continues. |
| `clear` | Delete `clone_project_to` and extract a fresh checkout before the step. |
| `engine_socket`| Give the step the Docker/Podman socket. Needs `CI_ENGINE_SOCKET=1`. |
| `build_image` | Build a Containerfile in a VM after `run_sh`, see below. |
| `publish_*` | `publish_file`, `publish_tar`, `publish_gzip`, `publish_zstd`, `publish_zip` |
### Environment
@@ -197,41 +201,92 @@ addresses.
## Building container images
Steps run in a plain container without an engine of their own. A step with
`engine_socket = true` gets the host engine's socket mounted at
`/run/hearthforge/engine.sock`, and `DOCKER_HOST` and `CONTAINER_HOST` point
at it. `docker build` and `podman build` then run on the host engine with the
full Dockerfile feature set and the engine's own layer cache.
A step with `build_image` builds a Containerfile and pushes the image to the
repository's namespace in the built-in registry. It runs after the step's
`run_sh`, so earlier steps can prepare the context. A step may have
`build_image` alone.
```toml
[[steps]]
name = "image"
engine_socket = true
run_sh = """
apt-get install -y -qq docker.io > /dev/null
echo "$REGISTRY_PASSWORD" | docker login "${CI_REGISTRY%%/*}" -u admin --password-stdin
docker build -t "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA" project
docker push "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA"
"""
[steps.build_image]
tags = ["$CI_COMMIT_SHORT_SHA", "latest"]
```
`REGISTRY_PASSWORD` is a CI secret holding the admin password.
The server must allow it with `CI_ENGINE_SOCKET=1`. Without that, a step
with `engine_socket = true` fails and the run stops.
The bind source is the socket path as Hearthforge sees it. When Hearthforge
runs in a container, mount the socket at the same path it has on the host,
for example `/run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock`.
The engine resolves the source on the host, so a different path inside the
container makes the step fail with a missing socket.
> **Warning:** a step with the socket controls the whole engine. It can start
> privileged containers on the host. That is the same access Hearthforge
> itself has, and only admins push CI configs, so the trust level does not
> change. The socket is mounted for the whole run once any step asks for it.
> Other steps do not get `DOCKER_HOST`, but they can still reach the socket
> file.
| Key | Default | Meaning |
|-----------|--------------------------------------|--------------------------------------------------------------|
| `context` | `clone_project_to` | Absolute directory in the CI container |
| `file` | `Containerfile`, then `Dockerfile` | Relative to `context` |
| `target` | last stage | Stage of a multi-stage build |
| `args` | none | Build args. `$VARS` are expanded |
| `secrets` | none | CI secret names, for `RUN --mount=type=secret,id=NAME` |
| `image` | none | Path below the repository: `<host>/<repo>/<image>` |
| `tags` | `["$CI_COMMIT_SHORT_SHA"]` | `$VARS` are expanded. A tag that expands to nothing is dropped |
`tags` and `args` expand the predefined variables and `[variables]`, but no
secrets. Tags are public, and an arg used in `RUN` is stored in the image
history. Pass a secret through `secrets` instead.
The step log lists every pushed reference and the manifest digest. Pull the
image with the admin password, see the registry section of the README.
### How a build runs
1. Hearthforge reads `context` out of the CI container.
2. It starts a build VM container from `CI_VM_IMAGE`. The container gets
`/dev/kvm`, its own memory and CPU limits, and `CI_NETWORK`. It gets no
bind mount, no engine socket and no privileges.
3. The container boots a QEMU microVM with KVM. The VM runs buildah, which
builds the context like `docker build`. Any valid Containerfile works:
multi-stage builds, heredocs, `RUN --mount` of type cache, secret and
bind, `HEALTHCHECK`, `ONBUILD`. The image is stored in Docker format with
one layer per instruction.
4. The VM writes the image to the container. Hearthforge checks every blob
against its digest and stores the image in the registry.
Each build starts cold. The VM keeps no layer cache and pulls its base
images again every time.
### Setup
The binary carries `vm/`. The first `build_image` step builds the image
`localhost/hearthforge-buildvm:<hash>` from it through the engine. That
takes a minute or two and needs internet access. The step log shows the
build output. Later runs reuse the image. A Hearthforge version that
changes `vm/` builds a new image under a new tag. Old tags stay until
`podman image prune` or `docker image prune -a` removes them.
`CI_VM_IMAGE` replaces the built image with your own, for example a pinned
image from a registry. Hearthforge pulls it when the engine lacks it.
The engine host needs `/dev/kvm`.
A cloud VM needs nested virtualization for that. There is no fallback
without KVM, because QEMU does not isolate a guest without it.
This works the same when Hearthforge itself runs in a container. The build
container is a sibling started through the engine socket. The Hearthforge
container needs no device and no extra mount.
Rootless Podman passes `/dev/kvm` when the user can open it. Hearthforge
asks crun to keep the user's supplementary groups, so membership in the
`kvm` group is enough. The build container always gets a CPU limit, so
rootless Podman needs the `cpu` cgroup controller delegated to the user.
### Isolation
| Layer | Stops |
|-------|-------|
| buildah's `RUN` container in the VM | normal build code |
| KVM | everything in the guest, its root and its kernel included |
| QEMU `-sandbox on` (seccomp) | most exploits of the emulated devices |
| The build container | a QEMU escape. The container holds no data and no socket |
| Disk sizes and `CI_MAX_IMAGE_BYTES` | filling the engine's disk |
The guest reaches the network through QEMU user networking inside the
build container. Its address for the host, `10.0.2.2`, is the container's
own loopback, where nothing listens. The guest still reaches everything the
container reaches, including host services on the bridge gateway. Block
those with a host firewall rule, as for any CI container.
## Caches
MREADME.md
@@ -102,8 +102,12 @@ All settings are environment variables:
| `CI_MAX_ARTIFACT_BYTES` | `536870912` | Max size of one published CI artifact (512 MiB) |
| `CI_DEFAULT_TIMEOUT` | `3600` | Default step timeout in seconds |
| `CI_MAX_CONCURRENT` | `2` | Advisory max concurrent runs |
| `CI_ENGINE_SOCKET` | `0` | Allow CI steps with `engine_socket = true` to use the engine socket |
| `CI_NETWORK` | _(engine default)_ | Engine network for CI containers, e.g. one created with IPv6 |
| `CI_VM_IMAGE` | _(built from `vm/`)_ | Image of the `build_image` VM container |
| `CI_VM_CPUS` | `2` | vCPUs of a build VM |
| `CI_VM_MEMORY_MB` | `2048` | RAM of a build VM, at least 1024 |
| `CI_VM_DISK_MB` | `20480` | Scratch disk of a build VM |
| `CI_MAX_IMAGE_BYTES` | `4294967296` | Largest image a `build_image` step may push (4 GiB) |
| `REGISTRY_PULL` | `admin` | Who may pull container images: `admin`, `users`, or `public` |
\* Set `TRUSTED_PROXY=1` only when Hearthforge is behind a reverse proxy that overwrites any incoming `X-Forwarded-For` from clients with the real client address. Without the header, the socket address is used. Caddy and Traefik do this by default; nginx requires `proxy_set_header X-Forwarded-For $remote_addr;` (rather than the common `$proxy_add_x_forwarded_for`, which appends to a client-supplied value). Setting `TRUSTED_PROXY=1` in front of a proxy that does not strip means rate limits and any audit logging are spoofable per request. Rate limits key IPv6 clients by their /64 prefix.
@@ -146,7 +150,7 @@ existing repository.
- `BASE_URL` must be HTTPS for Docker and Podman to talk to the registry
without an insecure-registry exception.
CI steps can build and push images through the engine socket, see
CI steps build images in a KVM microVM and push them here, see
[CI.md](CI.md#building-container-images).
## Development
Massets.go
@@ -1,4 +1,5 @@
// Package hearthforge embeds the static web assets into the binary.
// Package hearthforge embeds the static web assets and the build VM sources
// into the binary.
package hearthforge
import "embed"
@@ -7,3 +8,8 @@ import "embed"
//
//go:embed all:web/static
var StaticFS embed.FS
// VMFS holds vm/, the build context of the build VM image.
//
//go:embed vm
var VMFS embed.FS
Mcmd/hearthforge/main.go
@@ -108,6 +108,7 @@ func main() {
Git: git,
Patches: gitcmd.NewPatchCache(),
}
runner.ImportImage = srv.ImportImage
if err := srv.SyncRepos(ctx); err != nil {
log.Fatalf("repo sync: %v", err)
}
Ainternal/ci/build.go
@@ -0,0 +1,315 @@
package ci
import (
"archive/tar"
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"maps"
"net/http"
"net/url"
"os"
"path"
"slices"
"strconv"
"strings"
"time"
"hearthforge/internal/util"
)
// Exit codes of vm/run-vm.
const (
vmExitBuildFailed = 1
vmExitTooLarge = 3
)
// vmOverheadMB is the container memory on top of the VM's RAM, for QEMU
// itself and the job tar.
const vmOverheadMB = 512
func buildContainerName(runID int64) string {
return fmt.Sprintf("hearthforge-ci-%d-build", runID)
}
// buildImage runs one build_image step in a build VM container, a sibling
// of the CI container that boots QEMU from vm/. It returns the step log.
// vars expands tags and args. It holds no secrets, because tags are public
// and args end up in the image history.
func (r *Runner) buildImage(ctx context.Context, runID int64, ciContainerID, repoName string, cfg *Config,
spec *BuildImage, vars map[string]string, secrets []secret, onPartial func(string),
) (string, error) {
if r.ImportImage == nil {
return "", errors.New("no image store is configured")
}
tags, err := expandTags(spec.Tags, vars)
if err != nil {
return "", err
}
args := map[string]string{}
for name, v := range spec.Args {
args[name] = os.Expand(v, func(k string) string { return vars[k] })
}
secretFiles := map[string]string{}
for _, name := range spec.Secrets {
i := slices.IndexFunc(secrets, func(s secret) bool { return s.name == name })
if i < 0 {
return "", fmt.Errorf("secret %s is not set for this repository", name)
}
secretFiles[name] = secrets[i].value
}
contextPath := spec.Context
if contextPath == "" {
contextPath = cfg.CloneProjectTo
}
// The VM image build can take minutes, so its output goes to the step
// log as it arrives.
var setup logBuffer
var lastFlush time.Time
say := func(text string) {
setup.append(text)
if time.Since(lastFlush) >= 2*time.Second {
onPartial(setup.String())
lastFlush = time.Now()
}
}
vmImage, err := r.prepareVMImage(ctx, say)
if err != nil {
return setup.String(), err
}
id, err := r.createBuildContainer(ctx, runID, vmImage)
if err != nil {
return setup.String(), err
}
defer r.removeContainer(ctx, id)
if err := r.uploadBuildJob(ctx, id, ciContainerID, contextPath, spec, args, secretFiles); err != nil {
return setup.String(), err
}
if err := r.startContainer(ctx, id); err != nil {
return setup.String(), err
}
logResp, err := r.do(ctx, http.MethodGet, "/containers/"+id+"/logs?follow=1&stdout=1&stderr=1", nil, "")
if err != nil {
return setup.String(), err
}
if logResp.StatusCode >= 300 {
discard(logResp)
return setup.String(), fmt.Errorf("cannot read the build VM log: HTTP %d", logResp.StatusCode)
}
buildLog := setup.String() + readMuxFrames(logResp.Body, func(partial string) {
onPartial(setup.String() + partial)
})
discard(logResp)
if ctx.Err() != nil {
return buildLog, ctx.Err()
}
code, err := r.waitContainer(ctx, id)
if err != nil {
return buildLog, err
}
switch code {
case 0:
case vmExitBuildFailed:
return buildLog, errors.New("the build failed")
case vmExitTooLarge:
return buildLog, fmt.Errorf("the image is larger than CI_MAX_IMAGE_BYTES (%s)", formatBytes(r.cfg.CIMaxImageBytes))
default:
return buildLog, fmt.Errorf("the build VM did not run (exit code %d)", code)
}
digest, err := r.importBuiltImage(ctx, id, repoName, spec.Image, tags)
if err != nil {
return buildLog, err
}
ref := vars["CI_REGISTRY"]
if spec.Image != "" {
ref += "/" + spec.Image
}
for _, tag := range tags {
buildLog += fmt.Sprintf("Pushed %s:%s\n", ref, tag)
}
return buildLog + "Digest " + digest + "\n", nil
}
// expandTags expands each tag and drops the empty results, so a tag like
// $CI_COMMIT_TAG can sit in a config that also runs on branches.
func expandTags(tags []string, vars map[string]string) ([]string, error) {
if len(tags) == 0 {
tags = []string{"$CI_COMMIT_SHORT_SHA"}
}
var out []string
for _, t := range tags {
v := os.Expand(t, func(k string) string { return vars[k] })
if v == "" || slices.Contains(out, v) {
continue
}
if !util.ValidImageTag(v) {
return nil, fmt.Errorf("tag %q expands to %q, which is not a valid image tag", t, v)
}
out = append(out, v)
}
if len(out) == 0 {
return nil, errors.New("every tag expanded to an empty string")
}
return out, nil
}
func (r *Runner) createBuildContainer(ctx context.Context, runID int64, image string) (string, error) {
name := buildContainerName(runID)
// A retry reuses the name, see createContainer.
r.removeContainer(ctx, name)
hostConfig := map[string]any{
"Devices": []map[string]string{
{"PathOnHost": "/dev/kvm", "PathInContainer": "/dev/kvm", "CgroupPermissions": "rwm"},
},
// Rootless Podman drops supplementary groups, and /dev/kvm is often
// open to the kvm group only. crun reads this annotation, other
// runtimes ignore it.
"Annotations": map[string]string{"run.oci.keep_original_groups": "1"},
"Memory": (r.cfg.CIVMMemoryMB + vmOverheadMB) << 20,
"NanoCpus": int64(r.cfg.CIVMCPUs) * 1e9,
}
if r.cfg.CINetwork != "" {
hostConfig["NetworkMode"] = r.cfg.CINetwork
}
resp, body, err := r.doJSON(ctx, http.MethodPost, "/containers/create?name="+name, map[string]any{
"Image": image,
"Env": []string{
"HF_VM_CPUS=" + strconv.Itoa(r.cfg.CIVMCPUs),
"HF_VM_MEMORY_MB=" + strconv.FormatInt(r.cfg.CIVMMemoryMB, 10),
"HF_VM_DISK_BYTES=" + strconv.FormatInt(r.cfg.CIVMDiskMB<<20, 10),
"HF_VM_MAX_IMAGE_BYTES=" + strconv.FormatInt(r.cfg.CIMaxImageBytes, 10),
},
"HostConfig": hostConfig,
"Labels": ciContainerLabels,
})
if err != nil {
return "", err
}
if resp.StatusCode >= 300 {
return "", fmt.Errorf("failed to create the build VM container: %d %s",
resp.StatusCode, strings.TrimSpace(string(body)))
}
var data struct{ Id string }
if err := json.Unmarshal(body, &data); err != nil {
return "", err
}
return data.Id, nil
}
// uploadBuildJob fills /in of the build container: the context under
// /in/context/<its directory name>, and the build settings as one file each
// under /in/job. Files keep arbitrary values safe from shell quoting in the
// guest.
func (r *Runner) uploadBuildJob(ctx context.Context, buildID, ciContainerID, contextPath string,
spec *BuildImage, args, secrets map[string]string,
) error {
resp, err := r.do(ctx, http.MethodGet,
"/containers/"+ciContainerID+"/archive?path="+url.QueryEscape(path.Clean(contextPath)), nil, "")
if err != nil {
return err
}
defer discard(resp)
if resp.StatusCode >= 300 {
return fmt.Errorf("cannot read the context %s: HTTP %d", contextPath, resp.StatusCode)
}
if err := r.putBuildArchive(ctx, buildID, "/in/context", resp.Body); err != nil {
return err
}
files := map[string]string{}
if spec.File != "" {
files["job/file"] = spec.File
}
if spec.Target != "" {
files["job/target"] = spec.Target
}
for name, v := range args {
files["job/args/"+name] = v
}
for name, v := range secrets {
files["job/secrets/"+name] = v
}
var job bytes.Buffer
tw := tar.NewWriter(&job)
for _, dir := range []string{"job/", "job/args/", "job/secrets/"} {
if err := tw.WriteHeader(&tar.Header{Name: dir, Mode: 0o700, Typeflag: tar.TypeDir}); err != nil {
return err
}
}
for _, name := range slices.Sorted(maps.Keys(files)) {
hdr := &tar.Header{Name: name, Mode: 0o600, Size: int64(len(files[name])), Typeflag: tar.TypeReg}
if err := tw.WriteHeader(hdr); err != nil {
return err
}
if _, err := io.WriteString(tw, files[name]); err != nil {
return err
}
}
if err := tw.Close(); err != nil {
return err
}
return r.putBuildArchive(ctx, buildID, "/in", &job)
}
func (r *Runner) putBuildArchive(ctx context.Context, buildID, dest string, body io.Reader) error {
resp, data, err := r.putArchive(ctx, buildID, dest, body)
if err != nil {
return err
}
if resp.StatusCode >= 300 {
return fmt.Errorf("failed to upload the build job to %s: HTTP %d %s",
dest, resp.StatusCode, strings.TrimSpace(string(data)))
}
return nil
}
// waitContainer waits for the container to stop and returns its exit code.
func (r *Runner) waitContainer(ctx context.Context, id string) (int, error) {
resp, body, err := r.doJSON(ctx, http.MethodPost, "/containers/"+id+"/wait", nil)
if err != nil {
return 0, err
}
if resp.StatusCode >= 300 {
return 0, fmt.Errorf("wait for the build VM container: HTTP %d %s",
resp.StatusCode, strings.TrimSpace(string(body)))
}
var data struct{ StatusCode int }
if err := json.Unmarshal(body, &data); err != nil {
return 0, err
}
return data.StatusCode, nil
}
// importBuiltImage streams /out/image.tar out of the stopped build
// container into the registry. The archive endpoint wraps it in a tar of
// its own.
func (r *Runner) importBuiltImage(ctx context.Context, buildID, repoName, image string, tags []string) (string, error) {
resp, err := r.do(ctx, http.MethodGet, "/containers/"+buildID+"/archive?path=/out/image.tar", nil, "")
if err != nil {
return "", err
}
defer discard(resp)
if resp.StatusCode >= 300 {
return "", fmt.Errorf("cannot read the built image: HTTP %d", resp.StatusCode)
}
tr := tar.NewReader(resp.Body)
hdr, err := tr.Next()
if err != nil {
return "", fmt.Errorf("cannot read the built image: %w", err)
}
if hdr.Typeflag != tar.TypeReg {
return "", errors.New("the built image is not a file")
}
// run-vm enforces the limit too. This check does not trust it.
if hdr.Size > r.cfg.CIMaxImageBytes {
return "", fmt.Errorf("the image is larger than CI_MAX_IMAGE_BYTES (%s)", formatBytes(r.cfg.CIMaxImageBytes))
}
return r.ImportImage(ctx, repoName, image, tags, tr)
}
Minternal/ci/config.go
@@ -10,11 +10,12 @@ import (
"path"
"path/filepath"
"regexp"
"slices"
"strconv"
"strings"
"github.com/BurntSushi/toml"
"hearthforge/internal/util"
)
// VariableDef is one entry of the [variables] table. The manual run form
@@ -26,21 +27,35 @@ type VariableDef struct {
// Step is one [[steps]] entry.
type Step struct {
Name string `toml:"name"`
RunSh string `toml:"run_sh"`
RunIf string `toml:"run_if"`
Always bool `toml:"always"`
WarnOnFail bool `toml:"warn_on_fail"`
Clear bool `toml:"clear"`
// EngineSocket mounts the Docker/Podman socket and points DOCKER_HOST
// at it. Needs CI_ENGINE_SOCKET on the server.
EngineSocket bool `toml:"engine_socket"`
Timeout int `toml:"timeout"`
PublishFile StringList `toml:"publish_file"`
PublishTar StringList `toml:"publish_tar"`
PublishGzip StringList `toml:"publish_gzip"`
PublishZip StringList `toml:"publish_zip"`
PublishZstd StringList `toml:"publish_zstd"`
Name string `toml:"name"`
RunSh string `toml:"run_sh"`
RunIf string `toml:"run_if"`
Always bool `toml:"always"`
WarnOnFail bool `toml:"warn_on_fail"`
Clear bool `toml:"clear"`
Timeout int `toml:"timeout"`
BuildImage *BuildImage `toml:"build_image"`
PublishFile StringList `toml:"publish_file"`
PublishTar StringList `toml:"publish_tar"`
PublishGzip StringList `toml:"publish_gzip"`
PublishZip StringList `toml:"publish_zip"`
PublishZstd StringList `toml:"publish_zstd"`
}
// BuildImage builds a Containerfile in a build VM and pushes the result to
// the repository's registry namespace.
type BuildImage struct {
// Context is an absolute path in the CI container. Empty means
// clone_project_to.
Context string `toml:"context"`
// File is relative to Context. Empty lets buildah pick Containerfile or
// Dockerfile.
File string `toml:"file"`
Target string `toml:"target"`
Args map[string]string `toml:"args"`
Secrets []string `toml:"secrets"`
Image string `toml:"image"`
Tags []string `toml:"tags"`
}
// StringList accepts either a bare string or an array of strings.
@@ -149,11 +164,6 @@ func (p *PushSpec) UnmarshalTOML(v any) error {
return errors.New("on.push must be a boolean or an array of strings")
}
// WantsEngineSocket reports whether any step asked for the engine socket.
func (c *Config) WantsEngineSocket() bool {
return slices.ContainsFunc(c.Steps, func(s Step) bool { return s.EngineSocket })
}
// Config is a parsed .hearthforge-ci.toml.
type Config struct {
Image string `toml:"image"`
@@ -232,6 +242,14 @@ func ValidateCiConfig(cfg *Config) string {
)
}
for _, step := range cfg.Steps {
if step.BuildImage != nil {
if msg := validateBuildImage(cfg, step.BuildImage); msg != "" {
return fmt.Sprintf("step %q: build_image %s", step.Name, msg)
}
}
}
if cfg.CloneProjectTo == "" {
return ""
}
@@ -262,6 +280,43 @@ func ValidateCiConfig(cfg *Config) string {
return ""
}
var (
// envNameRe also keeps names safe as file names in the job tar.
envNameRe = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
buildTargetRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`)
)
// validateBuildImage checks the parts of build_image that do not depend on
// the run. Tags are checked after expansion, at run time.
func validateBuildImage(cfg *Config, b *BuildImage) string {
if b.Context == "" && cfg.CloneProjectTo == "" {
return "needs context when clone_project_to is not set"
}
if b.Context != "" && !path.IsAbs(b.Context) {
return fmt.Sprintf("context %q must be an absolute path", b.Context)
}
if b.File != "" && !filepath.IsLocal(b.File) {
return fmt.Sprintf("file %q must be a relative path inside the context", b.File)
}
if b.Target != "" && !buildTargetRe.MatchString(b.Target) {
return fmt.Sprintf("target %q is not a stage name", b.Target)
}
for name := range b.Args {
if !envNameRe.MatchString(name) {
return fmt.Sprintf("arg %q is not a valid name", name)
}
}
for _, name := range b.Secrets {
if !envNameRe.MatchString(name) {
return fmt.Sprintf("secret %q is not a valid name", name)
}
}
if !util.ValidImagePath(b.Image) {
return fmt.Sprintf("image %q must be lowercase path segments", b.Image)
}
return ""
}
// shouldTriggerPush reports whether a push to branch starts a run.
func shouldTriggerPush(cfg *Config, branch string) bool {
if cfg.On.Push.All {
Minternal/ci/config_test.go
@@ -339,25 +339,62 @@ func TestLogBufferTruncates(t *testing.T) {
}
}
func TestParseEngineSocket(t *testing.T) {
func TestParseBuildImage(t *testing.T) {
cfg := mustParse(t, `
image = "debian:latest"
[[steps]]
name = "test"
run_sh = "true"
clone_project_to = "/ci/project"
[[steps]]
name = "image"
engine_socket = true
run_sh = "docker build ."
build_image = { file = "deploy/Containerfile", target = "app", image = "web", tags = ["$CI_COMMIT_SHORT_SHA", "latest"], args = { VERSION = "$CI_COMMIT_TAG" }, secrets = ["NPM_TOKEN"] }
`)
if cfg.Steps[0].EngineSocket || !cfg.Steps[1].EngineSocket {
t.Fatalf("engine_socket parsed wrong: %+v", cfg.Steps)
want := &BuildImage{
File: "deploy/Containerfile", Target: "app", Image: "web",
Tags: []string{"$CI_COMMIT_SHORT_SHA", "latest"},
Args: map[string]string{"VERSION": "$CI_COMMIT_TAG"},
Secrets: []string{"NPM_TOKEN"},
}
if !reflect.DeepEqual(cfg.Steps[0].BuildImage, want) {
t.Fatalf("build_image = %+v", cfg.Steps[0].BuildImage)
}
if got := ValidateCiConfig(cfg); got != "" {
t.Fatalf("ValidateCiConfig = %q", got)
}
}
func TestValidateBuildImage(t *testing.T) {
bad := map[string]string{
"no context": `{}`,
"relative context": `{ context = "project" }`,
"absolute file": `{ context = "/p", file = "/etc/Containerfile" }`,
"file outside": `{ context = "/p", file = "../Containerfile" }`,
"bad target": `{ context = "/p", target = "-x" }`,
"bad arg": `{ context = "/p", args = { "A-B" = "x" } }`,
"bad secret": `{ context = "/p", secrets = ["../x"] }`,
"upper image": `{ context = "/p", image = "Web" }`,
"reserved image": `{ context = "/p", image = "a/blobs" }`,
}
for name, spec := range bad {
cfg := mustParse(t, "image = \"debian\"\n[[steps]]\nname = \"i\"\nbuild_image = "+spec)
if ValidateCiConfig(cfg) == "" {
t.Errorf("%s: accepted %s", name, spec)
}
}
}
func TestExpandTags(t *testing.T) {
vars := map[string]string{"CI_COMMIT_SHORT_SHA": "abc12345", "CI_COMMIT_TAG": ""}
got, err := expandTags(nil, vars)
if err != nil || !reflect.DeepEqual(got, []string{"abc12345"}) {
t.Errorf("default tags = %v, %v", got, err)
}
got, err = expandTags([]string{"$CI_COMMIT_TAG", "latest", "latest"}, vars)
if err != nil || !reflect.DeepEqual(got, []string{"latest"}) {
t.Errorf("tags = %v, %v", got, err)
}
if !cfg.WantsEngineSocket() {
t.Error("WantsEngineSocket = false")
if _, err := expandTags([]string{"$CI_COMMIT_TAG"}, vars); err == nil {
t.Error("all-empty tags accepted")
}
cfg.Steps[1].EngineSocket = false
if cfg.WantsEngineSocket() {
t.Error("WantsEngineSocket = true with no step asking")
if _, err := expandTags([]string{"a b"}, vars); err == nil {
t.Error("invalid tag accepted")
}
}
Minternal/ci/docker.go
@@ -306,24 +306,10 @@ func (r *Runner) enforceCacheLimits(ctx context.Context, repoName string, cache
// --- Containers ---
// engineSocketInContainer is where an engine_socket step finds the socket.
const engineSocketInContainer = "/run/hearthforge/engine.sock"
func (r *Runner) createContainer(ctx context.Context, runID int64, repoName string, cfg *Config, envVars []string) (string, error) {
// No bind for the repo: the daemon resolves bind sources on the host,
// where HearthForge's own paths need not exist. uploadCheckout copies it in.
binds := []string{}
// A step that asked for the socket on a server that forbids it fails in
// the step loop instead.
if r.cfg.CIEngineSocket && cfg.WantsEngineSocket() {
// The socket is a host path the engine itself listens on, so the
// engine can resolve it even when Hearthforge runs in a container.
sock, err := r.socketPath()
if err != nil {
return "", err
}
binds = append(binds, sock+":"+engineSocketInContainer)
}
for _, c := range cfg.Cache {
volName := cacheVolumeName(repoName, c.Path)
if err := r.ensureVolume(ctx, volName, repoName, c.Path); err != nil {
Minternal/ci/execute.go
@@ -5,7 +5,6 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"time"
@@ -166,24 +165,6 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
}
stepID := stepIDs[i]
if step.EngineSocket && !r.cfg.CIEngineSocket {
r.execSQL(ctx,
`UPDATE ci_steps SET status = 'failure', started_at = ?, finished_at = ?, log = ? WHERE id = ?`,
db.NowISO(), db.NowISO(),
"engine_socket is disabled on this server. Set CI_ENGINE_SOCKET=1 to allow it.\n", stepID)
runFailed = true
continue
}
// The socket file is visible to every step once mounted. Only
// steps that asked for it get the client pointed at it.
stepEnv := envArray
if step.EngineSocket {
stepEnv = slices.Concat(envArray, []string{
"DOCKER_HOST=unix://" + engineSocketInContainer,
"CONTAINER_HOST=unix://" + engineSocketInContainer,
})
}
timeout := step.Timeout
if timeout == 0 {
timeout = cfg.Timeout
@@ -195,7 +176,7 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
if step.RunIf != "" {
condCtx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
res, err := r.exec(condCtx, containerID, append(append([]string{}, shell...), step.RunIf),
cfg.WorkDir, stepEnv, nil)
cfg.WorkDir, envArray, nil)
cancel()
if ctx.Err() != nil {
return ctx.Err()
@@ -269,7 +250,7 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
}
stepCtx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
res, execErr := r.exec(stepCtx, containerID,
append(append([]string{}, shell...), command), cfg.WorkDir, stepEnv,
append(append([]string{}, shell...), command), cfg.WorkDir, envArray,
func(partial string) {
r.execSQL(ctx, `UPDATE ci_steps SET log = ? WHERE id = ?`,
maskLog(partial, secretValues, true), stepID)
@@ -304,6 +285,30 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
}
}
if step.BuildImage != nil && stepStatus == "success" {
prefix := stepLog
buildCtx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
buildLog, buildErr := r.buildImage(buildCtx, runID, containerID, repoName, cfg, step.BuildImage,
buildVars(envArray, secrets), secrets,
func(partial string) {
r.execSQL(ctx, `UPDATE ci_steps SET log = ? WHERE id = ?`,
maskLog(prefix+partial, secretValues, true), stepID)
})
timedOut := buildCtx.Err() == context.DeadlineExceeded
cancel()
stepLog = maskLog(prefix+buildLog, secretValues, false)
switch {
case ctx.Err() != nil:
return ctx.Err()
case timedOut:
stepStatus = "failure"
stepLog += fmt.Sprintf("Step timed out after %ds\n", timeout)
case buildErr != nil:
stepStatus = onFail
stepLog += fmt.Sprintf("Image build failed: %s\n", buildErr)
}
}
if stepStatus == "failure" {
runFailed = true
}
Minternal/ci/run.go
@@ -6,6 +6,7 @@ import (
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
@@ -49,6 +50,13 @@ type Runner struct {
cfg *config.Config
db *db.DB
// ImportImage stores an image from a build VM in the registry and
// returns its manifest digest. The web server provides it.
ImportImage func(ctx context.Context, repoName, image string, tags []string, src io.Reader) (string, error)
// vmImageSlot serialises builds of the default build VM image.
vmImageSlot chan struct{}
socketMu sync.Mutex
socket string
client *http.Client
@@ -62,7 +70,7 @@ type Runner struct {
}
func New(cfg *config.Config, database *db.DB) *Runner {
return &Runner{cfg: cfg, db: database, running: map[int64]*task{}}
return &Runner{cfg: cfg, db: database, running: map[int64]*task{}, vmImageSlot: make(chan struct{}, 1)}
}
func (r *Runner) repoPath(name string) string {
@@ -493,6 +501,20 @@ func buildEnvVars(runID int64, repoName, baseURL, host string, run runRow, cfg *
return envArray, secretValues
}
// buildVars maps the run environment without the secrets, for expanding
// build_image tags and args.
func buildVars(envArray []string, secrets []secret) map[string]string {
vars := map[string]string{}
for _, kv := range envArray {
k, v, _ := strings.Cut(kv, "=")
vars[k] = v
}
for _, s := range secrets {
delete(vars, s.name)
}
return vars
}
// maskSecrets replaces secret values in a log. It is a plain text match: a
// secret printed in encoded or split form is not caught.
func maskSecrets(text string, secrets []string) string {
Ainternal/ci/vmimage.go
@@ -0,0 +1,139 @@
package ci
import (
"archive/tar"
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"net/url"
"strings"
"sync"
"hearthforge"
)
// vmContext is the embedded vm/ directory as a tar, the build context of
// the default build VM image.
var vmContext = sync.OnceValues(func() ([]byte, error) {
sub, err := fs.Sub(hearthforge.VMFS, "vm")
if err != nil {
return nil, err
}
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
if err := tw.AddFS(sub); err != nil {
return nil, err
}
if err := tw.Close(); err != nil {
return nil, err
}
return buf.Bytes(), nil
})
// DefaultVMImage names the default build VM image after its sources, so an
// upgrade that changes vm/ builds a new image instead of reusing a stale one.
func DefaultVMImage() (string, error) {
data, err := vmContext()
if err != nil {
return "", err
}
sum := sha256.Sum256(data)
return "localhost/hearthforge-buildvm:" + hex.EncodeToString(sum[:])[:12], nil
}
// prepareVMImage returns the build VM image and makes sure the engine has
// it. A missing CI_VM_IMAGE is pulled. The default image is built from the
// embedded vm/ once per Hearthforge version. say receives progress for the
// step log.
func (r *Runner) prepareVMImage(ctx context.Context, say func(string)) (string, error) {
if image := r.cfg.CIVMImage; image != "" {
found, err := r.hasImage(ctx, image)
if err != nil || found {
return image, err
}
say("Pulling the build VM image " + image + "\n")
return image, r.pullImage(ctx, image)
}
image, err := DefaultVMImage()
if err != nil {
return "", err
}
// Concurrent runs would each start the same slow build. A channel, not a
// mutex, so a cancelled run stops waiting.
select {
case r.vmImageSlot <- struct{}{}:
default:
say("Waiting for another run to build the build VM image\n")
select {
case r.vmImageSlot <- struct{}{}:
case <-ctx.Done():
return "", ctx.Err()
}
}
defer func() { <-r.vmImageSlot }()
found, err := r.hasImage(ctx, image)
if err != nil || found {
return image, err
}
say("Building the build VM image " + image + ". This happens once per Hearthforge version.\n")
if err := r.buildVMImage(ctx, image, say); err != nil {
return "", fmt.Errorf("cannot build the build VM image: %w", err)
}
say("Built the build VM image " + image + "\n")
return image, nil
}
func (r *Runner) hasImage(ctx context.Context, image string) (bool, error) {
resp, _, err := r.doJSON(ctx, http.MethodGet, "/images/"+image+"/json", nil)
if err != nil {
return false, err
}
return resp.StatusCode == http.StatusOK, nil
}
// buildVMImage builds the embedded vm/ with the engine's classic build API.
// Docker and Podman both serve it without a BuildKit session.
func (r *Runner) buildVMImage(ctx context.Context, image string, say func(string)) error {
body, err := vmContext()
if err != nil {
return err
}
resp, err := r.do(ctx, http.MethodPost,
"/build?dockerfile=Containerfile&rm=1&forcerm=1&t="+url.QueryEscape(image),
bytes.NewReader(body), "application/x-tar")
if err != nil {
return err
}
defer discard(resp)
if resp.StatusCode >= 300 {
detail, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
return fmt.Errorf("HTTP %d %s", resp.StatusCode, strings.TrimSpace(string(detail)))
}
// A failed build still answers 200. The error arrives in the stream.
dec := json.NewDecoder(resp.Body)
for {
var msg struct {
Stream string `json:"stream"`
Error string `json:"error"`
}
err := dec.Decode(&msg)
if errors.Is(err, io.EOF) {
return nil
}
if err != nil {
return err
}
if msg.Error != "" {
return errors.New(strings.TrimSpace(msg.Error))
}
say(msg.Stream)
}
}
Minternal/config/config.go
@@ -45,8 +45,12 @@ type Config struct {
CIDefaultTimeout int
CIMaxConcurrent int
CIMaxArtifactBytes int64
CIEngineSocket bool
CINetwork string // engine network for CI containers, empty = engine default
CIVMImage string // empty = built from the embedded vm/
CIVMCPUs int
CIVMMemoryMB int64
CIVMDiskMB int64
CIMaxImageBytes int64
RegistryPull string // admin | users | public
MaxConcurrentArchives int
}
@@ -124,8 +128,12 @@ func Load() (*Config, error) {
CIDefaultTimeout: int(intEnv("CI_DEFAULT_TIMEOUT", 3600, 1)),
CIMaxConcurrent: int(intEnv("CI_MAX_CONCURRENT", 2, 1)),
CIMaxArtifactBytes: intEnv("CI_MAX_ARTIFACT_BYTES", 512<<20, 1),
CIEngineSocket: boolEnv("CI_ENGINE_SOCKET"),
CINetwork: os.Getenv("CI_NETWORK"),
CIVMImage: os.Getenv("CI_VM_IMAGE"),
CIVMCPUs: int(intEnv("CI_VM_CPUS", 2, 1)),
CIVMMemoryMB: intEnv("CI_VM_MEMORY_MB", 2048, 1024),
CIVMDiskMB: intEnv("CI_VM_DISK_MB", 20480, 1024),
CIMaxImageBytes: intEnv("CI_MAX_IMAGE_BYTES", 4<<30, 1),
RegistryPull: strEnv("REGISTRY_PULL", "admin"),
MaxConcurrentArchives: int(intEnv("MAX_CONCURRENT_ARCHIVE_JOBS", 2, 1)),
}
Minternal/util/util.go
@@ -174,3 +174,26 @@ func DisplayName(username, ownerDisplayName string) string {
var validUsername = regexp.MustCompile(`^[a-zA-Z0-9_-]+$`)
func ValidUsername(s string) bool { return validUsername.MatchString(s) }
var (
imageTagRe = regexp.MustCompile(`^[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}$`)
imagePathSegRe = regexp.MustCompile(`^[a-z0-9]+(?:[._-][a-z0-9]+)*$`)
)
// ValidImageTag reports whether tag is a valid container image tag.
func ValidImageTag(tag string) bool { return imageTagRe.MatchString(tag) }
// ValidImagePath reports whether image is a valid image path below a
// repository: empty, or lowercase segments joined by "/".
func ValidImagePath(image string) bool {
if image == "" {
return true
}
for _, seg := range strings.Split(image, "/") {
// The registry splits request paths on these words.
if !imagePathSegRe.MatchString(seg) || seg == "blobs" || seg == "manifests" || seg == "tags" {
return false
}
}
return true
}
Ainternal/web/e2e/ci_build_test.go
@@ -0,0 +1,391 @@
package e2e
import (
"archive/tar"
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"net/http"
"net/url"
"strconv"
"strings"
"testing"
"time"
"hearthforge/internal/ci"
)
// build_image runs in a build VM container. The mock engine plays that
// container: it prints a log, exits with a code, and serves the image tar.
const ciBuildTOML = `
image = "debian:latest"
clone_project_to = "/ci/project"
[on]
manual = true
[[steps]]
name = "image"
build_image = { image = "web", tags = ["$CI_COMMIT_SHORT_SHA", "latest", "$CI_COMMIT_TAG"], args = { REPO = "$CI_REPO_NAME" }, secrets = ["NPM_TOKEN"] }
`
// dirTar builds a directory tar as the archive endpoint returns it.
func dirTar(t *testing.T, top string, files map[string]string) []byte {
t.Helper()
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
if err := tw.WriteHeader(&tar.Header{Name: top + "/", Typeflag: tar.TypeDir, Mode: 0o755}); err != nil {
t.Fatal(err)
}
for name, body := range files {
hdr := &tar.Header{Name: top + "/" + name, Typeflag: tar.TypeReg, Mode: 0o644, Size: int64(len(body))}
if err := tw.WriteHeader(hdr); err != nil {
t.Fatal(err)
}
tw.Write([]byte(body))
}
tw.Close()
return buf.Bytes()
}
func hexOf(b []byte) string { s := sha256.Sum256(b); return hex.EncodeToString(s[:]) }
// extraBlob is a blob the archive carries but the manifest does not name.
var extraBlob = []byte("not-in-the-manifest")
// builtImage returns a tar of a containers-image dir: layout, as the build
// VM writes it, and the manifest digest. variant is "tamper" to corrupt the
// layer, "no-layer" to leave it out, or "" for a good image. Every variant
// carries extraBlob.
func builtImage(t *testing.T, variant string) ([]byte, string) {
t.Helper()
config := []byte(`{"architecture":"amd64","os":"linux","rootfs":{"type":"layers","diff_ids":[]}}`)
layer := []byte("layer-bytes")
manifest := fmtManifest(hexOf(config), len(config), hexOf(layer), len(layer))
files := map[string]string{
"manifest.json": string(manifest),
"version": "Directory Transport Version: 1.1\n",
hexOf(config): string(config),
hexOf(layer): string(layer),
hexOf(extraBlob): string(extraBlob),
}
switch variant {
case "tamper":
files[hexOf(layer)] = "other-bytes"
case "no-layer":
delete(files, hexOf(layer))
}
return dirTar(t, ".", files), regDigest(manifest)
}
// buildContainerRemoved reports whether the run's build VM container was
// deleted after it was created.
func buildContainerRemoved(m *mockDocker, runID int64) bool {
name := "hearthforge-ci-" + strconv.FormatInt(runID, 10) + "-build"
created := false
for _, req := range m.containerRequests() {
switch req {
case "POST create?name=" + name:
created = true
case "DELETE " + name:
if created {
return true
}
}
}
return false
}
func fmtManifest(config string, configSize int, layer string, layerSize int) []byte {
b, _ := json.Marshal(map[string]any{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": map[string]any{
"mediaType": "application/vnd.docker.container.image.v1+json",
"digest": "sha256:" + config, "size": configSize,
},
"layers": []map[string]any{{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"digest": "sha256:" + layer, "size": layerSize,
}},
})
return b
}
// ciBuildEnv seeds the build config, the secret it names, and the context
// directory in the CI container. extraEnv goes to ciEnv.
func ciBuildEnv(t *testing.T, extraEnv ...string) (*env, *mockDocker, *session, string) {
t.Helper()
e, m, admin := ciEnv(t, extraEnv...)
admin.post("/ci-repo/settings/ci-secrets", url.Values{
"name": {"NPM_TOKEN"}, "value": {"npm-s3cret"},
}).mustRedirect("/ci-repo/settings")
sha := ciSeedToml(e, ciBuildTOML)
m.reset()
m.setArchive("/ci/project", dirTar(t, "project", map[string]string{"Containerfile": "FROM scratch\n"}))
return e, m, admin, sha
}
func TestCIBuildImage(t *testing.T) {
e, m, admin, sha := ciBuildEnv(t)
image, digest := builtImage(t, "")
m.programBuild("STEP 1/1: FROM scratch\nnpm-s3cret\n", 0, image)
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "success" {
t.Fatalf("run status = %q, log %q", status, ciStep(e, runID, "image").Log)
}
step := ciStep(e, runID, "image")
host := strings.TrimPrefix(e.Base, "http://")
for _, want := range []string{
"STEP 1/1: FROM scratch",
"Pushed " + host + "/ci-repo/web:" + sha[:8],
"Pushed " + host + "/ci-repo/web:latest",
"Digest " + digest,
} {
if !strings.Contains(step.Log, want) {
t.Errorf("log lacks %q:\n%s", want, step.Log)
}
}
if strings.Contains(step.Log, "npm-s3cret") {
t.Error("the secret is not masked in the build log")
}
for _, tag := range []string{"latest", sha[:8]} {
r := regAdmin(t, e, http.MethodGet, "/v2/ci-repo/web/manifests/"+tag, nil).mustStatus(200)
if got := r.Header.Get("Docker-Content-Digest"); got != digest {
t.Errorf("%s digest = %s, want %s", tag, got, digest)
}
}
if tags := regTags(t, e, "ci-repo/web", ""); len(tags) != 2 {
t.Errorf("tags = %v, the empty $CI_COMMIT_TAG must be dropped", tags)
}
vmImage, err := ci.DefaultVMImage()
if err != nil {
t.Fatal(err)
}
t.Run("the VM image is built from the embedded vm/ and logged", func(t *testing.T) {
builds := m.builtImages()
if len(builds) != 1 || builds[0].tag != vmImage {
t.Fatalf("image builds = %v, want one of %s", builds, vmImage)
}
for _, f := range []string{"Containerfile", "run-vm", "guest/init"} {
if !contains(builds[0].files, f) {
t.Errorf("build context %v lacks %s", builds[0].files, f)
}
}
for _, want := range []string{"Building the build VM image " + vmImage, "Step 1/20 : ARG ALPINE"} {
if !strings.Contains(step.Log, want) {
t.Errorf("log lacks %q", want)
}
}
if len(m.pulledImages()) != 1 {
t.Errorf("pulls = %v, want only the CI image", m.pulledImages())
}
})
t.Run("the VM container gets KVM and nothing of the host", func(t *testing.T) {
body := m.buildBody()
if body["Image"] != vmImage {
t.Errorf("Image = %v", body["Image"])
}
host, _ := body["HostConfig"].(map[string]any)
devices, _ := json.Marshal(host["Devices"])
if !strings.Contains(string(devices), `"PathOnHost":"/dev/kvm"`) {
t.Errorf("Devices = %s", devices)
}
if host["Binds"] != nil || host["Privileged"] != nil {
t.Errorf("HostConfig = %v", host)
}
})
t.Run("the job carries the context, args and secrets", func(t *testing.T) {
if got := tarFiles(t, m.uploadTo("/in/context"))["project/Containerfile"]; got != "FROM scratch\n" {
t.Errorf("context Containerfile = %q", got)
}
files := tarFiles(t, m.uploadTo("/in"))
want := map[string]string{
"job/args/REPO": "ci-repo",
"job/secrets/NPM_TOKEN": "npm-s3cret",
}
for name, body := range want {
if files[name] != body {
t.Errorf("%s = %q, want %q", name, files[name], body)
}
}
})
t.Run("a blob the manifest does not name is not stored", func(t *testing.T) {
regAdmin(t, e, http.MethodGet, "/v2/ci-repo/web/blobs/sha256:"+hexOf(extraBlob), nil).mustStatus(404)
})
t.Run("the build VM container is removed", func(t *testing.T) {
if !buildContainerRemoved(m, runID) {
t.Errorf("requests = %v", m.containerRequests())
}
})
}
// tarFiles maps the regular files of a tar to their content.
func tarFiles(t *testing.T, data []byte) map[string]string {
t.Helper()
out := map[string]string{}
tr := tar.NewReader(bytes.NewReader(data))
for {
h, err := tr.Next()
if err != nil {
break
}
var b bytes.Buffer
b.ReadFrom(tr)
if h.Typeflag == tar.TypeReg {
out[h.Name] = b.String()
}
}
return out
}
func TestCIBuildImageFailures(t *testing.T) {
cases := []struct {
name string
exit int
variant string
env []string
wantLog string
}{
{"build fails", 1, "", nil, "Image build failed: the build failed"},
{"image too large", 3, "", nil, "Image build failed: the image is larger than CI_MAX_IMAGE_BYTES"},
{"VM does not start", 2, "", nil, "Image build failed: the build VM did not run (exit code 2)"},
{"layer does not match its digest", 0, "tamper", nil, "digest does not match"},
{"layer missing from the archive", 0, "no-layer", nil, "not uploaded"},
// run-vm enforces the cap too. This is the check that does not trust it.
{"image over the cap despite exit 0", 0, "", []string{"CI_MAX_IMAGE_BYTES", "64"}, "larger than CI_MAX_IMAGE_BYTES"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
e, m, admin, sha := ciBuildEnv(t, append([]string{"CI_VM_IMAGE", "localhost/test-vm:1"}, c.env...)...)
m.setLocalImages("localhost/test-vm:1")
image, _ := builtImage(t, c.variant)
m.programBuild("hearthforge: failure\n", c.exit, image)
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "failure" {
t.Fatalf("run status = %q", status)
}
if log := ciStep(e, runID, "image").Log; !strings.Contains(log, c.wantLog) {
t.Errorf("log = %q, want %q", log, c.wantLog)
}
regAdmin(t, e, http.MethodGet, "/v2/ci-repo/web/manifests/latest", nil).mustStatus(404)
if pulls := m.pulledImages(); contains(pulls, "localhost/test-vm") {
t.Errorf("pulls = %v, the local VM image must not be pulled", pulls)
}
if builds := m.builtImages(); len(builds) != 0 {
t.Errorf("image builds = %v, CI_VM_IMAGE must not be built", builds)
}
if !buildContainerRemoved(m, runID) {
t.Errorf("build VM container not removed: %v", m.containerRequests())
}
})
}
}
func TestCIBuildImageTimeout(t *testing.T) {
e, m, admin := ciEnv(t, "CI_VM_IMAGE", "localhost/test-vm:1")
sha := ciSeedToml(e, `
image = "debian:latest"
clone_project_to = "/ci/project"
[on]
manual = true
[[steps]]
name = "image"
timeout = 1
build_image = {}
`)
m.reset()
m.setLocalImages("localhost/test-vm:1")
m.setArchive("/ci/project", dirTar(t, "project", map[string]string{"Containerfile": "FROM scratch\n"}))
m.programBuild("STEP 1/1\n", 0, nil)
m.delayBuild(time.Minute)
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "failure" {
t.Fatalf("run status = %q", status)
}
if log := ciStep(e, runID, "image").Log; !strings.Contains(log, "Step timed out after 1s") {
t.Errorf("log = %q", log)
}
if !buildContainerRemoved(m, runID) {
t.Errorf("build VM container not removed: %v", m.containerRequests())
}
}
func TestCIBuildImageMissingSecret(t *testing.T) {
e, m, admin := ciEnv(t)
sha := ciSeedToml(e, ciBuildTOML)
m.reset()
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "failure" {
t.Fatalf("run status = %q", status)
}
if log := ciStep(e, runID, "image").Log; !strings.Contains(log, "secret NPM_TOKEN is not set") {
t.Errorf("log = %q", log)
}
if m.buildBody() != nil {
t.Error("a build VM container was created")
}
}
func TestCIBuildImageVMImage(t *testing.T) {
t.Run("a second run reuses the built VM image", func(t *testing.T) {
e, m, admin, sha := ciBuildEnv(t)
image, _ := builtImage(t, "")
for range 2 {
m.programBuild("", 0, image)
if status := ciWaitForRun(e, ciTrigger(e, admin, sha, nil)); status != "success" {
t.Fatalf("run status = %q", status)
}
}
if builds := m.builtImages(); len(builds) != 1 {
t.Errorf("image builds = %d, want 1", len(builds))
}
})
t.Run("a failed VM image build fails the step", func(t *testing.T) {
e, m, admin, sha := ciBuildEnv(t)
m.failImageBuild("apk: network unreachable")
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "failure" {
t.Fatalf("run status = %q", status)
}
log := ciStep(e, runID, "image").Log
if !strings.Contains(log, "cannot build the build VM image: apk: network unreachable") {
t.Errorf("log = %q", log)
}
if m.buildBody() != nil {
t.Error("a build VM container was created")
}
})
t.Run("a missing CI_VM_IMAGE is pulled, not built", func(t *testing.T) {
e, m, admin, sha := ciBuildEnv(t, "CI_VM_IMAGE", "registry.example.com/buildvm:1")
image, _ := builtImage(t, "")
m.programBuild("", 0, image)
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "success" {
t.Fatalf("run status = %q", status)
}
if !contains(m.pulledImages(), "registry.example.com/buildvm") || len(m.builtImages()) != 0 {
t.Errorf("pulls = %v, builds = %v", m.pulledImages(), m.builtImages())
}
if log := ciStep(e, runID, "image").Log; !strings.Contains(log, "Pulling the build VM image registry.example.com/buildvm:1") {
t.Errorf("log = %q", log)
}
})
}
Minternal/web/e2e/ci_mock_test.go
@@ -12,6 +12,7 @@ import (
"path"
"path/filepath"
"regexp"
"slices"
"strconv"
"strings"
"sync"
@@ -73,10 +74,32 @@ type mockDocker struct {
// execQueue is consumed in order as execs are created.
execQueue []execResp
// execCmds is every command run inside a container, in order.
execCmds [][]string
// execEnvs is the environment of every exec, aligned with execCmds.
execEnvs [][]string
execCmds [][]string
execCounter int
// archives answers GET /archive for these paths instead of the
// one-file default.
archives map[string][]byte
// localImages is what GET /images/*/json finds.
localImages []string
// buildCreateBody is the body of the last build VM container create.
buildCreateBody map[string]any
// buildLog and buildExit are the build VM container's output and exit
// code.
buildLog string
buildExit int
// buildDelay holds the build VM log open, so a step timeout can fire.
buildDelay time.Duration
// imageBuilds records POST /build: the tag and the context file names.
imageBuilds []ciImageBuild
// imageBuildError makes POST /build report this error in its stream.
imageBuildError string
}
// ciImageBuild is one recorded POST /build.
type ciImageBuild struct {
tag string
files []string
}
var (
@@ -85,6 +108,9 @@ var (
reExecStart = regexp.MustCompile(`/exec/([^/]+)/start$`)
reExecJSON = regexp.MustCompile(`/exec/([^/]+)/json$`)
reContainerArchive = regexp.MustCompile(`/containers/[^/]+/archive`)
reContainerLogs = regexp.MustCompile(`/containers/[^/]+/logs$`)
reContainerWait = regexp.MustCompile(`/containers/[^/]+/wait$`)
reImageInspect = regexp.MustCompile(`^/v1.47/images/(.+)/json$`)
)
// newMockDocker starts the mock on a unix socket. The socket lives in a short
@@ -124,7 +150,6 @@ func (m *mockDocker) reset() {
m.execMap = map[string]execResp{}
m.execQueue = nil
m.execCmds = nil
m.execEnvs = nil
m.execCounter = 0
m.uploads = nil
m.uploadError = ""
@@ -136,6 +161,73 @@ func (m *mockDocker) reset() {
m.lastCreateBody = nil
m.requests = nil
m.archiveDelay = 0
m.archives = map[string][]byte{}
m.localImages = nil
m.buildCreateBody = nil
m.buildLog = ""
m.buildExit = 0
m.buildDelay = 0
m.imageBuilds = nil
m.imageBuildError = ""
}
func (m *mockDocker) builtImages() []ciImageBuild {
m.mu.Lock()
defer m.mu.Unlock()
return append([]ciImageBuild(nil), m.imageBuilds...)
}
func (m *mockDocker) failImageBuild(msg string) {
m.mu.Lock()
defer m.mu.Unlock()
m.imageBuildError = msg
}
// programBuild sets what the next build VM container prints, exits with,
// and leaves at /out/image.tar. A nil image leaves nothing.
func (m *mockDocker) programBuild(log string, exit int, image []byte) {
m.mu.Lock()
defer m.mu.Unlock()
m.buildLog, m.buildExit = log, exit
if image != nil {
m.archives["/out/image.tar"] = makeTar("image.tar", string(image))
}
}
func (m *mockDocker) delayBuild(d time.Duration) {
m.mu.Lock()
defer m.mu.Unlock()
m.buildDelay = d
}
func (m *mockDocker) setArchive(path string, data []byte) {
m.mu.Lock()
defer m.mu.Unlock()
m.archives[path] = data
}
func (m *mockDocker) setLocalImages(names ...string) {
m.mu.Lock()
defer m.mu.Unlock()
m.localImages = names
}
func (m *mockDocker) buildBody() map[string]any {
m.mu.Lock()
defer m.mu.Unlock()
return m.buildCreateBody
}
// uploadTo returns the last archive uploaded to path.
func (m *mockDocker) uploadTo(path string) []byte {
m.mu.Lock()
defer m.mu.Unlock()
for i := len(m.uploads) - 1; i >= 0; i-- {
if m.uploads[i].path == path {
return m.uploads[i].body
}
}
return nil
}
func (m *mockDocker) containerRequests() []string {
@@ -216,19 +308,6 @@ func (m *mockDocker) createBody() map[string]any {
return m.lastCreateBody
}
// envForCommand returns the environment of the first exec whose command
// contains sub, or nil when no command matches.
func (m *mockDocker) envForCommand(sub string) []string {
m.mu.Lock()
defer m.mu.Unlock()
for i, c := range m.execCmds {
if strings.Contains(strings.Join(c, " "), sub) {
return m.execEnvs[i]
}
}
return nil
}
func (m *mockDocker) commands() [][]string {
m.mu.Lock()
defer m.mu.Unlock()
@@ -258,6 +337,33 @@ func (m *mockDocker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
case r.Method == http.MethodGet && p == "/v1.47/info":
writeJSON(w, map[string]string{"ServerVersion": "mock"})
// Inspect image.
case r.Method == http.MethodGet && reImageInspect.MatchString(p):
m.mu.Lock()
found := slices.Contains(m.localImages, reImageInspect.FindStringSubmatch(p)[1])
m.mu.Unlock()
if !found {
http.Error(w, "no such image", http.StatusNotFound)
return
}
writeJSON(w, map[string]string{"Id": "sha256:mock"})
// Build an image. A success makes it local, like the real engine.
case r.Method == http.MethodPost && p == "/v1.47/build":
body, _ := io.ReadAll(r.Body)
tag := qs.Get("t")
m.mu.Lock()
m.imageBuilds = append(m.imageBuilds, ciImageBuild{tag: tag, files: tarNames(body)})
fail := m.imageBuildError
if fail == "" {
m.localImages = append(m.localImages, tag)
}
m.mu.Unlock()
writeJSON(w, map[string]string{"stream": "Step 1/20 : ARG ALPINE\n"})
if fail != "" {
writeJSON(w, map[string]string{"error": fail})
}
// Pull image.
case r.Method == http.MethodPost && strings.HasPrefix(p, "/v1.47/images/create"):
m.mu.Lock()
@@ -278,13 +384,40 @@ func (m *mockDocker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
m.mu.Unlock()
// A copy creates its own source container, so the run's container
// must keep its identity.
if !strings.Contains(name, "-copy-") {
switch {
case strings.HasSuffix(name, "-build"):
m.mu.Lock()
m.buildCreateBody = body
m.mu.Unlock()
case !strings.Contains(name, "-copy-"):
m.mu.Lock()
m.lastCreateBody = body
m.mu.Unlock()
}
writeJSON(w, map[string]string{"Id": name})
// Build VM container output.
case r.Method == http.MethodGet && reContainerLogs.MatchString(p):
m.mu.Lock()
out, delay := m.buildLog, m.buildDelay
m.mu.Unlock()
if out != "" {
_, _ = w.Write(muxFrame(out))
}
if delay > 0 {
w.(http.Flusher).Flush()
select {
case <-time.After(delay):
case <-r.Context().Done():
}
}
case r.Method == http.MethodPost && reContainerWait.MatchString(p):
m.mu.Lock()
code := m.buildExit
m.mu.Unlock()
writeJSON(w, map[string]int{"StatusCode": code})
// Start container.
case r.Method == http.MethodPost && reContainerStart.MatchString(p):
w.WriteHeader(http.StatusNoContent)
@@ -293,14 +426,12 @@ func (m *mockDocker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
case r.Method == http.MethodPost && reContainerExec.MatchString(p):
var body struct {
Cmd []string
Env []string
}
_ = json.NewDecoder(r.Body).Decode(&body)
m.mu.Lock()
m.execCounter++
id := "mock-exec-" + strconv.Itoa(m.execCounter)
m.execCmds = append(m.execCmds, body.Cmd)
m.execEnvs = append(m.execEnvs, body.Env)
resp := execResp{}
if len(m.execQueue) > 0 {
resp, m.execQueue = m.execQueue[0], m.execQueue[1:]
@@ -355,6 +486,14 @@ func (m *mockDocker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
}
filePath := qs.Get("path")
m.mu.Lock()
programmed, ok := m.archives[filePath]
m.mu.Unlock()
if ok {
w.Header().Set("Content-Type", "application/x-tar")
_, _ = w.Write(programmed)
return
}
if filePath == "" {
filePath = "file.txt"
}
@@ -470,6 +609,19 @@ func tarHeaders(t *testing.T, data []byte) []tarEntry {
return out
}
// tarNames lists the entry names of a tar, as far as it reads.
func tarNames(data []byte) []string {
var out []string
tr := tar.NewReader(bytes.NewReader(data))
for {
h, err := tr.Next()
if err != nil {
return out
}
out = append(out, h.Name)
}
}
// tarEntryNames lists the file names of an uncompressed tar.
func tarEntryNames(t *testing.T, data []byte) []string {
t.Helper()
Ainternal/web/e2e/ci_network_test.go
@@ -0,0 +1,38 @@
package e2e
import "testing"
// TestCINetwork checks that CI_NETWORK reaches the container create call.
// An engine network is how a CI container gets IPv6 or an isolated segment.
func TestCINetwork(t *testing.T) {
networkMode := func(m *mockDocker) any {
host, _ := m.createBody()["HostConfig"].(map[string]any)
return host["NetworkMode"]
}
t.Run("unset leaves the engine default", func(t *testing.T) {
e, m, admin := ciEnv(t)
m.queueExec(execResp{output: "hi\n"})
sha := ciSeedToml(e, ciSimpleTOML)
runID := ciTrigger(e, admin, sha, nil)
if got := ciWaitForRun(e, runID); got != "success" {
t.Fatalf("status = %q", got)
}
if got := networkMode(m); got != nil {
t.Errorf("NetworkMode = %v, want absent", got)
}
})
t.Run("set joins that network", func(t *testing.T) {
e, m, admin := ciEnv(t, "CI_NETWORK", "hearthforge-ci")
m.queueExec(execResp{output: "hi\n"})
sha := ciSeedToml(e, ciSimpleTOML)
runID := ciTrigger(e, admin, sha, nil)
if got := ciWaitForRun(e, runID); got != "success" {
t.Fatalf("status = %q", got)
}
if got := networkMode(m); got != "hearthforge-ci" {
t.Errorf("NetworkMode = %v", got)
}
})
}
Dinternal/web/e2e/ci_socket_test.go-151
@@ -1,151 +0,0 @@
package e2e
import (
"strings"
"testing"
)
// engine_socket lets a step talk to the container engine that runs the
// pipeline. The server must opt in with CI_ENGINE_SOCKET.
const ciEngineSocketTOML = `
image = "debian:latest"
[on]
manual = true
[[steps]]
name = "build-image"
run_sh = "engine-build ."
engine_socket = true
[[steps]]
name = "plain"
run_sh = "echo plain"
`
// ciBinds returns the HostConfig.Binds of the last created container.
func ciBinds(t *testing.T, m *mockDocker) []string {
t.Helper()
body := m.createBody()
if body == nil {
t.Fatal("no container was created")
}
host, _ := body["HostConfig"].(map[string]any)
raw, _ := host["Binds"].([]any)
out := make([]string, 0, len(raw))
for _, b := range raw {
s, _ := b.(string)
out = append(out, s)
}
return out
}
func TestCIEngineSocketDisabled(t *testing.T) {
e, m, admin := ciEnv(t)
sha := ciSeedToml(e, ciEngineSocketTOML)
m.reset()
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "failure" {
t.Errorf("run status = %q, want failure", status)
}
step := ciStep(e, runID, "build-image")
if step.Status != "failure" {
t.Errorf("build-image status = %q, want failure", step.Status)
}
if !strings.Contains(step.Log, "CI_ENGINE_SOCKET") {
t.Errorf("build-image log = %q", step.Log)
}
if got := ciStep(e, runID, "plain").Status; got != "skipped" {
t.Errorf("plain status = %q, want skipped", got)
}
if binds := ciBinds(t, m); contains(binds, "engine.sock") {
t.Errorf("binds = %v, want no engine socket", binds)
}
if strings.Contains(m.allCommandText(), "engine-build") {
t.Error("the disabled step still ran")
}
}
func TestCIEngineSocketEnabled(t *testing.T) {
e, m, admin := ciEnv(t, "CI_ENGINE_SOCKET", "1")
sha := ciSeedToml(e, ciEngineSocketTOML)
m.reset()
m.queueExec(execResp{output: "built\n"})
m.queueExec(execResp{output: "plain\n"})
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "success" {
t.Fatalf("run status = %q, want success", status)
}
if got := ciStep(e, runID, "build-image").Status; got != "success" {
t.Errorf("build-image status = %q", got)
}
want := m.sock + ":/run/hearthforge/engine.sock"
if binds := ciBinds(t, m); !contains(binds, want) {
t.Errorf("binds = %v, want %q", binds, want)
}
socketEnv := m.envForCommand("engine-build")
if socketEnv == nil {
t.Fatal("the engine_socket step did not run")
}
if !contains(socketEnv, "DOCKER_HOST=unix:///run/hearthforge/engine.sock") {
t.Errorf("engine_socket step env = %v", socketEnv)
}
if !contains(socketEnv, "CONTAINER_HOST=unix:///run/hearthforge/engine.sock") {
t.Errorf("CONTAINER_HOST missing from %v", socketEnv)
}
// CI_REGISTRY points at this server's registry path for the repo.
wantRegistry := "CI_REGISTRY=" + strings.TrimPrefix(e.Base, "http://") + "/ci-repo"
if !contains(socketEnv, wantRegistry) {
t.Errorf("env has no %q: %v", wantRegistry, socketEnv)
}
plainEnv := m.envForCommand("echo plain")
if plainEnv == nil {
t.Fatal("the plain step did not run")
}
if contains(plainEnv, "DOCKER_HOST") {
t.Errorf("plain step env = %v, want no DOCKER_HOST", plainEnv)
}
}
// TestCINetwork checks that CI_NETWORK reaches the container create call.
// An engine network is how a CI container gets IPv6 or an isolated segment.
func TestCINetwork(t *testing.T) {
networkMode := func(m *mockDocker) any {
host, _ := m.createBody()["HostConfig"].(map[string]any)
return host["NetworkMode"]
}
t.Run("unset leaves the engine default", func(t *testing.T) {
e, m, admin := ciEnv(t)
m.queueExec(execResp{output: "hi\n"})
sha := ciSeedToml(e, ciSimpleTOML)
runID := ciTrigger(e, admin, sha, nil)
if got := ciWaitForRun(e, runID); got != "success" {
t.Fatalf("status = %q", got)
}
if got := networkMode(m); got != nil {
t.Errorf("NetworkMode = %v, want absent", got)
}
})
t.Run("set joins that network", func(t *testing.T) {
e, m, admin := ciEnv(t, "CI_NETWORK", "hearthforge-ci")
m.queueExec(execResp{output: "hi\n"})
sha := ciSeedToml(e, ciSimpleTOML)
runID := ciTrigger(e, admin, sha, nil)
if got := ciWaitForRun(e, runID); got != "success" {
t.Fatalf("status = %q", got)
}
if got := networkMode(m); got != "hearthforge-ci" {
t.Errorf("NetworkMode = %v", got)
}
})
}
Minternal/web/e2e/harness_test.go
@@ -94,6 +94,7 @@ func newEnv(t *testing.T, extraEnv ...string) *env {
Git: git,
Patches: gitcmd.NewPatchCache(),
}
runner.ImportImage = srv.ImportImage
if err := srv.SyncRepos(ctx); err != nil {
t.Fatal(err)
}
Minternal/web/registry.go
@@ -1,10 +1,12 @@
package web
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"io"
"net/http"
"os"
@@ -36,10 +38,8 @@ const (
)
var (
digestRe = regexp.MustCompile(`^sha256:[a-f0-9]{64}$`)
uploadIDRe = regexp.MustCompile(`^[a-f0-9]{32}$`)
tagRe = regexp.MustCompile(`^[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}$`)
imagePathRe = regexp.MustCompile(`^[a-z0-9]+(?:[._-][a-z0-9]+)*$`)
digestRe = regexp.MustCompile(`^sha256:[a-f0-9]{64}$`)
uploadIDRe = regexp.MustCompile(`^[a-f0-9]{32}$`)
)
// registryAuthLimiter counts argon2 checks per IP. Clients send Basic auth
@@ -187,16 +187,7 @@ func (s *Server) registry(w http.ResponseWriter, r *http.Request) {
// It reports false for an unknown repo or a path the spec would reject.
func (s *Server) registryName(r *http.Request, name string) (*db.Repo, string, bool) {
repoName, image, _ := strings.Cut(name, "/")
for _, seg := range strings.Split(image, "/") {
if seg != "" && !imagePathRe.MatchString(seg) {
return nil, "", false
}
// The path parser splits on these words, so they cannot be segments.
if seg == "blobs" || seg == "manifests" || seg == "tags" {
return nil, "", false
}
}
if image != "" && strings.Contains(image, "//") {
if !util.ValidImagePath(image) {
return nil, "", false
}
repo, err := s.DB.RepoByNameFold(r.Context(), repoName)
@@ -394,30 +385,39 @@ func (s *Server) finishUpload(w http.ResponseWriter, r *http.Request, repo *db.R
registryError(w, http.StatusBadRequest, "DIGEST_INVALID", "digest must be sha256:<hex>")
return
}
size, actual, err := fileDigest(path)
if err != nil {
_ = os.Remove(path)
if err := s.commitBlob(r.Context(), repo.ID, image, path, digest); err != nil {
if errors.Is(err, errDigestMismatch) {
registryError(w, http.StatusBadRequest, "DIGEST_INVALID", err.Error())
return
}
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if actual != digest {
w.Header().Set("Location", base+"/blobs/"+digest)
w.Header().Set("Docker-Content-Digest", digest)
w.WriteHeader(http.StatusCreated)
}
var errDigestMismatch = errors.New("digest does not match uploaded content")
// commitBlob checks the file at path against digest, moves it into the blob
// store, and links it to the image. The file is gone afterwards either way.
func (s *Server) commitBlob(ctx context.Context, repoID int64, image, path, digest string) error {
size, actual, err := fileDigest(path)
if err == nil && actual != digest {
err = errDigestMismatch
}
if err != nil {
_ = os.Remove(path)
registryError(w, http.StatusBadRequest, "DIGEST_INVALID", "digest does not match uploaded content")
return
return err
}
s.registryMu.Lock()
err = s.storeBlob(path, digest)
if err == nil {
err = s.DB.LinkBlob(r.Context(), repo.ID, image, digest, size)
}
s.registryMu.Unlock()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
defer s.registryMu.Unlock()
if err := s.storeBlob(path, digest); err != nil {
_ = os.Remove(path)
return err
}
w.Header().Set("Location", base+"/blobs/"+digest)
w.Header().Set("Docker-Content-Digest", digest)
w.WriteHeader(http.StatusCreated)
return s.DB.LinkBlob(ctx, repoID, image, digest, size)
}
// storeBlob moves a verified file into place. An existing blob with the
@@ -533,7 +533,7 @@ type manifestRefs struct {
func (s *Server) registryManifest(w http.ResponseWriter, r *http.Request, repo *db.Repo, image, ref string) {
isDigest := digestRe.MatchString(ref)
if !isDigest && !tagRe.MatchString(ref) {
if !isDigest && !util.ValidImageTag(ref) {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "invalid reference")
return
}
@@ -606,25 +606,54 @@ func (s *Server) putManifest(w http.ResponseWriter, r *http.Request, repo *db.Re
registryError(w, http.StatusRequestEntityTooLarge, "MANIFEST_INVALID", "manifest too large")
return
}
sum := sha256.Sum256(body)
digest := "sha256:" + hex.EncodeToString(sum[:])
if isDigest && ref != digest {
if isDigest && ref != bodyDigest(body) {
registryError(w, http.StatusBadRequest, "DIGEST_INVALID", "reference digest does not match body")
return
}
mediaType, _, _ := strings.Cut(r.Header.Get("Content-Type"), ";")
tag := ""
if !isDigest {
tag = ref
}
digest, err := s.storeManifest(r.Context(), repo, image, body, strings.TrimSpace(mediaType), tag)
if err != nil {
var refused *manifestRefused
if errors.As(err, &refused) {
registryError(w, http.StatusBadRequest, refused.code, refused.msg)
return
}
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.Header().Set("Location", imageBase(repo, image)+"/manifests/"+digest)
w.Header().Set("Docker-Content-Digest", digest)
w.WriteHeader(http.StatusCreated)
}
func bodyDigest(body []byte) string {
sum := sha256.Sum256(body)
return "sha256:" + hex.EncodeToString(sum[:])
}
// manifestRefused is a manifest the registry rejects, with its spec code.
type manifestRefused struct{ code, msg string }
func (e *manifestRefused) Error() string { return e.msg }
// storeManifest stores a manifest after checking that everything it points
// at is already in this image. That is what makes a pull reliable. An empty
// mediaType falls back to the one in the body. An empty tag stores the
// manifest by digest only.
func (s *Server) storeManifest(ctx context.Context, repo *db.Repo, image string, body []byte, mediaType, tag string) (string, error) {
var refs manifestRefs
if err := json.Unmarshal(body, &refs); err != nil {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "manifest is not valid JSON")
return
return "", &manifestRefused{"MANIFEST_INVALID", "manifest is not valid JSON"}
}
mediaType, _, _ := strings.Cut(r.Header.Get("Content-Type"), ";")
mediaType = strings.TrimSpace(mediaType)
if mediaType == "" {
mediaType = refs.MediaType
}
if mediaType == "" {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "manifest has no media type")
return
return "", &manifestRefused{"MANIFEST_INVALID", "manifest has no media type"}
}
// Every layer and config blob must be linked, every child manifest
@@ -638,53 +667,37 @@ func (s *Server) putManifest(w http.ResponseWriter, r *http.Request, repo *db.Re
}
for _, d := range blobs {
if !digestRe.MatchString(d) {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "unsupported digest "+d)
return
return "", &manifestRefused{"MANIFEST_INVALID", "unsupported digest " + d}
}
linked, err := s.DB.BlobLinked(r.Context(), repo.ID, image, d)
linked, err := s.DB.BlobLinked(ctx, repo.ID, image, d)
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
return "", err
}
if !linked {
registryError(w, http.StatusBadRequest, "MANIFEST_BLOB_UNKNOWN", "blob "+d+" not uploaded")
return
return "", &manifestRefused{"MANIFEST_BLOB_UNKNOWN", "blob " + d + " not uploaded"}
}
}
for _, c := range refs.Manifests {
if !digestRe.MatchString(c.Digest) {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "unsupported digest "+c.Digest)
return
return "", &manifestRefused{"MANIFEST_INVALID", "unsupported digest " + c.Digest}
}
child, err := s.DB.ManifestByDigest(r.Context(), repo.ID, image, c.Digest)
child, err := s.DB.ManifestByDigest(ctx, repo.ID, image, c.Digest)
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
return "", err
}
if child == nil {
registryError(w, http.StatusBadRequest, "MANIFEST_BLOB_UNKNOWN", "manifest "+c.Digest+" not uploaded")
return
return "", &manifestRefused{"MANIFEST_BLOB_UNKNOWN", "manifest " + c.Digest + " not uploaded"}
}
}
tag := ""
if !isDigest {
tag = ref
}
digest := bodyDigest(body)
m := db.Manifest{Digest: digest, MediaType: mediaType}
s.registryMu.Lock()
err = s.writeBlob(digest, body)
if err == nil {
err = s.DB.PutManifest(r.Context(), repo.ID, image, m, tag, db.NowISO())
}
s.registryMu.Unlock()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
defer s.registryMu.Unlock()
if err := s.writeBlob(digest, body); err != nil {
return "", err
}
w.Header().Set("Location", imageBase(repo, image)+"/manifests/"+digest)
w.Header().Set("Docker-Content-Digest", digest)
w.WriteHeader(http.StatusCreated)
return digest, s.DB.PutManifest(ctx, repo.ID, image, m, tag, db.NowISO())
}
// writeBlob stores small content (a manifest) by digest.
Ainternal/web/registry_import.go
@@ -0,0 +1,119 @@
package web
import (
"archive/tar"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path"
"regexp"
)
// dirBlobRe matches a blob file of a containers-image dir: layout. The
// file name is the sha256 hex digest of its content.
var dirBlobRe = regexp.MustCompile(`^[a-f0-9]{64}$`)
// maxImportEntries caps the files of one imported image. A real image has
// a few dozen.
const maxImportEntries = 1000
// ImportImage stores an image that a CI build VM wrote and points tags at
// it. src is a tar of a containers-image dir: layout. The VM is untrusted:
// blobs are staged first, and only those the manifest names are checked
// against their digest and linked. It returns the manifest digest.
func (s *Server) ImportImage(ctx context.Context, repoName, image string, tags []string, src io.Reader) (string, error) {
repo, err := s.DB.RepoByNameFold(ctx, repoName)
if err != nil {
return "", err
}
if repo == nil {
return "", fmt.Errorf("repository %s not found", repoName)
}
// staged maps a digest to its upload file. Whatever is left in it at
// the end was never committed.
staged := map[string]string{}
defer func() {
for _, p := range staged {
_ = os.Remove(p)
}
}()
var manifest []byte
tr := tar.NewReader(src)
for entries := 0; ; entries++ {
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return "", fmt.Errorf("read image archive: %w", err)
}
if entries >= maxImportEntries {
return "", fmt.Errorf("image archive has more than %d entries", maxImportEntries)
}
if hdr.Typeflag != tar.TypeReg {
continue
}
switch name := path.Clean(hdr.Name); {
case name == "manifest.json":
manifest, err = io.ReadAll(io.LimitReader(tr, maxManifestBytes+1))
if err != nil {
return "", err
}
if len(manifest) > maxManifestBytes {
return "", errors.New("manifest too large")
}
case dirBlobRe.MatchString(name):
id, err := s.newUpload()
if err != nil {
return "", err
}
if old, dup := staged["sha256:"+name]; dup {
_ = os.Remove(old)
}
staged["sha256:"+name] = s.uploadPath(id)
if _, err := appendUpload(s.uploadPath(id), tr); err != nil {
return "", err
}
}
}
if manifest == nil {
return "", errors.New("image archive has no manifest.json")
}
var refs manifestRefs
if err := json.Unmarshal(manifest, &refs); err != nil {
return "", errors.New("manifest is not valid JSON")
}
var digests []string
if refs.Config != nil {
digests = append(digests, refs.Config.Digest)
}
for _, l := range refs.Layers {
digests = append(digests, l.Digest)
}
// A digest missing from the archive is left to storeManifest, which
// accepts it only when this image already links it.
for _, d := range digests {
p, ok := staged[d]
if !ok {
continue
}
delete(staged, d)
if err := s.commitBlob(ctx, repo.ID, image, p, d); err != nil {
return "", fmt.Errorf("blob %s: %w", d, err)
}
}
var digest string
for _, tag := range tags {
digest, err = s.storeManifest(ctx, repo, image, manifest, "", tag)
if err != nil {
return "", err
}
}
return digest, nil
}
Minternal/web/views/ci.go
@@ -135,7 +135,7 @@ var ciStepOptions = [][2]string{
{"always", "run even after an earlier step failed"},
{"warn_on_fail", "step warns instead of failing; the run reports warning"},
{"clear", "re-extract a clean checkout before the step"},
{"engine_socket", "mount the Docker/Podman socket for image builds; the server must set CI_ENGINE_SOCKET=1"},
{"build_image", "build a Containerfile in a VM and push it to this repository's registry"},
{"timeout", "per-step timeout in seconds"},
}
Avm/Containerfile
@@ -0,0 +1,48 @@
# The build VM image. Hearthforge starts one container of it per build_image
# step. The container runs QEMU, and the VM inside runs buildah. See CI.md.
# Hearthforge embeds this directory and builds the image on first use with
# the engine's classic build API, so it must not need BuildKit.
ARG ALPINE=docker.io/library/alpine:3.24
# The system that boots inside the VM.
FROM ${ALPINE} AS guest
RUN apk add --no-cache buildah crun netavark e2fsprogs tar ca-certificates
COPY guest/ /
# The embedded copy loses file modes.
RUN chmod 755 /init
# --no-scripts skips the mkinitfs trigger. Its initramfs is not used.
FROM ${ALPINE} AS kernel
RUN apk add --no-cache --no-scripts linux-virt kmod
# Modules are stored uncompressed and numbered in load order, so the guest
# init needs only busybox insmod.
RUN set -eu; \
kver=$(ls /lib/modules); \
mkdir -p /out/modules; \
for m in virtio_blk virtio_net ext4 overlay virtio_rng; do \
modprobe -S "$kver" --show-depends "$m"; \
done | awk '$1 == "insmod" && !seen[$2]++ { print $2 }' > /tmp/modules; \
i=0; \
while read -r ko; do \
i=$((i + 1)); \
gunzip -c "$ko" > "/out/modules/$(printf %02d $i)-$(basename "$ko" .gz)"; \
done < /tmp/modules; \
cp /boot/vmlinuz-virt /out/vmlinuz
FROM ${ALPINE} AS initramfs
RUN apk add --no-cache cpio
COPY --from=guest / /rootfs/
COPY --from=kernel /out/modules/ /rootfs/lib/hf-modules/
RUN set -eu; \
cd /rootfs; \
mkdir -p proc sys dev tmp run var/lib/containers; \
find . -print0 | cpio --null --quiet -o -H newc | gzip -1 > /initramfs.gz
FROM ${ALPINE}
RUN apk add --no-cache "qemu-system-$(apk --print-arch)" tar
COPY --from=kernel /out/vmlinuz /vm/vmlinuz
COPY --from=initramfs /initramfs.gz /vm/initramfs.gz
COPY run-vm /usr/local/bin/run-vm
RUN chmod 755 /usr/local/bin/run-vm && mkdir -p /in/context /out /work
ENTRYPOINT ["/usr/local/bin/run-vm"]
Avm/guest/etc/containers/containers.conf
@@ -0,0 +1,5 @@
# The guest has no systemd and no journald.
[engine]
cgroup_manager = "cgroupfs"
events_logger = "file"
runtime = "crun"
Avm/guest/etc/containers/registries.conf
@@ -0,0 +1,4 @@
# Docker resolves "debian" to docker.io/library/debian. A single search
# registry gives the same result, and there is no TTY to prompt on.
unqualified-search-registries = ["docker.io"]
short-name-mode = "permissive"
Avm/guest/etc/containers/storage.conf
@@ -0,0 +1,4 @@
[storage]
driver = "overlay"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers/storage"
Avm/guest/init
@@ -0,0 +1,94 @@
#!/bin/sh
# PID 1 of the build VM. It builds the context from the job disk with
# buildah and writes the image to the "hf.image" port as a tar of a
# containers-image dir: layout.
# Every exit path reboots, and QEMU runs with -no-reboot, so it exits.
finish() {
echo "$1"
sync
reboot -f
}
# crun cannot pivot_root out of the initramfs, so the rest runs on a tmpfs.
if [ ! -e /.hf-tmpfs ]; then
mount -t tmpfs -o mode=0755 tmpfs /mnt
tar -c -C / --exclude=./mnt . | tar -x -C /mnt
touch /mnt/.hf-tmpfs
exec switch_root /mnt /init
fi
mount -t devtmpfs dev /dev
exec </dev/null >/dev/console 2>&1
mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t cgroup2 cgroup2 /sys/fs/cgroup
mkdir -p /dev/pts /dev/shm
mount -t devpts devpts /dev/pts
mount -t tmpfs tmpfs /dev/shm
mount -t tmpfs tmpfs /tmp
mount -t tmpfs tmpfs /run
for ko in /lib/hf-modules/*.ko; do
insmod "$ko" || finish "hearthforge: cannot load $ko"
done
# QEMU user networking: fixed guest address, gateway and DNS.
ip link set lo up
ip link set eth0 up
ip addr add 10.0.2.15/24 dev eth0
ip route add default via 10.0.2.2
echo "nameserver 10.0.2.3" > /etc/resolv.conf
echo "127.0.0.1 localhost" > /etc/hosts
job=
scratch=
for b in /sys/block/vd*; do
case $(cat "$b/serial" 2>/dev/null) in
hfjob) job=/dev/${b##*/} ;;
hfscratch) scratch=/dev/${b##*/} ;;
esac
done
[ -n "$job" ] && [ -n "$scratch" ] || finish "hearthforge: disks missing"
mkfs.ext4 -q -F -E lazy_itable_init=1,lazy_journal_init=1 "$scratch" ||
finish "hearthforge: cannot format the scratch disk"
mount -o noatime "$scratch" /var/lib/containers ||
finish "hearthforge: cannot mount the scratch disk"
# The root is a RAM tmpfs. The job and buildah's layer staging in /var/tmp
# go to the scratch disk instead.
hf=/var/lib/containers/hf
mkdir -p "$hf" /var/lib/containers/tmp
mount --bind /var/lib/containers/tmp /var/tmp
tar -x -f "$job" -C "$hf" || finish "hearthforge: cannot read the job"
# The engine extracts the context under its own directory name.
ctx=$(find "$hf/context" -mindepth 1 -maxdepth 1)
[ -d "$ctx" ] || finish "hearthforge: the build context is not a single directory"
out=
for port in /sys/class/virtio-ports/*; do
[ "$(cat "$port/name" 2>/dev/null)" = hf.image ] && out=/dev/${port##*/}
done
[ -n "$out" ] || finish "hearthforge: image port missing"
set --
[ -f "$hf/job/file" ] && set -- "$@" -f "$ctx/$(cat "$hf/job/file")"
[ -f "$hf/job/target" ] && set -- "$@" --target "$(cat "$hf/job/target")"
for f in "$hf"/job/args/*; do
[ -f "$f" ] && set -- "$@" --build-arg "${f##*/}=$(cat "$f")"
done
for f in "$hf"/job/secrets/*; do
[ -f "$f" ] && set -- "$@" --secret "id=${f##*/},src=$f"
done
# Docker format keeps HEALTHCHECK, SHELL and ONBUILD, which OCI drops.
# --layers gives one layer per instruction, like docker build.
buildah build --format docker --layers --network host "$@" \
-t localhost/hf-build "$ctx" ||
finish "hearthforge: build failed"
buildah push --quiet --format v2s2 --compression-format gzip \
localhost/hf-build dir:/var/lib/containers/hf-out ||
finish "hearthforge: cannot export the image"
tar -c -f "$out" -C /var/lib/containers/hf-out . ||
finish "hearthforge: cannot write the image"
finish "hearthforge: image written"
Avm/run-vm
@@ -0,0 +1,86 @@
#!/bin/sh
# Entrypoint of the build VM container. It packs /in into the job disk, boots
# the VM, and leaves the image at /out/image.tar.
#
# Exit codes: 0 image written, 1 build failed, 2 the VM did not run,
# 3 image over HF_VM_MAX_IMAGE_BYTES.
set -eu
cpus=${HF_VM_CPUS:-2}
mem=${HF_VM_MEMORY_MB:-2048}
disk=${HF_VM_DISK_BYTES:-21474836480}
max=${HF_VM_MAX_IMAGE_BYTES:-4294967296}
if [ ! -r /dev/kvm ] || [ ! -w /dev/kvm ]; then
echo "hearthforge: /dev/kvm is missing or not writable in the build VM container"
exit 2
fi
tar -c -f /work/job.tar -C /in .
truncate -s "$disk" /work/scratch.img
: > /out/image.tar
arch=$(uname -m)
case $arch in
x86_64)
# acpi=off: QEMU puts a large initramfs over the microvm ACPI tables, and
# the kernel hangs while it parses them.
machine=microvm,accel=kvm,acpi=off,pit=off,pic=off,isa-serial=on,rtc=on
console=ttyS0
;;
aarch64)
machine=virt,accel=kvm,gic-version=host
console=ttyAMA0
;;
*)
echo "hearthforge: unsupported architecture $arch"
exit 2
;;
esac
# The serial numbers let the guest find its disks whatever order the
# virtio-mmio transports probe in.
qemu-system-"$arch" \
-nodefaults -no-user-config -display none -no-reboot \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-machine "$machine" -cpu host -smp "$cpus" -m "$mem" \
-kernel /vm/vmlinuz -initrd /vm/initramfs.gz \
-append "console=$console quiet panic=-1 reboot=t" \
-serial stdio \
-drive if=none,id=job,file=/work/job.tar,format=raw,readonly=on \
-device virtio-blk-device,drive=job,serial=hfjob \
-drive if=none,id=scratch,file=/work/scratch.img,format=raw,cache=unsafe,discard=unmap \
-device virtio-blk-device,drive=scratch,serial=hfscratch \
-netdev user,id=net -device virtio-net-device,netdev=net \
-device virtio-rng-device \
-device virtio-serial-device \
-chardev file,id=image,path=/out/image.tar \
-device virtserialport,chardev=image,name=hf.image &
qemu=$!
# The guest controls how much it writes. Polling lets it overshoot by about
# one second of writes, which the engine's disk absorbs.
over=0
while kill -0 "$qemu" 2>/dev/null; do
if [ "$(wc -c < /out/image.tar)" -gt "$max" ]; then
over=1
# QEMU may have exited since the check.
kill "$qemu" 2>/dev/null || true
break
fi
sleep 1
done
status=0
wait "$qemu" || status=$?
size=$(wc -c < /out/image.tar)
if [ "$over" = 1 ] || [ "$size" -gt "$max" ]; then
: > /out/image.tar
echo "hearthforge: the image is larger than $max bytes"
exit 3
fi
if [ "$status" != 0 ]; then
echo "hearthforge: QEMU exited with status $status"
exit 2
fi
[ "$size" -gt 0 ] || exit 1
Mweb/static/assets/hearthforge-ci-template.toml
@@ -96,11 +96,14 @@ run_sh = "rm -rf /ci/build/scratch"
# [[steps]]
# name = "image"
# # engine_socket gives this step the host engine. Needs CI_ENGINE_SOCKET=1 on
# # the server. $CI_REGISTRY is "<host>/<repo>" on the built-in registry.
# engine_socket = true
# run_sh = """
# echo "$REGISTRY_PASSWORD" | docker login "${CI_REGISTRY%%/*}" -u admin --password-stdin
# docker build -t "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA" project
# docker push "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA"
# """
# # build_image builds a Containerfile in a VM after run_sh and pushes the
# # image to this repository's registry, at <host>/<repo>[/<image>]:<tag>.
# # Any valid Containerfile works. The engine host needs /dev/kvm.
# [steps.build_image]
# context = "/ci/build/project" # directory in this container, default clone_project_to
# file = "Containerfile" # relative to context, default Containerfile or Dockerfile
# target = "runtime" # stage of a multi-stage build
# image = "web" # path below the repository's image name
# tags = ["$CI_COMMIT_SHORT_SHA", "$CI_COMMIT_TAG"] # $VARS expanded, empty results dropped
# secrets = ["NPM_TOKEN"] # CI secrets for RUN --mount=type=secret,id=NPM_TOKEN
# args = { VERSION = "$CI_COMMIT_REF_NAME" } # build args, $VARS expanded