Add a container registry, an Images tab, and engine access for CI
Registry: an OCI Distribution server under /v2/ with the standard library only. Image names map to repositories, "<repo>" or "<repo>/<path>", matched case-insensitively. Blobs and manifests are content-addressed files under DATA_DIR/registry; the per-image index lives in SQLite. The admin pushes with Basic auth. REGISTRY_PULL decides who may pull public images: admin, users, or public. Private repos are admin-only and look unknown to everyone else. Failed logins are rate limited per IP; successful requests do not count, so multi-layer pulls are not throttled. Abandoned uploads are swept after a day. One mutex orders file moves against index writes. Repo name "v2" is reserved. Images tab: lists images and tags with the same access rule as pulls. Admins delete a tag, an image path, or all images. Deleting a repo removes its image files unless another repo shares them. CI: a step may set engine_socket = true to get the Docker/Podman socket at /run/hearthforge/engine.sock with DOCKER_HOST and CONTAINER_HOST set. The server must allow it with CI_ENGINE_SOCKET=1, otherwise the step fails. CI_REGISTRY names the image prefix for the run's repo. CI images need fewer tools. Archives for publish_* are built on the host from the engine's tar stream, zstd in-process like releases. The checkout and copy targets are uploaded as rooted tars, so no mkdir runs in the container. A shell, sleep, and rm for clear are all that remain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
MCI.md
@@ -15,16 +15,23 @@ reference, and a link to the full [template](public/assets/hearthforge-ci-templa
first.
3. The commit is exported with `git archive` on the host and uploaded into
`clone_project_to`. The image needs no git. No `.git` directory reaches the
container.
container. Directories are created by the upload itself, so the image
needs no `mkdir` either.
4. Steps run in file order, in that one container, so files and installed
packages persist from step to step. Each step is one `docker exec` of
`shell` with `shell_setup` prepended.
5. Published files are copied out of the container into
`$DATA_DIR/ci/artifacts/<run id>/`. The container is removed. Cache volumes
stay.
5. Published files are streamed out of the container into
`$DATA_DIR/ci/artifacts/<run id>/`. Archives are built on the host, so the
image needs no tar, gzip, zstd or zip. The container is removed. Cache
volumes stay.
Everything up to the first step is shown as the step `pipeline setup`.
The image must provide a POSIX shell at `/bin/sh` (or whatever `shell`
names), `sleep` for the container's main process, and `rm` plus `mkdir` if a
step uses `clear`. Any image with busybox or coreutils qualifies. Nothing
else is required.
## Setup
Hearthforge talks to the container engine over its Unix socket. It checks,
@@ -69,6 +76,7 @@ containerised Hearthforge works the same as a host install.
| `CI_MAX_CONCURRENT` | `2` | Runs executed at once. Further runs wait in a queue. |
| `CI_MAX_HISTORY` | `50` | Runs kept per repository, artifacts included |
| `CI_DEFAULT_TIMEOUT` | `3600` | Step timeout in seconds when the config sets none |
| `CI_ENGINE_SOCKET` | `0` | Allow `engine_socket = true` steps |
## Example
@@ -126,6 +134,7 @@ The template in the Pipelines tab lists every key with a comment.
| `always` | Run even after an earlier step failed. |
| `warn_on_fail` | A failure marks the step `warning` and the run continues. |
| `clear` | Delete `clone_project_to` and extract a fresh checkout before the step. |
| `engine_socket`| Give the step the Docker/Podman socket. Needs `CI_ENGINE_SOCKET=1`. |
| `publish_*` | `publish_file`, `publish_tar`, `publish_gzip`, `publish_zstd`, `publish_zip` |
### Environment
@@ -143,6 +152,7 @@ Every step sees `CI=true` plus:
| `CI_COMMIT_BRANCH` | branch name, empty for tags |
| `CI_COMMIT_TAG` | tag name, empty for branches |
| `CI_COMMIT_REF_NAME` | branch or tag name |
| `CI_REGISTRY` | `<BASE_URL host>/<repo>`, the image name prefix for the built-in registry |
`[variables]` defaults come next, then overrides from the **Run pipeline**
form, then secrets. A later source wins on a name clash.
@@ -160,6 +170,44 @@ or split form is not caught.
```
## Building container images
Steps run in a plain container without an engine of their own. A step with
`engine_socket = true` gets the host engine's socket mounted at
`/run/hearthforge/engine.sock`, and `DOCKER_HOST` and `CONTAINER_HOST` point
at it. `docker build` and `podman build` then run on the host engine with the
full Dockerfile feature set and the engine's own layer cache.
```toml
[[steps]]
name = "image"
engine_socket = true
run_sh = """
apt-get install -y -qq docker.io > /dev/null
echo "$REGISTRY_PASSWORD" | docker login "${CI_REGISTRY%%/*}" -u admin --password-stdin
docker build -t "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA" project
docker push "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA"
"""
```
`REGISTRY_PASSWORD` is a CI secret holding the admin password.
The server must allow it with `CI_ENGINE_SOCKET=1`. Without that, a step
with `engine_socket = true` fails and the run stops.
The bind source is the socket path as Hearthforge sees it. When Hearthforge
runs in a container, mount the socket at the same path it has on the host,
for example `/run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock`.
The engine resolves the source on the host, so a different path inside the
container makes the step fail with a missing socket.
> **Warning:** a step with the socket controls the whole engine. It can start
> privileged containers on the host. That is the same access Hearthforge
> itself has, and only admins push CI configs, so the trust level does not
> change. The socket is mounted for the whole run once any step asks for it.
> Other steps do not get `DOCKER_HOST`, but they can still reach the socket
> file.
## Caches
Each `cache` path becomes a named volume, `hearthforge-ci-cache-<hash>`,
@@ -189,20 +237,9 @@ report volume sizes never triggers the cap.
- **`clone_project_to` must be absolute**, and no `cache` path may lie inside
it or above it. The checkout is extracted over that directory and a `clear`
step deletes it. The config is rejected otherwise.
- **The image needs no git, and the checkout has no `.git`.** `git describe`
or a version from the log needs a step that installs git and fetches the
history itself, for example over the forge's HTTP clone URL. Pushing back
from a pipeline needs the admin's credentials, kept as a secret.
- **The image is pulled on every run.** Even with a local copy, the daemon
contacts the registry. There is no registry authentication, so private
images must already be present and pullable without login. Pin a tag; a
moving tag changes the toolchain under the pipeline.
- **Match the libc for `[[copy]]`.** A binary from an `-alpine` image is
musl and fails on a glibc image with an error that looks like a missing
file.
- **`publish_*` uses tools inside the image.** `publish_gzip`, `publish_tar`,
and `publish_zstd` need `tar` (plus `zstd`). `publish_zip` needs `zip`.
`publish_file` needs nothing.
- **`publish_zip` drops symlinks.** The zip format has no symlink entry, so
the archive keeps regular files and directories only. Use `publish_tar`
or `publish_gzip` when links matter.
- **Steps share one container.** A file left behind by one step is visible to
the next. Use `clear` on a step that must start from a clean tree. Do not
`rm -rf` `work_dir` itself: the container's working directory is gone and
MREADME.md
@@ -102,6 +102,8 @@ All settings are environment variables:
| `CI_MAX_ARTIFACT_BYTES` | `536870912` | Max size of one published CI artifact (512 MiB) |
| `CI_DEFAULT_TIMEOUT` | `3600` | Default step timeout in seconds |
| `CI_MAX_CONCURRENT` | `2` | Advisory max concurrent runs |
| `CI_ENGINE_SOCKET` | `0` | Allow CI steps with `engine_socket = true` to use the engine socket |
| `REGISTRY_PULL` | `admin` | Who may pull container images: `admin`, `users`, or `public` |
\* Set `TRUSTED_PROXY=1` only when Hearthforge is behind a reverse proxy that strips any incoming `X-Forwarded-For` from clients. Caddy and Traefik do this by default; nginx requires `proxy_set_header X-Forwarded-For $remote_addr;` (rather than the common `$proxy_add_x_forwarded_for`, which appends to a client-supplied value). Setting `TRUSTED_PROXY=1` in front of a proxy that does not strip means rate limits and any audit logging are spoofable per request.
@@ -125,6 +127,27 @@ Hearthforge includes a built-in CI/CD system that runs pipelines in Docker or Po
configured via a `.hearthforge-ci.toml` file at the root of your repository. The Pipelines tab contains a small tutorial and
an example file. Setup, the run model, caches, and known pitfalls are documented in [CI.md](CI.md).
## Container registry
Hearthforge serves an OCI container registry under `/v2/`. Image names map to
repositories: `your-forge.example.com/REPO_NAME:tag` or
`your-forge.example.com/REPO_NAME/sub-image:tag`. The first segment must be an
existing repository.
- Only the admin may push, with the admin password as HTTP Basic auth.
`docker login your-forge.example.com` or `podman login` stores it.
- `REGISTRY_PULL` decides who may pull from public repositories: `admin`,
any signed-in user (`users`), or anyone (`public`). Images of private
repositories are always admin-only.
- Blobs are stored once by digest under `DATA_DIR/registry/`. Deleting a
repository removes its image index. Blob files shared with other images
stay.
- `BASE_URL` must be HTTPS for Docker and Podman to talk to the registry
without an insecure-registry exception.
CI steps can build and push images through the engine socket, see
[CI.md](CI.md#building-container-images).
## Development
```bash
Ainternal/ci/archive.go
@@ -0,0 +1,176 @@
package ci
import (
"archive/tar"
"archive/zip"
"bytes"
"compress/gzip"
"context"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path"
"strings"
"github.com/klauspost/compress/zstd"
)
// The container image needs no tar, gzip, zstd or zip. The engine streams
// any path out of a container as a tar, and Hearthforge converts that
// stream on the host. Directories are created the same way in reverse: a
// tar with directory entries, extracted at /, needs no mkdir in the image.
// mkdirTar builds a tar that creates dir and its parents when extracted at /.
func mkdirTar(dir string) []byte {
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
clean := strings.TrimPrefix(path.Clean(dir), "/")
if clean != "" && clean != "." {
parts := strings.Split(clean, "/")
for i := range parts {
_ = tw.WriteHeader(&tar.Header{
Name: strings.Join(parts[:i+1], "/") + "/", Mode: 0o755, Typeflag: tar.TypeDir,
})
}
}
_ = tw.Close()
return buf.Bytes()
}
// mkdirInContainer creates dir and its parents without running a command.
func (r *Runner) mkdirInContainer(ctx context.Context, containerID, dir string) error {
resp, body, err := r.putArchive(ctx, containerID, "/", bytes.NewReader(mkdirTar(dir)))
if err != nil {
return err
}
if resp.StatusCode >= 300 {
return fmt.Errorf("cannot create %s in the container: HTTP %d %s",
dir, resp.StatusCode, strings.TrimSpace(string(body)))
}
return nil
}
// archiveFormats maps a publish_* option to its file extension. The order
// is the order artifacts are collected in.
var archiveFormats = []struct {
pick func(Step) StringList
ext string
}{
{func(s Step) StringList { return s.PublishTar }, ".tar"},
{func(s Step) StringList { return s.PublishGzip }, ".tar.gz"},
{func(s Step) StringList { return s.PublishZstd }, ".tar.zst"},
{func(s Step) StringList { return s.PublishZip }, ".zip"},
}
// storeArchive streams srcPath out of the container and writes it to
// destPath in the format named by ext. maxBytes caps the written file.
func (r *Runner) storeArchive(ctx context.Context, containerID, srcPath, destPath, ext string, maxBytes int64) (int64, error) {
resp, err := r.do(ctx, http.MethodGet,
"/containers/"+containerID+"/archive?path="+url.QueryEscape(srcPath), nil, "")
if err != nil {
return 0, err
}
defer discard(resp)
if resp.StatusCode >= 300 {
return 0, fmt.Errorf("cannot read %s: HTTP %d", srcPath, resp.StatusCode)
}
f, err := os.Create(destPath)
if err != nil {
return 0, err
}
lw := &limitedWriter{w: f, left: maxBytes}
err = writeArchive(ext, resp.Body, lw)
if cerr := f.Close(); err == nil {
err = cerr
}
if err != nil {
os.Remove(destPath)
return 0, err
}
return maxBytes - lw.left, nil
}
// writeArchive converts a tar stream into the wanted format.
func writeArchive(ext string, src io.Reader, dst io.Writer) error {
switch ext {
case ".tar":
_, err := io.Copy(dst, src)
return err
case ".tar.gz":
gz := gzip.NewWriter(dst)
if _, err := io.Copy(gz, src); err != nil {
return err
}
return gz.Close()
case ".tar.zst":
enc, err := zstd.NewWriter(dst)
if err != nil {
return err
}
if _, err := io.Copy(enc, src); err != nil {
return err
}
return enc.Close()
case ".zip":
return tarToZip(src, dst)
}
return fmt.Errorf("unknown archive format %q", ext)
}
// tarToZip re-packs regular files and directories. Symlinks and devices
// have no zip equivalent and are skipped.
func tarToZip(src io.Reader, dst io.Writer) error {
tr := tar.NewReader(src)
zw := zip.NewWriter(dst)
for {
hdr, err := tr.Next()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return err
}
switch hdr.Typeflag {
case tar.TypeDir:
if _, err := zw.CreateHeader(&zip.FileHeader{
Name: strings.TrimSuffix(hdr.Name, "/") + "/", Modified: hdr.ModTime,
}); err != nil {
return err
}
case tar.TypeReg:
w, err := zw.CreateHeader(&zip.FileHeader{
Name: hdr.Name, Method: zip.Deflate, Modified: hdr.ModTime,
})
if err != nil {
return err
}
// The input tar is not compressed and the output is size-capped
// by limitedWriter, so no decompression bomb is possible here.
if _, err := io.Copy(w, tr); err != nil { //nolint:gosec
return err
}
}
}
return zw.Close()
}
// errArtifactTooLarge is what a capped write reports.
var errArtifactTooLarge = errors.New("artifact exceeds CI_MAX_ARTIFACT_BYTES")
// limitedWriter fails once more than left bytes were written.
type limitedWriter struct {
w io.Writer
left int64
}
func (l *limitedWriter) Write(p []byte) (int, error) {
if int64(len(p)) > l.left {
return 0, errArtifactTooLarge
}
n, err := l.w.Write(p)
l.left -= int64(n)
return n, err
}
Minternal/ci/config.go
@@ -10,6 +10,7 @@ import (
"path"
"path/filepath"
"regexp"
"slices"
"strconv"
"strings"
@@ -25,18 +26,21 @@ type VariableDef struct {
// Step is one [[steps]] entry.
type Step struct {
Name string `toml:"name"`
RunSh string `toml:"run_sh"`
RunIf string `toml:"run_if"`
Always bool `toml:"always"`
WarnOnFail bool `toml:"warn_on_fail"`
Clear bool `toml:"clear"`
Timeout int `toml:"timeout"`
PublishFile StringList `toml:"publish_file"`
PublishTar StringList `toml:"publish_tar"`
PublishGzip StringList `toml:"publish_gzip"`
PublishZip StringList `toml:"publish_zip"`
PublishZstd StringList `toml:"publish_zstd"`
Name string `toml:"name"`
RunSh string `toml:"run_sh"`
RunIf string `toml:"run_if"`
Always bool `toml:"always"`
WarnOnFail bool `toml:"warn_on_fail"`
Clear bool `toml:"clear"`
// EngineSocket mounts the Docker/Podman socket and points DOCKER_HOST
// at it. Needs CI_ENGINE_SOCKET on the server.
EngineSocket bool `toml:"engine_socket"`
Timeout int `toml:"timeout"`
PublishFile StringList `toml:"publish_file"`
PublishTar StringList `toml:"publish_tar"`
PublishGzip StringList `toml:"publish_gzip"`
PublishZip StringList `toml:"publish_zip"`
PublishZstd StringList `toml:"publish_zstd"`
}
// StringList accepts either a bare string or an array of strings.
@@ -145,6 +149,11 @@ func (p *PushSpec) UnmarshalTOML(v any) error {
return errors.New("on.push must be a boolean or an array of strings")
}
// WantsEngineSocket reports whether any step asked for the engine socket.
func (c *Config) WantsEngineSocket() bool {
return slices.ContainsFunc(c.Steps, func(s Step) bool { return s.EngineSocket })
}
// Config is a parsed .hearthforge-ci.toml.
type Config struct {
Image string `toml:"image"`
Minternal/ci/config_test.go
@@ -338,3 +338,26 @@ func TestLogBufferTruncates(t *testing.T) {
t.Fatal("appended after truncation")
}
}
func TestParseEngineSocket(t *testing.T) {
cfg := mustParse(t, `
image = "debian:latest"
[[steps]]
name = "test"
run_sh = "true"
[[steps]]
name = "image"
engine_socket = true
run_sh = "docker build ."
`)
if cfg.Steps[0].EngineSocket || !cfg.Steps[1].EngineSocket {
t.Fatalf("engine_socket parsed wrong: %+v", cfg.Steps)
}
if !cfg.WantsEngineSocket() {
t.Error("WantsEngineSocket = false")
}
cfg.Steps[1].EngineSocket = false
if cfg.WantsEngineSocket() {
t.Error("WantsEngineSocket = true with no step asking")
}
}
Minternal/ci/docker.go
@@ -306,10 +306,24 @@ func (r *Runner) enforceCacheLimits(ctx context.Context, repoName string, cache
// --- Containers ---
// engineSocketInContainer is where an engine_socket step finds the socket.
const engineSocketInContainer = "/run/hearthforge/engine.sock"
func (r *Runner) createContainer(ctx context.Context, runID int64, repoName string, cfg *Config, envVars []string) (string, error) {
// No bind for the repo: the daemon resolves bind sources on the host,
// where HearthForge's own paths need not exist. uploadCheckout copies it in.
binds := []string{}
// A step that asked for the socket on a server that forbids it fails in
// the step loop instead.
if r.cfg.CIEngineSocket && cfg.WantsEngineSocket() {
// The socket is a host path the engine itself listens on, so the
// engine can resolve it even when Hearthforge runs in a container.
sock, err := r.socketPath()
if err != nil {
return "", err
}
binds = append(binds, sock+":"+engineSocketInContainer)
}
for _, c := range cfg.Cache {
volName := cacheVolumeName(repoName, c.Path)
if err := r.ensureVolume(ctx, volName, repoName, c.Path); err != nil {
@@ -519,12 +533,8 @@ func (r *Runner) copyFromImage(ctx context.Context, containerID string, spec Cop
defer r.removeContainer(context.WithoutCancel(ctx), created.Id)
// Docker rejects the upload unless the destination already exists.
mk, err := r.exec(ctx, containerID, []string{"mkdir", "-p", spec.To}, "", nil, nil)
if err != nil {
return err
}
if mk.exitCode != 0 {
return fmt.Errorf("copy: cannot create %s: %s", spec.To, strings.TrimSpace(mk.log))
if err := r.mkdirInContainer(ctx, containerID, spec.To); err != nil {
return fmt.Errorf("copy: %w", err)
}
get, err := r.do(ctx, http.MethodGet,
Minternal/ci/execute.go
@@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"time"
@@ -74,7 +75,7 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
return fmt.Errorf("invalid overrides: %w", err)
}
}
envArray, secretValues := buildEnvVars(runID, repoName, r.cfg.BaseURL, run, cfg, overrides, secrets)
envArray, secretValues := buildEnvVars(runID, repoName, r.cfg.BaseURL, r.cfg.PublicHost, run, cfg, overrides, secrets)
// Step names need not be unique, so a name cannot identify a row.
// Keep the ids in config order instead.
@@ -104,7 +105,7 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
}
if cfg.WorkDir != "" {
if _, err := r.exec(ctx, containerID, []string{"mkdir", "-p", cfg.WorkDir}, "", nil, nil); err != nil {
if err := r.mkdirInContainer(ctx, containerID, cfg.WorkDir); err != nil {
return fmt.Errorf("create workdir: %w", err)
}
}
@@ -154,9 +155,27 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
}
stepID := stepIDs[i]
if step.EngineSocket && !r.cfg.CIEngineSocket {
r.execSQL(ctx,
`UPDATE ci_steps SET status = 'failure', started_at = ?, finished_at = ?, log = ? WHERE id = ?`,
db.NowISO(), db.NowISO(),
"engine_socket is disabled on this server. Set CI_ENGINE_SOCKET=1 to allow it.\n", stepID)
runFailed = true
continue
}
// The socket file is visible to every step once mounted. Only
// steps that asked for it get the client pointed at it.
stepEnv := envArray
if step.EngineSocket {
stepEnv = slices.Concat(envArray, []string{
"DOCKER_HOST=unix://" + engineSocketInContainer,
"CONTAINER_HOST=unix://" + engineSocketInContainer,
})
}
if step.RunIf != "" {
res, err := r.exec(ctx, containerID, append(append([]string{}, shell...), step.RunIf),
cfg.WorkDir, envArray, nil)
cfg.WorkDir, stepEnv, nil)
if err != nil {
// The engine went away. That is a run failure, not a
// condition that did not hold.
@@ -224,7 +243,7 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
}
stepCtx, cancel := context.WithTimeout(ctx, time.Duration(timeout)*time.Second)
res, execErr := r.exec(stepCtx, containerID,
append(append([]string{}, shell...), command), cfg.WorkDir, envArray,
append(append([]string{}, shell...), command), cfg.WorkDir, stepEnv,
func(partial string) {
r.execSQL(ctx, `UPDATE ci_steps SET log = ? WHERE id = ?`,
maskSecrets(partial, secretValues), stepID)
@@ -266,7 +285,7 @@ func (r *Runner) executeRun(ctx context.Context, runID int64, t *task) {
sawWarning = true
}
if stepStatus != "failure" {
r.collectArtifacts(ctx, runID, containerID, step, envArray)
r.collectArtifacts(ctx, runID, containerID, step)
}
r.execSQL(ctx, `UPDATE ci_steps SET status = ?, finished_at = ?, log = ? WHERE id = ?`,
Minternal/ci/run.go
@@ -394,7 +394,7 @@ func (r *Runner) pruneHistory(ctx context.Context, repoID int64) {
// buildEnvVars assembles the container environment. Later sources win on a
// name clash: config defaults, then manual overrides, then secrets.
func buildEnvVars(runID int64, repoName, baseURL string, run runRow, cfg *Config,
func buildEnvVars(runID int64, repoName, baseURL, host string, run runRow, cfg *Config,
overrides map[string]string, secrets []secret,
) (envArray, secretValues []string) {
order := []string{}
@@ -424,6 +424,7 @@ func buildEnvVars(runID int64, repoName, baseURL string, run runRow, cfg *Config
set("CI_COMMIT_BRANCH", run.CommitBranch)
set("CI_COMMIT_TAG", run.CommitTag)
set("CI_COMMIT_REF_NAME", refName)
set("CI_REGISTRY", host+"/"+repoName)
for _, name := range cfg.VariableOrder {
if def, ok := cfg.Variables[name]; ok && def.Default != "" {
@@ -505,17 +506,12 @@ func (r *Runner) loadRun(ctx context.Context, runID int64) (runRow, error) {
// entry for the archive root, so the destination keeps the ownership the
// container gave it. No .git reaches the container.
func (r *Runner) uploadCheckout(ctx context.Context, containerID, repoName, commitSha, destPath string) error {
mk, err := r.exec(ctx, containerID, []string{"mkdir", "-p", destPath}, "", nil, nil)
if err != nil {
return err
}
if mk.exitCode != 0 {
return fmt.Errorf("failed to create %s in the container: %s", destPath, strings.TrimSpace(mk.log))
}
// The archive carries destPath as its prefix and is extracted at /, so
// the engine creates the directories and the image needs no mkdir.
prefix := strings.TrimPrefix(path.Clean(destPath), "/") + "/"
// --end-of-options: commit_sha is unvalidated text from the push.
cmd := exec.CommandContext(ctx, "git", "-C", r.repoPath(repoName), "archive",
"--format=tar", "--end-of-options", commitSha)
"--format=tar", "--prefix="+prefix, "--end-of-options", commitSha)
cmd.Env = gitcmd.Env()
var stderr strings.Builder
cmd.Stderr = &stderr
@@ -526,7 +522,7 @@ func (r *Runner) uploadCheckout(ctx context.Context, containerID, repoName, comm
if err := cmd.Start(); err != nil {
return err
}
resp, body, putErr := r.putArchive(ctx, containerID, destPath, stdout)
resp, body, putErr := r.putArchive(ctx, containerID, "/", stdout)
// The PUT stopped reading, so git would block writing into a full pipe.
stdout.Close()
waitErr := cmd.Wait()
@@ -548,11 +544,9 @@ func (r *Runner) uploadCheckout(ctx context.Context, containerID, repoName, comm
// --- Artifacts ---
// collectArtifacts copies this step's published files out of the container.
//
// Archive commands run with the source's parent directory as the working
// directory and name the source by its basename, so a user-controlled path
// never becomes part of an interpolated shell string.
func (r *Runner) collectArtifacts(ctx context.Context, runID int64, containerID string, step Step, envVars []string) {
// Archives are built on the host from the engine's tar stream, so the image
// needs no archive tools.
func (r *Runner) collectArtifacts(ctx context.Context, runID int64, containerID string, step Step) {
dir := r.artifactDir(runID)
if err := os.MkdirAll(dir, 0o755); err != nil {
return
@@ -580,26 +574,21 @@ func (r *Runner) collectArtifacts(ctx context.Context, runID int64, containerID
store(path.Base(srcPath), srcPath)
}
formats := []struct {
paths []string
ext string
cmd func(name, dst string) []string
}{
{step.PublishTar, ".tar", func(n, d string) []string { return []string{"tar", "-cf", d, n} }},
{step.PublishGzip, ".tar.gz", func(n, d string) []string { return []string{"tar", "-czf", d, n} }},
{step.PublishZstd, ".tar.zst", func(n, d string) []string { return []string{"tar", "--zstd", "-cf", d, n} }},
{step.PublishZip, ".zip", func(n, d string) []string { return []string{"zip", "-r", d, n} }},
}
archiveIndex := 0
for _, f := range formats {
for _, srcPath := range f.paths {
archiveIndex++
tmpPath := fmt.Sprintf("/tmp/hf-artifact-%d-%d%s", runID, archiveIndex, f.ext)
res, err := r.exec(ctx, containerID, f.cmd(path.Base(srcPath), tmpPath), path.Dir(srcPath), envVars, nil)
if err != nil || res.exitCode != 0 {
for _, f := range archiveFormats {
for _, srcPath := range f.pick(step) {
filename := path.Base(srcPath) + f.ext
dest := filepath.Join(dir, filename)
if _, err := os.Stat(dest); err == nil {
log.Printf("[ci] run %d: artifact %s already published, skipping %s", runID, filename, srcPath)
continue
}
size, err := r.storeArchive(ctx, containerID, srcPath, dest, f.ext, r.cfg.CIMaxArtifactBytes)
if err != nil {
log.Printf("[ci] run %d: artifact %s: %v", runID, srcPath, err)
continue
}
store(path.Base(srcPath)+f.ext, tmpPath)
r.execSQL(ctx, `INSERT INTO ci_artifacts (run_id, filename, size) VALUES (?, ?, ?)`,
runID, filename, size)
}
}
}
Minternal/config/config.go
@@ -19,6 +19,7 @@ type Config struct {
BaseURL string
PublicHTTPS bool
PublicOrigin string
PublicHost string // host[:port] of BaseURL, what image names start with
RegistrationType string // enabled | disabled | queue
RegisterQuestion string
MaxUploadBytes int64
@@ -43,6 +44,8 @@ type Config struct {
CIDefaultTimeout int
CIMaxConcurrent int
CIMaxArtifactBytes int64
CIEngineSocket bool
RegistryPull string // admin | users | public
MaxConcurrentArchives int
}
@@ -119,6 +122,8 @@ func Load() (*Config, error) {
CIDefaultTimeout: int(intEnv("CI_DEFAULT_TIMEOUT", 3600, 1)),
CIMaxConcurrent: int(intEnv("CI_MAX_CONCURRENT", 2, 1)),
CIMaxArtifactBytes: intEnv("CI_MAX_ARTIFACT_BYTES", 512<<20, 1),
CIEngineSocket: boolEnv("CI_ENGINE_SOCKET"),
RegistryPull: strEnv("REGISTRY_PULL", "admin"),
MaxConcurrentArchives: int(intEnv("MAX_CONCURRENT_ARCHIVE_JOBS", 2, 1)),
}
switch c.RegistrationType {
@@ -126,16 +131,34 @@ func Load() (*Config, error) {
default:
return nil, fmt.Errorf("REGISTRATION_TYPE %q must be enabled, disabled or queue", c.RegistrationType)
}
switch c.RegistryPull {
case "admin", "users", "public":
default:
return nil, fmt.Errorf("REGISTRY_PULL %q must be admin, users or public", c.RegistryPull)
}
u, err := url.Parse(c.BaseURL)
if err != nil || u.Host == "" {
return nil, fmt.Errorf("BASE_URL %q is not a valid URL", c.BaseURL)
}
c.PublicHTTPS = u.Scheme == "https"
c.PublicOrigin = u.Scheme + "://" + u.Host
c.PublicHost = u.Host
c.CommitterEmail = strEnv("COMMITTER_EMAIL", owner+"@"+u.Hostname())
return c, nil
}
// CanPullImages applies REGISTRY_PULL. Images of private repositories are
// admin-only whatever the setting says.
func (c *Config) CanPullImages(isPrivate, authed, isAdmin bool) bool {
if isAdmin {
return true
}
if isPrivate {
return false
}
return c.RegistryPull == "public" || (c.RegistryPull == "users" && authed)
}
// Derived paths under DataDir.
func (c *Config) DBPath() string { return filepath.Join(c.DataDir, "hearthforge.db") }
func (c *Config) ReposDir() string { return filepath.Join(c.DataDir, "repos") }
@@ -143,3 +166,4 @@ func (c *Config) AvatarsDir() string { return filepath.Join(c.DataDir, "
func (c *Config) ReleasesDir() string { return filepath.Join(c.DataDir, "releases") }
func (c *Config) AllowedSignersPath() string { return filepath.Join(c.DataDir, "allowed_signers") }
func (c *Config) CIArtifactsDir() string { return filepath.Join(c.DataDir, "ci", "artifacts") }
func (c *Config) RegistryDir() string { return filepath.Join(c.DataDir, "registry") }
Ainternal/db/registry.go
@@ -0,0 +1,238 @@
package db
import (
"context"
"database/sql"
"errors"
)
// Manifest is one stored image manifest or index.
type Manifest struct {
Digest string
MediaType string
}
// LinkBlob records that image (repo_id, image) references digest.
func (d *DB) LinkBlob(ctx context.Context, repoID int64, image, digest string, size int64) error {
_, err := d.ExecContext(ctx,
`INSERT OR IGNORE INTO registry_blobs (repo_id, image, digest, size) VALUES (?, ?, ?, ?)`,
repoID, image, digest, size)
return err
}
// BlobLinked reports whether image references digest.
func (d *DB) BlobLinked(ctx context.Context, repoID int64, image, digest string) (bool, error) {
var n int
err := d.QueryRowContext(ctx,
`SELECT COUNT(*) FROM registry_blobs WHERE repo_id = ? AND image = ? AND digest = ?`,
repoID, image, digest).Scan(&n)
return n > 0, err
}
// UnlinkBlob drops one image's reference to digest.
func (d *DB) UnlinkBlob(ctx context.Context, repoID int64, image, digest string) error {
_, err := d.ExecContext(ctx,
`DELETE FROM registry_blobs WHERE repo_id = ? AND image = ? AND digest = ?`,
repoID, image, digest)
return err
}
// DigestReferenced reports whether any image still uses the digest as a
// blob or as a manifest. Blobs and manifest bodies share one file store, so
// both tables count before a file is removed.
func (d *DB) DigestReferenced(ctx context.Context, digest string) (bool, error) {
var n int
err := d.QueryRowContext(ctx,
`SELECT (SELECT COUNT(*) FROM registry_blobs WHERE digest = ?) +
(SELECT COUNT(*) FROM registry_manifests WHERE digest = ?)`,
digest, digest).Scan(&n)
return n > 0, err
}
// PutManifest stores the manifest row and, for a tag reference, points the
// tag at it.
func (d *DB) PutManifest(ctx context.Context, repoID int64, image string, m Manifest, tag, now string) error {
if _, err := d.ExecContext(ctx,
`INSERT OR REPLACE INTO registry_manifests (repo_id, image, digest, media_type, created_at)
VALUES (?, ?, ?, ?, ?)`,
repoID, image, m.Digest, m.MediaType, now); err != nil {
return err
}
if tag == "" {
return nil
}
_, err := d.ExecContext(ctx,
`INSERT OR REPLACE INTO registry_tags (repo_id, image, tag, digest, updated_at) VALUES (?, ?, ?, ?, ?)`,
repoID, image, tag, m.Digest, now)
return err
}
// ManifestByDigest returns nil when the image has no such manifest.
func (d *DB) ManifestByDigest(ctx context.Context, repoID int64, image, digest string) (*Manifest, error) {
m := &Manifest{Digest: digest}
err := d.QueryRowContext(ctx,
`SELECT media_type FROM registry_manifests WHERE repo_id = ? AND image = ? AND digest = ?`,
repoID, image, digest).Scan(&m.MediaType)
if errors.Is(err, sql.ErrNoRows) {
return nil, nil
}
return m, err
}
// ManifestByTag resolves a tag. It returns nil when the tag is unknown.
func (d *DB) ManifestByTag(ctx context.Context, repoID int64, image, tag string) (*Manifest, error) {
m := &Manifest{}
err := d.QueryRowContext(ctx,
`SELECT m.digest, m.media_type FROM registry_tags t
JOIN registry_manifests m ON m.repo_id = t.repo_id AND m.image = t.image AND m.digest = t.digest
WHERE t.repo_id = ? AND t.image = ? AND t.tag = ?`,
repoID, image, tag).Scan(&m.Digest, &m.MediaType)
if errors.Is(err, sql.ErrNoRows) {
return nil, nil
}
return m, err
}
// DeleteManifest removes the manifest and every tag pointing at it. It
// reports whether the manifest existed.
func (d *DB) DeleteManifest(ctx context.Context, repoID int64, image, digest string) (bool, error) {
if _, err := d.ExecContext(ctx,
`DELETE FROM registry_tags WHERE repo_id = ? AND image = ? AND digest = ?`,
repoID, image, digest); err != nil {
return false, err
}
res, err := d.ExecContext(ctx,
`DELETE FROM registry_manifests WHERE repo_id = ? AND image = ? AND digest = ?`,
repoID, image, digest)
if err != nil {
return false, err
}
n, _ := res.RowsAffected()
return n > 0, nil
}
// DeleteTag removes one tag and reports whether it existed.
func (d *DB) DeleteTag(ctx context.Context, repoID int64, image, tag string) (bool, error) {
res, err := d.ExecContext(ctx,
`DELETE FROM registry_tags WHERE repo_id = ? AND image = ? AND tag = ?`, repoID, image, tag)
if err != nil {
return false, err
}
n, _ := res.RowsAffected()
return n > 0, nil
}
// ListTags returns the image's tags in lexical order.
func (d *DB) ListTags(ctx context.Context, repoID int64, image string) ([]string, error) {
rows, err := d.QueryContext(ctx,
`SELECT tag FROM registry_tags WHERE repo_id = ? AND image = ? ORDER BY tag`, repoID, image)
if err != nil {
return nil, err
}
defer rows.Close()
var out []string
for rows.Next() {
var t string
if err := rows.Scan(&t); err != nil {
return nil, err
}
out = append(out, t)
}
return out, rows.Err()
}
// ImageTag is one row of the Images tab.
type ImageTag struct {
Image string
Tag string
Digest string
UpdatedAt string
}
// ListImageTags returns every tag of the repo, grouped by image path.
func (d *DB) ListImageTags(ctx context.Context, repoID int64) ([]ImageTag, error) {
rows, err := d.QueryContext(ctx,
`SELECT image, tag, digest, updated_at FROM registry_tags WHERE repo_id = ? ORDER BY image, tag`, repoID)
if err != nil {
return nil, err
}
defer rows.Close()
var out []ImageTag
for rows.Next() {
var t ImageTag
if err := rows.Scan(&t.Image, &t.Tag, &t.Digest, &t.UpdatedAt); err != nil {
return nil, err
}
out = append(out, t)
}
return out, rows.Err()
}
// ImageSizes sums the linked blob sizes per image path.
func (d *DB) ImageSizes(ctx context.Context, repoID int64) (map[string]int64, error) {
rows, err := d.QueryContext(ctx,
`SELECT image, SUM(size) FROM registry_blobs WHERE repo_id = ? GROUP BY image`, repoID)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]int64{}
for rows.Next() {
var image string
var size int64
if err := rows.Scan(&image, &size); err != nil {
return nil, err
}
out[image] = size
}
return out, rows.Err()
}
// DeleteUntaggedManifest removes the manifest when no tag of the image
// points at it any more. It reports whether a row went.
func (d *DB) DeleteUntaggedManifest(ctx context.Context, repoID int64, image, digest string) (bool, error) {
res, err := d.ExecContext(ctx,
`DELETE FROM registry_manifests WHERE repo_id = ?1 AND image = ?2 AND digest = ?3
AND NOT EXISTS (SELECT 1 FROM registry_tags WHERE repo_id = ?1 AND image = ?2 AND digest = ?3)`,
repoID, image, digest)
if err != nil {
return false, err
}
n, _ := res.RowsAffected()
return n > 0, nil
}
// RepoDigests lists every blob and manifest digest the repo references.
// image narrows it to one image path; pass nil for the whole repo.
func (d *DB) RepoDigests(ctx context.Context, repoID int64, image *string) ([]string, error) {
// A nil image binds as NULL and matches every row.
rows, err := d.QueryContext(ctx,
`SELECT digest FROM registry_blobs WHERE repo_id = ?1 AND (?2 IS NULL OR image = ?2)
UNION SELECT digest FROM registry_manifests WHERE repo_id = ?1 AND (?2 IS NULL OR image = ?2)`,
repoID, image)
if err != nil {
return nil, err
}
defer rows.Close()
var out []string
for rows.Next() {
var dg string
if err := rows.Scan(&dg); err != nil {
return nil, err
}
out = append(out, dg)
}
return out, rows.Err()
}
// DeleteImages removes the index rows of one image path, or of every image
// when image is nil. Files are the caller's job.
func (d *DB) DeleteImages(ctx context.Context, repoID int64, image *string) error {
for _, table := range []string{"registry_tags", "registry_manifests", "registry_blobs"} {
if _, err := d.ExecContext(ctx,
`DELETE FROM `+table+` WHERE repo_id = ?1 AND (?2 IS NULL OR image = ?2)`, repoID, image); err != nil {
return err
}
}
return nil
}
Minternal/db/repos.go
@@ -46,6 +46,15 @@ func (d *DB) RepoByName(ctx context.Context, name string) (*Repo, error) {
return scanRepo(d.QueryRowContext(ctx, `SELECT `+repoColumns+` FROM repositories WHERE name = ?`, name))
}
// RepoByNameFold matches the name case-insensitively. Image names are
// lowercase on the wire, repo names need not be. With two repos that only
// differ in case the first by name wins.
func (d *DB) RepoByNameFold(ctx context.Context, name string) (*Repo, error) {
return scanRepo(d.QueryRowContext(ctx,
`SELECT `+repoColumns+` FROM repositories WHERE name = ? OR lower(name) = lower(?) ORDER BY name = ? DESC, name LIMIT 1`,
name, name, name))
}
// GetRepo looks a repo up by name and hides private repos from non-admins.
func (d *DB) GetRepo(ctx context.Context, name string, isAdmin bool) (*Repo, error) {
repo, err := d.RepoByName(ctx, name)
Minternal/db/schema.sql
@@ -207,6 +207,34 @@ CREATE TABLE IF NOT EXISTS ci_run_counters (
last_run_id INTEGER NOT NULL DEFAULT 0
);
-- Container registry. Blob and manifest bodies live under DATA_DIR/registry
-- keyed by digest; these tables hold the per-image index.
CREATE TABLE IF NOT EXISTS registry_blobs (
repo_id INTEGER NOT NULL REFERENCES repositories(id) ON DELETE CASCADE,
image TEXT NOT NULL,
digest TEXT NOT NULL,
size INTEGER NOT NULL,
PRIMARY KEY (repo_id, image, digest)
);
CREATE TABLE IF NOT EXISTS registry_manifests (
repo_id INTEGER NOT NULL REFERENCES repositories(id) ON DELETE CASCADE,
image TEXT NOT NULL,
digest TEXT NOT NULL,
media_type TEXT NOT NULL,
created_at TEXT NOT NULL,
PRIMARY KEY (repo_id, image, digest)
);
CREATE TABLE IF NOT EXISTS registry_tags (
repo_id INTEGER NOT NULL REFERENCES repositories(id) ON DELETE CASCADE,
image TEXT NOT NULL,
tag TEXT NOT NULL,
digest TEXT NOT NULL,
updated_at TEXT NOT NULL,
PRIMARY KEY (repo_id, image, tag)
);
-- Indexes for common query patterns
CREATE INDEX IF NOT EXISTS idx_issues_repo_id ON issues(repo_id);
CREATE INDEX IF NOT EXISTS idx_issues_author_id ON issues(author_id);
Minternal/gitcmd/gitcmd.go
@@ -47,7 +47,8 @@ var validRepoName = regexp.MustCompile(`^[a-zA-Z0-9._-]+$`)
// ValidRepoName mirrors VALID_REPO_NAME_RE plus the traversal guard.
func ValidRepoName(name string) bool {
return name != "" && !strings.Contains(name, "..") && validRepoName.MatchString(name)
// "v2" is the container registry prefix.
return name != "" && name != "v2" && !strings.Contains(name, "..") && validRepoName.MatchString(name)
}
// ValidRef rejects names git would read as options or path traversal.
Minternal/ratelimit/ratelimit.go
@@ -60,6 +60,16 @@ func (l *Limiter) Allow(key string) bool {
return true
}
// Blocked reports whether key is over the limit without recording an event.
// Callers that only want to count failures check this first and call Allow
// after a failure.
func (l *Limiter) Blocked(key string) bool {
l.mu.Lock()
defer l.mu.Unlock()
b := l.buckets[key]
return b != nil && time.Now().Before(b.resetAt) && b.count >= l.max
}
// ClientIP returns the address to rate-limit on.
// With a trusted proxy it takes the first X-Forwarded-For entry, because the
// proxy appends the real client there. Otherwise the header is attacker
Minternal/web/e2e/ci_mock_test.go
@@ -51,6 +51,8 @@ type mockDocker struct {
mu sync.Mutex
sock string
// uploadError makes every PUT /archive answer 500 with this text.
uploadError string
uploads []ciUpload
pulls []string
volumesCreated []ciVolume
@@ -66,7 +68,9 @@ type mockDocker struct {
// execQueue is consumed in order as execs are created.
execQueue []execResp
// execCmds is every command run inside a container, in order.
execCmds [][]string
execCmds [][]string
// execEnvs is the environment of every exec, aligned with execCmds.
execEnvs [][]string
execCounter int
}
@@ -115,8 +119,10 @@ func (m *mockDocker) reset() {
m.execMap = map[string]execResp{}
m.execQueue = nil
m.execCmds = nil
m.execEnvs = nil
m.execCounter = 0
m.uploads = nil
m.uploadError = ""
m.pulls = nil
m.volumesCreated = nil
m.volumesDeleted = nil
@@ -147,13 +153,21 @@ func (m *mockDocker) uploadedPaths() []string {
return out
}
func (m *mockDocker) uploadsTo(dest string) []ciUpload {
// uploadsInto lists uploads whose tar holds at least one entry under dir.
// Directories are created by the upload itself, so the PUT path is "/" and
// the interesting part is inside the archive.
func (m *mockDocker) uploadsInto(t *testing.T, dir string) []ciUpload {
t.Helper()
prefix := strings.TrimPrefix(dir, "/") + "/"
m.mu.Lock()
defer m.mu.Unlock()
var out []ciUpload
for _, u := range m.uploads {
if u.path == dest {
out = append(out, u)
for _, h := range tarHeaders(t, u.body) {
if strings.HasPrefix(h.name, prefix) && h.name != prefix {
out = append(out, u)
break
}
}
}
return out
@@ -189,6 +203,19 @@ func (m *mockDocker) createBody() map[string]any {
return m.lastCreateBody
}
// envForCommand returns the environment of the first exec whose command
// contains sub, or nil when no command matches.
func (m *mockDocker) envForCommand(sub string) []string {
m.mu.Lock()
defer m.mu.Unlock()
for i, c := range m.execCmds {
if strings.Contains(strings.Join(c, " "), sub) {
return m.execEnvs[i]
}
}
return nil
}
func (m *mockDocker) commands() [][]string {
m.mu.Lock()
defer m.mu.Unlock()
@@ -248,12 +275,16 @@ func (m *mockDocker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// Create exec: take the next queued response and bind it to this id.
case r.Method == http.MethodPost && reContainerExec.MatchString(p):
var body struct{ Cmd []string }
var body struct {
Cmd []string
Env []string
}
_ = json.NewDecoder(r.Body).Decode(&body)
m.mu.Lock()
m.execCounter++
id := "mock-exec-" + strconv.Itoa(m.execCounter)
m.execCmds = append(m.execCmds, body.Cmd)
m.execEnvs = append(m.execEnvs, body.Env)
resp := execResp{}
if len(m.execQueue) > 0 {
resp, m.execQueue = m.execQueue[0], m.execQueue[1:]
@@ -286,7 +317,12 @@ func (m *mockDocker) ServeHTTP(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
m.mu.Lock()
m.uploads = append(m.uploads, ciUpload{path: qs.Get("path"), body: body})
fail := m.uploadError
m.mu.Unlock()
if fail != "" {
http.Error(w, fail, http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
// Archive download: artifact collection and [[copy]].
Ainternal/web/e2e/ci_socket_test.go
@@ -0,0 +1,116 @@
package e2e
import (
"strings"
"testing"
)
// engine_socket lets a step talk to the container engine that runs the
// pipeline. The server must opt in with CI_ENGINE_SOCKET.
const ciEngineSocketTOML = `
image = "debian:latest"
[on]
manual = true
[[steps]]
name = "build-image"
run_sh = "engine-build ."
engine_socket = true
[[steps]]
name = "plain"
run_sh = "echo plain"
`
// ciBinds returns the HostConfig.Binds of the last created container.
func ciBinds(t *testing.T, m *mockDocker) []string {
t.Helper()
body := m.createBody()
if body == nil {
t.Fatal("no container was created")
}
host, _ := body["HostConfig"].(map[string]any)
raw, _ := host["Binds"].([]any)
out := make([]string, 0, len(raw))
for _, b := range raw {
s, _ := b.(string)
out = append(out, s)
}
return out
}
func TestCIEngineSocketDisabled(t *testing.T) {
e, m, admin := ciEnv(t)
sha := ciSeedToml(e, ciEngineSocketTOML)
m.reset()
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "failure" {
t.Errorf("run status = %q, want failure", status)
}
step := ciStep(e, runID, "build-image")
if step.Status != "failure" {
t.Errorf("build-image status = %q, want failure", step.Status)
}
if !strings.Contains(step.Log, "CI_ENGINE_SOCKET") {
t.Errorf("build-image log = %q", step.Log)
}
if got := ciStep(e, runID, "plain").Status; got != "skipped" {
t.Errorf("plain status = %q, want skipped", got)
}
if binds := ciBinds(t, m); contains(binds, "engine.sock") {
t.Errorf("binds = %v, want no engine socket", binds)
}
if strings.Contains(m.allCommandText(), "engine-build") {
t.Error("the disabled step still ran")
}
}
func TestCIEngineSocketEnabled(t *testing.T) {
e, m, admin := ciEnv(t, "CI_ENGINE_SOCKET", "1")
sha := ciSeedToml(e, ciEngineSocketTOML)
m.reset()
m.queueExec(execResp{output: "built\n"})
m.queueExec(execResp{output: "plain\n"})
runID := ciTrigger(e, admin, sha, nil)
if status := ciWaitForRun(e, runID); status != "success" {
t.Fatalf("run status = %q, want success", status)
}
if got := ciStep(e, runID, "build-image").Status; got != "success" {
t.Errorf("build-image status = %q", got)
}
want := m.sock + ":/run/hearthforge/engine.sock"
if binds := ciBinds(t, m); !contains(binds, want) {
t.Errorf("binds = %v, want %q", binds, want)
}
socketEnv := m.envForCommand("engine-build")
if socketEnv == nil {
t.Fatal("the engine_socket step did not run")
}
if !contains(socketEnv, "DOCKER_HOST=unix:///run/hearthforge/engine.sock") {
t.Errorf("engine_socket step env = %v", socketEnv)
}
if !contains(socketEnv, "CONTAINER_HOST=unix:///run/hearthforge/engine.sock") {
t.Errorf("CONTAINER_HOST missing from %v", socketEnv)
}
// CI_REGISTRY points at this server's registry path for the repo.
wantRegistry := "CI_REGISTRY=" + strings.TrimPrefix(e.Base, "http://") + "/ci-repo"
if !contains(socketEnv, wantRegistry) {
t.Errorf("env has no %q: %v", wantRegistry, socketEnv)
}
plainEnv := m.envForCommand("echo plain")
if plainEnv == nil {
t.Fatal("the plain step did not run")
}
if contains(plainEnv, "DOCKER_HOST") {
t.Errorf("plain step env = %v, want no DOCKER_HOST", plainEnv)
}
}
Minternal/web/e2e/ci_test.go
@@ -1,9 +1,14 @@
package e2e
import (
"archive/tar"
"archive/zip"
"bytes"
"compress/gzip"
"context"
"database/sql"
"encoding/json"
"io"
"net/http"
"net/url"
"os"
@@ -12,6 +17,8 @@ import (
"strings"
"testing"
"time"
"github.com/klauspost/compress/zstd"
)
// The CI suite drives the real pipeline runner against a mock Docker Engine
@@ -43,10 +50,11 @@ publish_file = ["/ci/output.txt"]
`
// ciEnv starts a server wired to a fresh mock engine and seeds "ci-repo".
func ciEnv(t *testing.T) (*env, *mockDocker, *session) {
// extraEnv holds further environment pairs for newEnv.
func ciEnv(t *testing.T, extraEnv ...string) (*env, *mockDocker, *session) {
t.Helper()
m := newMockDocker(t)
e := newEnv(t, "CI_DOCKER_SOCKET", m.sock)
e := newEnv(t, append([]string{"CI_DOCKER_SOCKET", m.sock}, extraEnv...)...)
admin := e.admin()
e.createRepo(admin, "ci-repo")
e.seedRepo("ci-repo", nil)
@@ -846,11 +854,8 @@ func TestCIRepoUpload(t *testing.T) {
if got := ciWaitForRun(e, runID); got != "success" {
t.Fatalf("status = %q", got)
}
if !contains(m.uploadedPaths(), "/ci/build/project") {
t.Fatalf("uploads = %v", m.uploadedPaths())
}
if len(m.uploadsTo("/ci/build/project")[0].body) == 0 {
t.Error("the checkout upload is empty")
if len(m.uploadsInto(t, "/ci/build/project")) == 0 {
t.Fatalf("no upload carries files under the clone directory; uploads = %v", m.uploadedPaths())
}
binds, _ := json.Marshal(m.createBody()["HostConfig"])
if strings.Contains(string(binds), e.DataDir) {
@@ -871,8 +876,7 @@ func TestCIRepoUpload(t *testing.T) {
t.Run("a failing checkout fails the run before any step runs", func(t *testing.T) {
m.reset()
m.queueExec(execResp{}) // mkdir work_dir
m.queueExec(execResp{output: "mkdir: read-only\n", exitCode: 1}) // mkdir dest
m.uploadError = "read-only file system"
runID := trigger()
if got := ciWaitForRun(e, runID); got != "failure" {
@@ -885,7 +889,7 @@ func TestCIRepoUpload(t *testing.T) {
if setup.Status != "failure" {
t.Errorf("setup status = %q", setup.Status)
}
if !strings.Contains(setup.Log, "mkdir: read-only") {
if !strings.Contains(setup.Log, "read-only file system") {
t.Errorf("setup log = %q", setup.Log)
}
})
@@ -968,30 +972,36 @@ run_sh = "echo hi"
if got := ciWaitForRun(e, runID); got != "success" {
t.Fatalf("status = %q", got)
}
ups := m.uploadsTo("/ci/build/project")
ups := m.uploadsInto(t, "/ci/build/project")
if len(ups) == 0 {
t.Fatal("no upload to the clone directory")
t.Fatal("no upload into the clone directory")
}
// The archive is extracted at / and carries the clone directory as
// its prefix. It must hold the CI config at the triggered commit,
// and no .git. A dropped commit argument would still produce a
// valid tar.
if ups[0].path != "/" {
t.Errorf("upload path = %q, want /", ups[0].path)
}
// The archive must hold the CI config at the triggered commit, and no
// .git. A dropped commit argument would still produce a valid tar.
names := tarEntryNames(t, ups[0].body)
if !contains(names, ".hearthforge-ci.toml") {
if !contains(names, "ci/build/project/.hearthforge-ci.toml") {
t.Errorf("entries = %v", names)
}
for _, n := range names {
if strings.HasPrefix(n, ".git/") {
t.Errorf("archive contains %q", n)
// git archive adds a pax header carrying the commit id.
if n == "pax_global_header" {
continue
}
if strings.Contains(n, ".git/") || !strings.HasPrefix(n, "ci/build/project/") {
t.Errorf("unexpected entry %q", n)
}
}
// git archive writes uid 0 and no entry for the archive root, so the
// destination keeps the mode the container gave it.
// git archive writes uid 0, so the files belong to root in the
// container.
for _, h := range tarHeaders(t, ups[0].body) {
if h.uid != 0 {
t.Errorf("entry %q has uid %d", h.name, h.uid)
}
if h.name == "./" {
t.Error("archive contains a root entry")
}
}
})
}
@@ -1020,7 +1030,6 @@ func TestCICopyFromAnotherImage(t *testing.T) {
t.Run("pulls the source image and uploads its files", func(t *testing.T) {
sha := ciSeedToml(e, ciCopyTOML)
m.reset()
m.queueExec(execResp{}) // mkdir of the copy target
m.queueExec(execResp{output: "1.4.0\n"}) // the step
runID := ciTrigger(e, admin, sha, nil)
@@ -1030,7 +1039,7 @@ func TestCICopyFromAnotherImage(t *testing.T) {
if !contains(m.pulledImages(), "docker.io/oven/bun") {
t.Errorf("pulls = %v", m.pulledImages())
}
if !contains(m.uploadedPaths(), "/usr/local/bin") {
if !contains(m.uploadedPaths(), "/usr/local/bin") || len(m.uploadsInto(t, "/usr/local")) == 0 {
t.Errorf("uploads = %v", m.uploadedPaths())
}
})
@@ -1269,8 +1278,6 @@ func TestCIClearFailuresAreRecorded(t *testing.T) {
t.Run("a clear failure lands on the step, not the console", func(t *testing.T) {
sha := ciSeedToml(e, ciClearTOML)
m.reset()
m.queueExec(execResp{}) // mkdir work_dir
m.queueExec(execResp{}) // mkdir clone_project_to
m.queueExec(execResp{output: "boom\n", exitCode: 1}) // first, fails
m.queueExec(execResp{output: "rm: device busy\n", exitCode: 1}) // clear
@@ -1291,19 +1298,16 @@ func TestCIClearFailuresAreRecorded(t *testing.T) {
t.Run("a clear step re-extracts the checkout", func(t *testing.T) {
sha := ciSeedToml(e, ciClearTOML)
m.reset()
m.queueExec(execResp{}) // mkdir work_dir
m.queueExec(execResp{}) // mkdir clone_project_to
m.queueExec(execResp{output: "ok\n"}) // first
m.queueExec(execResp{}) // clear: rm -rf
m.queueExec(execResp{}) // mkdir clone_project_to again
m.queueExec(execResp{output: "hi\n"}) // second
runID := ciTrigger(e, admin, sha, nil)
if got := ciWaitForRun(e, runID); got != "success" {
t.Fatalf("status = %q", got)
}
if n := len(m.uploadsTo("/ci/build/project")); n != 2 {
t.Errorf("uploads to the clone directory = %d, want 2", n)
if n := len(m.uploadsInto(t, "/ci/build/project")); n != 2 {
t.Errorf("uploads into the clone directory = %d, want 2", n)
}
if strings.Contains(m.allCommandText(), "git") {
t.Errorf("commands = %v", m.commands())
@@ -1553,3 +1557,114 @@ func TestCICacheSizeCaps(t *testing.T) {
}
})
}
// ── host-built archives ─────────────────────────────────────────────────
const ciArchiveTOML = `
image = "debian:latest"
work_dir = "/ci"
[on]
manual = true
[[steps]]
name = "build"
run_sh = "echo building"
publish_tar = ["/ci/dist"]
publish_gzip = ["/ci/dist"]
publish_zstd = ["/ci/dist"]
publish_zip = ["/ci/dist"]
`
// TestCIArchivesBuiltOnHost checks that publish_* archives are built from
// the engine's tar stream. The mock runs no tar, gzip, zstd or zip, so any
// exec for them would fail the test.
func TestCIArchivesBuiltOnHost(t *testing.T) {
e, m, admin := ciEnv(t)
m.queueExec(execResp{output: "building\n"})
sha := ciSeedToml(e, ciArchiveTOML)
runID := ciTrigger(e, admin, sha, nil)
if got := ciWaitForRun(e, runID); got != "success" {
t.Fatalf("status = %q", got)
}
for _, c := range m.commands() {
if len(c) > 0 && (c[0] == "tar" || c[0] == "zip") {
t.Errorf("archive tool run in the container: %v", c)
}
}
artifacts := map[string]int64{}
rows, err := e.DB.QueryContext(context.Background(),
`SELECT id, filename FROM ci_artifacts WHERE run_id = ?`, runID)
if err != nil {
t.Fatal(err)
}
for rows.Next() {
var id int64
var name string
if err := rows.Scan(&id, &name); err != nil {
t.Fatal(err)
}
artifacts[name] = id
}
rows.Close()
if len(artifacts) != 4 {
t.Fatalf("artifacts = %v", artifacts)
}
download := func(name string) []byte {
t.Helper()
id, ok := artifacts[name]
if !ok {
t.Fatalf("artifact %s missing from %v", name, artifacts)
}
return admin.get(ciRunPath(runID) + "/artifacts/" + strconv.FormatInt(id, 10)).mustStatus(200).Body
}
// The mock answers every archive request with one file "dist" holding
// artifact-content-123.
checkTar := func(name string, r io.Reader) {
t.Helper()
tr := tar.NewReader(r)
h, err := tr.Next()
if err != nil || h.Name != "dist" {
t.Fatalf("%s: first entry %v, err %v", name, h, err)
}
data, _ := io.ReadAll(tr)
if string(data) != "artifact-content-123" {
t.Errorf("%s: content = %q", name, data)
}
}
t.Run("tar", func(t *testing.T) {
checkTar("dist.tar", bytes.NewReader(download("dist.tar")))
})
t.Run("gzip", func(t *testing.T) {
gz, err := gzip.NewReader(bytes.NewReader(download("dist.tar.gz")))
if err != nil {
t.Fatal(err)
}
checkTar("dist.tar.gz", gz)
})
t.Run("zstd", func(t *testing.T) {
dec, err := zstd.NewReader(bytes.NewReader(download("dist.tar.zst")))
if err != nil {
t.Fatal(err)
}
defer dec.Close()
checkTar("dist.tar.zst", dec)
})
t.Run("zip", func(t *testing.T) {
data := download("dist.zip")
zr, err := zip.NewReader(bytes.NewReader(data), int64(len(data)))
if err != nil {
t.Fatal(err)
}
if len(zr.File) != 1 || zr.File[0].Name != "dist" {
t.Fatalf("zip entries = %v", zr.File)
}
f, _ := zr.File[0].Open()
body, _ := io.ReadAll(f)
if string(body) != "artifact-content-123" {
t.Errorf("zip content = %q", body)
}
})
}
Ainternal/web/e2e/images_test.go
@@ -0,0 +1,124 @@
package e2e
import (
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"testing"
)
func (e *env) blobFile(digest string) string {
return filepath.Join(e.DataDir, "registry", "blobs", strings.Replace(digest, ":", "/", 1))
}
func TestImagesTab(t *testing.T) {
e := newEnv(t, "REGISTRY_PULL", "users")
admin := e.admin()
alice := e.register("alice", "password123")
e.createRepo(admin, "img-repo")
e.createRepo(admin, "other-repo")
cfgA, layA, _ := pushImage(t, e, "img-repo", "v1", "a")
_, _, manB := pushImage(t, e, "img-repo", "v2", "b")
pushImage(t, e, "img-repo/web", "latest", "c")
// Shared layer: other-repo references the same bytes as img-repo v1.
sharedCfg, sharedLay, _ := pushImage(t, e, "other-repo", "v1", "a")
if sharedCfg != cfgA || sharedLay != layA {
t.Fatal("expected identical digests for identical content")
}
t.Run("tab visible and lists images with tags", func(t *testing.T) {
r := admin.get("/img-repo/images").mustStatus(200)
if !contains(admin.get("/img-repo").Texts(".repo-tab"), "Images") {
t.Error("Images tab missing")
}
titles := r.Texts(".release-item-title")
if len(titles) != 2 || !contains(titles, "img-repo") || !contains(titles, "img-repo/web") {
t.Errorf("images = %v", titles)
}
if tags := r.Texts(".image-tag .badge"); len(tags) != 3 {
t.Errorf("tags = %v", tags)
}
})
t.Run("access follows REGISTRY_PULL", func(t *testing.T) {
alice.get("/img-repo/images").mustStatus(200)
if e.anon().get("/img-repo/images").Code != 403 {
t.Error("anonymous could open the Images tab with REGISTRY_PULL=users")
}
if contains(e.anon().get("/img-repo").Texts(".repo-tab"), "Images") {
t.Error("Images tab shown to anonymous")
}
if alice.get("/img-repo/images").Has(`form[action="/img-repo/images/delete"]`) {
t.Error("delete form shown to non-admin")
}
alice.post("/img-repo/images/delete-all", nil).mustStatus(403)
})
t.Run("delete tag removes an untagged manifest", func(t *testing.T) {
// v2 is unique to this image, so its manifest file must go.
admin.post("/img-repo/images/delete", url.Values{"image": {""}, "tag": {"v2"}}).mustRedirect("/img-repo/images")
if got := regTags(t, e, "img-repo", ""); len(got) != 1 || got[0] != "v1" {
t.Errorf("tags = %v", got)
}
regAdmin(t, e, http.MethodGet, "/v2/img-repo/manifests/"+manB, nil).mustStatus(404)
if _, err := os.Stat(e.blobFile(manB)); !os.IsNotExist(err) {
t.Error("manifest file still on disk")
}
// The v1 layer stays: other-repo links the same bytes.
if _, err := os.Stat(e.blobFile(layA)); err != nil {
t.Error("shared layer file removed")
}
})
t.Run("delete image removes only its unshared files", func(t *testing.T) {
admin.post("/img-repo/images/delete", url.Values{"image": {""}}).mustRedirect("/img-repo/images")
regAdmin(t, e, http.MethodGet, "/v2/img-repo/tags/list", nil).mustStatus(200)
if got := regTags(t, e, "img-repo", ""); len(got) != 0 {
t.Errorf("tags = %v", got)
}
if _, err := os.Stat(e.blobFile(layA)); err != nil {
t.Error("layer shared with other-repo was removed")
}
if titles := admin.get("/img-repo/images").Texts(".release-item-title"); len(titles) != 1 {
t.Errorf("images after delete = %v", titles)
}
})
t.Run("delete all empties the tab", func(t *testing.T) {
admin.post("/img-repo/images/delete-all", nil).mustRedirect("/img-repo/images")
r := admin.get("/img-repo/images")
if r.Count(".release-item-title") != 0 || !r.Contains("No images yet") {
t.Error("images remain after delete all")
}
})
t.Run("deleting a repo removes its unshared image files", func(t *testing.T) {
_, layD, manD := pushImage(t, e, "other-repo", "v2", "d")
admin.post("/other-repo/settings/delete", nil).mustRedirect("/")
for _, d := range []string{layD, manD} {
if _, err := os.Stat(e.blobFile(d)); !os.IsNotExist(err) {
t.Errorf("file %s survived repo delete", d)
}
}
})
}
func TestImagesTabPublicAndPrivate(t *testing.T) {
e := newEnv(t, "REGISTRY_PULL", "public")
admin := e.admin()
e.createRepo(admin, "pub-img")
e.createRepo(admin, "priv-img", "is_private", "1")
pushImage(t, e, "pub-img", "v1", "p")
pushImage(t, e, "priv-img", "v1", "q")
if !contains(e.anon().get("/pub-img").Texts(".repo-tab"), "Images") {
t.Error("Images tab hidden from anonymous with REGISTRY_PULL=public")
}
e.anon().get("/pub-img/images").mustStatus(200)
if e.anon().get("/priv-img/images").Code == 200 {
t.Error("private repo images visible to anonymous")
}
admin.get("/priv-img/images").mustStatus(200)
}
Ainternal/web/e2e/registry_test.go
@@ -0,0 +1,431 @@
package e2e
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"testing"
"hearthforge/internal/db"
)
// The registry suite drives the OCI Distribution endpoints under /v2/.
// Registry clients use HTTP Basic auth, not the session cookie, so these
// tests build their own requests instead of using a session.
const (
ociManifestType = "application/vnd.oci.image.manifest.v1+json"
registryRealm = `Basic realm="Hearthforge registry"`
)
// regReq sends one registry request and reads the whole response. An empty
// user sends no Authorization header. Redirects are not followed.
func regReq(t *testing.T, e *env, method, path string, body []byte, user, pass string, headers ...string) *response {
t.Helper()
var rd io.Reader
if body != nil {
rd = bytes.NewReader(body)
}
req, err := http.NewRequest(method, e.Base+path, rd)
if err != nil {
t.Fatal(err)
}
if user != "" {
req.SetBasicAuth(user, pass)
}
for i := 0; i+1 < len(headers); i += 2 {
req.Header.Set(headers[i], headers[i+1])
}
res, err := e.anon().client.Do(req)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
defer res.Body.Close()
data, err := io.ReadAll(res.Body)
if err != nil {
t.Fatal(err)
}
return &response{t: t, Code: res.StatusCode, Header: res.Header, Body: data}
}
// regAdmin sends a request signed in as the admin.
func regAdmin(t *testing.T, e *env, method, path string, body []byte, headers ...string) *response {
t.Helper()
return regReq(t, e, method, path, body, db.AdminUsername, adminPass, headers...)
}
// regDigest is the content digest the registry expects.
func regDigest(b []byte) string {
sum := sha256.Sum256(b)
return "sha256:" + hex.EncodeToString(sum[:])
}
// regErrCode returns the first error code of a registry error envelope.
func regErrCode(r *response) string {
var body struct {
Errors []struct{ Code string } `json:"errors"`
}
if err := json.Unmarshal(r.Body, &body); err != nil || len(body.Errors) == 0 {
return ""
}
return body.Errors[0].Code
}
// regUpload pushes one blob to an image in a single request.
func regUpload(t *testing.T, e *env, image string, data []byte) string {
t.Helper()
d := regDigest(data)
regAdmin(t, e, http.MethodPost, "/v2/"+image+"/blobs/uploads/?digest="+d, data).mustStatus(201)
return d
}
// pushImage uploads two blobs and a manifest under image:tag. It returns the
// digests of the config blob, layer blob and manifest.
func pushImage(t *testing.T, e *env, image, tag, seed string) (config, layer, manifest string) {
t.Helper()
cfg := []byte(`{"cfg":"` + seed + `"}`)
lay := []byte("layer-" + seed)
config = regUpload(t, e, image, cfg)
layer = regUpload(t, e, image, lay)
body := ociManifest(config, len(cfg), layer, len(lay))
regAdmin(t, e, http.MethodPut, "/v2/"+image+"/manifests/"+tag, body,
"Content-Type", ociManifestType).mustStatus(201)
return config, layer, regDigest(body)
}
// regTags reads the tag list. query is appended to the URL, e.g. "?n=1".
func regTags(t *testing.T, e *env, image, query string) []string {
t.Helper()
r := regAdmin(t, e, http.MethodGet, "/v2/"+image+"/tags/list"+query, nil).mustStatus(200)
var body struct {
Name string `json:"name"`
Tags []string `json:"tags"`
}
if err := json.Unmarshal(r.Body, &body); err != nil {
t.Fatalf("tags list %q: %v (%s)", image, err, r.BodyString())
}
if body.Name != image {
t.Errorf("tags list name = %q, want %q", body.Name, image)
}
return body.Tags
}
// ociManifest builds a minimal image manifest pointing at two blobs.
func ociManifest(config string, configSize int, layer string, layerSize int) []byte {
return fmt.Appendf(nil,
`{"schemaVersion":2,"mediaType":%q,`+
`"config":{"mediaType":"application/vnd.oci.image.config.v1+json","digest":%q,"size":%d},`+
`"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":%q,"size":%d}]}`,
ociManifestType, config, configSize, layer, layerSize)
}
func TestRegistry(t *testing.T) {
e := newEnv(t)
admin := e.admin()
e.createRepo(admin, "reg-repo")
e.register("alice", "password123")
configBlob := []byte(`{"architecture":"amd64","os":"linux"}`)
layerBlob := []byte("layer-bytes-0123456789")
configDigest := regDigest(configBlob)
layerDigest := regDigest(layerBlob)
manifest := ociManifest(configDigest, len(configBlob), layerDigest, len(layerBlob))
manifestDigest := regDigest(manifest)
t.Run("version check challenges an anonymous client", func(t *testing.T) {
r := regReq(t, e, http.MethodGet, "/v2/", nil, "", "").mustStatus(401)
if got := r.Header.Get("WWW-Authenticate"); got != registryRealm {
t.Errorf("WWW-Authenticate = %q, want %q", got, registryRealm)
}
if got := r.Header.Get("Docker-Distribution-API-Version"); got != "registry/2.0" {
t.Errorf("API version header = %q", got)
}
if code := regErrCode(r); code != "UNAUTHORIZED" {
t.Errorf("error code = %q, body %s", code, r.BodyString())
}
})
t.Run("version check succeeds for the admin", func(t *testing.T) {
regAdmin(t, e, http.MethodGet, "/v2/", nil).mustStatus(200)
})
t.Run("a chunked upload stores a blob", func(t *testing.T) {
start := regAdmin(t, e, http.MethodPost, "/v2/reg-repo/blobs/uploads/", nil).mustStatus(202)
loc := start.Header.Get("Location")
if loc == "" || start.Header.Get("Docker-Upload-UUID") == "" {
t.Fatalf("Location = %q, UUID = %q", loc, start.Header.Get("Docker-Upload-UUID"))
}
patch := regAdmin(t, e, http.MethodPatch, loc, configBlob).mustStatus(202)
wantRange := "0-" + strconv.Itoa(len(configBlob)-1)
if got := patch.Header.Get("Range"); got != wantRange {
t.Errorf("Range = %q, want %q", got, wantRange)
}
done := regAdmin(t, e, http.MethodPut, loc+"?digest="+configDigest, nil).mustStatus(201)
if got := done.Header.Get("Docker-Content-Digest"); got != configDigest {
t.Errorf("Docker-Content-Digest = %q, want %q", got, configDigest)
}
head := regAdmin(t, e, http.MethodHead, "/v2/reg-repo/blobs/"+configDigest, nil).mustStatus(200)
if got := head.Header.Get("Content-Length"); got != strconv.Itoa(len(configBlob)) {
t.Errorf("Content-Length = %q, want %d", got, len(configBlob))
}
if got := head.Header.Get("Docker-Content-Digest"); got != configDigest {
t.Errorf("Docker-Content-Digest = %q", got)
}
})
t.Run("a monolithic upload stores a blob", func(t *testing.T) {
if got := regUpload(t, e, "reg-repo", layerBlob); got != layerDigest {
t.Fatalf("digest = %q", got)
}
r := regAdmin(t, e, http.MethodGet, "/v2/reg-repo/blobs/"+layerDigest, nil).mustStatus(200)
if !bytes.Equal(r.Body, layerBlob) {
t.Errorf("blob body = %q", r.BodyString())
}
})
t.Run("a manifest is readable by tag and by digest", func(t *testing.T) {
put := regAdmin(t, e, http.MethodPut, "/v2/reg-repo/manifests/v1", manifest,
"Content-Type", ociManifestType).mustStatus(201)
if got := put.Header.Get("Docker-Content-Digest"); got != manifestDigest {
t.Errorf("Docker-Content-Digest = %q, want %q", got, manifestDigest)
}
byTag := regAdmin(t, e, http.MethodGet, "/v2/reg-repo/manifests/v1", nil).mustStatus(200)
if !bytes.Equal(byTag.Body, manifest) {
t.Errorf("manifest body = %q", byTag.BodyString())
}
if got := byTag.Header.Get("Content-Type"); got != ociManifestType {
t.Errorf("Content-Type = %q, want %q", got, ociManifestType)
}
if got := byTag.Header.Get("Docker-Content-Digest"); got != manifestDigest {
t.Errorf("Docker-Content-Digest = %q, want %q", got, manifestDigest)
}
byDigest := regAdmin(t, e, http.MethodGet, "/v2/reg-repo/manifests/"+manifestDigest, nil).mustStatus(200)
if !bytes.Equal(byDigest.Body, manifest) {
t.Errorf("manifest by digest = %q", byDigest.BodyString())
}
regAdmin(t, e, http.MethodHead, "/v2/reg-repo/manifests/v1", nil).mustStatus(200)
regAdmin(t, e, http.MethodGet, "/v2/reg-repo/manifests/nosuchtag", nil).mustStatus(404)
})
t.Run("the tag list is sorted and paginated", func(t *testing.T) {
if got := regTags(t, e, "reg-repo", ""); !eqStrings(got, []string{"v1"}) {
t.Fatalf("tags = %v", got)
}
regAdmin(t, e, http.MethodPut, "/v2/reg-repo/manifests/latest", manifest,
"Content-Type", ociManifestType).mustStatus(201)
if got := regTags(t, e, "reg-repo", ""); !eqStrings(got, []string{"latest", "v1"}) {
t.Errorf("tags = %v", got)
}
if got := regTags(t, e, "reg-repo", "?n=1"); !eqStrings(got, []string{"latest"}) {
t.Errorf("tags?n=1 = %v", got)
}
if got := regTags(t, e, "reg-repo", "?last=latest"); !eqStrings(got, []string{"v1"}) {
t.Errorf("tags?last=latest = %v", got)
}
})
t.Run("a manifest referencing an unknown blob is rejected", func(t *testing.T) {
missing := regDigest([]byte("never uploaded"))
bad := ociManifest(configDigest, len(configBlob), missing, 14)
r := regAdmin(t, e, http.MethodPut, "/v2/reg-repo/manifests/broken", bad,
"Content-Type", ociManifestType).mustStatus(400)
if code := regErrCode(r); code != "MANIFEST_BLOB_UNKNOWN" {
t.Errorf("error code = %q, body %s", code, r.BodyString())
}
regAdmin(t, e, http.MethodGet, "/v2/reg-repo/manifests/broken", nil).mustStatus(404)
})
t.Run("a wrong digest discards the upload", func(t *testing.T) {
data := []byte("content that does not match")
claimed := regDigest([]byte("something else"))
loc := regAdmin(t, e, http.MethodPost, "/v2/reg-repo/blobs/uploads/", nil).
mustStatus(202).Header.Get("Location")
regAdmin(t, e, http.MethodPatch, loc, data).mustStatus(202)
r := regAdmin(t, e, http.MethodPut, loc+"?digest="+claimed, nil).mustStatus(400)
if code := regErrCode(r); code != "DIGEST_INVALID" {
t.Errorf("error code = %q, body %s", code, r.BodyString())
}
regAdmin(t, e, http.MethodHead, "/v2/reg-repo/blobs/"+claimed, nil).mustStatus(404)
regAdmin(t, e, http.MethodHead, "/v2/reg-repo/blobs/"+regDigest(data), nil).mustStatus(404)
})
t.Run("a sub-path image keeps its own blobs", func(t *testing.T) {
r := regAdmin(t, e, http.MethodHead, "/v2/reg-repo/frontend/blobs/"+configDigest, nil).mustStatus(404)
if code := regErrCode(r); code != "" && code != "BLOB_UNKNOWN" {
t.Errorf("error code = %q", code)
}
regUpload(t, e, "reg-repo/frontend", configBlob)
regAdmin(t, e, http.MethodHead, "/v2/reg-repo/frontend/blobs/"+configDigest, nil).mustStatus(200)
if got := regTags(t, e, "reg-repo/frontend", ""); len(got) != 0 {
t.Errorf("sub-image tags = %v", got)
}
})
t.Run("an unknown repository is not found", func(t *testing.T) {
r := regAdmin(t, e, http.MethodGet, "/v2/nope/tags/list", nil).mustStatus(404)
if code := regErrCode(r); code != "NAME_UNKNOWN" {
t.Errorf("error code = %q, body %s", code, r.BodyString())
}
})
t.Run("only the admin may push", func(t *testing.T) {
r := regReq(t, e, http.MethodPost, "/v2/reg-repo/blobs/uploads/", nil, "alice", "password123").
mustStatus(403)
if code := regErrCode(r); code != "DENIED" {
t.Errorf("error code = %q, body %s", code, r.BodyString())
}
regReq(t, e, http.MethodPost, "/v2/reg-repo/blobs/uploads/", nil, "", "").mustStatus(401)
})
t.Run("pull is admin only by default", func(t *testing.T) {
r := regReq(t, e, http.MethodGet, "/v2/reg-repo/manifests/v1", nil, "alice", "password123").
mustStatus(403)
if code := regErrCode(r); code != "DENIED" {
t.Errorf("error code = %q, body %s", code, r.BodyString())
}
anon := regReq(t, e, http.MethodGet, "/v2/reg-repo/manifests/v1", nil, "", "").mustStatus(401)
if got := anon.Header.Get("WWW-Authenticate"); got != registryRealm {
t.Errorf("WWW-Authenticate = %q", got)
}
})
t.Run("deleting a tag keeps the other tags", func(t *testing.T) {
regAdmin(t, e, http.MethodDelete, "/v2/reg-repo/manifests/latest", nil).mustStatus(202)
if got := regTags(t, e, "reg-repo", ""); !eqStrings(got, []string{"v1"}) {
t.Errorf("tags = %v", got)
}
regAdmin(t, e, http.MethodDelete, "/v2/reg-repo/manifests/latest", nil).mustStatus(404)
})
t.Run("deleting a manifest by digest drops its tags", func(t *testing.T) {
regAdmin(t, e, http.MethodDelete, "/v2/reg-repo/manifests/"+manifestDigest, nil).mustStatus(202)
regAdmin(t, e, http.MethodGet, "/v2/reg-repo/manifests/v1", nil).mustStatus(404)
if got := regTags(t, e, "reg-repo", ""); len(got) != 0 {
t.Errorf("tags = %v", got)
}
})
t.Run("deleting a blob unlinks it from the image", func(t *testing.T) {
regAdmin(t, e, http.MethodDelete, "/v2/reg-repo/blobs/"+layerDigest, nil).mustStatus(202)
r := regAdmin(t, e, http.MethodHead, "/v2/reg-repo/blobs/"+layerDigest, nil).mustStatus(404)
if code := regErrCode(r); code != "" && code != "BLOB_UNKNOWN" {
t.Errorf("error code = %q", code)
}
})
t.Run("a repository cannot be named v2", func(t *testing.T) {
r := admin.post("/new", url.Values{"name": {"v2"}, "default_branch": {"main"}}).mustStatus(200)
if !r.Contains("Invalid repository name") {
t.Error("error message missing")
}
if r.Location() != "" {
t.Errorf("redirected to %q", r.Location())
}
})
t.Run("blob uploads ignore the request body limit", func(t *testing.T) {
// MAX_UPLOAD_BYTES defaults to 10 MiB. A layer is routinely larger.
big := bytes.Repeat([]byte("0123456789abcdef"), 11<<20/16)
digest := regUpload(t, e, "reg-repo", big)
head := regAdmin(t, e, http.MethodHead, "/v2/reg-repo/blobs/"+digest, nil).mustStatus(200)
if got := head.Header.Get("Content-Length"); got != strconv.Itoa(len(big)) {
t.Errorf("Content-Length = %q, want %d", got, len(big))
}
})
t.Run("image names are matched case-insensitively", func(t *testing.T) {
e.createRepo(admin, "MixedCase")
d := regUpload(t, e, "mixedcase", []byte("mixed"))
regAdmin(t, e, http.MethodHead, "/v2/mixedcase/blobs/"+d, nil).mustStatus(200)
if got := regTags(t, e, "mixedcase", ""); len(got) != 0 {
t.Errorf("tags = %v", got)
}
})
t.Run("segments that clash with the path keywords are rejected", func(t *testing.T) {
regAdmin(t, e, http.MethodPost, "/v2/reg-repo/blobs/blobs/uploads/", nil).mustStatus(404)
})
}
// TestRegistryPullUsers checks REGISTRY_PULL=users: any signed-in user may
// pull a public repo's images, anonymous clients may not.
func TestRegistryPullUsers(t *testing.T) {
e := newEnv(t, "REGISTRY_PULL", "users")
admin := e.admin()
e.createRepo(admin, "pub-repo")
e.createRepo(admin, "priv-repo", "is_private", "1")
e.register("alice", "password123")
_, _, manifest := pushImage(t, e, "pub-repo", "v1", "shared")
pushImage(t, e, "priv-repo", "v1", "shared")
t.Run("a signed-in user may pull a public image", func(t *testing.T) {
r := regReq(t, e, http.MethodGet, "/v2/pub-repo/manifests/v1", nil, "alice", "password123").
mustStatus(200)
if regDigest(r.Body) != manifest {
t.Errorf("manifest = %q", r.BodyString())
}
regReq(t, e, http.MethodGet, "/v2/", nil, "alice", "password123").mustStatus(200)
})
t.Run("an anonymous client is challenged", func(t *testing.T) {
regReq(t, e, http.MethodGet, "/v2/pub-repo/manifests/v1", nil, "", "").mustStatus(401)
regReq(t, e, http.MethodGet, "/v2/", nil, "", "").mustStatus(401)
})
t.Run("a private repository looks unknown to other users", func(t *testing.T) {
// Same answer as for a repo that does not exist, so the name cannot
// be probed through the status code.
r := regReq(t, e, http.MethodGet, "/v2/priv-repo/manifests/v1", nil, "alice", "password123").
mustStatus(404)
if code := regErrCode(r); code != "NAME_UNKNOWN" {
t.Errorf("error code = %q, body %s", code, r.BodyString())
}
regReq(t, e, http.MethodGet, "/v2/no-such-repo/manifests/v1", nil, "alice", "password123").mustStatus(404)
regAdmin(t, e, http.MethodGet, "/v2/priv-repo/manifests/v1", nil).mustStatus(200)
})
}
// TestRegistryPullPublic checks REGISTRY_PULL=public: anyone may pull a
// public repo's images without credentials.
func TestRegistryPullPublic(t *testing.T) {
e := newEnv(t, "REGISTRY_PULL", "public")
admin := e.admin()
e.createRepo(admin, "pub-repo")
e.createRepo(admin, "priv-repo", "is_private", "1")
_, layerDigest, manifest := pushImage(t, e, "pub-repo", "v1", "shared")
pushImage(t, e, "priv-repo", "v1", "shared")
t.Run("the version check needs no credentials", func(t *testing.T) {
regReq(t, e, http.MethodGet, "/v2/", nil, "", "").mustStatus(200)
})
t.Run("anyone may pull a public image", func(t *testing.T) {
m := regReq(t, e, http.MethodGet, "/v2/pub-repo/manifests/v1", nil, "", "").mustStatus(200)
if regDigest(m.Body) != manifest {
t.Errorf("manifest = %q", m.BodyString())
}
b := regReq(t, e, http.MethodGet, "/v2/pub-repo/blobs/"+layerDigest, nil, "", "").mustStatus(200)
if regDigest(b.Body) != layerDigest {
t.Errorf("blob = %q", b.BodyString())
}
})
t.Run("a private repository looks unknown to anonymous", func(t *testing.T) {
regReq(t, e, http.MethodGet, "/v2/priv-repo/manifests/v1", nil, "", "").mustStatus(404)
regReq(t, e, http.MethodGet, "/v2/no-such-repo/manifests/v1", nil, "", "").mustStatus(404)
})
t.Run("pushing still needs the admin", func(t *testing.T) {
regReq(t, e, http.MethodPost, "/v2/pub-repo/blobs/uploads/", nil, "", "").mustStatus(401)
})
}
Ainternal/web/images.go
@@ -0,0 +1,129 @@
package web
import (
"net/http"
"github.com/go-chi/chi/v5"
"hearthforge/internal/web/views"
)
// imageRoutes registers the Images tab. Reads follow REGISTRY_PULL like the
// registry itself; deletes are admin-only.
func (s *Server) imageRoutes(r chi.Router) {
r.Get("/{repo}/images", s.imageList)
r.Group(func(r chi.Router) {
r.Use(s.requireAdmin)
r.Post("/{repo}/images/delete", s.deleteImage)
r.Post("/{repo}/images/delete-all", s.deleteAllImages)
})
}
func (s *Server) imageList(w http.ResponseWriter, r *http.Request) {
repo, ok := s.visibleRepo(w, r)
if !ok {
return
}
u := User(r)
if !s.Cfg.CanPullImages(repo.IsPrivate, u != nil, u != nil && u.IsAdmin) {
http.Error(w, "Images are not visible to you", http.StatusForbidden)
return
}
tags, err := s.DB.ListImageTags(r.Context(), repo.ID)
if err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
sizes, err := s.DB.ImageSizes(r.Context(), repo.ID)
if err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
views.Render(w, http.StatusOK, views.ImageList(s.Cfg, u, repo, tags, sizes))
}
// deleteImage removes one tag, or the whole image path when no tag is given.
// A manifest left without tags goes too. Layer blobs stay linked to the
// image until the image itself is deleted.
func (s *Server) deleteImage(w http.ResponseWriter, r *http.Request) {
repo, ok := s.adminRepo(w, r)
if !ok {
return
}
image := r.FormValue("image")
tag := r.FormValue("tag")
if tag == "" {
if err := s.purgeImages(r, repo.ID, &image); err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
redirectTo(w, r, "/"+repo.Name+"/images")
return
}
m, err := s.DB.ManifestByTag(r.Context(), repo.ID, image, tag)
if err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
if m != nil {
if _, err := s.DB.DeleteTag(r.Context(), repo.ID, image, tag); err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
s.registryMu.Lock()
if gone, err := s.DB.DeleteUntaggedManifest(r.Context(), repo.ID, image, m.Digest); err == nil && gone {
s.removeUnreferenced(r, m.Digest)
}
s.registryMu.Unlock()
}
redirectTo(w, r, "/"+repo.Name+"/images")
}
func (s *Server) deleteAllImages(w http.ResponseWriter, r *http.Request) {
repo, ok := s.adminRepo(w, r)
if !ok {
return
}
if err := s.purgeImages(r, repo.ID, nil); err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
redirectTo(w, r, "/"+repo.Name+"/images")
}
// purgeImages drops one image path, or every image when image is nil, and
// removes the files nothing else uses.
func (s *Server) purgeImages(r *http.Request, repoID int64, image *string) error {
s.registryMu.Lock()
defer s.registryMu.Unlock()
digests, err := s.DB.RepoDigests(r.Context(), repoID, image)
if err != nil {
return err
}
if err := s.DB.DeleteImages(r.Context(), repoID, image); err != nil {
return err
}
for _, d := range digests {
s.removeUnreferenced(r, d)
}
return nil
}
// deleteRepoRow removes a repository row and its image files. The cascade
// drops the index rows; the files nothing else uses go afterwards. Every
// place that deletes a repo goes through here.
func (s *Server) deleteRepoRow(r *http.Request, repoID int64) error {
s.registryMu.Lock()
defer s.registryMu.Unlock()
digests, err := s.DB.RepoDigests(r.Context(), repoID, nil)
if err != nil {
return err
}
if err := s.DB.DeleteRepo(r.Context(), repoID); err != nil {
return err
}
for _, d := range digests {
s.removeUnreferenced(r, d)
}
return nil
}
Ainternal/web/registry.go
@@ -0,0 +1,735 @@
package web
import (
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"io"
"net/http"
"os"
"path/filepath"
"regexp"
"strconv"
"strings"
"time"
"github.com/go-chi/chi/v5"
"hearthforge/internal/db"
"hearthforge/internal/ratelimit"
)
// The registry implements the OCI Distribution Spec under /v2/. An image
// name is "<repo>" or "<repo>/<path>"; the first segment must be an existing
// repository. Blob and manifest bodies are content-addressed files under
// DATA_DIR/registry, the per-image index lives in SQLite.
//
// Admins push and delete. Pull access follows REGISTRY_PULL, except that
// images of private repositories are always admin-only.
const (
// maxManifestBytes bounds a manifest body. Real ones are a few KiB.
maxManifestBytes = 4 << 20
registryRealm = `Basic realm="Hearthforge registry"`
)
var (
digestRe = regexp.MustCompile(`^sha256:[a-f0-9]{64}$`)
uploadIDRe = regexp.MustCompile(`^[a-f0-9]{32}$`)
tagRe = regexp.MustCompile(`^[a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}$`)
imagePathRe = regexp.MustCompile(`^[a-z0-9]+(?:[._-][a-z0-9]+)*$`)
)
// registryAuthLimiter counts failed Basic auth attempts per IP. Successful
// pulls make one request per layer and must not count against it.
var registryAuthLimiter = ratelimit.New(10, time.Minute)
func (s *Server) registryRoutes(r chi.Router) {
r.HandleFunc("/v2", s.registry)
r.HandleFunc("/v2/", s.registry)
r.HandleFunc("/v2/*", s.registry)
}
// registryError writes the spec's JSON error envelope.
func registryError(w http.ResponseWriter, status int, code, msg string) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(map[string]any{
"errors": []map[string]string{{"code": code, "message": msg}},
})
}
// registryUser resolves Basic auth. ok is false when the header is missing
// or wrong. The limiter blocks an IP after repeated failures.
func (s *Server) registryUser(r *http.Request) (username string, isAdmin, ok bool) {
username, password, found := r.BasicAuth()
if !found {
return "", false, false
}
ip := ratelimit.ClientIP(r, s.Cfg.TrustedProxy)
if !s.Cfg.RateLimitDisabled && registryAuthLimiter.Blocked(ip) {
return "", false, false
}
if len(password) <= s.Cfg.MaxPasswordBytes {
hash, exists, err := s.DB.ActivePasswordHash(r.Context(), username)
if err == nil && exists {
if valid, err := db.VerifyPassword(hash, password); err == nil && valid {
return username, username == db.AdminUsername, true
}
}
}
if !s.Cfg.RateLimitDisabled {
registryAuthLimiter.Allow(ip)
}
return "", false, false
}
// challenge answers 401 with the Basic scheme so clients retry with
// credentials.
func challenge(w http.ResponseWriter) {
w.Header().Set("WWW-Authenticate", registryRealm)
registryError(w, http.StatusUnauthorized, "UNAUTHORIZED", "authentication required")
}
// registry dispatches one /v2/ request. Names may contain slashes, so the
// path is split on the fixed keywords instead of chi params.
func (s *Server) registry(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Docker-Distribution-API-Version", "registry/2.0")
_, isAdmin, authed := s.registryUser(r)
write := r.Method != http.MethodGet && r.Method != http.MethodHead
rest := strings.Trim(strings.TrimPrefix(r.URL.Path, "/v2"), "/")
if rest == "" {
// The version check doubles as the auth probe for clients.
if write || !s.Cfg.CanPullImages(false, authed, isAdmin) {
challenge(w)
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte("{}"))
return
}
var name, kind, ref string
switch {
case strings.Contains(rest, "/blobs/uploads/") || strings.HasSuffix(rest, "/blobs/uploads"):
i := strings.LastIndex(rest, "/blobs/uploads")
name, kind = rest[:i], "upload"
ref = strings.TrimPrefix(rest[i+len("/blobs/uploads"):], "/")
case strings.Contains(rest, "/blobs/"):
i := strings.LastIndex(rest, "/blobs/")
name, kind, ref = rest[:i], "blob", rest[i+len("/blobs/"):]
case strings.Contains(rest, "/manifests/"):
i := strings.LastIndex(rest, "/manifests/")
name, kind, ref = rest[:i], "manifest", rest[i+len("/manifests/"):]
case strings.HasSuffix(rest, "/tags/list"):
name, kind = strings.TrimSuffix(rest, "/tags/list"), "tags"
default:
registryError(w, http.StatusNotFound, "UNSUPPORTED", "unknown endpoint")
return
}
repo, image, ok := s.registryName(r, name)
// A private repo must look exactly like an unknown one to non-admins,
// or its name leaks through the status code.
if ok && repo.IsPrivate && !isAdmin {
ok = false
}
if !ok {
// Do not reveal whether the repo exists before auth.
if !authed && (write || !s.Cfg.CanPullImages(false, false, false)) {
challenge(w)
return
}
registryError(w, http.StatusNotFound, "NAME_UNKNOWN", "repository name not known to registry")
return
}
if write && !isAdmin {
if !authed {
challenge(w)
return
}
registryError(w, http.StatusForbidden, "DENIED", "only the admin may push")
return
}
if !write && !s.Cfg.CanPullImages(repo.IsPrivate, authed, isAdmin) {
if !authed {
challenge(w)
return
}
registryError(w, http.StatusForbidden, "DENIED", "pull access denied")
return
}
switch kind {
case "upload":
s.registryUpload(w, r, repo, image, ref)
case "blob":
s.registryBlob(w, r, repo, image, ref)
case "manifest":
s.registryManifest(w, r, repo, image, ref)
case "tags":
s.registryTags(w, r, repo, image)
}
}
// registryName maps "<repo>[/<path>]" to the repository row and image path.
// It reports false for an unknown repo or a path the spec would reject.
func (s *Server) registryName(r *http.Request, name string) (*db.Repo, string, bool) {
repoName, image, _ := strings.Cut(name, "/")
for _, seg := range strings.Split(image, "/") {
if seg != "" && !imagePathRe.MatchString(seg) {
return nil, "", false
}
// The path parser splits on these words, so they cannot be segments.
if seg == "blobs" || seg == "manifests" || seg == "tags" {
return nil, "", false
}
}
if image != "" && strings.Contains(image, "//") {
return nil, "", false
}
repo, err := s.DB.RepoByNameFold(r.Context(), repoName)
if err != nil || repo == nil {
return nil, "", false
}
return repo, image, true
}
// imageBase is the URL prefix of an image. Image names are lowercase on the
// wire, so the repo name is lowered even when the repository is not.
func imageBase(repo *db.Repo, image string) string {
return "/v2/" + strings.TrimSuffix(strings.ToLower(repo.Name)+"/"+image, "/")
}
// --- storage paths ---
func (s *Server) blobPath(digest string) string {
return filepath.Join(s.Cfg.RegistryDir(), "blobs", strings.Replace(digest, ":", "/", 1))
}
func (s *Server) uploadPath(id string) string {
return filepath.Join(s.Cfg.RegistryDir(), "uploads", id)
}
// --- blob uploads ---
func (s *Server) registryUpload(w http.ResponseWriter, r *http.Request, repo *db.Repo, image, id string) {
base := imageBase(repo, image)
switch {
case r.Method == http.MethodPost && id == "":
// A cross-repo mount request falls through to a normal upload, which
// the spec allows. The client then uploads the blob.
if digest := r.URL.Query().Get("digest"); digest != "" && r.URL.Query().Get("mount") == "" {
// Monolithic upload in one request.
tmp, err := s.newUpload()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if _, err := appendUpload(s.uploadPath(tmp), r.Body); err != nil {
_ = os.Remove(s.uploadPath(tmp))
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
s.finishUpload(w, r, repo, image, tmp, digest, base)
return
}
id, err := s.newUpload()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.Header().Set("Location", base+"/blobs/uploads/"+id)
w.Header().Set("Docker-Upload-UUID", id)
w.Header().Set("Range", "0-0")
w.WriteHeader(http.StatusAccepted)
case id == "" || !uploadIDRe.MatchString(id):
registryError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "upload not found")
case r.Method == http.MethodGet:
st, err := os.Stat(s.uploadPath(id))
if err != nil {
registryError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "upload not found")
return
}
w.Header().Set("Location", base+"/blobs/uploads/"+id)
w.Header().Set("Docker-Upload-UUID", id)
w.Header().Set("Range", rangeHeader(st.Size()))
w.WriteHeader(http.StatusNoContent)
case r.Method == http.MethodPatch:
path := s.uploadPath(id)
st, err := os.Stat(path)
if err != nil {
registryError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "upload not found")
return
}
// Chunks must arrive in order. A stated start that is not the
// current end means the client and server disagree on the state.
if cr := r.Header.Get("Content-Range"); cr != "" {
start, _, _ := strings.Cut(cr, "-")
if n, err := strconv.ParseInt(start, 10, 64); err != nil || n != st.Size() {
w.Header().Set("Location", base+"/blobs/uploads/"+id)
w.Header().Set("Range", rangeHeader(st.Size()))
registryError(w, http.StatusRequestedRangeNotSatisfiable, "BLOB_UPLOAD_INVALID", "chunk out of order")
return
}
}
n, err := appendUpload(path, r.Body)
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.Header().Set("Location", base+"/blobs/uploads/"+id)
w.Header().Set("Docker-Upload-UUID", id)
w.Header().Set("Range", rangeHeader(st.Size()+n))
w.WriteHeader(http.StatusAccepted)
case r.Method == http.MethodPut:
path := s.uploadPath(id)
if _, err := os.Stat(path); err != nil {
registryError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "upload not found")
return
}
if _, err := appendUpload(path, r.Body); err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
s.finishUpload(w, r, repo, image, id, r.URL.Query().Get("digest"), base)
case r.Method == http.MethodDelete:
if err := os.Remove(s.uploadPath(id)); err != nil {
registryError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "upload not found")
return
}
w.WriteHeader(http.StatusNoContent)
default:
registryError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "method not allowed")
}
}
func rangeHeader(size int64) string {
if size == 0 {
return "0-0"
}
return "0-" + strconv.FormatInt(size-1, 10)
}
// staleUploadAge is how long a partial upload may sit before it is removed.
const staleUploadAge = 24 * time.Hour
// sweepUploads removes upload files nobody finished. A client that
// disconnects mid-push never sends the final request, so nothing else
// would ever delete them.
func (s *Server) sweepUploads() {
entries, err := os.ReadDir(filepath.Dir(s.uploadPath("x")))
if err != nil {
return
}
cutoff := time.Now().Add(-staleUploadAge)
for _, e := range entries {
if info, err := e.Info(); err == nil && info.ModTime().Before(cutoff) {
_ = os.Remove(s.uploadPath(e.Name()))
}
}
}
// newUpload creates an empty upload file and returns its id.
func (s *Server) newUpload() (string, error) {
s.sweepUploads()
var b [16]byte
if _, err := rand.Read(b[:]); err != nil {
return "", err
}
id := hex.EncodeToString(b[:])
if err := os.MkdirAll(filepath.Dir(s.uploadPath(id)), 0o755); err != nil {
return "", err
}
f, err := os.OpenFile(s.uploadPath(id), os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
if err != nil {
return "", err
}
return id, f.Close()
}
// appendUpload appends the body and returns how many bytes it added.
func appendUpload(path string, body io.Reader) (int64, error) {
f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o600)
if err != nil {
return 0, err
}
n, err := io.Copy(f, body)
if err != nil {
f.Close()
return n, err
}
return n, f.Close()
}
// finishUpload verifies the digest, moves the file into the blob store, and
// links it to the image.
func (s *Server) finishUpload(w http.ResponseWriter, r *http.Request, repo *db.Repo, image, id, digest, base string) {
// Claim the file under a private name first. A late PATCH on the upload
// id then finds nothing, so the bytes hashed are the bytes stored.
path := s.uploadPath(id) + ".final"
if err := os.Rename(s.uploadPath(id), path); err != nil {
registryError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "upload not found")
return
}
if !digestRe.MatchString(digest) {
_ = os.Remove(path)
registryError(w, http.StatusBadRequest, "DIGEST_INVALID", "digest must be sha256:<hex>")
return
}
size, actual, err := fileDigest(path)
if err != nil {
_ = os.Remove(path)
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if actual != digest {
_ = os.Remove(path)
registryError(w, http.StatusBadRequest, "DIGEST_INVALID", "digest does not match uploaded content")
return
}
s.registryMu.Lock()
err = s.storeBlob(path, digest)
if err == nil {
err = s.DB.LinkBlob(r.Context(), repo.ID, image, digest, size)
}
s.registryMu.Unlock()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.Header().Set("Location", base+"/blobs/"+digest)
w.Header().Set("Docker-Content-Digest", digest)
w.WriteHeader(http.StatusCreated)
}
// storeBlob moves a verified file into place. An existing blob with the
// same digest has identical content, so the upload is simply dropped.
func (s *Server) storeBlob(src, digest string) error {
dst := s.blobPath(digest)
if _, err := os.Stat(dst); err == nil {
return os.Remove(src)
}
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
return err
}
return os.Rename(src, dst)
}
func fileDigest(path string) (int64, string, error) {
f, err := os.Open(path)
if err != nil {
return 0, "", err
}
defer f.Close()
h := sha256.New()
n, err := io.Copy(h, f)
if err != nil {
return 0, "", err
}
return n, "sha256:" + hex.EncodeToString(h.Sum(nil)), nil
}
// --- blobs ---
func (s *Server) registryBlob(w http.ResponseWriter, r *http.Request, repo *db.Repo, image, digest string) {
if !digestRe.MatchString(digest) {
registryError(w, http.StatusBadRequest, "DIGEST_INVALID", "digest must be sha256:<hex>")
return
}
linked, err := s.DB.BlobLinked(r.Context(), repo.ID, image, digest)
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if !linked {
registryError(w, http.StatusNotFound, "BLOB_UNKNOWN", "blob unknown to registry")
return
}
switch r.Method {
case http.MethodGet, http.MethodHead:
s.serveDigest(w, r, digest, "application/octet-stream")
case http.MethodDelete:
s.registryMu.Lock()
err = s.DB.UnlinkBlob(r.Context(), repo.ID, image, digest)
if err == nil {
s.removeUnreferenced(r, digest)
}
s.registryMu.Unlock()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.WriteHeader(http.StatusAccepted)
default:
registryError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "method not allowed")
}
}
// removeUnreferenced deletes the file behind digest once no image uses it.
// The caller holds registryMu. The lookup failing keeps the file; a stray
// file is cheaper than a broken pull.
func (s *Server) removeUnreferenced(r *http.Request, digest string) {
used, err := s.DB.DigestReferenced(r.Context(), digest)
if err == nil && !used {
_ = os.Remove(s.blobPath(digest))
}
}
// serveDigest streams a content-addressed file. ServeContent handles HEAD
// and Range requests.
func (s *Server) serveDigest(w http.ResponseWriter, r *http.Request, digest, contentType string) {
f, err := os.Open(s.blobPath(digest))
if err != nil {
registryError(w, http.StatusNotFound, "BLOB_UNKNOWN", "blob file missing")
return
}
defer f.Close()
st, err := f.Stat()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Docker-Content-Digest", digest)
// Content-Disposition keeps a browser from rendering a layer or
// manifest inline. The raw endpoint's sandbox CSP does not apply here.
w.Header().Set("Content-Disposition", "attachment")
http.ServeContent(w, r, "", st.ModTime(), f)
}
// --- manifests ---
// manifestRefs is the part of a manifest or index the registry checks.
type manifestRefs struct {
MediaType string `json:"mediaType"`
Config *struct {
Digest string `json:"digest"`
} `json:"config"`
Layers []struct {
Digest string `json:"digest"`
} `json:"layers"`
Manifests []struct {
Digest string `json:"digest"`
} `json:"manifests"`
}
func (s *Server) registryManifest(w http.ResponseWriter, r *http.Request, repo *db.Repo, image, ref string) {
isDigest := digestRe.MatchString(ref)
if !isDigest && !tagRe.MatchString(ref) {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "invalid reference")
return
}
switch r.Method {
case http.MethodPut:
s.putManifest(w, r, repo, image, ref, isDigest)
return
case http.MethodDelete:
if isDigest {
s.registryMu.Lock()
found, err := s.DB.DeleteManifest(r.Context(), repo.ID, image, ref)
if err == nil && found {
s.removeUnreferenced(r, ref)
}
s.registryMu.Unlock()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if !found {
registryError(w, http.StatusNotFound, "MANIFEST_UNKNOWN", "manifest unknown")
return
}
} else {
found, err := s.DB.DeleteTag(r.Context(), repo.ID, image, ref)
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if !found {
registryError(w, http.StatusNotFound, "MANIFEST_UNKNOWN", "tag unknown")
return
}
}
w.WriteHeader(http.StatusAccepted)
return
case http.MethodGet, http.MethodHead:
default:
registryError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "method not allowed")
return
}
var m *db.Manifest
var err error
if isDigest {
m, err = s.DB.ManifestByDigest(r.Context(), repo.ID, image, ref)
} else {
m, err = s.DB.ManifestByTag(r.Context(), repo.ID, image, ref)
}
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if m == nil {
registryError(w, http.StatusNotFound, "MANIFEST_UNKNOWN", "manifest unknown")
return
}
s.serveDigest(w, r, m.Digest, m.MediaType)
}
// putManifest stores a manifest after checking that everything it points
// at is already in this image. That is what makes a pull reliable.
func (s *Server) putManifest(w http.ResponseWriter, r *http.Request, repo *db.Repo, image, ref string, isDigest bool) {
body, err := io.ReadAll(io.LimitReader(r.Body, maxManifestBytes+1))
if err != nil {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "could not read body")
return
}
if len(body) > maxManifestBytes {
registryError(w, http.StatusRequestEntityTooLarge, "MANIFEST_INVALID", "manifest too large")
return
}
sum := sha256.Sum256(body)
digest := "sha256:" + hex.EncodeToString(sum[:])
if isDigest && ref != digest {
registryError(w, http.StatusBadRequest, "DIGEST_INVALID", "reference digest does not match body")
return
}
var refs manifestRefs
if err := json.Unmarshal(body, &refs); err != nil {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "manifest is not valid JSON")
return
}
mediaType, _, _ := strings.Cut(r.Header.Get("Content-Type"), ";")
mediaType = strings.TrimSpace(mediaType)
if mediaType == "" {
mediaType = refs.MediaType
}
if mediaType == "" {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "manifest has no media type")
return
}
// Every layer and config blob must be linked, every child manifest
// stored. Otherwise a client could pull a manifest whose parts 404.
var blobs []string
if refs.Config != nil {
blobs = append(blobs, refs.Config.Digest)
}
for _, l := range refs.Layers {
blobs = append(blobs, l.Digest)
}
for _, d := range blobs {
if !digestRe.MatchString(d) {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "unsupported digest "+d)
return
}
linked, err := s.DB.BlobLinked(r.Context(), repo.ID, image, d)
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if !linked {
registryError(w, http.StatusBadRequest, "MANIFEST_BLOB_UNKNOWN", "blob "+d+" not uploaded")
return
}
}
for _, c := range refs.Manifests {
if !digestRe.MatchString(c.Digest) {
registryError(w, http.StatusBadRequest, "MANIFEST_INVALID", "unsupported digest "+c.Digest)
return
}
child, err := s.DB.ManifestByDigest(r.Context(), repo.ID, image, c.Digest)
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if child == nil {
registryError(w, http.StatusBadRequest, "MANIFEST_BLOB_UNKNOWN", "manifest "+c.Digest+" not uploaded")
return
}
}
tag := ""
if !isDigest {
tag = ref
}
m := db.Manifest{Digest: digest, MediaType: mediaType}
s.registryMu.Lock()
err = s.writeBlob(digest, body)
if err == nil {
err = s.DB.PutManifest(r.Context(), repo.ID, image, m, tag, db.NowISO())
}
s.registryMu.Unlock()
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.Header().Set("Location", imageBase(repo, image)+"/manifests/"+digest)
w.Header().Set("Docker-Content-Digest", digest)
w.WriteHeader(http.StatusCreated)
}
// writeBlob stores small content (a manifest) by digest.
func (s *Server) writeBlob(digest string, body []byte) error {
dst := s.blobPath(digest)
if _, err := os.Stat(dst); err == nil {
return nil
}
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(dst), ".manifest-*")
if err != nil {
return err
}
if _, err := tmp.Write(body); err != nil {
tmp.Close()
_ = os.Remove(tmp.Name())
return err
}
if err := tmp.Close(); err != nil {
_ = os.Remove(tmp.Name())
return err
}
return os.Rename(tmp.Name(), dst)
}
// --- tags ---
func (s *Server) registryTags(w http.ResponseWriter, r *http.Request, repo *db.Repo, image string) {
if r.Method != http.MethodGet && r.Method != http.MethodHead {
registryError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "method not allowed")
return
}
tags, err := s.DB.ListTags(r.Context(), repo.ID, image)
if err != nil {
registryError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
// Pagination: `last` is exclusive, `n` caps the page.
if last := r.URL.Query().Get("last"); last != "" {
for len(tags) > 0 && tags[0] <= last {
tags = tags[1:]
}
}
if n, err := strconv.Atoi(r.URL.Query().Get("n")); err == nil && n >= 0 && n < len(tags) {
tags = tags[:n]
}
if tags == nil {
tags = []string{}
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]any{
"name": strings.TrimPrefix(imageBase(repo, image), "/v2/"),
"tags": tags,
})
}
Minternal/web/repos.go
@@ -422,7 +422,7 @@ func (s *Server) deleteRepo(w http.ResponseWriter, r *http.Request) {
http.Error(w, "Failed to delete repository", http.StatusInternalServerError)
return
}
if err := s.DB.DeleteRepo(r.Context(), repo.ID); err != nil {
if err := s.deleteRepoRow(r, repo.ID); err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
Minternal/web/routes.go
@@ -8,6 +8,7 @@ func (s *Server) routes(r chi.Router) {
r.Group(func(r chi.Router) {
r.Use(s.withSession)
s.gitRoutes(r)
s.registryRoutes(r)
s.authRoutes(r)
s.settingsRoutes(r)
s.avatarRoutes(r)
@@ -15,6 +16,7 @@ func (s *Server) routes(r chi.Router) {
s.issueRoutes(r)
s.patchRoutes(r)
s.releaseRoutes(r)
s.imageRoutes(r)
s.repoRoutes(r)
})
}
Minternal/web/server.go
@@ -7,6 +7,7 @@ import (
"net/http"
"net/url"
"strings"
"sync"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
@@ -40,6 +41,12 @@ type Server struct {
CI *ci.Runner
Git *gitcmd.Git
Patches *gitcmd.PatchCache
// registryMu orders registry file moves against index writes, so a
// delete cannot judge a file unreferenced while a push is still linking
// it. ponytail: one lock for the whole store, per-digest locks if pushes
// ever contend.
registryMu sync.Mutex
}
// Router builds the chi router with global middleware. Route groups are
@@ -115,13 +122,14 @@ func (s *Server) csrf(next http.Handler) http.Handler {
})
}
// bodyLimit caps request bodies at MaxUploadBytes. git push is exempt: it is
// admin-only behind Basic auth, streams to git's stdin, and SSH push has no
// cap either.
// bodyLimit caps request bodies at MaxUploadBytes. git push and registry
// blob uploads are exempt: both are admin-only behind Basic auth, and SSH
// push has no cap either.
func (s *Server) bodyLimit(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
isPush := r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/git-receive-pack")
if r.Body != nil && s.Cfg.MaxUploadBytes > 0 && !isPush {
isBlobUpload := strings.HasPrefix(r.URL.Path, "/v2/") && strings.Contains(r.URL.Path, "/blobs/uploads")
if r.Body != nil && s.Cfg.MaxUploadBytes > 0 && !isPush && !isBlobUpload {
r.Body = http.MaxBytesReader(w, r.Body, s.Cfg.MaxUploadBytes)
}
next.ServeHTTP(w, r)
Minternal/web/settings.go
@@ -490,7 +490,7 @@ func (s *Server) adminDropRepo(w http.ResponseWriter, r *http.Request) {
// Re-check on disk so a repo restored after the page rendered survives.
for _, m := range missing {
if m.ID == id {
if err := s.DB.DeleteRepo(r.Context(), id); err != nil {
if err := s.deleteRepoRow(r, id); err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
@@ -508,7 +508,7 @@ func (s *Server) adminDropAllRepos(w http.ResponseWriter, r *http.Request) {
return
}
for _, m := range missing {
if err := s.DB.DeleteRepo(r.Context(), m.ID); err != nil {
if err := s.deleteRepoRow(r, m.ID); err != nil {
http.Error(w, "Database error", http.StatusInternalServerError)
return
}
Minternal/web/views/ci.go
@@ -127,6 +127,7 @@ var ciStepOptions = [][2]string{
{"always", "run even after an earlier step failed"},
{"warn_on_fail", "step warns instead of failing; the run reports warning"},
{"clear", "re-extract a clean checkout before the step"},
{"engine_socket", "mount the Docker/Podman socket for image builds; the server must set CI_ENGINE_SOCKET=1"},
{"timeout", "per-step timeout in seconds"},
}
@@ -229,7 +230,7 @@ func CiHistory(cfg *config.Config, user *db.SessionUser, repo *db.Repo, runs []C
g.If(isRunning, Meta(g.Attr("http-equiv", "refresh"), Content("4"))),
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "ci", user),
RepoNav(cfg, repo, "ci", user),
g.If(success != "", P(Class("form-success"), g.Text(success))),
g.If(errMsg != "", P(Class("form-error"), g.Text(errMsg))),
Div(Class("list-header"),
@@ -365,7 +366,7 @@ func CiRunDetail(cfg *config.Config, user *db.SessionUser, repo *db.Repo, run *d
g.If(isActive && autoRefresh, Meta(g.Attr("http-equiv", "refresh"), Content("3"))),
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "ci", user),
RepoNav(cfg, repo, "ci", user),
Div(Class("ci-run-header"),
Div(
Ainternal/web/views/images.go
@@ -0,0 +1,100 @@
package views
import (
"strings"
"hearthforge/internal/config"
"hearthforge/internal/db"
"hearthforge/internal/util"
g "maragu.dev/gomponents"
. "maragu.dev/gomponents/html"
)
// ImageList renders the Images tab: one block per image path with its tags.
// host is what a client puts in front of the image name.
func ImageList(cfg *config.Config, user *db.SessionUser, repo *db.Repo,
tags []db.ImageTag, sizes map[string]int64,
) g.Node {
host := cfg.PublicHost
isAdmin := user != nil && user.IsAdmin
// Group tags by image path, keeping the sorted order from the query.
var order []string
byImage := map[string][]db.ImageTag{}
for _, t := range tags {
if _, seen := byImage[t.Image]; !seen {
order = append(order, t.Image)
}
byImage[t.Image] = append(byImage[t.Image], t)
}
// Clients lowercase image names, so the tab shows the name they use.
lowerRepo := strings.ToLower(repo.Name)
fullName := func(image string) string { return strings.TrimSuffix(lowerRepo+"/"+image, "/") }
return Layout(Page{Title: "Images — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(cfg, repo, "images", user),
Div(Class("list-header"),
H2(Class("list-heading"), g.Text("Images")),
g.If(isAdmin && len(order) > 0, Details(Class("confirm-details"),
Summary(Class("btn btn-sm btn-danger"), g.Text("Delete all")),
Div(Class("confirm-popup"),
g.Text("Delete every image and tag of this repository?"),
Form(Method("POST"), Action("/"+repo.Name+"/images/delete-all"), Class("inline-form"),
Button(Type("submit"), Class("btn btn-sm btn-danger"), g.Text("Yes, delete all")),
),
),
)),
),
g.If(len(order) == 0, Div(Class("empty-state"),
P(g.Text("No images yet.")),
g.If(isAdmin, P(Class("text-muted"),
g.Text("Push one with "),
Code(g.Text("docker push "+host+"/"+lowerRepo+":TAG")),
g.Text("."))),
)),
g.If(len(order) > 0, Ul(Class("issue-list"),
g.Map(order, func(image string) g.Node {
return Li(Class("issue-item"),
Div(Class("release-item-header"),
Div(Class("release-item-main"),
Code(Class("release-item-title"), g.Text(host+"/"+fullName(image))),
Div(Class("release-item-meta"),
Span(g.Text(countLabel(len(byImage[image]), "tag", "tags"))),
Span(g.Text(util.FormatBytes(sizes[image]))),
),
),
g.If(isAdmin, Details(Class("confirm-details"),
Summary(Class("btn btn-sm btn-danger"), g.Text("Delete image")),
Div(Class("confirm-popup"),
g.Text("Delete every tag and layer of "+fullName(image)+"?"),
Form(Method("POST"), Action("/"+repo.Name+"/images/delete"), Class("inline-form"),
Input(Type("hidden"), Name("image"), Value(image)),
Button(Type("submit"), Class("btn btn-sm btn-danger"), g.Text("Yes, delete")),
),
),
)),
),
Ul(Class("image-tags"),
g.Map(byImage[image], func(t db.ImageTag) g.Node {
return Li(Class("image-tag"),
Span(Class("badge"), g.Text(t.Tag)),
// Digests passed the registry's regex, so 19 chars is "sha256:" plus 12 hex.
Code(Class("image-digest"), Title(t.Digest), g.Text(t.Digest[:19])),
g.El("time", Class("text-muted"), DateTime(t.UpdatedAt), g.Text(util.FormatDate(t.UpdatedAt))),
g.If(isAdmin, Form(Method("POST"), Action("/"+repo.Name+"/images/delete"),
Class("inline-form image-tag-delete"),
Input(Type("hidden"), Name("image"), Value(image)),
Input(Type("hidden"), Name("tag"), Value(t.Tag)),
Button(Type("submit"), Class("btn btn-xs"), g.Text("Delete tag")),
)),
)
}),
),
)
}),
)),
),
)
}
Minternal/web/views/issues.go
@@ -53,7 +53,7 @@ func IssueList(cfg *config.Config, user *db.SessionUser, repo *db.Repo, issues [
return Layout(Page{Title: "Issues — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "issues", user),
RepoNav(cfg, repo, "issues", user),
Div(Class("list-header"),
Div(Class("list-header-tabs"),
tab("open", "Open"),
@@ -128,7 +128,7 @@ func NewIssue(cfg *config.Config, user *db.SessionUser, repo *db.Repo, errMsg, t
return Layout(Page{Title: "New issue — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "issues", user),
RepoNav(cfg, repo, "issues", user),
H2(Class("section-title"), g.Text("New issue")),
g.If(errMsg != "", P(Class("form-error"), g.Text(errMsg))),
Form(Method("POST"), Action("/"+repo.Name+"/issues"), Class("form-card"),
@@ -183,7 +183,7 @@ func IssueDetail(cfg *config.Config, user *db.SessionUser, repo *db.Repo, issue
return Layout(Page{Title: issue.Title + " — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "issues", user),
RepoNav(cfg, repo, "issues", user),
Div(Class("issue-detail"),
Div(Class("issue-detail-header"),
Span(Class("issue-number"), g.Text("#"+strconv.FormatInt(issue.Number, 10))),
Minternal/web/views/patches.go
@@ -36,7 +36,7 @@ func PatchList(cfg *config.Config, user *db.SessionUser, repo *db.Repo, patches
return Layout(Page{Title: "Patches — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "patches", user),
RepoNav(cfg, repo, "patches", user),
Div(Class("list-header"),
Div(Class("list-header-tabs"),
tab("open", "Open"),
@@ -110,7 +110,7 @@ func NewPatch(cfg *config.Config, user *db.SessionUser, repo *db.Repo, errMsg, t
return Layout(Page{Title: "New patch — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "patches", user),
RepoNav(cfg, repo, "patches", user),
H2(Class("section-title"), g.Text("Upload patch")),
g.If(errMsg != "", P(Class("form-error"), g.Text(errMsg))),
Form(Method("POST"), Action("/"+repo.Name+"/patches"),
@@ -175,7 +175,7 @@ func PatchDetail(cfg *config.Config, user *db.SessionUser, repo *db.Repo, patch
return Layout(Page{Title: patch.Title + " — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "patches", user),
RepoNav(cfg, repo, "patches", user),
Div(Class("issue-detail"),
Div(Class("issue-detail-header"),
Minternal/web/views/releases.go
@@ -45,7 +45,7 @@ func ReleaseList(cfg *config.Config, user *db.SessionUser, repo *db.Repo,
return Layout(Page{Title: "Releases — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "releases", user),
RepoNav(cfg, repo, "releases", user),
Div(Class("list-header"),
H2(Class("list-heading"), g.Text("Releases")),
g.If(user != nil && user.IsAdmin,
@@ -128,7 +128,7 @@ func NewRelease(cfg *config.Config, user *db.SessionUser, repo *db.Repo,
return Layout(Page{Title: "New Release — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "releases", user),
RepoNav(cfg, repo, "releases", user),
Div(Class("form-page"),
H2(Class("page-title"), g.Text("New Release")),
g.If(errMsg != "", Div(Class("form-error"), g.Text(errMsg))),
@@ -205,7 +205,7 @@ func ReleaseDetail(cfg *config.Config, user *db.SessionUser, repo *db.Repo, rele
return Layout(Page{Title: release.Name + " — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "releases", user),
RepoNav(cfg, repo, "releases", user),
Div(Class("release-detail"),
Div(Class("release-header"),
Div(Class("release-item-header"),
Minternal/web/views/repos.go
@@ -39,7 +39,7 @@ func CommitLog(cfg *config.Config, user *db.SessionUser, repo *db.Repo, logRef s
return Layout(Page{Title: "Commits — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "commits", user),
RepoNav(cfg, repo, "commits", user),
Div(Class("commits-header"),
H2(Class("section-title"), g.Text("Commits")),
BranchSelector(repo.Name, branches, tags, logRef, "commits", ""),
@@ -111,7 +111,7 @@ func CommitDetail(cfg *config.Config, user *db.SessionUser, repo *db.Repo, sha s
return Layout(Page{Title: short + " — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "commits", user),
RepoNav(cfg, repo, "commits", user),
Div(Class("commit-page-top"),
A(Href("/"+repo.Name+"/commits/"+EscapePath(repo.DefaultBranch)), Class("btn btn-sm"),
g.Text("← Back to log")),
@@ -184,7 +184,7 @@ func BranchList(cfg *config.Config, user *db.SessionUser, repo *db.Repo, branche
return Layout(Page{Title: "Branches — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "branches", user),
RepoNav(cfg, repo, "branches", user),
g.If(success != "", P(Class("form-success"), g.Text(success))),
g.If(errMsg != "", P(Class("form-error"), g.Text(errMsg))),
Div(Class("list-header"),
@@ -275,7 +275,7 @@ func TagList(cfg *config.Config, user *db.SessionUser, repo *db.Repo, tags []git
return Layout(Page{Title: "Tags — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "tags", user),
RepoNav(cfg, repo, "tags", user),
g.If(success != "", P(Class("form-success"), g.Text(success))),
g.If(errMsg != "", P(Class("form-error"), g.Text(errMsg))),
Div(Class("list-header"),
Minternal/web/views/repos_files.go
@@ -180,7 +180,7 @@ func RepoHome(cfg *config.Config, user *db.SessionUser, repo *db.Repo, entries [
RepoHeader(repo),
g.If(deref(repo.Description) != "",
P(Class("repo-description"), g.Text(deref(repo.Description)))),
RepoNav(repo, "code", user),
RepoNav(cfg, repo, "code", user),
g.If(!hasContent, Div(Class("empty-state"),
H2(g.Text("This repository is empty.")),
P(g.Text("Push your first commit to get started:")),
@@ -230,7 +230,7 @@ func FileTree(cfg *config.Config, user *db.SessionUser, repo *db.Repo, treeRef,
return Layout(Page{Title: title, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "code", user),
RepoNav(cfg, repo, "code", user),
Div(Class("tree-toolbar"),
BranchSelector(repo.Name, branches, tags, treeRef, "tree", subpath),
g.If(slicesContains(branches, treeRef) && user != nil && user.IsAdmin,
@@ -259,7 +259,7 @@ func FileBlob(cfg *config.Config, user *db.SessionUser, repo *db.Repo, blobRef,
return Layout(Page{Title: repo.Name + "/" + filePath, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "code", user),
RepoNav(cfg, repo, "code", user),
g.If(errMsg != "", P(Class("form-error"), g.Text(errMsg))),
breadcrumb(repo.Name, blobRef, filePath, true),
Div(Class("file-blob-header"),
@@ -337,7 +337,7 @@ func FileEdit(cfg *config.Config, user *db.SessionUser, repo *db.Repo, editRef,
return Layout(Page{Title: "Edit " + repo.Name + "/" + filePath, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "code", user),
RepoNav(cfg, repo, "code", user),
breadcrumb(repo.Name, editRef, filePath, true),
P(Class("form-hint"),
g.Text(`WARNING: Line endings are normalized to LF (\n) on save.`)),
@@ -389,7 +389,7 @@ func NewFileForm(cfg *config.Config, user *db.SessionUser, repo *db.Repo, treeRe
return Layout(Page{Title: "New file — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "code", user),
RepoNav(cfg, repo, "code", user),
g.If(errMsg != "", P(Class("form-error"), g.Text(errMsg))),
Form(Method("POST"), Action("/"+repo.Name+"/new-file/"+EscapePath(treeRef)),
Div(Class("file-blob-header"),
Minternal/web/views/repos_shell.go
@@ -126,7 +126,7 @@ func RepoHeader(repo *db.Repo) g.Node {
// RepoNav renders the repository tab bar. active is the key of the current tab,
// one of code, branches, tags, commits, issues, patches, releases, ci, settings.
func RepoNav(repo *db.Repo, active string, user *db.SessionUser) g.Node {
func RepoNav(cfg *config.Config, repo *db.Repo, active string, user *db.SessionUser) g.Node {
tabs := []struct{ key, label, href string }{
{"code", "Code", "/" + repo.Name},
{"branches", "Branches", "/" + repo.Name + "/branches"},
@@ -137,6 +137,10 @@ func RepoNav(repo *db.Repo, active string, user *db.SessionUser) g.Node {
{"releases", "Releases", "/" + repo.Name + "/releases"},
{"ci", "Pipelines", "/" + repo.Name + "/ci"},
}
// The Images tab follows the registry pull rules.
if cfg.CanPullImages(repo.IsPrivate, user != nil, user != nil && user.IsAdmin) {
tabs = append(tabs, struct{ key, label, href string }{"images", "Images", "/" + repo.Name + "/images"})
}
tabClass := func(key string) string {
if active == key {
return "repo-tab active"
@@ -216,7 +220,7 @@ func RepoSettings(cfg *config.Config, user *db.SessionUser, repo *db.Repo, branc
return Layout(Page{Title: "Settings — " + repo.Name, User: user, Cfg: cfg},
Div(Class("container"),
RepoHeader(repo),
RepoNav(repo, "settings", user),
RepoNav(cfg, repo, "settings", user),
g.If(success != "", P(Class("form-success"), g.Text(success))),
g.If(errMsg != "", P(Class("form-error"), g.Text(errMsg))),
Form(Method("POST"), Action("/"+repo.Name+"/settings"), Class("form-card"),
Mweb/static/assets/css/components.css
@@ -1349,6 +1349,30 @@
margin-left: auto;
}
/* --- Images tab --- */
.image-tags {
list-style: none;
margin: var(--space-2) 0 0;
padding: 0;
display: flex;
flex-direction: column;
gap: var(--space-1);
}
.image-tag {
display: flex;
flex-wrap: wrap;
align-items: center;
gap: var(--space-2) var(--space-3);
font-size: var(--text-sm);
}
.image-digest {
color: var(--color-text-muted);
font-size: var(--text-xs);
}
.image-tag-delete {
margin-left: auto;
}
/* --- Danger zone --- */
.danger-zone {
margin-top: var(--space-8);
Mweb/static/assets/hearthforge-ci-template.toml
@@ -2,6 +2,11 @@
# Place this file at .hearthforge-ci.toml in your repository root.
image = "docker.io/debian:stable"
# Tools the image must provide: a POSIX shell at /bin/sh (or `shell` below),
# `sleep` for the container's main process, and `rm` + `mkdir` when a step
# uses `clear`. Any image with busybox or coreutils has them. Hearthforge does
# the checkout, [[copy]], and publish_* archives itself, so no git, tar, gzip,
# zstd or zip is needed inside the image.
work_dir = "/ci/build"
# Must be absolute. The image needs no git. HearthForge exports the commit
# and uploads it. No .git reaches the container, so `git describe` needs a
@@ -73,13 +78,14 @@ run_sh = "make -C project dist"
# Publish artifacts — these can appear in any step
# publish_file: copy a single file directly
# publish_file = ["/ci/build/project/dist/my-binary"]
# publish_gzip: create a .tar.gz archive (requires tar in image)
# Archives are built by Hearthforge on the host; the image needs no tools.
# publish_gzip: create a .tar.gz archive
# publish_gzip = ["/ci/build/project/dist/"]
# publish_tar: create a plain .tar archive
# publish_tar = ["/ci/build/project/dist/"]
# publish_zip: create a .zip archive (requires zip in image)
# publish_zip: create a .zip archive (symlinks are dropped)
# publish_zip = ["/ci/build/project/dist/"]
# publish_zstd: create a .tar.zst archive (requires tar + zstd in image)
# publish_zstd: create a .tar.zst archive
# publish_zstd = ["/ci/build/project/dist/"]
[[steps]]
@@ -87,3 +93,14 @@ name = "cleanup"
# always runs the step even when an earlier one failed.
always = true
run_sh = "rm -rf /ci/build/scratch"
# [[steps]]
# name = "image"
# # engine_socket gives this step the host engine. Needs CI_ENGINE_SOCKET=1 on
# # the server. $CI_REGISTRY is "<host>/<repo>" on the built-in registry.
# engine_socket = true
# run_sh = """
# echo "$REGISTRY_PASSWORD" | docker login "${CI_REGISTRY%%/*}" -u admin --password-stdin
# docker build -t "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA" project
# docker push "$CI_REGISTRY:$CI_COMMIT_SHORT_SHA"
# """