manage yt-dlp and a JS runtime; move the image to Debian
yt-dlp releases roughly weekly and the image installed it at build time, so every release meant an image rebuild. VidArchive now installs and updates it itself. - Tools have two owners. Without VIDARCHIVE_YTDLP_PATH the binary is managed: installed into <data>/bin, updated from the Settings page or a daily check. With it set the operator owns the binary and VidArchive leaves it alone. VIDARCHIVE_DENO_PATH works the same way. This changes the default, which used to be "yt-dlp" and resolved through PATH. Set VIDARCHIVE_YTDLP_PATH=yt-dlp to keep that behaviour. - Updating delegates to `yt-dlp -U` and `deno upgrade`. Both already check the version, verify the download and swap the binary by rename, so none of that is reimplemented. The rename leaves a running download on the old inode, so updating mid-download is safe. - Nothing detects the C library. Each architecture lists candidate assets and a candidate is run once before it is installed, so a glibc build on musl fails to exec and the next one is tried. A loader glob was tried first and misfired on hosts carrying a musl cross-toolchain beside glibc. - Downloads are verified against the release's SHA-256 sums and fail closed. The binary is staged, trial-run, then renamed into place, so a candidate that fails never lands. - A JS runtime (deno) covers sites that answer with a JavaScript challenge. It is off by default at ~130 MB unpacked. yt-dlp finds it because main prepends the managed directory to PATH, which every child inherits. - VIDARCHIVE_UPDATE_EXTERNAL_TOOLS grants update rights without install rights. It is the way out for an architecture with no mapped build: install the binary by hand once, then manage it from the UI. VidArchive still never writes over a path the operator chose. - The runtime image moves from Alpine to debian-slim, because deno ships glibc builds only. Python and the pip venv drop out with it. - toolVersions owns its probe deadline now. It holds the version cache mutex across the probe, so a caller without a deadline would block /healthz past the timeout that endpoint was written around. Tests cover the candidate fallback, a failed checksum, a missing deno checksum, an unexecutable binary being replaced, and the daily update gate. TestLiveInstall runs the real downloads behind VIDARCHIVE_ONLINE_TESTS=1, so a renamed or dropped release asset fails there instead of on a user's first start.
MContainerfile
@@ -20,16 +20,16 @@ FROM scratch AS prebuilt
COPY ci-bin/vidarchive /vidarchive
# ── runtime ──────────────────────────────────────────────────────────────────
FROM alpine:3.24
# Debian, not Alpine: the deno JS runtime ships glibc builds only. VidArchive
# downloads yt-dlp and deno into /data/bin at first start, so no Python here.
FROM debian:trixie-slim
# re-declare: args set before FROM do not carry into stages
ARG BIN_STAGE
RUN apk --no-cache add ca-certificates ffmpeg python3 py3-pip \
&& python3 -m venv /opt/ytdlp \
&& /opt/ytdlp/bin/pip install --no-cache-dir --upgrade yt-dlp
ENV PATH="/opt/ytdlp/bin:${PATH}"
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates ffmpeg curl \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
@@ -44,9 +44,9 @@ VOLUME ["/data"]
EXPOSE 8080
# /healthz reports 503 when the database is unreachable. start-period covers
# migrations on first boot so they don't count as failures.
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
CMD wget -q -O /dev/null "http://127.0.0.1:${VIDARCHIVE_PORT}/healthz" || exit 1
# migrations and the first yt-dlp download so they don't count as failures.
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
CMD curl -fsS -o /dev/null "http://127.0.0.1:${VIDARCHIVE_PORT}/healthz" || exit 1
# exec form, so the app is PID 1 and gets SIGTERM directly — it needs that for a
# graceful shutdown (drain requests, kill yt-dlp children, checkpoint the DB).
MREADME.md
@@ -13,10 +13,12 @@ templates. No JavaScript, no build step for the front end.
## Requirements
- `yt-dlp` on `PATH` (or set `VIDARCHIVE_YTDLP_PATH`)
- `ffmpeg` and `ffprobe` for thumbnails, subtitle conversion, and media probing
- Go 1.26+ to build from source
yt-dlp is downloaded and kept up to date by VidArchive itself. See
[Managed tools](#managed-tools).
Missing tools are reported at startup and on `/healthz`; the app still starts.
## Run
@@ -29,12 +31,9 @@ htpasswd -bnBC 12 "" 'your-password' | tr -d ':\n' # paste the output into .en
docker compose up -d # or: podman-compose up -d
```
The compose file mounts `./data` and publishes port 8080. The image installs
yt-dlp at build time, so rebuild to update it:
```sh
docker compose build --pull --no-cache vidarchive
```
The compose file mounts `./data` and publishes port 8080. yt-dlp is downloaded
into `./data/bin` on first start and updated from the Settings page, so the
image does not have to be rebuilt for a new yt-dlp release.
From source:
@@ -57,7 +56,10 @@ embedded in the binary.
| `VIDARCHIVE_DB_PATH` | `<data>/vidarchive.db` | SQLite database |
| `VIDARCHIVE_LIBRARY_DIR` | `<data>/library` | Imported media |
| `VIDARCHIVE_TEMP_DIR` | `<data>/temp` | Download scratch space |
| `VIDARCHIVE_YTDLP_PATH` | `yt-dlp` | yt-dlp binary |
| `VIDARCHIVE_BIN_DIR` | `<data>/bin` | Where managed tools are installed |
| `VIDARCHIVE_YTDLP_PATH` | — | yt-dlp binary; setting it turns off management |
| `VIDARCHIVE_DENO_PATH` | — | deno binary; setting it turns off management |
| `VIDARCHIVE_UPDATE_EXTERNAL_TOOLS` | `0` | Let VidArchive update a tool it did not install |
| `VIDARCHIVE_FFMPEG_PATH` | `ffmpeg` | ffmpeg binary |
| `VIDARCHIVE_FFPROBE_PATH` | `ffprobe` | ffprobe binary |
| `VIDARCHIVE_WORKERS` | `2` | Concurrent downloads (minimum 1) |
@@ -68,6 +70,31 @@ embedded in the binary.
| `VIDARCHIVE_USERNAME` | — | **Required.** Login user |
| `VIDARCHIVE_PASSWORD_HASH` | — | **Required.** bcrypt hash of that user's password |
## Managed tools
yt-dlp and the optional JS runtime have three modes.
| `VIDARCHIVE_YTDLP_PATH` | `VIDARCHIVE_UPDATE_EXTERNAL_TOOLS` | Mode | Behaviour |
| --- | --- | --- | --- |
| unset | — | managed | Installed to `<bin>/yt-dlp` on first start, updated by VidArchive |
| set | `0` | external | The path is used as given; the binary is never touched |
| set | `1` | external, updates enabled | You install it once, VidArchive updates it from then on |
`VIDARCHIVE_DENO_PATH` works the same way, and the opt-in covers both tools.
Use the third mode on a platform VidArchive has no build for. To keep the old
behaviour of picking yt-dlp up from `PATH`, set `VIDARCHIVE_YTDLP_PATH=yt-dlp`.
**Updating.** Settings has an *Update Now* button and a daily auto-update
checkbox. Both run the tool's own updater (`yt-dlp -U`, `deno upgrade`).
**JS runtime.** Some sites answer with a JavaScript challenge that yt-dlp
cannot solve alone. Enable *Install a JS runtime* in Settings to download deno.
At around 130 MB unpacked it is off by default.
> [!note]
> Managed builds cover Linux amd64 and arm64, glibc and musl. Deno has no musl
> build, so a musl host must supply its own and set `VIDARCHIVE_DENO_PATH`.
## Authentication
Every route needs a login. Only `/login`, `/healthz` and the static assets are
Mcmd/vidarchive/main.go
@@ -17,6 +17,7 @@ import (
"vidarchive/internal/repository"
"vidarchive/internal/server"
"vidarchive/internal/service"
"vidarchive/internal/tools"
"vidarchive/internal/worker"
)
@@ -36,12 +37,15 @@ func main() {
cfg.SetupLogging()
checkCredentials(cfg)
// Before checkDependencies, so the lookup sees the managed directory too.
addToolsToPath(cfg.BinDir)
checkDependencies(cfg)
for _, dir := range []string{
cfg.DataDir,
cfg.LibraryDir,
cfg.TempDir,
cfg.BinDir,
filepath.Join(cfg.DataDir, "archives"),
} {
if err := os.MkdirAll(dir, 0o755); err != nil {
@@ -65,13 +69,16 @@ func main() {
subscriptionSvc := service.NewSubscriptionService(subscriptionRepo, cfg)
downloadSvc := service.NewDownloadService(downloadRepo, librarySvc, presetSvc, settingsSvc, subscriptionSvc, cfg)
toolMgr := tools.New(cfg)
installTools(ctx, toolMgr, settingsSvc)
workerPool := worker.New(downloadSvc, cfg.Workers)
workerPool.Start()
scheduler := worker.NewScheduler(subscriptionSvc, downloadSvc, workerPool, time.Duration(cfg.SchedulerInterval)*time.Second)
scheduler := worker.NewScheduler(subscriptionSvc, downloadSvc, settingsSvc, toolMgr, workerPool, time.Duration(cfg.SchedulerInterval)*time.Second)
scheduler.Start()
h, err := handler.New(cfg, presetSvc, downloadSvc, librarySvc, settingsSvc, subscriptionSvc, workerPool)
h, err := handler.New(cfg, presetSvc, downloadSvc, librarySvc, settingsSvc, subscriptionSvc, workerPool, toolMgr)
if err != nil {
fatal("failed to initialize handler", "err", err)
}
@@ -102,6 +109,36 @@ func main() {
}
}
// addToolsToPath puts the managed directory on PATH for VidArchive and every
// child it starts, because yt-dlp finds its JS runtime there.
func addToolsToPath(binDir string) {
path := binDir
// A trailing separator would leave an empty element, which POSIX reads as
// the working directory.
if old := os.Getenv("PATH"); old != "" {
path += string(os.PathListSeparator) + old
}
if err := os.Setenv("PATH", path); err != nil {
slog.Error("failed to add the managed tool directory to PATH", "dir", binDir, "err", err)
}
}
// installTools fetches any missing managed binary in the background: a 40 MB
// download must not hold up the listener. A download queued before it finishes
// fails the same way a missing yt-dlp always has.
func installTools(ctx context.Context, toolMgr *tools.Manager, settingsSvc *service.SettingsService) {
settings, err := settingsSvc.GetAll()
if err != nil {
slog.Error("failed to read settings for tool install", "err", err)
return
}
go func() {
if err := toolMgr.Ensure(ctx, settings.JSRuntimeEnabled); err != nil {
slog.Error("tool install failed", "err", err)
}
}()
}
// fatal exists because slog has no Fatal, and log.Fatalf would report at info
// level once slog owns the log package.
func fatal(msg string, args ...any) {
@@ -127,14 +164,23 @@ func checkCredentials(cfg *config.Config) {
// checkDependencies warns without aborting, so a missing yt-dlp or ffmpeg shows
// up at startup rather than as an opaque per-download failure later.
func checkDependencies(cfg *config.Config) {
deps := []struct{ label, path string }{
{"yt-dlp", cfg.YTDLPPath},
{"ffmpeg", cfg.FFmpegPath},
{"ffprobe", cfg.FFprobePath},
deps := []struct {
label string
path string
managed bool
}{
{"yt-dlp", cfg.YTDLPPath, cfg.YTDLPManaged},
{"ffmpeg", cfg.FFmpegPath, false},
{"ffprobe", cfg.FFprobePath, false},
}
for _, dep := range deps {
if _, err := exec.LookPath(dep.path); err != nil {
slog.Warn("dependency not found in PATH, related features will fail until it is installed", "dependency", dep.label, "path", dep.path)
if _, err := exec.LookPath(dep.path); err == nil {
continue
}
if dep.managed {
slog.Info("managed dependency missing, a download will be attempted", "dependency", dep.label, "path", dep.path)
continue
}
slog.Warn("dependency not found in PATH, related features will fail until it is installed", "dependency", dep.label, "path", dep.path)
}
}
Mcompose.yml
@@ -1,7 +1,7 @@
services:
vidarchive:
# yt-dlp is installed when the image is built. Rebuild the image to get a
# newer one: `docker compose build --pull --no-cache vidarchive`.
# yt-dlp is downloaded into ./data/bin on first start and updated from the
# Settings page. The image does not carry it.
build:
context: .
dockerfile: Containerfile
@@ -20,8 +20,11 @@ services:
- VIDARCHIVE_SCHEDULER_INTERVAL=60
# Uncomment to run behind HTTPS proxy:
# - VIDARCHIVE_BASE_URL=https://vidarchive.example.com
# Uncomment to use a custom yt-dlp path:
# Uncomment to manage yt-dlp yourself; VidArchive then never touches it.
# - VIDARCHIVE_YTDLP_PATH=/usr/bin/yt-dlp
# Add this to still let VidArchive run the updater of a binary you
# installed yourself, e.g. on an architecture it has no build for.
# - VIDARCHIVE_UPDATE_EXTERNAL_TOOLS=1
restart: unless-stopped
# Give in-flight downloads and the database checkpoint time to finish before
# the container is killed; the default 10s can cut a shutdown short.
Minternal/config/config.go
@@ -10,42 +10,58 @@ import (
)
type Config struct {
Port int
DataDir string
DBPath string
LibraryDir string
TempDir string
YTDLPPath string
FFmpegPath string
FFprobePath string
BaseURL string
Workers int
SchedulerInterval int
LogLevel string
LogFormat string
Username string
PasswordHash string
Port int
DataDir string
DBPath string
LibraryDir string
TempDir string
BinDir string
YTDLPPath string
YTDLPManaged bool
DenoPath string
DenoManaged bool
// UpdateExternalTools lets VidArchive run the updater of a tool it did not
// install. It is the escape hatch for a platform with no mapped release
// build: install the binary once by hand, then manage it from here.
UpdateExternalTools bool
FFmpegPath string
FFprobePath string
BaseURL string
Workers int
SchedulerInterval int
LogLevel string
LogFormat string
Username string
PasswordHash string
}
func New() *Config {
dataDir := getEnv("VIDARCHIVE_DATA_DIR", "./data")
binDir := getEnv("VIDARCHIVE_BIN_DIR", filepath.Join(dataDir, "bin"))
ytdlpPath, ytdlpManaged := managedPath("VIDARCHIVE_YTDLP_PATH", binDir, "yt-dlp")
denoPath, denoManaged := managedPath("VIDARCHIVE_DENO_PATH", binDir, "deno")
return &Config{
Port: getEnvInt("VIDARCHIVE_PORT", 8080, 1),
DataDir: dataDir,
DBPath: getEnv("VIDARCHIVE_DB_PATH", filepath.Join(dataDir, "vidarchive.db")),
LibraryDir: getEnv("VIDARCHIVE_LIBRARY_DIR", filepath.Join(dataDir, "library")),
TempDir: getEnv("VIDARCHIVE_TEMP_DIR", filepath.Join(dataDir, "temp")),
YTDLPPath: getEnv("VIDARCHIVE_YTDLP_PATH", "yt-dlp"),
FFmpegPath: getEnv("VIDARCHIVE_FFMPEG_PATH", "ffmpeg"),
FFprobePath: getEnv("VIDARCHIVE_FFPROBE_PATH", "ffprobe"),
BaseURL: getEnv("VIDARCHIVE_BASE_URL", ""),
Workers: getEnvInt("VIDARCHIVE_WORKERS", 2, 1),
SchedulerInterval: getEnvInt("VIDARCHIVE_SCHEDULER_INTERVAL", 60, 1),
LogLevel: getEnv("VIDARCHIVE_LOG_LEVEL", "info"),
LogFormat: getEnv("VIDARCHIVE_LOG_FORMAT", "text"),
Username: os.Getenv("VIDARCHIVE_USERNAME"),
PasswordHash: os.Getenv("VIDARCHIVE_PASSWORD_HASH"),
Port: getEnvInt("VIDARCHIVE_PORT", 8080, 1),
DataDir: dataDir,
DBPath: getEnv("VIDARCHIVE_DB_PATH", filepath.Join(dataDir, "vidarchive.db")),
LibraryDir: getEnv("VIDARCHIVE_LIBRARY_DIR", filepath.Join(dataDir, "library")),
TempDir: getEnv("VIDARCHIVE_TEMP_DIR", filepath.Join(dataDir, "temp")),
BinDir: binDir,
YTDLPPath: ytdlpPath,
YTDLPManaged: ytdlpManaged,
DenoPath: denoPath,
DenoManaged: denoManaged,
UpdateExternalTools: getEnvBool("VIDARCHIVE_UPDATE_EXTERNAL_TOOLS"),
FFmpegPath: getEnv("VIDARCHIVE_FFMPEG_PATH", "ffmpeg"),
FFprobePath: getEnv("VIDARCHIVE_FFPROBE_PATH", "ffprobe"),
BaseURL: getEnv("VIDARCHIVE_BASE_URL", ""),
Workers: getEnvInt("VIDARCHIVE_WORKERS", 2, 1),
SchedulerInterval: getEnvInt("VIDARCHIVE_SCHEDULER_INTERVAL", 60, 1),
LogLevel: getEnv("VIDARCHIVE_LOG_LEVEL", "info"),
LogFormat: getEnv("VIDARCHIVE_LOG_FORMAT", "text"),
Username: os.Getenv("VIDARCHIVE_USERNAME"),
PasswordHash: os.Getenv("VIDARCHIVE_PASSWORD_HASH"),
}
}
@@ -70,6 +86,20 @@ func (c *Config) IsHTTPS() bool {
return strings.HasPrefix(c.BaseURL, "https://")
}
// VidArchive installs only what it owns, but it updates anything the operator
// lets it update.
func (c *Config) CanUpdateYTDLP() bool { return c.YTDLPManaged || c.UpdateExternalTools }
func (c *Config) CanUpdateDeno() bool { return c.DenoManaged || c.UpdateExternalTools }
// managedPath decides who owns a tool's binary. An explicit VIDARCHIVE_*_PATH
// means the operator does, so VidArchive never installs over it.
func managedPath(key, binDir, name string) (string, bool) {
if v := os.Getenv(key); v != "" {
return v, false
}
return filepath.Join(binDir, name), true
}
func getEnv(key, defaultVal string) string {
if v := os.Getenv(key); v != "" {
return v
@@ -77,6 +107,11 @@ func getEnv(key, defaultVal string) string {
return defaultVal
}
func getEnvBool(key string) bool {
v, _ := strconv.ParseBool(os.Getenv(key))
return v
}
// getEnvInt falls back to the default below min: a zero or negative worker
// count, tick interval or port stalls downloads, spins the scheduler or fails to
// bind.
Minternal/handler/handler.go
@@ -17,6 +17,7 @@ import (
"vidarchive/internal/config"
"vidarchive/internal/models"
"vidarchive/internal/service"
"vidarchive/internal/tools"
"vidarchive/internal/util"
"vidarchive/internal/worker"
)
@@ -30,16 +31,17 @@ type Handler struct {
settingsSvc *service.SettingsService
subscriptionSvc *service.SubscriptionService
workerPool *worker.Pool
tools *tools.Manager
loginSem chan struct{}
// sessionSecret is cached here because every guarded request reads it; Logout
// rotates it.
sessionMu sync.RWMutex
sessionSecret string
tools toolCache
toolCache toolCache
}
func New(cfg *config.Config, presetSvc *service.PresetService, downloadSvc *service.DownloadService, librarySvc *service.LibraryService, settingsSvc *service.SettingsService, subscriptionSvc *service.SubscriptionService, workerPool *worker.Pool) (*Handler, error) {
func New(cfg *config.Config, presetSvc *service.PresetService, downloadSvc *service.DownloadService, librarySvc *service.LibraryService, settingsSvc *service.SettingsService, subscriptionSvc *service.SubscriptionService, workerPool *worker.Pool, toolMgr *tools.Manager) (*Handler, error) {
tmpl, err := loadTemplates(presetSvc)
if err != nil {
return nil, fmt.Errorf("load templates: %w", err)
@@ -59,6 +61,7 @@ func New(cfg *config.Config, presetSvc *service.PresetService, downloadSvc *serv
settingsSvc: settingsSvc,
subscriptionSvc: subscriptionSvc,
workerPool: workerPool,
tools: toolMgr,
loginSem: make(chan struct{}, maxConcurrentLogins),
sessionSecret: sessionSecret,
}, nil
Minternal/handler/health.go
@@ -9,7 +9,9 @@ import (
"strings"
"sync"
"time"
"unicode"
"vidarchive/internal/tools"
"vidarchive/internal/util"
)
@@ -72,26 +74,45 @@ func (h *Handler) Health(w http.ResponseWriter, r *http.Request) {
}
func (h *Handler) toolVersions(ctx context.Context) map[string]string {
h.tools.mu.Lock()
defer h.tools.mu.Unlock()
h.toolCache.mu.Lock()
defer h.toolCache.mu.Unlock()
if h.tools.results != nil && time.Since(h.tools.at) < h.tools.ttl {
return h.tools.results
if h.toolCache.results != nil && time.Since(h.toolCache.at) < h.toolCache.ttl {
return h.toolCache.results
}
// The mutex is held across the probe, so a caller without its own deadline
// would block every other one. /healthz would then miss its own timeout
// while waiting for the lock.
ctx, cancel := context.WithTimeout(ctx, healthProbeTimeout)
defer cancel()
results := map[string]string{
"yt-dlp": toolVersion(ctx, h.cfg.YTDLPPath, "--version"),
"ffmpeg": toolVersion(ctx, h.cfg.FFmpegPath, "-version"),
"ffprobe": toolVersion(ctx, h.cfg.FFprobePath, "-version"),
}
// The JS runtime is optional. A permanent "not installed" would look like a
// fault on a setup that never asked for it.
if !h.cfg.DenoManaged || tools.Installed(h.cfg.DenoPath) {
results["deno"] = toolVersion(ctx, h.cfg.DenoPath, "--version")
}
ttl := toolCacheTTL
if slices.Contains(slices.Collect(maps.Values(results)), "timed out") {
ttl = toolRetryTTL
}
h.tools.results, h.tools.at, h.tools.ttl = results, time.Now(), ttl
h.toolCache.results, h.toolCache.at, h.toolCache.ttl = results, time.Now(), ttl
return results
}
// invalidateToolVersions forces the next probe to re-run, so the settings page
// shows the new version straight after an update instead of the cached old one.
func (h *Handler) invalidateToolVersions() {
h.toolCache.mu.Lock()
defer h.toolCache.mu.Unlock()
h.toolCache.results = nil
}
// toolVersion reports "not installed" for a missing binary and "timed out" when
// the tool does not answer within ctx.
func toolVersion(ctx context.Context, path, versionArg string) string {
@@ -103,9 +124,16 @@ func toolVersion(ctx context.Context, path, versionArg string) string {
return "not installed"
}
line, _, _ := strings.Cut(strings.TrimSpace(string(out)), "\n")
fields := strings.Fields(line)
switch {
// ffmpeg prints "ffmpeg version N-x ..."; keep the version token only.
if fields := strings.Fields(line); len(fields) >= 3 && fields[1] == "version" {
case len(fields) >= 3 && fields[1] == "version":
return fields[2]
// deno prints "deno 2.x.y (release, ...)". The digit check stops some other
// second word from being reported as a version.
case len(fields) >= 2 && fields[1] != "" && unicode.IsDigit(rune(fields[1][0])):
return fields[1]
}
// yt-dlp prints the bare version.
return line
}
Minternal/handler/settings.go
@@ -1,16 +1,47 @@
package handler
import (
"context"
"fmt"
"log/slog"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"vidarchive/internal/models"
"vidarchive/internal/tools"
)
// toolOpTimeout bounds an install or update. A slow mirror should not pin a
// request or a goroutine forever.
const toolOpTimeout = 10 * time.Minute
type ToolsView struct {
YTDLPVersion string
YTDLPPath string
YTDLPMode string
DenoVersion string
DenoPath string
DenoMode string
AutoUpdate bool
JSRuntime bool
CheckedAt time.Time
}
// toolMode is the label for who owns a binary and who may update it.
func toolMode(managed, updatable bool) string {
switch {
case managed:
return "managed"
case updatable:
return "external, updates enabled"
default:
return "external"
}
}
func (h *Handler) Settings(w http.ResponseWriter, r *http.Request) {
presets, err := h.presetSvc.GetAll()
if err != nil {
@@ -24,19 +55,102 @@ func (h *Handler) Settings(w http.ResponseWriter, r *http.Request) {
return
}
versions := h.toolVersions(r.Context())
toolsView := ToolsView{
YTDLPVersion: versions["yt-dlp"],
YTDLPPath: h.cfg.YTDLPPath,
YTDLPMode: toolMode(h.cfg.YTDLPManaged, h.cfg.CanUpdateYTDLP()),
DenoVersion: versions["deno"],
DenoPath: h.cfg.DenoPath,
DenoMode: toolMode(h.cfg.DenoManaged, h.cfg.CanUpdateDeno()),
AutoUpdate: settings.ToolAutoUpdate,
JSRuntime: settings.JSRuntimeEnabled,
CheckedAt: settings.ToolsCheckedAt,
}
if toolsView.DenoVersion == "" {
toolsView.DenoVersion = "not installed"
}
h.renderWithRequest(w, r, "settings", PageData{
Title: "Settings",
ActiveTab: "settings",
Data: struct {
Presets []*models.Preset
Settings *models.Settings
Tools ToolsView
}{
Presets: presets,
Settings: settings,
Tools: toolsView,
},
})
}
// UpdateTools runs the managed tools' own updaters and waits for them. The app
// has no JavaScript, so a blocking POST is the only way to report a real result.
//
// ponytail: blocks one request for up to toolOpTimeout. Move to a queued job if
// operators start updating from flaky connections.
func (h *Handler) UpdateTools(w http.ResponseWriter, r *http.Request) {
// Deliberately not r.Context(): a closed tab must not cancel a replacement
// half way through.
ctx, cancel := context.WithTimeout(context.Background(), toolOpTimeout)
defer cancel()
summary, err := h.tools.Update(ctx)
h.invalidateToolVersions()
if err != nil {
// summary still holds whatever did update. Dropping it would invite a
// second, redundant click.
message := "Tool update failed: " + err.Error()
if summary != "" {
message = summary + " " + message
}
redirectWithError(w, r, "/settings", message, err)
return
}
redirectWithSuccess(w, r, "/settings", summary)
}
// UpdateToolSettings owns its own form. Folding it into the main settings form
// would let a submit from either one clear the other's checkboxes.
func (h *Handler) UpdateToolSettings(w http.ResponseWriter, r *http.Request) {
if !parseForm(w, r) {
return
}
jsRuntime := r.FormValue("js_runtime_enabled") == "1"
if err := h.settingsSvc.SetToolAutoUpdate(r.FormValue("tool_auto_update") == "1"); err != nil {
redirectWithError(w, r, "/settings", "Couldn't save the tool settings.", err)
return
}
if err := h.settingsSvc.SetJSRuntimeEnabled(jsRuntime); err != nil {
redirectWithError(w, r, "/settings", "Couldn't save the tool settings.", err)
return
}
if jsRuntime && h.cfg.DenoManaged && !tools.Installed(h.cfg.DenoPath) {
h.installJSRuntime()
redirectWithSuccess(w, r, "/settings", "Tool settings saved. The JS runtime is downloading in the background.")
return
}
redirectWithSuccess(w, r, "/settings", "Tool settings saved.")
}
// installJSRuntime detaches because deno is a large download and the request
// should not wait for it. The settings page shows the version once it lands.
func (h *Handler) installJSRuntime() {
go func() {
ctx, cancel := context.WithTimeout(context.Background(), toolOpTimeout)
defer cancel()
if err := h.tools.Ensure(ctx, true); err != nil {
slog.Error("JS runtime install failed", "err", err)
return
}
h.invalidateToolVersions()
}()
}
func (h *Handler) CreatePreset(w http.ResponseWriter, r *http.Request) {
if !parseForm(w, r) {
return
Minternal/models/models.go
@@ -89,6 +89,11 @@ type Settings struct {
AutoRefreshLibrary bool
AutoRefreshDownloads bool
Cookies string
// ToolAutoUpdate lets the scheduler run the managed tools' own updaters once
// a day. ToolsCheckedAt is when it last did, zero if never.
ToolAutoUpdate bool
ToolsCheckedAt time.Time
JSRuntimeEnabled bool
}
type FormatInfo struct {
Minternal/repository/settings.go
@@ -3,6 +3,7 @@ package repository
import (
"database/sql"
"strconv"
"time"
"vidarchive/internal/models"
)
@@ -44,6 +45,14 @@ func (r *SettingsRepository) GetAll() (*models.Settings, error) {
settings.AutoRefreshDownloads = value == "1" || value == "true"
case "cookies":
settings.Cookies = value
case "tool_auto_update":
settings.ToolAutoUpdate = value == "1" || value == "true"
case "tools_checked_at":
if t, err := time.Parse(time.RFC3339, value); err == nil {
settings.ToolsCheckedAt = t
}
case "js_runtime_enabled":
settings.JSRuntimeEnabled = value == "1" || value == "true"
}
}
return settings, rows.Err()
Minternal/server/server.go
@@ -85,6 +85,8 @@ func (s *Server) setupRoutes() {
s.router.Post("/settings/presets/{id}", s.handler.UpdatePreset)
s.router.Post("/settings/presets/{id}/delete", s.handler.DeletePreset)
s.router.Post("/settings", s.handler.UpdateSettings)
s.router.Post("/settings/tools", s.handler.UpdateToolSettings)
s.router.Post("/settings/tools/update", s.handler.UpdateTools)
s.router.Post("/theme", s.handler.Theme)
}
Minternal/server/server_test.go
@@ -19,6 +19,7 @@ import (
"vidarchive/internal/handler"
"vidarchive/internal/repository"
"vidarchive/internal/service"
"vidarchive/internal/tools"
"vidarchive/internal/worker"
)
@@ -87,7 +88,7 @@ func setupTestServerDB(t *testing.T) (testServer, *config.Config, *sql.DB, func(
downloadSvc := service.NewDownloadService(downloadRepo, librarySvc, presetSvc, settingsSvc, subscriptionSvc, cfg)
workerPool := worker.New(downloadSvc, cfg.Workers)
h, err := handler.New(cfg, presetSvc, downloadSvc, librarySvc, settingsSvc, subscriptionSvc, workerPool)
h, err := handler.New(cfg, presetSvc, downloadSvc, librarySvc, settingsSvc, subscriptionSvc, workerPool, tools.New(cfg))
if err != nil {
t.Fatalf("init handler: %v", err)
}
Minternal/service/settings.go
@@ -1,6 +1,8 @@
package service
import (
"time"
"vidarchive/internal/models"
"vidarchive/internal/repository"
)
@@ -22,19 +24,32 @@ func (s *SettingsService) SetRefreshInterval(interval string) error {
}
func (s *SettingsService) SetAutoRefreshLibrary(enabled bool) error {
val := "0"
if enabled {
val = "1"
}
return s.repo.Set("auto_refresh_library", val)
return s.repo.Set("auto_refresh_library", boolValue(enabled))
}
func (s *SettingsService) SetAutoRefreshDownloads(enabled bool) error {
val := "0"
if enabled {
val = "1"
return s.repo.Set("auto_refresh_downloads", boolValue(enabled))
}
func boolValue(b bool) string {
if b {
return "1"
}
return s.repo.Set("auto_refresh_downloads", val)
return "0"
}
func (s *SettingsService) SetToolAutoUpdate(enabled bool) error {
return s.repo.Set("tool_auto_update", boolValue(enabled))
}
func (s *SettingsService) SetJSRuntimeEnabled(enabled bool) error {
return s.repo.Set("js_runtime_enabled", boolValue(enabled))
}
// SetToolsCheckedAt is written before the update runs, not after: a crash mid
// update must not make the scheduler retry on every tick.
func (s *SettingsService) SetToolsCheckedAt(t time.Time) error {
return s.repo.Set("tools_checked_at", t.Format(time.RFC3339))
}
func (s *SettingsService) SetCookies(cookies string) error {
Ainternal/tools/tools.go
@@ -0,0 +1,445 @@
// Package tools installs and updates the external binaries VidArchive drives.
//
// It only ever touches binaries it owns. When the operator set the matching
// VIDARCHIVE_*_PATH, the tool is external and every method here skips it.
package tools
import (
"archive/zip"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"path"
"path/filepath"
"regexp"
"runtime"
"strings"
"sync"
"time"
"vidarchive/internal/config"
"vidarchive/internal/util"
)
const (
ytdlpRepo = "yt-dlp/yt-dlp"
denoRepo = "denoland/deno"
)
// maxAssetSize bounds both the download and the unpacked binary, so a corrupt
// or hostile release cannot fill the data volume.
const maxAssetSize = 512 << 20
// Candidate release assets per architecture, best first: the installer keeps
// the first one that runs here. armv7 is absent because yt-dlp only ships it
// zipped.
var (
ytdlpAssets = map[string][]string{
"amd64": {"yt-dlp_linux", "yt-dlp_musllinux"},
"arm64": {"yt-dlp_linux_aarch64", "yt-dlp_musllinux_aarch64"},
}
// Deno publishes one build per architecture, glibc only.
denoAssets = map[string][]string{
"amd64": {"deno-x86_64-unknown-linux-gnu.zip"},
"arm64": {"deno-aarch64-unknown-linux-gnu.zip"},
}
)
// trialRunTimeout: the standalone yt-dlp build unpacks itself on every start,
// which is slow the first time.
const trialRunTimeout = time.Minute
type Manager struct {
cfg *config.Config
// mu serializes installs and updates: they rewrite the same files, and two
// at once would race on the rename.
mu sync.Mutex
// baseURL is the GitHub release host. Tests point it at a local server.
baseURL string
}
func New(cfg *config.Config) *Manager {
return &Manager{cfg: cfg, baseURL: "https://github.com"}
}
// Installed reports whether path holds something that could be executed. The
// permission check matters: a truncated or 0644 file left by a volume restore
// would otherwise block the install forever and fail every download instead.
func Installed(path string) bool {
info, err := os.Stat(path)
return err == nil && info.Mode().IsRegular() && info.Mode().Perm()&0o111 != 0
}
// Ensure installs every managed tool that is missing. It is safe to call on
// every start, because updating an existing tool is a separate operation.
// wantJSRuntime comes from the settings toggle, because deno is a large
// download nobody should pay for unless they use it.
func (m *Manager) Ensure(ctx context.Context, wantJSRuntime bool) error {
// TryLock, not Lock: a blocked caller would sit on the mutex without any way
// to honour its context, and there is nothing to do once another install of
// the same files is already running.
if !m.mu.TryLock() {
slog.Info("tool install skipped, another one is already running")
return nil
}
defer m.mu.Unlock()
if err := os.MkdirAll(m.cfg.BinDir, 0o755); err != nil {
return fmt.Errorf("create bin dir: %w", err)
}
cleanTemps(m.cfg.BinDir)
var errs []error
if m.cfg.YTDLPManaged && !Installed(m.cfg.YTDLPPath) {
slog.Info("installing yt-dlp", "path", m.cfg.YTDLPPath)
if err := m.installYTDLP(ctx); err != nil {
errs = append(errs, fmt.Errorf("install yt-dlp: %w", err))
} else {
slog.Info("yt-dlp installed", "path", m.cfg.YTDLPPath)
}
}
if wantJSRuntime && m.cfg.DenoManaged && !Installed(m.cfg.DenoPath) {
slog.Info("installing deno JS runtime", "path", m.cfg.DenoPath)
if err := m.installDeno(ctx); err != nil {
errs = append(errs, fmt.Errorf("install deno: %w", err))
} else {
slog.Info("deno installed", "path", m.cfg.DenoPath)
}
}
return errors.Join(errs...)
}
// Update brings the managed tools up to date and returns a one-line summary for
// the UI. Both yt-dlp and deno ship an updater that checks the version, verifies
// the download and replaces the binary atomically, so none of that is
// reimplemented here. A tool that is not installed yet is installed instead.
func (m *Manager) Update(ctx context.Context) (string, error) {
// An update takes minutes and the page has no JavaScript, so a second click
// is likely. Queueing it behind the mutex would spend its whole context
// waiting, then kill the child and report a failure for an update that
// worked.
if !m.mu.TryLock() {
return "An install or update is already running.", nil
}
defer m.mu.Unlock()
if err := os.MkdirAll(m.cfg.BinDir, 0o755); err != nil {
return "", fmt.Errorf("create bin dir: %w", err)
}
var parts []string
var errs []error
if m.cfg.CanUpdateYTDLP() {
summary, err := m.updateOne(ctx, "yt-dlp", m.cfg.YTDLPPath, m.cfg.YTDLPManaged, m.installYTDLP, "-U")
if err != nil {
errs = append(errs, err)
} else {
parts = append(parts, summary)
}
}
// deno is only updated when it is already there. Installing it is the
// settings toggle's job, not the update button's.
if m.cfg.CanUpdateDeno() && Installed(m.cfg.DenoPath) {
summary, err := m.updateOne(ctx, "deno", m.cfg.DenoPath, m.cfg.DenoManaged, m.installDeno, "upgrade")
if err != nil {
errs = append(errs, err)
} else {
parts = append(parts, summary)
}
}
if len(parts) == 0 && len(errs) == 0 {
return "Nothing to update: no tool is under VidArchive's control.", nil
}
return strings.Join(parts, " "), errors.Join(errs...)
}
func (m *Manager) updateOne(ctx context.Context, name, path string, managed bool, install func(context.Context) error, updateArg string) (string, error) {
if !Installed(path) {
// Writing a fresh binary to a path the operator chose is not this
// program's business.
if !managed {
return "", fmt.Errorf("%s is not installed at %s; install it there first", name, path)
}
if err := install(ctx); err != nil {
return "", fmt.Errorf("install %s: %w", name, err)
}
return name + ": installed.", nil
}
out, err := util.KillableCommand(ctx, path, updateArg).CombinedOutput()
text := strings.TrimSpace(string(out))
if err != nil {
slog.Error("tool update failed", "tool", name, "err", err, "output", text)
return "", fmt.Errorf("update %s: %w: %s", name, err, lastLine(text))
}
slog.Info("tool update finished", "tool", name, "output", text)
return name + ": " + lastLine(text), nil
}
// ansiColor matches the SGR escapes deno writes even with NO_COLOR set. They
// would reach the browser as literal control characters in the flash message.
var ansiColor = regexp.MustCompile("\x1b\\[[0-9;]*m")
// lastLine is what the updaters put their verdict on ("up to date", "Updated to
// ..."). The lines before it are progress noise, which the log already has.
func lastLine(s string) string {
lines := strings.Split(s, "\n")
for i := len(lines) - 1; i >= 0; i-- {
if t := strings.TrimSpace(lines[i]); t != "" {
return ansiColor.ReplaceAllString(t, "")
}
}
return "done"
}
func (m *Manager) installYTDLP(ctx context.Context) error {
candidates := ytdlpAssets[runtime.GOARCH]
if len(candidates) == 0 {
return fmt.Errorf("no yt-dlp release build for %s/%s; set VIDARCHIVE_YTDLP_PATH", runtime.GOOS, runtime.GOARCH)
}
// yt-dlp publishes SHA2-256SUMS in every release, so a missing or
// unparseable sums file means something is wrong and the install stops.
sums, err := m.get(ctx, ytdlpRepo, "SHA2-256SUMS")
if err != nil {
return err
}
var errs []error
for _, asset := range candidates {
want, ok := parseChecksum(sums, asset)
if !ok {
errs = append(errs, fmt.Errorf("no checksum for %s in SHA2-256SUMS", asset))
continue
}
err := m.tryCandidate(ctx, ytdlpRepo, asset, want, m.cfg.YTDLPPath, placeBinary)
if err == nil {
return nil
}
errs = append(errs, err)
}
return errors.Join(errs...)
}
func (m *Manager) installDeno(ctx context.Context) error {
candidates := denoAssets[runtime.GOARCH]
if len(candidates) == 0 {
return fmt.Errorf("no deno release build for %s/%s; set VIDARCHIVE_DENO_PATH", runtime.GOOS, runtime.GOARCH)
}
unzip := func(src, dest string) error { return extractBinary(src, "deno", dest) }
var errs []error
for _, asset := range candidates {
// Fail closed. Deno publishes a sums file for every Linux asset, so a
// missing one means something between here and GitHub is wrong. The
// binary is executed and left on PATH, so it is never taken unverified.
sums, err := m.get(ctx, denoRepo, asset+".sha256sum")
if err != nil {
errs = append(errs, fmt.Errorf("checksum file for %s: %w", asset, err))
continue
}
want, ok := parseChecksum(sums, asset)
if !ok {
errs = append(errs, fmt.Errorf("no checksum for %s in %s.sha256sum", asset, asset))
continue
}
err = m.tryCandidate(ctx, denoRepo, asset, want, m.cfg.DenoPath, unzip)
if err == nil {
return nil
}
errs = append(errs, err)
}
return errors.Join(errs...)
}
// tryCandidate downloads one release asset, unpacks it and runs it once before
// moving it into place. That trial run replaces libc and architecture
// detection: a build for the wrong platform fails to exec, and the caller moves
// on to the next candidate.
func (m *Manager) tryCandidate(ctx context.Context, repo, asset, wantSum, dest string, unpack func(src, dest string) error) error {
tmp, err := m.fetchAsset(ctx, repo, asset, wantSum)
if err != nil {
return err
}
defer os.Remove(tmp)
staged := dest + ".tmp"
defer os.Remove(staged)
if err := unpack(tmp, staged); err != nil {
return fmt.Errorf("%s: %w", asset, err)
}
if err := trialRun(ctx, staged); err != nil {
return fmt.Errorf("%s: %w", asset, err)
}
// Rename is atomic, and on Linux it leaves a currently running copy of the
// old binary untouched, so a download in flight is not disturbed.
return os.Rename(staged, dest)
}
// trialRun checks that the downloaded binary starts on this system. Both tools
// answer --version without touching the network.
func trialRun(ctx context.Context, path string) error {
ctx, cancel := context.WithTimeout(ctx, trialRunTimeout)
defer cancel()
out, err := util.KillableCommand(ctx, path, "--version").CombinedOutput()
if err != nil {
return fmt.Errorf("does not run on this system: %w: %s", err, lastLine(string(out)))
}
return nil
}
// assetURL uses the "latest" alias for both the asset and its checksum file, so
// the two always come from the same release without asking the GitHub API.
func (m *Manager) assetURL(repo, asset string) string {
return fmt.Sprintf("%s/%s/releases/latest/download/%s", m.baseURL, repo, asset)
}
func (m *Manager) get(ctx context.Context, repo, asset string) ([]byte, error) {
resp, err := m.open(ctx, repo, asset)
if err != nil {
return nil, err
}
defer resp.Body.Close()
return io.ReadAll(io.LimitReader(resp.Body, 1<<20))
}
func (m *Manager) open(ctx context.Context, repo, asset string) (*http.Response, error) {
url := m.assetURL(repo, asset)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, fmt.Errorf("fetch %s: %w", url, err)
}
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
return nil, fmt.Errorf("fetch %s: %s", url, resp.Status)
}
return resp, nil
}
// fetchAsset downloads a release asset into BinDir and verifies its SHA-256
// while writing. The caller removes the returned file.
func (m *Manager) fetchAsset(ctx context.Context, repo, asset, wantSum string) (string, error) {
resp, err := m.open(ctx, repo, asset)
if err != nil {
return "", err
}
defer resp.Body.Close()
f, err := os.CreateTemp(m.cfg.BinDir, "download-*")
if err != nil {
return "", err
}
tmp := f.Name()
sum := sha256.New()
n, err := io.Copy(io.MultiWriter(f, sum), io.LimitReader(resp.Body, maxAssetSize+1))
closeErr := f.Close()
switch {
case err != nil:
os.Remove(tmp)
return "", fmt.Errorf("download %s: %w", asset, err)
case closeErr != nil:
os.Remove(tmp)
return "", closeErr
case n > maxAssetSize:
os.Remove(tmp)
return "", fmt.Errorf("download %s: larger than %d bytes", asset, int64(maxAssetSize))
}
if got := hex.EncodeToString(sum.Sum(nil)); got != wantSum {
os.Remove(tmp)
return "", fmt.Errorf("checksum mismatch for %s: got %s, want %s", asset, got, wantSum)
}
return tmp, nil
}
// parseChecksum reads sha256sum output: one "<hex> <filename>" line per file.
// Both repositories publish that format.
func parseChecksum(data []byte, asset string) (string, bool) {
for _, line := range strings.Split(string(data), "\n") {
fields := strings.Fields(line)
if len(fields) >= 2 && path.Base(fields[1]) == asset {
return strings.ToLower(fields[0]), true
}
}
return "", false
}
// cleanTemps removes downloads a hard kill interrupted. fetchAsset cannot, and
// a partial file can be as large as maxAssetSize. The age check spares a
// download another process may still be writing.
func cleanTemps(dir string) {
matches, _ := filepath.Glob(filepath.Join(dir, "download-*"))
for _, match := range matches {
info, err := os.Stat(match)
if err != nil || time.Since(info.ModTime()) < time.Hour {
continue
}
if err := os.Remove(match); err != nil {
slog.Warn("failed to remove an orphaned download", "path", match, "err", err)
}
}
}
// placeBinary is the unpack step for an asset that is already the binary.
func placeBinary(src, dest string) error {
if err := os.Chmod(src, 0o755); err != nil {
return err
}
return os.Rename(src, dest)
}
func extractBinary(archive, entry, dest string) error {
zr, err := zip.OpenReader(archive)
if err != nil {
return fmt.Errorf("open %s: %w", filepath.Base(archive), err)
}
defer zr.Close()
for _, f := range zr.File {
if path.Base(f.Name) != entry {
continue
}
rc, err := f.Open()
if err != nil {
return err
}
defer rc.Close()
out, err := os.OpenFile(dest, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755)
if err != nil {
return err
}
n, err := io.Copy(out, io.LimitReader(rc, maxAssetSize+1))
if cErr := out.Close(); err == nil {
err = cErr
}
if err == nil && n > maxAssetSize {
err = fmt.Errorf("%s in archive is larger than %d bytes", entry, int64(maxAssetSize))
}
if err != nil {
os.Remove(dest)
return err
}
// Not left to the OpenFile mode: umask reduces it, and it is ignored
// outright when a leftover file from a crashed install already exists.
return os.Chmod(dest, 0o755)
}
return fmt.Errorf("%s not found in %s", entry, filepath.Base(archive))
}
Ainternal/tools/tools_live_test.go
@@ -0,0 +1,52 @@
package tools
import (
"context"
"os"
"path/filepath"
"testing"
"time"
"vidarchive/internal/config"
"vidarchive/internal/util"
)
// TestLiveInstall downloads the real releases. It catches the failure the unit
// tests cannot: an asset renamed or dropped from a release, which turns the
// install into a 404 or a missing checksum entry.
//
// VIDARCHIVE_ONLINE_TESTS=1 go test ./internal/tools -run Live -v
func TestLiveInstall(t *testing.T) {
if os.Getenv("VIDARCHIVE_ONLINE_TESTS") != "1" {
t.Skip("set VIDARCHIVE_ONLINE_TESTS=1 to run")
}
dir := t.TempDir()
cfg := &config.Config{
BinDir: dir,
YTDLPPath: filepath.Join(dir, "yt-dlp"),
YTDLPManaged: true,
DenoPath: filepath.Join(dir, "deno"),
DenoManaged: true,
}
m := New(cfg)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
if err := m.Ensure(ctx, true); err != nil {
t.Fatalf("Ensure: %v", err)
}
for _, tool := range []struct{ path, arg string }{
{cfg.YTDLPPath, "--version"},
{cfg.DenoPath, "--version"},
} {
out, err := util.KillableCommand(ctx, tool.path, tool.arg).CombinedOutput()
if err != nil {
t.Errorf("%s %s: %v: %s", tool.path, tool.arg, err, out)
continue
}
t.Logf("%s -> %s", filepath.Base(tool.path), out)
}
}
Ainternal/tools/tools_test.go
@@ -0,0 +1,406 @@
package tools
import (
"archive/zip"
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"vidarchive/internal/config"
)
// fakeRelease serves the two GitHub paths the installer uses: the asset itself
// and the checksum file next to it.
func fakeRelease(t *testing.T, assets map[string][]byte) string {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
name := filepath.Base(r.URL.Path)
body, ok := assets[name]
if !ok {
http.NotFound(w, r)
return
}
_, _ = w.Write(body)
}))
t.Cleanup(srv.Close)
return srv.URL
}
func sumsFile(asset string, body []byte) []byte {
sum := sha256.Sum256(body)
return []byte(fmt.Sprintf("%s %s\n", hex.EncodeToString(sum[:]), asset))
}
func testManager(t *testing.T, assets map[string][]byte) *Manager {
t.Helper()
dir := t.TempDir()
cfg := &config.Config{
BinDir: dir,
YTDLPPath: filepath.Join(dir, "yt-dlp"),
YTDLPManaged: true,
DenoPath: filepath.Join(dir, "deno"),
DenoManaged: true,
}
return &Manager{cfg: cfg, baseURL: fakeRelease(t, assets)}
}
// requireArch returns the candidate list for this architecture, skipping when
// the platform has no mapped release build.
func requireArch(t *testing.T, assets map[string][]string) []string {
t.Helper()
candidates, ok := assets[runtime.GOARCH]
if !ok {
t.Skipf("no release asset mapped for %s", runtime.GOARCH)
}
return candidates
}
func requireYTDLPAsset(t *testing.T) string {
t.Helper()
return requireArch(t, ytdlpAssets)[0]
}
// script is a tiny executable stand-in for a real release binary.
func script(body string) []byte {
return []byte("#!/bin/sh\n" + body + "\n")
}
func TestEnsureInstallsYTDLP(t *testing.T) {
asset := requireYTDLPAsset(t)
body := script("echo 2025.09.05")
m := testManager(t, map[string][]byte{
asset: body,
"SHA2-256SUMS": sumsFile(asset, body),
})
if err := m.Ensure(context.Background(), false); err != nil {
t.Fatalf("Ensure: %v", err)
}
got, err := os.ReadFile(m.cfg.YTDLPPath)
if err != nil {
t.Fatalf("read installed binary: %v", err)
}
if !bytes.Equal(got, body) {
t.Errorf("installed content = %q, want %q", got, body)
}
info, err := os.Stat(m.cfg.YTDLPPath)
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm()&0o111 == 0 {
t.Errorf("installed binary is not executable, mode %v", info.Mode())
}
}
func TestEnsureRejectsBadChecksum(t *testing.T) {
asset := requireYTDLPAsset(t)
m := testManager(t, map[string][]byte{
asset: []byte("tampered"),
"SHA2-256SUMS": sumsFile(asset, []byte("original")),
})
err := m.Ensure(context.Background(), false)
if err == nil || !strings.Contains(err.Error(), "checksum mismatch") {
t.Fatalf("Ensure err = %v, want a checksum mismatch", err)
}
if Installed(m.cfg.YTDLPPath) {
t.Error("a binary that failed verification was installed anyway")
}
// A failed download must not leave scratch files in the managed directory.
entries, err := os.ReadDir(m.cfg.BinDir)
if err != nil {
t.Fatal(err)
}
if len(entries) != 0 {
t.Errorf("bin dir not clean after failure: %v", entries)
}
}
func TestEnsureInstallsDenoFromZip(t *testing.T) {
ytdlpAsset := requireYTDLPAsset(t)
denoAsset := requireArch(t, denoAssets)[0]
body := script("echo deno 2.1.4")
var archive bytes.Buffer
zw := zip.NewWriter(&archive)
f, err := zw.Create("deno")
if err != nil {
t.Fatal(err)
}
if _, err := f.Write(body); err != nil {
t.Fatal(err)
}
if err := zw.Close(); err != nil {
t.Fatal(err)
}
ytdlpBody := script("echo 2025.09.05")
m := testManager(t, map[string][]byte{
ytdlpAsset: ytdlpBody,
"SHA2-256SUMS": sumsFile(ytdlpAsset, ytdlpBody),
denoAsset: archive.Bytes(),
denoAsset + ".sha256sum": sumsFile(denoAsset, archive.Bytes()),
})
if err := m.Ensure(context.Background(), true); err != nil {
t.Fatalf("Ensure: %v", err)
}
got, err := os.ReadFile(m.cfg.DenoPath)
if err != nil {
t.Fatalf("read deno: %v", err)
}
if !bytes.Equal(got, body) {
t.Errorf("extracted deno = %q, want %q", got, body)
}
}
// An external tool is the operator's to manage, so nothing is downloaded and
// nothing is overwritten.
func TestEnsureSkipsExternalTools(t *testing.T) {
dir := t.TempDir()
m := &Manager{
cfg: &config.Config{
BinDir: dir,
YTDLPPath: "/usr/bin/yt-dlp",
YTDLPManaged: false,
DenoPath: "/usr/bin/deno",
DenoManaged: false,
},
// Any request would fail: the installer must not make one.
baseURL: "http://127.0.0.1:1",
}
if err := m.Ensure(context.Background(), true); err != nil {
t.Fatalf("Ensure: %v", err)
}
summary, err := m.Update(context.Background())
if err != nil {
t.Fatalf("Update: %v", err)
}
if !strings.Contains(summary, "Nothing to update") {
t.Errorf("Update summary = %q, want a note that nothing is under control", summary)
}
}
func TestParseChecksum(t *testing.T) {
multi := []byte("aa yt-dlp\nbb yt-dlp_linux\n")
if got, ok := parseChecksum(multi, "yt-dlp_linux"); !ok || got != "bb" {
t.Errorf("multi-line = %q,%v, want bb,true", got, ok)
}
if _, ok := parseChecksum(multi, "yt-dlp_macos"); ok {
t.Error("missing asset reported as found")
}
// A hex with no filename is not sha256sum format and must not be trusted
// for an arbitrary asset.
bare := []byte(strings.Repeat("a", 64) + "\n")
if _, ok := parseChecksum(bare, "deno.zip"); ok {
t.Error("a checksum line with no filename was accepted")
}
}
func TestLastLineStripsColor(t *testing.T) {
out := "Looking up stable version\nLocal deno version \x1b[32m2.9.7\x1b[39m is the most recent release\n"
if got, want := lastLine(out), "Local deno version 2.9.7 is the most recent release"; got != want {
t.Errorf("lastLine = %q, want %q", got, want)
}
}
// A build for the wrong C library or architecture downloads fine and then fails
// to exec. The installer must fall through to the next candidate instead of
// leaving a binary that cannot run.
func TestEnsureFallsBackWhenCandidateDoesNotRun(t *testing.T) {
candidates := requireArch(t, ytdlpAssets)
if len(candidates) < 2 {
t.Skipf("only one yt-dlp candidate for %s", runtime.GOARCH)
}
broken, working := candidates[0], candidates[1]
brokenBody := script("exit 1")
workingBody := script("echo 2025.09.05")
sums := append(sumsFile(broken, brokenBody), sumsFile(working, workingBody)...)
m := testManager(t, map[string][]byte{
broken: brokenBody,
working: workingBody,
"SHA2-256SUMS": sums,
})
if err := m.Ensure(context.Background(), false); err != nil {
t.Fatalf("Ensure: %v", err)
}
got, err := os.ReadFile(m.cfg.YTDLPPath)
if err != nil {
t.Fatalf("read installed binary: %v", err)
}
if !bytes.Equal(got, workingBody) {
t.Errorf("installed the candidate that does not run: %q", got)
}
}
// Every candidate failing must leave nothing behind, not a half-installed
// binary that fails on the first download instead.
func TestEnsureInstallsNothingWhenNoCandidateRuns(t *testing.T) {
candidates := requireArch(t, ytdlpAssets)
assets := map[string][]byte{}
var sums []byte
for _, asset := range candidates {
body := script("exit 1")
assets[asset] = body
sums = append(sums, sumsFile(asset, body)...)
}
assets["SHA2-256SUMS"] = sums
m := testManager(t, assets)
err := m.Ensure(context.Background(), false)
if err == nil || !strings.Contains(err.Error(), "does not run on this system") {
t.Fatalf("Ensure err = %v, want a trial-run failure", err)
}
if Installed(m.cfg.YTDLPPath) {
t.Error("a binary that never ran was installed")
}
entries, err := os.ReadDir(m.cfg.BinDir)
if err != nil {
t.Fatal(err)
}
if len(entries) != 0 {
t.Errorf("bin dir not clean after failure: %v", entries)
}
}
// VIDARCHIVE_UPDATE_EXTERNAL_TOOLS is the escape hatch for a platform with no
// mapped release build: the operator installs the binary once, VidArchive runs
// its updater from then on.
func TestUpdateRunsUpdaterForOptedInExternalTool(t *testing.T) {
dir := t.TempDir()
external := filepath.Join(dir, "my-yt-dlp")
if err := os.WriteFile(external, script(`echo "yt-dlp is up to date (stable@2026.01.01)"`), 0o755); err != nil {
t.Fatal(err)
}
m := &Manager{
cfg: &config.Config{
BinDir: dir,
YTDLPPath: external,
YTDLPManaged: false,
DenoPath: filepath.Join(dir, "deno"),
DenoManaged: false,
UpdateExternalTools: true,
},
// Any release request would fail: an external tool is never downloaded.
baseURL: "http://127.0.0.1:1",
}
summary, err := m.Update(context.Background())
if err != nil {
t.Fatalf("Update: %v", err)
}
if !strings.Contains(summary, "up to date") {
t.Errorf("Update summary = %q, want the updater's own output", summary)
}
}
// The opt-in grants update rights, not install rights. A missing external
// binary must be reported, never downloaded over the operator's chosen path.
func TestUpdateDoesNotInstallOverExternalPath(t *testing.T) {
dir := t.TempDir()
external := filepath.Join(dir, "my-yt-dlp")
m := &Manager{
cfg: &config.Config{
BinDir: dir,
YTDLPPath: external,
YTDLPManaged: false,
DenoPath: filepath.Join(dir, "deno"),
DenoManaged: false,
UpdateExternalTools: true,
},
baseURL: "http://127.0.0.1:1",
}
_, err := m.Update(context.Background())
if err == nil || !strings.Contains(err.Error(), "is not installed at") {
t.Fatalf("Update err = %v, want a not-installed report", err)
}
if Installed(external) {
t.Error("an external path was populated by the updater")
}
}
// The checksum is the only integrity control on a binary that gets executed
// and then left on PATH, so a missing sums file must stop the install.
func TestEnsureFailsClosedWithoutDenoChecksum(t *testing.T) {
ytdlpAsset := requireYTDLPAsset(t)
denoAsset := requireArch(t, denoAssets)[0]
ytdlpBody := script("echo 2025.09.05")
m := testManager(t, map[string][]byte{
ytdlpAsset: ytdlpBody,
"SHA2-256SUMS": sumsFile(ytdlpAsset, ytdlpBody),
// The asset is served, its .sha256sum is not.
denoAsset: []byte("PK\x03\x04 not a real archive"),
})
err := m.Ensure(context.Background(), true)
if err == nil || !strings.Contains(err.Error(), "checksum file") {
t.Fatalf("Ensure err = %v, want a missing-checksum failure", err)
}
if Installed(m.cfg.DenoPath) {
t.Error("deno was installed without a verified checksum")
}
}
// A file that cannot be executed is not an install. Treating it as one would
// block the repair forever and fail every download instead.
func TestEnsureReplacesUnexecutableBinary(t *testing.T) {
asset := requireYTDLPAsset(t)
body := script("echo 2025.09.05")
m := testManager(t, map[string][]byte{
asset: body,
"SHA2-256SUMS": sumsFile(asset, body),
})
if err := os.WriteFile(m.cfg.YTDLPPath, []byte("truncated"), 0o644); err != nil {
t.Fatal(err)
}
if Installed(m.cfg.YTDLPPath) {
t.Fatal("a 0644 file counted as installed")
}
if err := m.Ensure(context.Background(), false); err != nil {
t.Fatalf("Ensure: %v", err)
}
got, err := os.ReadFile(m.cfg.YTDLPPath)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(got, body) {
t.Errorf("unexecutable file was not replaced: %q", got)
}
}
// A second caller must be told, not queued: queueing spends its context waiting
// and then reports a failure for an update that actually worked.
func TestUpdateDoesNotQueueBehindARunningOne(t *testing.T) {
m := testManager(t, nil)
m.mu.Lock()
defer m.mu.Unlock()
summary, err := m.Update(context.Background())
if err != nil {
t.Fatalf("Update: %v", err)
}
if !strings.Contains(summary, "already running") {
t.Errorf("Update summary = %q, want a note that one is already running", summary)
}
}
Minternal/worker/scheduler.go
@@ -9,6 +9,7 @@ import (
"vidarchive/internal/models"
"vidarchive/internal/service"
"vidarchive/internal/tools"
)
// Scheduler enqueues downloads for subscriptions whose next run is due. It
@@ -17,6 +18,8 @@ import (
type Scheduler struct {
subscriptionSvc *service.SubscriptionService
downloadSvc *service.DownloadService
settingsSvc *service.SettingsService
tools *tools.Manager
pool *Pool
interval time.Duration
ctx context.Context
@@ -24,7 +27,7 @@ type Scheduler struct {
wg sync.WaitGroup
}
func NewScheduler(subscriptionSvc *service.SubscriptionService, downloadSvc *service.DownloadService, pool *Pool, interval time.Duration) *Scheduler {
func NewScheduler(subscriptionSvc *service.SubscriptionService, downloadSvc *service.DownloadService, settingsSvc *service.SettingsService, toolMgr *tools.Manager, pool *Pool, interval time.Duration) *Scheduler {
if interval <= 0 {
interval = time.Minute
}
@@ -32,6 +35,8 @@ func NewScheduler(subscriptionSvc *service.SubscriptionService, downloadSvc *ser
return &Scheduler{
subscriptionSvc: subscriptionSvc,
downloadSvc: downloadSvc,
settingsSvc: settingsSvc,
tools: toolMgr,
pool: pool,
interval: interval,
ctx: ctx,
@@ -65,6 +70,7 @@ func (s *Scheduler) loop() {
select {
case <-ticker.C:
s.checkDue()
s.checkToolUpdate(time.Now())
case <-s.ctx.Done():
return
}
@@ -108,6 +114,45 @@ func (s *Scheduler) checkDue() {
}
}
// toolUpdateInterval: yt-dlp releases roughly weekly, and its own updater is a
// network round trip, so a daily check is plenty.
const toolUpdateInterval = 24 * time.Hour
func (s *Scheduler) checkToolUpdate(now time.Time) {
if s.tools == nil || s.settingsSvc == nil {
return
}
settings, err := s.settingsSvc.GetAll()
if err != nil {
slog.Error("scheduler: failed to read settings for tool update", "err", err)
return
}
if !settings.ToolAutoUpdate || now.Sub(settings.ToolsCheckedAt) < toolUpdateInterval {
return
}
// Stamped before the run, so a failure waits a day instead of retrying every
// tick. It also keeps the goroutine below off the database, which matters
// because shutdown does not wait for it.
if err := s.settingsSvc.SetToolsCheckedAt(now); err != nil {
slog.Error("scheduler: failed to record tool check time", "err", err)
return
}
// An update takes minutes. Blocking the tick would delay every subscription,
// so it runs detached. Shutdown cancels ctx and kills the child. Both
// updaters replace the binary by rename, so a killed run leaves the old one
// intact.
go func() {
summary, err := s.tools.Update(s.ctx)
if err != nil {
slog.Error("scheduled tool update failed", "err", err)
return
}
slog.Info("scheduled tool update finished", "result", summary)
}()
}
func (s *Scheduler) run(sub *models.Subscription, now time.Time) {
next, err := s.subscriptionSvc.ComputeNextRun(sub, now)
if err != nil {
Minternal/worker/scheduler_test.go
@@ -11,14 +11,16 @@ import (
"vidarchive/internal/models"
"vidarchive/internal/repository"
"vidarchive/internal/service"
"vidarchive/internal/tools"
)
type schedEnv struct {
sched *Scheduler
pool *Pool
subSvc *service.SubscriptionService
subRepo *repository.SubscriptionRepository
repo *repository.DownloadRepository
sched *Scheduler
pool *Pool
subSvc *service.SubscriptionService
settingsSvc *service.SettingsService
subRepo *repository.SubscriptionRepository
repo *repository.DownloadRepository
}
// newTestScheduler builds a scheduler over a real database. The pool is created
@@ -52,22 +54,24 @@ func newTestScheduler(t *testing.T) *schedEnv {
downloadRepo := repository.NewDownloadRepository(db)
subRepo := repository.NewSubscriptionRepository(db)
subSvc := service.NewSubscriptionService(subRepo, cfg)
settingsSvc := service.NewSettingsService(repository.NewSettingsRepository(db))
downloadSvc := service.NewDownloadService(
downloadRepo,
service.NewLibraryService(cfg.LibraryDir, cfg.FFmpegPath, cfg.FFprobePath),
service.NewPresetService(repository.NewPresetRepository(db)),
service.NewSettingsService(repository.NewSettingsRepository(db)),
settingsSvc,
subSvc,
cfg,
)
pool := New(downloadSvc, 1)
return &schedEnv{
sched: NewScheduler(subSvc, downloadSvc, pool, time.Minute),
pool: pool,
subSvc: subSvc,
subRepo: subRepo,
repo: downloadRepo,
sched: NewScheduler(subSvc, downloadSvc, settingsSvc, tools.New(cfg), pool, time.Minute),
pool: pool,
subSvc: subSvc,
settingsSvc: settingsSvc,
subRepo: subRepo,
repo: downloadRepo,
}
}
@@ -278,9 +282,56 @@ func stopWithin(t *testing.T, s *Scheduler, d time.Duration) {
func TestNewSchedulerRejectsNonPositiveInterval(t *testing.T) {
e := newTestScheduler(t)
for _, interval := range []time.Duration{0, -time.Second} {
s := NewScheduler(e.subSvc, nil, e.pool, interval)
s := NewScheduler(e.subSvc, nil, nil, nil, e.pool, interval)
if s.interval != time.Minute {
t.Errorf("interval %v became %v, want 1m", interval, s.interval)
}
}
}
// A comparison flipped the wrong way turns the daily check into one network
// round trip per tick, with nothing in the UI to show for it.
//
// The scheduler's config leaves both tools external and not opted in, so Update
// finds nothing to do and the test never reaches the network.
func TestCheckToolUpdateDailyGate(t *testing.T) {
now := time.Date(2026, 3, 1, 12, 0, 0, 0, time.UTC)
cases := []struct {
name string
autoUpdate bool
lastCheck time.Time
wantStamp bool
}{
{"opted out stays quiet", false, now.Add(-25 * time.Hour), false},
{"checked an hour ago waits", true, now.Add(-time.Hour), false},
{"never checked runs", true, time.Time{}, true},
{"checked 25 hours ago runs", true, now.Add(-25 * time.Hour), true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
e := newTestScheduler(t)
if err := e.settingsSvc.SetToolAutoUpdate(tc.autoUpdate); err != nil {
t.Fatal(err)
}
if !tc.lastCheck.IsZero() {
if err := e.settingsSvc.SetToolsCheckedAt(tc.lastCheck); err != nil {
t.Fatal(err)
}
}
e.sched.checkToolUpdate(now)
settings, err := e.settingsSvc.GetAll()
if err != nil {
t.Fatal(err)
}
// The stamp is written before the run, so it is what says whether the
// gate opened.
if got := settings.ToolsCheckedAt.Equal(now); got != tc.wantStamp {
t.Errorf("stamped = %v, want %v (last check %v)", got, tc.wantStamp, settings.ToolsCheckedAt)
}
})
}
}
Mweb/static/style.css
@@ -1283,3 +1283,27 @@ input[name="cron_expr"] {
justify-content: safe center;
}
}
.tools-table {
width: 100%;
border-collapse: collapse;
font-size: 0.9rem;
margin-bottom: 1rem;
}
/* Pico gives cells their own background and a bottom rule. Both are dropped:
the first column has no left padding so it lines up with the card heading,
which puts any cell edge right against the first character. */
.tools-table th,
.tools-table td {
padding: 0.3rem 0.6rem 0.3rem 0;
border: none;
background: none;
text-align: left;
vertical-align: top;
font-weight: normal;
}
.tools-table code {
word-break: break-all;
}
Mweb/templates/settings.html
@@ -33,6 +33,55 @@
</form>
</article>
<article>
<h3>Tools</h3>
<table class="tools-table">
<tbody>
<tr>
<th scope="row">yt-dlp</th>
<td>{{.Data.Tools.YTDLPVersion}}</td>
<td><code>{{.Data.Tools.YTDLPPath}}</code></td>
<td>{{.Data.Tools.YTDLPMode}}</td>
</tr>
<tr>
<th scope="row">JS runtime (deno)</th>
<td>{{.Data.Tools.DenoVersion}}</td>
<td><code>{{.Data.Tools.DenoPath}}</code></td>
<td>{{.Data.Tools.DenoMode}}</td>
</tr>
</tbody>
</table>
<form method="post" action="/settings/tools">
<label>
<input type="checkbox" name="tool_auto_update" value="1"
{{if .Data.Tools.AutoUpdate}}checked{{end}}>
Check for tool updates once a day
</label>
<label>
<input type="checkbox" name="js_runtime_enabled" value="1"
{{if .Data.Tools.JSRuntime}}checked{{end}}>
Install a JS runtime for sites with JavaScript challenges
</label>
<small>Deno is about 130 MB unpacked. yt-dlp picks it up from the
managed directory automatically.</small>
<div class="form-actions">
<button type="submit" class="primary">Save Tool Settings</button>
</div>
</form>
<form method="post" action="/settings/tools/update">
<small>Last automatic check:
{{if .Data.Tools.CheckedAt.IsZero}}never{{else}}{{.Data.Tools.CheckedAt.Format "2006-01-02 15:04"}}{{end}}</small>
<div class="form-actions">
<button type="submit" class="secondary">Update Now</button>
</div>
<small>Runs each managed tool's own updater. This can take a few minutes.</small>
</form>
</article>
<article>
<h3>Theme</h3>
<div class="theme-selector">