diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 903d081..6332947 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -25,6 +25,7 @@ Workspaces are declared in the root `package.json` (`workspaces`). Each lives in | `packages/vue` | `@data-fair/lib-vue` | Composables and utilities for Vue 3 applications (e.g. `useSession`, ws). | | `packages/vuetify` | `@data-fair/lib-vuetify` | UI components built on Vuetify. | | `packages/processing-dev` | `@data-fair/lib-processing-dev` | Utilities for the development environment of a processing. | +| `packages/dev` | `@data-fair/lib-dev` | Dev environment tooling of the services (`df-dev`, `df-worktree`, `df-mem-cap`, `df-status`). | ## Prerequisites diff --git a/README.md b/README.md index d9ea806..1dda4e5 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,7 @@ This is an npm-workspaces monorepo. Each package is published independently in t | [`@data-fair/lib-vue`](./packages/vue) | Composables and utilities for Vue 3 applications: session, fetch, reactive search params, websockets, notifications… | | [`@data-fair/lib-vuetify`](./packages/vuetify) | Vuetify components shared by the stack UIs: personal menu, owner pickers, theme and language switchers… | | [`@data-fair/lib-processing-dev`](./packages/processing-dev) | Development and test utilities for processing plugins. | +| [`@data-fair/lib-dev`](./packages/dev) | The dev environment of the services, for humans and agents: `df-dev` (setup, zellij session, status, logs, restart), `df-worktree`, `df-mem-cap`. | ## Coding agent skills diff --git a/package-lock.json b/package-lock.json index 6ca4671..9889f85 100644 --- a/package-lock.json +++ b/package-lock.json @@ -10,6 +10,7 @@ "license": "MIT", "workspaces": [ "packages/common-types", + "packages/dev", "packages/express", "packages/node", "packages/processing-dev", @@ -427,6 +428,10 @@ "resolved": "packages/common-types", "link": true }, + "node_modules/@data-fair/lib-dev": { + "resolved": "packages/dev", + "link": true + }, "node_modules/@data-fair/lib-express": { "resolved": "packages/express", "link": true @@ -8336,6 +8341,21 @@ } } }, + "packages/dev": { + "name": "@data-fair/lib-dev", + "version": "0.1.0", + "license": "MIT", + "dependencies": { + "commander": "^12.1.0" + }, + "bin": { + "df-dev": "cli.js", + "df-mem-cap": "mem-cap.sh", + "df-status": "status-pane.sh", + "df-worktree": "worktree.sh", + "df-worktree-delete": "worktree-delete.sh" + } + }, "packages/express": { "name": "@data-fair/lib-express", "version": "1.31.0", diff --git a/package.json b/package.json index 33b601e..eaab003 100644 --- a/package.json +++ b/package.json @@ -59,6 +59,7 @@ }, "workspaces": [ "packages/common-types", + "packages/dev", "packages/express", "packages/node", "packages/processing-dev", diff --git a/packages/dev/README.md b/packages/dev/README.md new file mode 100644 index 0000000..d711a65 --- /dev/null +++ b/packages/dev/README.md @@ -0,0 +1,39 @@ +# @data-fair/lib-dev + +Dev environment of a data-fair service, for humans and agents. Needs zellij >= 0.44 and docker compose v2. + +## In a service + +`dev/dev.json` declares what is specific to the repo: + +```json +{ + "ports": ["NGINX_PORT", "DEV_API_PORT", "DEV_UI_PORT", "MONGO_PORT"], + "panes": { "api": { "ready": "http://localhost:${DEV_API_PORT}/api/ping" } }, + "url": "http://${DEV_HOST}:${NGINX_PORT}/portals-manager/dev", + "optional": { "events": { "privateEventsUrl": "" } } +} +``` + +- `ports`: names of the port variables written to `.env`, each a free port drawn in 20000-29999, so worktrees never collide. `.env` also gets `DEV_HOST` (`.localhost`) and `SSRF_PUBLIC_IPS`; `init-env --force` draws them again and keeps the variables added by hand. +- `panes`: zellij pane titles; `ready` is polled by `restart`. +- `url`: the page to open, the UI's `/dev` page when it has one (its dev navigation bar), printed by `df-dev status` so an agent knows where to point a browser (the footer pane of the layout prints it with a plain `echo`, no node process). +- `setup`: repo-specific commands `df-dev setup` runs before build-types (e.g. `npm -w parquet-writer run build`). +- `optional`: one key per optional compose service (a service with a profile), its value the config overrides to apply **while that service does not run**, e.g. `{ "events": { "privateEventsUrl": "" }, "clamav": { "clamav": { "active": false } } }`: without events the API does not call it. Values must pass the config schema. `df-dev absent-config` merges the overrides of every stopped service; the api pane does `export NODE_CONFIG=$(df-dev absent-config)` (node-config merges it over the config files, no local file to write). +- No key for the UI: `setup` builds the `ui` workspace when `ui/dist/index.html` is missing (the API's serve-spa reads it at startup, and the build generates the auto-import declarations `check-types` needs). + +## Commands + +| Command | | +|---|---| +| `npx --no df-worktree ` / `npx --no df-worktree-delete [-f]` | from the main checkout: create `../_` (npm ci + setup) / delete it with its containers, volumes and zellij session; branch names: letters, digits, `.`, `_`, `-`, no `/`, and `_` (the zellij session) at most 60 chars | +| `npx --no df-dev setup` | first install / new worktree: `.env` if missing, `setup` commands, build-types, the `ui` build when `ui/dist/index.html` is missing | +| `npx --no df-dev init-env [--force]` | `.env` only | +| `npm run dev-zellij -- --services agents,events` | attach or create the session of this worktree (`df-dev zellij`) | +| `npx --no df-dev status` | for agents: containers, panes (running / exited / not ready), dev processes, memory | +| `df-status [--start-deps]` | the status pane: containers like `compose up --wait` and one RAM line, every 3 s, in bash (~4 MB); `--start-deps` starts the containers first | +| `npx --no df-dev logs [-n 100]` | last lines of a pane | +| `npx --no df-dev restart ` | Ctrl+C, rerun, wait for readiness | +| `npx --no df-dev up ` / `down ` | optional compose services | +| `npx --no df-dev wait-deps` | first command of the panes that need the containers | +| `df-mem-cap ` | run a dev server under a 4G cgroup cap (`DEV_MEMORY_MAX`) | diff --git a/packages/dev/cli.test.ts b/packages/dev/cli.test.ts new file mode 100644 index 0000000..1dbdd66 --- /dev/null +++ b/packages/dev/cli.test.ts @@ -0,0 +1,46 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { execFileSync, spawnSync } from 'node:child_process' +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, chmodSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' + +const cli = path.join(import.meta.dirname, 'cli.js') + +describe('df-dev cli', () => { + it('lists its commands', () => { + const out = execFileSync('node', [cli, '--help'], { encoding: 'utf8' }) + for (const cmd of ['setup', 'init-env', 'zellij', 'status', 'logs', 'restart', 'up', 'down', 'wait-deps']) assert.match(out, new RegExp(cmd)) + }) + + it('runs init-env from a sub-directory of a repo', () => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + mkdirSync(path.join(root, 'dev')) + mkdirSync(path.join(root, 'api')) + writeFileSync(path.join(root, 'dev/dev.json'), JSON.stringify({ ports: ['NGINX_PORT'] })) + const res = spawnSync('node', [cli, 'init-env'], { cwd: path.join(root, 'api'), encoding: 'utf8' }) + assert.match(readFileSync(path.join(root, '.env'), 'utf8'), /NGINX_PORT=2\d{4}/) + assert.equal(res.stderr, '', 'outside a git repo, git errors must not leak') + }) + + it('fails with a readable message, not a stack trace', () => { + try { + execFileSync('node', [cli, 'status'], { cwd: tmpdir(), encoding: 'utf8', stdio: 'pipe' }) + assert.fail('should have failed') + } catch (err: any) { + assert.match(err.stderr, /dev\/dev\.json not found/) + assert.ok(!err.stderr.includes(' at ')) + } + }) + + it('absent-config prints the overrides of the optional services that do not run', () => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + mkdirSync(path.join(root, 'dev')) + writeFileSync(path.join(root, 'dev/dev.json'), JSON.stringify({ optional: { events: { privateEventsUrl: null }, clamav: { clamav: { active: false } } } })) + const bin = mkdtempSync(path.join(tmpdir(), 'fakebin-')) + writeFileSync(path.join(bin, 'docker'), '#!/usr/bin/env bash\nprintf "mongodb\\nevents\\n"\n') + chmodSync(path.join(bin, 'docker'), 0o755) + const res = spawnSync('node', [cli, 'absent-config'], { cwd: root, encoding: 'utf8', env: { ...process.env, PATH: bin + ':' + process.env.PATH } }) + assert.deepEqual(JSON.parse(res.stdout), { clamav: { active: false } }) + }) +}) diff --git a/packages/dev/cli.ts b/packages/dev/cli.ts new file mode 100644 index 0000000..bff131f --- /dev/null +++ b/packages/dev/cli.ts @@ -0,0 +1,73 @@ +#!/usr/bin/env node +import { program } from 'commander' +import { findRoot, readDevConfig, readEnvFile } from './config.js' +import { initEnv } from './env.js' +import { setup } from './setup.js' +import { launch } from './launch.js' +import { logs, restart } from './panes.js' +import { composeUp, composeDown, waitDeps } from './compose.js' +import { collectStatus, renderStatus, checkReadiness } from './status.js' +import { absentConfig } from './optional.js' +import { realExec } from './exec.js' + +const root = () => findRoot(process.cwd()) + +program.name('df-dev').description('Dev environment of a data-fair service: setup, zellij session, status, logs and restart.') + +program.command('setup').description('create .env if missing, build types, build the ui workspace if missing (first install and new worktrees)') + .action(async () => { const r = root(); await setup(r, readDevConfig(r)) }) + +program.command('init-env').description('write .env with free ports if missing').option('--force', 'draw the ports and DEV_HOST again, keep the variables added by hand') + .action(async (opts) => { + const r = root() + const res = await initEnv(r, readDevConfig(r), { force: opts.force }) + console.log(res.written ? `wrote ${res.file}` : `${res.file} exists, use --force to regenerate it`) + }) + +program.command('zellij').description('attach to the session of this worktree, or create it').option('--services ', 'optional compose services to start, comma separated', '') + .action((opts) => { process.exitCode = launch(root(), opts.services.split(',').filter(Boolean)) }) + +program.command('status').description('containers, panes (with readiness) and dev processes memory, for agents (the status pane is df-status)') + .action(async () => { + const r = root() + const data = collectStatus(r) + try { + data.panes = await checkReadiness(data.panes, readDevConfig(r), readEnvFile(r)) + } catch (err: any) { + data.panesError = `readiness not checked: ${err.message}` + } + console.log(renderStatus(data, false)) + }) + +program.command('absent-config').description('JSON config overrides (dev.json "optional") for the optional services that do not run, for NODE_CONFIG') + .action(() => { + const r = root() + const running = realExec('docker', ['compose', '--project-directory', r, 'ps', '--services', '--status', 'running']).split('\n').filter(Boolean) + console.log(JSON.stringify(absentConfig(readDevConfig(r).optional ?? {}, running))) + }) + +program.command('logs ').description('last lines of a zellij pane, plain text').option('-n, --lines ', 'number of lines', '100') + .action((pane, opts) => { console.log(logs(root(), pane, Number(opts.lines))) }) + +program.command('restart ').description('stop and rerun a zellij pane, then wait for its readiness URL') + .action(async (pane) => { + const r = root() + const res = await restart(r, pane, readDevConfig(r)) + console.log(res.message) + if (res.logs) console.log('--- last logs ---\n' + res.logs) + if (!res.ok) process.exitCode = 1 + }) + +program.command('up ').description('start optional compose services in the running env') + .action((services) => { composeUp(root(), services) }) + +program.command('down ').description('stop optional compose services') + .action((services) => { composeDown(root(), services) }) + +program.command('wait-deps').description('wait until the containers of the active profiles are healthy (first command of process panes; the status pane starts them)') + .action(async () => { await waitDeps(root()) }) + +program.parseAsync().catch((err) => { + console.error(`df-dev: ${err.message}`) + process.exitCode = 1 +}) diff --git a/packages/dev/compose.test.ts b/packages/dev/compose.test.ts new file mode 100644 index 0000000..9b76c60 --- /dev/null +++ b/packages/dev/compose.test.ts @@ -0,0 +1,77 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { composeUp, composeDown, waitDeps, pendingDeps } from './compose.js' +import type { Exec } from './exec.js' + +const recorder = () => { + const calls: string[] = [] + return { calls, run: (cmd: string, args: string[]) => { calls.push([cmd, ...args].join(' ')) } } +} + +describe('compose', () => { + it('starts named services and waits for them', () => { + const r = recorder() + composeUp('/x', ['agents', 'events'], r.run) + assert.deepEqual(r.calls, ['docker compose up -d --wait agents events']) + }) + + it('refuses to stop the whole stack', () => { + assert.throws(() => composeDown('/x', [], recorder().run), /name the services/) + }) + + it('stops only the named services', () => { + const r = recorder() + composeDown('/x', ['agents'], r.run) + assert.deepEqual(r.calls, ['docker compose stop agents']) + }) + + it('lists the services of the active profiles that are not ready yet', () => { + const rows = [ + { Service: 'mongo', State: 'running', Health: 'healthy' }, + { Service: 'elasticsearch', State: 'running', Health: 'starting' }, + { Service: 'nginx', State: 'running', Health: '' } + ] + assert.deepEqual(pendingDeps(['elasticsearch', 'mongo', 'nginx', 'simple-directory'], rows), ['elasticsearch (starting)', 'simple-directory (not created)']) + }) + + it('counts a one-shot container that exited with 0 as done (s3-init)', () => { + assert.deepEqual(pendingDeps(['s3-init', 'job'], [{ Service: 's3-init', State: 'exited', ExitCode: 0 }, { Service: 'job', State: 'exited', ExitCode: 1 }]), ['job (exited 1)']) + }) + + it('waits for the containers without ever starting them (no race between panes)', async () => { + let t = 0 + const calls: string[] = [] + const exec: Exec = (cmd, args) => { + calls.push(args.join(' ')) + if (args.includes('config')) return 'mongo\n' + return t < 3000 ? '' : JSON.stringify({ Service: 'mongo', State: 'running', Health: 'healthy' }) + '\n' + } + await waitDeps('/x', { exec, sleep: async (ms) => { t += ms }, now: () => t, log: () => {} }) + assert.ok(t >= 3000) + assert.ok(!calls.some(c => c.includes(' up')), calls.join(' | ')) + }) + + it('fails at once when a dependency stays exited with an error code', async () => { + let t = 0 + const exec: Exec = (cmd, args) => args.includes('config') ? 'mongo\n' : JSON.stringify({ Service: 'mongo', State: 'exited', ExitCode: 1 }) + '\n' + await assert.rejects(waitDeps('/x', { exec, sleep: async (ms) => { t += ms }, now: () => t, log: () => {} }), /containers failed: mongo \(exited 1\).*docker compose logs mongo/) + assert.ok(t < 10_000, `waited ${t} ms`) + }) + + it('keeps waiting for a container that restarts after a failed attempt', async () => { + let t = 0 + const states = [{ State: 'exited', ExitCode: 1 }, { State: 'restarting' }, { State: 'running', Health: 'healthy' }] + const exec: Exec = (cmd, args) => { + if (args.includes('config')) return 'data-fair\n' + return JSON.stringify({ Service: 'data-fair', ...states[Math.min(Math.floor(t / 1000), states.length - 1)] }) + '\n' + } + await waitDeps('/x', { exec, sleep: async (ms) => { t += ms }, now: () => t, log: () => {} }) + assert.equal(t, 2000) + }) + + it('gives up after 3 minutes and names what is not ready', async () => { + let t = 0 + const exec: Exec = (cmd, args) => args.includes('config') ? 'mongo\n' : '' + await assert.rejects(waitDeps('/x', { exec, sleep: async (ms) => { t += ms }, now: () => t, log: () => {} }), /not ready after 180s: mongo \(not created\).*status pane/) + }) +}) diff --git a/packages/dev/compose.ts b/packages/dev/compose.ts new file mode 100644 index 0000000..156fd97 --- /dev/null +++ b/packages/dev/compose.ts @@ -0,0 +1,61 @@ +import { run as realRun, realExec, jsonLines, type Exec } from './exec.js' + +// optional services live in a compose profile named after them; naming a service starts it whatever its profile +export const composeUp = (root: string, services: string[], runFn = realRun) => { + runFn('docker', ['compose', 'up', '-d', '--wait', ...services], root) +} + +export const composeDown = (root: string, services: string[], runFn = realRun) => { + if (!services.length) throw new Error('name the services to stop, the whole stack is stopped by the user when closing zellij') + runFn('docker', ['compose', 'stop', ...services], root) +} + +type PsRow = { Service: string, State: string, Health?: string, ExitCode?: number } + +export const pendingDeps = (services: string[], rows: PsRow[]) => services.flatMap(service => { + const row = rows.find(r => r.Service === service) + if (!row) return [`${service} (not created)`] + if (row.State === 'running' && (!row.Health || row.Health === 'healthy')) return [] + if (row.State === 'exited' && row.ExitCode === 0) return [] // one-shot init container (s3-init…) that completed + return [`${service} (${row.Health || row.State}${row.State === 'exited' ? ` ${row.ExitCode}` : ''})`] +}) + +/** Dependencies that exited with an error code: they will not become ready by waiting. */ +export const failedDeps = (services: string[], rows: PsRow[]) => rows + .filter(r => services.includes(r.Service) && r.State === 'exited' && r.ExitCode !== 0) + .map(r => `${r.Service} (exited ${r.ExitCode})`) + +const WAIT_TIMEOUT = 180_000 +// a container with a restart policy is briefly "exited" between two attempts: fail on a lasting exit only +const FAILED_POLLS = 3 + +/** + * First command of the panes that need the containers. It only waits: the status pane starts + * the containers, and two panes running "compose up" at the same time conflict on container names. + * The services are those of the active profiles (COMPOSE_PROFILES comes from the zellij session env). + */ +export const waitDeps = async (root: string, deps: { exec?: Exec, sleep?: (ms: number) => Promise, now?: () => number, log?: (msg: string) => void } = {}) => { + const exec = deps.exec ?? realExec + const sleep = deps.sleep ?? ((ms: number) => new Promise(resolve => setTimeout(resolve, ms))) + const now = deps.now ?? Date.now + const log = deps.log ?? console.log + const compose = ['compose', '--project-directory', root] + const services = exec('docker', [...compose, 'config', '--services']).split('\n').filter(Boolean).sort() + const start = now() + let lastLog = -Infinity + let failedPolls = 0 + while (true) { + const rows: PsRow[] = jsonLines(exec('docker', [...compose, 'ps', '-a', '--format', 'json'])) + const pending = pendingDeps(services, rows) + if (!pending.length) return + const failed = failedDeps(services, rows) + failedPolls = failed.length ? failedPolls + 1 : 0 + if (failedPolls >= FAILED_POLLS) throw new Error(`containers failed: ${failed.join(', ')} (see "docker compose logs ${failed.map(f => f.split(' ')[0]).join(' ')}")`) + if (now() - start >= WAIT_TIMEOUT) throw new Error(`containers not ready after 180s: ${pending.join(', ')} (see the status pane)`) + if (now() - lastLog >= 10_000) { + log(`waiting for containers: ${pending.join(', ')}`) + lastLog = now() + } + await sleep(1000) + } +} diff --git a/packages/dev/config.test.ts b/packages/dev/config.test.ts new file mode 100644 index 0000000..e3219aa --- /dev/null +++ b/packages/dev/config.test.ts @@ -0,0 +1,51 @@ +/* eslint-disable no-template-curly-in-string -- ${VAR} placeholders are the dev/dev.json syntax */ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { mkdtempSync, mkdirSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' +import { findRoot, readDevConfig, interpolate, readEnvFile } from './config.js' + +const makeRepo = (devJson: object) => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + mkdirSync(path.join(root, 'dev')) + mkdirSync(path.join(root, 'api')) + writeFileSync(path.join(root, 'dev/dev.json'), JSON.stringify(devJson)) + return root +} + +describe('config', () => { + it('finds the repo root from a sub-directory', () => { + const root = makeRepo({}) + assert.equal(findRoot(path.join(root, 'api')), root) + }) + + it('fails clearly outside a repo using the convention', () => { + assert.throws(() => findRoot(tmpdir()), /dev\/dev\.json not found/) + }) + + it('fills defaults for missing sections', () => { + const root = makeRepo({ ports: ['NGINX_PORT'] }) + assert.deepEqual(readDevConfig(root), { ports: ['NGINX_PORT'], panes: {}, url: undefined, setup: [], optional: {} }) + }) + + it('interpolates variables', () => { + assert.equal(interpolate('http://localhost:${DEV_API_PORT}/api/ping', { DEV_API_PORT: '21000' }), 'http://localhost:21000/api/ping') + }) + + it('names a variable missing from .env and suggests regenerating it', () => { + assert.throws(() => interpolate('http://localhost:${DEV_PORTAL_PORT}/', {}), /DEV_PORTAL_PORT.*df-dev init-env --force/) + }) + + it('reads .env, ignoring comments and blank lines', () => { + const root = makeRepo({}) + writeFileSync(path.join(root, '.env'), '# comment\nA=1\n\nB=x=y\n') + assert.deepEqual(readEnvFile(root), { A: '1', B: 'x=y' }) + }) + + it('parses .env like dotenv (quotes, inline comments)', () => { + const root = makeRepo({}) + writeFileSync(path.join(root, '.env'), 'A="a b"\nB=x # note\nC=\'single\'\n') + assert.deepEqual(readEnvFile(root), { A: 'a b', B: 'x', C: 'single' }) + }) +}) diff --git a/packages/dev/config.ts b/packages/dev/config.ts new file mode 100644 index 0000000..890913b --- /dev/null +++ b/packages/dev/config.ts @@ -0,0 +1,41 @@ +import { readFileSync, existsSync } from 'node:fs' +import path from 'node:path' +import { parseEnv } from 'node:util' + +export type PaneOptions = { ready?: string } +export type DevConfig = { + ports: string[] + panes: Record + url?: string + setup?: string[] + optional?: Record> +} + +export const findRoot = (start: string): string => { + let dir = path.resolve(start) + while (true) { + if (existsSync(path.join(dir, 'dev/dev.json'))) return dir + const parent = path.dirname(dir) + if (parent === dir) throw new Error(`dev/dev.json not found in ${start} or its parents: this repo does not use the df-dev convention`) + dir = parent + } +} + +export const readDevConfig = (root: string): DevConfig => { + const raw = JSON.parse(readFileSync(path.join(root, 'dev/dev.json'), 'utf8')) + return { ports: raw.ports ?? [], panes: raw.panes ?? {}, url: raw.url, setup: raw.setup ?? [], optional: raw.optional ?? {} } +} + +export const interpolate = (template: string, env: Record) => + template.replace(/\$\{(\w+)\}/g, (_, name: string) => { + const value = env[name] + if (value === undefined) throw new Error(`variable ${name} is not defined in .env (added to dev/dev.json after .env was created?), regenerate it with "npx --no df-dev init-env --force"`) + return value + }) + +export const readEnvFile = (root: string): Record => { + const file = path.join(root, '.env') + if (!existsSync(file)) return {} + // same parser as node --env-file, so panes and df-dev read the same values + return parseEnv(readFileSync(file, 'utf8')) as Record +} diff --git a/packages/dev/env.test.ts b/packages/dev/env.test.ts new file mode 100644 index 0000000..6ea6bf2 --- /dev/null +++ b/packages/dev/env.test.ts @@ -0,0 +1,117 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' +import { pickPorts, hostLabel, initEnv, otherWorktreePorts, PORT_MIN, PORT_MAX } from './env.js' +import { readEnvFile } from './config.js' + +const config = { ports: ['NGINX_PORT', 'DEV_API_PORT'], panes: {} } + +describe('pickPorts', () => { + it('draws distinct free ports in range', async () => { + const ports = await pickPorts(config.ports, async () => true) + for (const name of ['NGINX_PORT', 'DEV_API_PORT']) { + assert.ok(ports[name] >= PORT_MIN && ports[name] <= PORT_MAX) + } + assert.notEqual(ports.NGINX_PORT, ports.DEV_API_PORT) + }) + + it('skips ports that are busy', async () => { + const busy = new Set([PORT_MIN]) + let n = 0 + const random = () => (n++ === 0 ? 0 : 0.5) // first draw hits the busy PORT_MIN + const ports = await pickPorts(['A'], async port => !busy.has(port), random) + assert.notEqual(ports.A, PORT_MIN) + }) +}) + +describe('hostLabel', () => { + it('turns any branch name into a valid hostname label', () => { + assert.equal(hostLabel('feat/X_y'), 'feat-x-y') + assert.equal(hostLabel('master'), 'master') + assert.equal(hostLabel('--weird--'), 'weird') + }) + + it('caps the label at 63 characters (DNS limit)', () => { + const label = hostLabel('a'.repeat(70) + '-b') + assert.equal(label.length, 63) + assert.ok(!label.endsWith('-')) + }) +}) + +describe('initEnv', () => { + it('writes ports, DEV_HOST and SSRF_PUBLIC_IPS', async () => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + const res = await initEnv(root, config, { branch: 'feat/X', isFree: async () => true }) + assert.equal(res.written, true) + const env = readEnvFile(root) + assert.equal(env.DEV_HOST, 'feat-x.localhost') + assert.equal(env.SSRF_PUBLIC_IPS, '127.0.0.1,::1') + assert.ok(Number(env.DEV_API_PORT) >= PORT_MIN) + }) + + it('never overwrites an existing .env without force', async () => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + writeFileSync(path.join(root, '.env'), 'NGINX_PORT=1\n') + const res = await initEnv(root, config, { branch: 'x', isFree: async () => true }) + assert.equal(res.written, false) + assert.equal(readFileSync(path.join(root, '.env'), 'utf8'), 'NGINX_PORT=1\n') + }) + + it('regenerates with force', async () => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + writeFileSync(path.join(root, '.env'), 'NGINX_PORT=1\n') + await initEnv(root, config, { force: true, branch: 'x', isFree: async () => true }) + assert.notEqual(readEnvFile(root).NGINX_PORT, '1') + }) + + it('keeps the variables added by hand when regenerating with force', async () => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + writeFileSync(path.join(root, '.env'), 'NGINX_PORT=1\nDEV_HOST=old.localhost\nMY_TOKEN=secret\n') + await initEnv(root, config, { force: true, branch: 'x', isFree: async () => true }) + const env = readEnvFile(root) + assert.equal(env.MY_TOKEN, 'secret') + assert.equal(env.DEV_HOST, 'x.localhost') + assert.notEqual(env.NGINX_PORT, '1') + }) + + it('keeps hand-added values that need quotes intact through a regeneration', async () => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + writeFileSync(path.join(root, '.env'), 'NGINX_PORT=1\nSECRET=\'abc#def\'\nMSG="a b"\nQUOTE=`it\'s`\nCERT=\'line1\nline2\'\nPLAIN=x\n') + await initEnv(root, config, { force: true, branch: 'x', isFree: async () => true, reserved: new Set() }) + await initEnv(root, config, { force: true, branch: 'x', isFree: async () => true, reserved: new Set() }) + const env = readEnvFile(root) + assert.equal(env.SECRET, 'abc#def') + assert.equal(env.MSG, 'a b') + assert.equal(env.QUOTE, "it's") + assert.equal(env.CERT, 'line1\nline2') + assert.equal(env.PLAIN, 'x') + assert.match(readFileSync(path.join(root, '.env'), 'utf8'), /^PLAIN=x$/m) + }) + + it('never draws a port written in the .env of another worktree', async () => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + // the first draw falls on a port another worktree uses, the second on a fresh one + const draws = [PORT_MIN + 5, PORT_MIN + 6].map(p => (p - PORT_MIN + 0.5) / (PORT_MAX - PORT_MIN + 1)) + let i = 0 + await initEnv(root, { ...config, ports: ['NGINX_PORT'] }, { branch: 'x', isFree: async () => true, reserved: new Set([PORT_MIN + 5]), random: () => draws[i++] }) + assert.equal(readEnvFile(root).NGINX_PORT, String(PORT_MIN + 6)) + assert.equal(i, 2) + }) + + it('reads the ports of the other worktrees of the repository', () => { + const tmp = mkdtempSync(path.join(tmpdir(), 'df-dev-wt-')) + const root = path.join(tmp, 'repo') + const other = path.join(tmp, 'repo_feat-a') + mkdirSync(root) + mkdirSync(other) + writeFileSync(path.join(other, '.env'), `NGINX_PORT=${PORT_MIN + 5}\nDEV_HOST=feat-a.localhost\nOTHER=80\n`) + writeFileSync(path.join(root, '.env'), `NGINX_PORT=${PORT_MIN + 6}\n`) + // output of git worktree list --porcelain, given as text: no git in this test + const list = () => `worktree ${root}\nHEAD abc\nbranch refs/heads/master\n\nworktree ${other}\nHEAD def\nbranch refs/heads/feat-a\n` + assert.deepEqual([...otherWorktreePorts(root, list)], [PORT_MIN + 5]) + assert.deepEqual([...otherWorktreePorts(other, list)], [PORT_MIN + 6]) + assert.deepEqual([...otherWorktreePorts(root, () => { throw new Error('not a git repository') })], []) + }) +}) diff --git a/packages/dev/env.ts b/packages/dev/env.ts new file mode 100644 index 0000000..c294450 --- /dev/null +++ b/packages/dev/env.ts @@ -0,0 +1,93 @@ +import { existsSync, writeFileSync } from 'node:fs' +import net from 'node:net' +import path from 'node:path' +import { execFileSync } from 'node:child_process' +import { readEnvFile, type DevConfig } from './config.js' + +// below the kernel ephemeral range (32768+), so a port is not taken later by an outgoing connection +export const PORT_MIN = 20000 +export const PORT_MAX = 29999 + +export const isPortFree = (port: number) => new Promise(resolve => { + const server = net.createServer() + server.once('error', () => resolve(false)) + server.listen(port, '127.0.0.1', () => server.close(() => resolve(true))) +}) + +export const pickPorts = async (ports: string[], isFree = isPortFree, random = Math.random) => { + const result: Record = {} + const taken = new Set() + for (const name of ports) { + for (let attempt = 0; ; attempt++) { + if (attempt >= 100) throw new Error(`no free port found for ${name} in ${PORT_MIN}-${PORT_MAX}`) + const port = PORT_MIN + Math.floor(random() * (PORT_MAX - PORT_MIN + 1)) + if (!taken.has(port) && await isFree(port)) { + result[name] = port + taken.add(port) + break + } + } + } + return result +} + +/** Ports already written in the .env of the other worktrees of the repository: free now while their env is stopped, taken later. */ +const gitWorktreeList = (root: string) => execFileSync('git', ['worktree', 'list', '--porcelain'], { cwd: root, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }) + +export const otherWorktreePorts = (root: string, worktreeList = gitWorktreeList) => { + const ports = new Set() + let list = '' + try { + list = worktreeList(root) + } catch { + return ports // not a git checkout: nothing to avoid + } + for (const line of list.split('\n')) { + if (!line.startsWith('worktree ')) continue + const dir = line.slice('worktree '.length) + if (path.resolve(dir) === path.resolve(root)) continue + for (const value of Object.values(readEnvFile(dir))) { + const port = Number(value) + if (Number.isInteger(port) && port >= PORT_MIN && port <= PORT_MAX) ports.add(port) + } + } + return ports +} + +// quoted only when parseEnv would otherwise cut or change the value (#, spaces, quotes, new lines) +export const formatEnvValue = (value: string | number) => { + const s = String(value) + if (/^[\w.,:/@%+=-]*$/.test(s)) return s + // single quotes are read literally by parseEnv, backticks for a value that contains one + for (const quote of ["'", '`', '"']) if (!s.includes(quote)) return quote + s + quote + throw new Error(`cannot write a .env value that contains ', \` and ": ${s}`) +} + +export const hostLabel = (branch: string) => + branch.toLowerCase().replace(/[^a-z0-9-]+/g, '-').replace(/-+/g, '-').replace(/^-|-$/g, '').slice(0, 63).replace(/-$/, '') || 'dev' + +const currentBranch = (root: string) => { + try { + return execFileSync('git', ['branch', '--show-current'], { cwd: root, encoding: 'utf8', stdio: ['ignore', 'pipe', 'ignore'] }).trim() || 'detached' + } catch { + return 'dev' + } +} + +export const initEnv = async (root: string, config: DevConfig, opts: { force?: boolean, branch?: string, isFree?: (port: number) => Promise, reserved?: Set, random?: () => number } = {}) => { + const file = path.join(root, '.env') + if (existsSync(file) && !opts.force) return { written: false, file } + const reserved = opts.reserved ?? otherWorktreePorts(root) + const isFree = opts.isFree ?? isPortFree + const ports = await pickPorts(config.ports, async port => !reserved.has(port) && await isFree(port), opts.random) + const vars: Record = { + // with --force, variables added by hand survive; the generated ones are drawn again + ...readEnvFile(root), + DEV_HOST: `${hostLabel(opts.branch ?? currentBranch(root))}.localhost`, + // all dev services listen on the loopback, the SSRF protection of the default http agents must accept it + SSRF_PUBLIC_IPS: '127.0.0.1,::1', + ...ports + } + writeFileSync(file, Object.entries(vars).map(([k, v]) => `${k}=${formatEnvValue(v)}`).join('\n') + '\n') + return { written: true, file } +} diff --git a/packages/dev/exec.test.ts b/packages/dev/exec.test.ts new file mode 100644 index 0000000..ccee394 --- /dev/null +++ b/packages/dev/exec.test.ts @@ -0,0 +1,10 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { realExec } from './exec.js' + +describe('realExec', () => { + it('reads outputs larger than 1 MiB (a full pane scrollback)', () => { + const out = realExec('node', ['-e', "process.stdout.write('x'.repeat(3 * 1024 * 1024))"]) + assert.equal(out.length, 3 * 1024 * 1024) + }) +}) diff --git a/packages/dev/exec.ts b/packages/dev/exec.ts new file mode 100644 index 0000000..7439485 --- /dev/null +++ b/packages/dev/exec.ts @@ -0,0 +1,14 @@ +import { execFileSync, spawnSync } from 'node:child_process' + +export type Exec = (cmd: string, args: string[]) => string + +export const realExec: Exec = (cmd, args) => execFileSync(cmd, args, { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 256 * 1024 * 1024 }) // a full pane scrollback exceeds the 1 MiB default + +/** Run a command with the terminal attached (output visible to the user), throw if it fails. */ +export const run = (cmd: string, args: string[], cwd: string, env: NodeJS.ProcessEnv = process.env) => { + const result = spawnSync(cmd, args, { cwd, env, stdio: 'inherit' }) + if (result.status !== 0) throw new Error(`${cmd} ${args.join(' ')} failed with exit code ${result.status}`) +} + +// docker compose prints one JSON object per line (older versions: a single JSON array) +export const jsonLines = (out: string): any[] => out.split('\n').filter(l => l.trim()).flatMap(l => JSON.parse(l)) diff --git a/packages/dev/launch.test.ts b/packages/dev/launch.test.ts new file mode 100644 index 0000000..1f4e71b --- /dev/null +++ b/packages/dev/launch.test.ts @@ -0,0 +1,22 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { launchPlan } from './launch.js' + +describe('launchPlan', () => { + it('creates a missing session with the layout', () => { + assert.deepEqual(launchPlan('absent', 'portals', []), { deleteFirst: false, args: ['--session', 'portals', '--new-session-with-layout', '.zellij.kdl'] }) + }) + + it('deletes an EXITED session before creating it again', () => { + assert.equal(launchPlan('exited', 'portals', []).deleteFirst, true) + }) + + it('attaches to a running session and never kills it', () => { + const plan = launchPlan('running', 'portals', []) + assert.deepEqual(plan, { deleteFirst: false, args: ['attach', 'portals'] }) + }) + + it('warns that --services is ignored when attaching to a running session', () => { + assert.match(launchPlan('running', 'portals', ['agents']).warning ?? '', /df-dev up agents/) + }) +}) diff --git a/packages/dev/launch.ts b/packages/dev/launch.ts new file mode 100644 index 0000000..226930e --- /dev/null +++ b/packages/dev/launch.ts @@ -0,0 +1,33 @@ +import { existsSync } from 'node:fs' +import path from 'node:path' +import { spawnSync } from 'node:child_process' +import { readEnvFile } from './config.js' +import { realExec, type Exec } from './exec.js' +import { sessionName, sessionState, checkVersion, type SessionState } from './zellij.js' + +export const launchPlan = (state: SessionState, session: string, services: string[]) => { + if (state === 'running') { + const plan: { deleteFirst: boolean, args: string[], warning?: string } = { deleteFirst: false, args: ['attach', session] } + if (services.length) plan.warning = `session ${session} is already running, --services ignored: add them with "npx --no df-dev up ${services.join(' ')}"` + return plan + } + // an EXITED session keeps its name reserved and makes the creation fail; + // --layout with --session would add a tab to an existing session instead of creating one + return { deleteFirst: state === 'exited', args: ['--session', session, '--new-session-with-layout', '.zellij.kdl'] } +} + +const realSpawn = (args: string[], env: NodeJS.ProcessEnv) => spawnSync('zellij', args, { stdio: 'inherit', env }).status ?? 1 + +export const launch = (root: string, services: string[], deps: { exec?: Exec, spawn?: typeof realSpawn } = {}) => { + const exec = deps.exec ?? realExec + if (!existsSync(path.join(root, '.env'))) throw new Error('no .env in this worktree, run "npx --no df-dev setup" first') + checkVersion(exec) + const session = sessionName(root) + const plan = launchPlan(sessionState(exec, session), session, services) + if (plan.warning) console.warn(plan.warning) + if (plan.deleteFirst) exec('zellij', ['delete-session', session]) + const env = { ...process.env, ...readEnvFile(root) } + if (services.length) env.COMPOSE_PROFILES = services.join(',') + process.chdir(root) + return (deps.spawn ?? realSpawn)(plan.args, env) +} diff --git a/packages/dev/mem-cap.sh b/packages/dev/mem-cap.sh new file mode 100755 index 0000000..28be9a7 --- /dev/null +++ b/packages/dev/mem-cap.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Run a dev process under a hard memory cap, so a runaway one is killed by its own cgroup +# instead of driving the whole host out of memory. A V8 heap limit is not enough: most of +# vite's memory is rolldown's native memory. No-op where systemd-run is unavailable. +# Usage: df-mem-cap vite --port 5173 (DEV_MEMORY_MAX=4G by default) +set -euo pipefail + +: "${DEV_MEMORY_MAX:=4G}" + +if command -v systemd-run >/dev/null 2>&1 && [ -n "${XDG_RUNTIME_DIR:-}" ] && [ -S "${XDG_RUNTIME_DIR}/systemd/private" ]; then + # systemd-run does not see the PATH additions of npm (node_modules/.bin) + cmd=$(command -v "$1") || { echo "df-mem-cap: command not found: $1" >&2; exit 127; } + shift + # inside a capped cgroup node sizes its heap from the cap (about half of it: 2.2 GB for 4G), + # so vite died of a V8 heap OOM long before the cgroup limit; give the heap 3/4 of the cap + # and leave the rest to native memory (rolldown), unless the caller already chose a size + if [[ "${NODE_OPTIONS:-}" != *--max-old-space-size* ]]; then + case "$DEV_MEMORY_MAX" in + *[Gg]) cap_mb=$(( ${DEV_MEMORY_MAX%[Gg]} * 1024 )) ;; + *[Mm]) cap_mb=${DEV_MEMORY_MAX%[Mm]} ;; + *) cap_mb= ;; + esac + [ -n "$cap_mb" ] && export NODE_OPTIONS="${NODE_OPTIONS:+$NODE_OPTIONS }--max-old-space-size=$(( cap_mb * 3 / 4 ))" + fi + # --collect: otherwise every OOM-killed run leaves a failed scope unit to reset by hand + exec systemd-run --user --scope --quiet --collect \ + --unit="dev-$(basename "$cmd")-$$" \ + -p MemoryMax="$DEV_MEMORY_MAX" \ + -p MemorySwapMax=0 \ + -- "$cmd" "$@" +fi + +exec "$@" diff --git a/packages/dev/mem-cap.test.ts b/packages/dev/mem-cap.test.ts new file mode 100644 index 0000000..4dbf83a --- /dev/null +++ b/packages/dev/mem-cap.test.ts @@ -0,0 +1,43 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { spawnSync } from 'node:child_process' +import { existsSync, mkdtempSync, writeFileSync, chmodSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' + +const script = path.join(import.meta.dirname, 'mem-cap.sh') +const runtimeDir = process.env.XDG_RUNTIME_DIR +// the cap only applies with a user systemd instance: the script checks its socket +const hasSystemd = !!runtimeDir && existsSync(path.join(runtimeDir, 'systemd/private')) + +// fake systemd-run on the PATH: prints the NODE_OPTIONS the capped command would get +const fakeBin = () => { + const bin = mkdtempSync(path.join(tmpdir(), 'fakebin-')) + writeFileSync(path.join(bin, 'systemd-run'), '#!/usr/bin/env bash\necho "$NODE_OPTIONS"\n') + chmodSync(path.join(bin, 'systemd-run'), 0o755) + return bin +} + +const nodeOptions = (env: Record) => { + const res = spawnSync('bash', [script, 'node', '-e', ''], { + env: { ...process.env, PATH: `${fakeBin()}:${process.env.PATH}`, NODE_OPTIONS: '', ...env }, + encoding: 'utf8' + }) + assert.equal(res.status, 0, res.stderr) + return res.stdout.trim() +} + +describe('df-mem-cap', { skip: !hasSystemd && 'no user systemd instance' }, () => { + it('gives the V8 heap 3/4 of the memory cap', () => { + assert.equal(nodeOptions({ DEV_MEMORY_MAX: '4G' }), '--max-old-space-size=3072') + assert.equal(nodeOptions({ DEV_MEMORY_MAX: '1536M' }), '--max-old-space-size=1152') + }) + + it('keeps the other node options', () => { + assert.equal(nodeOptions({ DEV_MEMORY_MAX: '2G', NODE_OPTIONS: '--no-warnings' }), '--no-warnings --max-old-space-size=1536') + }) + + it('never overrides a heap size chosen by the caller', () => { + assert.equal(nodeOptions({ NODE_OPTIONS: '--max-old-space-size=1000' }), '--max-old-space-size=1000') + }) +}) diff --git a/packages/dev/optional.test.ts b/packages/dev/optional.test.ts new file mode 100644 index 0000000..1705e2f --- /dev/null +++ b/packages/dev/optional.test.ts @@ -0,0 +1,27 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { absentConfig } from './optional.js' + +const optional = { + events: { privateEventsUrl: null }, + clamav: { clamav: { active: false } }, + agents: { privateAgentsUrl: null, agents: { chat: false } } +} + +describe('absentConfig', () => { + it('merges the overrides of the optional services that are not running', () => { + assert.deepEqual(absentConfig(optional, ['mongodb', 'es', 'events']), { + clamav: { active: false }, + privateAgentsUrl: null, + agents: { chat: false } + }) + }) + + it('is empty when every optional service runs', () => { + assert.deepEqual(absentConfig(optional, ['events', 'clamav', 'agents']), {}) + }) + + it('deep-merges overrides that touch the same object', () => { + assert.deepEqual(absentConfig({ a: { x: { one: 1 } }, b: { x: { two: 2 } } }, []), { x: { one: 1, two: 2 } }) + }) +}) diff --git a/packages/dev/optional.ts b/packages/dev/optional.ts new file mode 100644 index 0000000..83e5d4a --- /dev/null +++ b/packages/dev/optional.ts @@ -0,0 +1,23 @@ +type Obj = Record + +const isObj = (v: unknown): v is Obj => typeof v === 'object' && v !== null && !Array.isArray(v) + +const merge = (target: Obj, source: Obj) => { + for (const [key, value] of Object.entries(source)) { + target[key] = isObj(value) && isObj(target[key]) ? merge({ ...target[key] }, value) : value + } + return target +} + +/** + * Config overrides (dev/dev.json "optional") of the optional compose services that are not running. + * Exported as NODE_CONFIG by the api pane: node-config merges it over the config files, like a + * local-development file but without touching one the developer may already have. + */ +export const absentConfig = (optional: Record, running: string[]) => { + const result: Obj = {} + for (const [service, overrides] of Object.entries(optional)) { + if (!running.includes(service)) merge(result, overrides) + } + return result +} diff --git a/packages/dev/package.json b/packages/dev/package.json new file mode 100644 index 0000000..16a055f --- /dev/null +++ b/packages/dev/package.json @@ -0,0 +1,31 @@ +{ + "name": "@data-fair/lib-dev", + "version": "0.1.0", + "description": "Dev environment tooling of the data-fair services: setup, zellij session, status, logs and restart for humans and agents.", + "files": [ + "**/*.js", + "**/*.js.map", + "**/*.d.ts", + "mem-cap.sh", + "worktree.sh", + "worktree-delete.sh", + "status-pane.sh" + ], + "author": "", + "license": "MIT", + "scripts": { + "prepublishOnly": "cd .. && npm run prepublishOnly", + "build": "cd .. && npm run build" + }, + "dependencies": { + "commander": "^12.1.0" + }, + "type": "module", + "bin": { + "df-dev": "cli.js", + "df-mem-cap": "mem-cap.sh", + "df-worktree": "worktree.sh", + "df-worktree-delete": "worktree-delete.sh", + "df-status": "status-pane.sh" + } +} diff --git a/packages/dev/panes.test.ts b/packages/dev/panes.test.ts new file mode 100644 index 0000000..df41682 --- /dev/null +++ b/packages/dev/panes.test.ts @@ -0,0 +1,100 @@ +/* eslint-disable no-template-curly-in-string -- ${VAR} placeholders are the dev/dev.json syntax */ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { stripAnsi, tailLines, restart } from './panes.js' +import type { Exec } from './exec.js' + +const config = { ports: [], panes: { api: { ready: 'http://localhost:${DEV_API_PORT}/api/ping' } } } + +// a fake zellij whose api pane stops on Ctrl c (unless stubborn) and runs again on Enter +const makeZellij = (opts: { stubborn?: boolean, retitle?: boolean, dumpFails?: boolean } = {}) => { + const state = { exited: false, keys: [] as string[], title: 'api' } + const exec: Exec = (cmd, args) => { + if (args[0] === '--version') return 'zellij 0.45.1' + if (args[0] === 'list-sessions') return 'portals [Created 1m ago]\n' + const action = args[3] + if (action === 'list-panes') return JSON.stringify([{ id: 3, title: state.title, exited: state.exited, exit_status: state.exited ? 0 : null, is_plugin: false, terminal_command: 'node --watch index.ts' }]) + if (action === 'send-keys') { + const keys = args[6] + state.keys.push(keys) + if (keys === 'Ctrl c' && !opts.stubborn) state.exited = true + if (keys === 'Ctrl c' && opts.retitle) state.title = 'node (exited)' + if (keys === 'Enter') state.exited = false + return '' + } + if (action === 'dump-screen') { + if (opts.dumpFails) throw new Error('spawnSync zellij ENOBUFS') + return 'line1\nAPI crashed\n\n' + } + throw new Error('unexpected ' + args.join(' ')) + } + return { exec, state } +} + +const clock = () => { + let t = 0 + return { now: () => t, sleep: async (ms: number) => { t += ms } } +} + +describe('logs helpers', () => { + it('strips ANSI codes', () => { + assert.equal(stripAnsi('\x1b[32mVITE\x1b[0m ready'), 'VITE ready') + }) + it('keeps the last n non-trailing-blank lines', () => { + assert.equal(tailLines('a\nb\nc\n\n\n', 2), 'b\nc') + }) +}) + +describe('restart', () => { + it('stops the pane, reruns it and waits for readiness', async () => { + const { exec, state } = makeZellij() + const { now, sleep } = clock() + let checks = 0 + const res = await restart('/x/portals', 'api', config, { exec, now, sleep, env: { DEV_API_PORT: '21000' }, ready: async (url) => { assert.equal(url, 'http://localhost:21000/api/ping'); return ++checks >= 3 } }) + assert.deepEqual(state.keys, ['Ctrl c', 'Enter']) + assert.equal(res.ok, true) + assert.match(res.message, /ready after/) + }) + + it('reports a pane that ignores Ctrl c instead of pressing Enter on it', async () => { + const { exec, state } = makeZellij({ stubborn: true }) + const { now, sleep } = clock() + const res = await restart('/x/portals', 'api', config, { exec, now, sleep, env: { DEV_API_PORT: '21000' }, ready: async () => true }) + assert.equal(res.ok, false) + assert.match(res.message, /did not stop within 10s/) + assert.deepEqual(state.keys, ['Ctrl c']) + }) + + it('returns the last logs when the process never gets ready', async () => { + const { exec } = makeZellij() + const { now, sleep } = clock() + const res = await restart('/x/portals', 'api', config, { exec, now, sleep, env: { DEV_API_PORT: '21000' }, ready: async () => false }) + assert.equal(res.ok, false) + assert.match(res.message, /not ready after 60s/) + assert.match(res.logs ?? '', /API crashed/) + }) + + it('reruns an already exited pane without sending Ctrl c', async () => { + const { exec, state } = makeZellij() + state.exited = true + const { now, sleep } = clock() + await restart('/x/portals', 'api', config, { exec, now, sleep, env: { DEV_API_PORT: '21000' }, ready: async () => true }) + assert.deepEqual(state.keys, ['Enter']) + }) + + it('follows the pane by id when its title changes after Ctrl c', async () => { + const { exec, state } = makeZellij({ retitle: true }) + const { now, sleep } = clock() + const res = await restart('/x/portals', 'api', config, { exec, now, sleep, env: { DEV_API_PORT: '21000' }, ready: async () => true }) + assert.deepEqual(state.keys, ['Ctrl c', 'Enter']) + assert.equal(res.ok, true) + }) + + it('keeps the failure message when the logs cannot be read', async () => { + const { exec } = makeZellij({ dumpFails: true }) + const { now, sleep } = clock() + const res = await restart('/x/portals', 'api', config, { exec, now, sleep, env: { DEV_API_PORT: '21000' }, ready: async () => false }) + assert.match(res.message, /not ready after 60s/) + assert.match(res.logs ?? '', /logs unavailable: .*ENOBUFS/) + }) +}) diff --git a/packages/dev/panes.ts b/packages/dev/panes.ts new file mode 100644 index 0000000..fa23b12 --- /dev/null +++ b/packages/dev/panes.ts @@ -0,0 +1,77 @@ +import type { DevConfig } from './config.js' +import { interpolate, readEnvFile } from './config.js' +import { realExec, type Exec } from './exec.js' +import { requireSession, findPane, listPanes, dumpScreen, sendKeys, type Pane } from './zellij.js' + +// eslint-disable-next-line no-control-regex -- matching ANSI escape sequences is the point +export const stripAnsi = (text: string) => text.replace(/\x1b\[[0-9;?]*[ -/]*[@-~]/g, '') + +export const tailLines = (text: string, n: number) => text.replace(/\s+$/, '').split('\n').slice(-n).join('\n') + +export const logs = (root: string, title: string, n: number, exec: Exec = realExec) => { + const session = requireSession(exec, root) + return tailLines(stripAnsi(dumpScreen(exec, session, findPane(exec, session, title))), n) +} + +export type RestartResult = { ok: boolean, message: string, logs?: string } + +export const checkReady = async (url: string) => { + try { + return (await fetch(url, { signal: AbortSignal.timeout(2000) })).ok + } catch { + return false + } +} + +const STOP_TIMEOUT = 10_000 +const READY_TIMEOUT = 60_000 + +export const restart = async (root: string, title: string, config: DevConfig, deps: { + exec?: Exec, + ready?: (url: string) => Promise, + sleep?: (ms: number) => Promise, + now?: () => number, + env?: Record +} = {}): Promise => { + const exec = deps.exec ?? realExec + const ready = deps.ready ?? checkReady + const sleep = deps.sleep ?? ((ms: number) => new Promise(resolve => setTimeout(resolve, ms))) + const now = deps.now ?? Date.now + const session = requireSession(exec, root) + let pane = findPane(exec, session, title) + // the failure message must survive a failing dump + const lastLogs = () => { + try { + return tailLines(stripAnsi(dumpScreen(exec, session, pane)), 20) + } catch (err: any) { + return `logs unavailable: ${err.message}` + } + } + // follow the pane by id: a process can retitle its pane while exiting + const current = (): Pane => { + const found = listPanes(exec, session).find(p => p.id === pane.id) + if (!found) throw new Error(`pane "${title}" disappeared from session ${session}`) + return found + } + if (!pane.terminal_command) throw new Error(`pane "${title}" has no command to rerun`) + // resolve the URL first, so a broken .env fails before anything is stopped + const readyUrl = config.panes[title]?.ready && interpolate(config.panes[title].ready!, deps.env ?? readEnvFile(root)) + + if (!pane.exited) { + sendKeys(exec, session, pane, 'Ctrl c') + const deadline = now() + STOP_TIMEOUT + while (!(pane = current()).exited) { + if (now() > deadline) return { ok: false, message: `pane "${title}" did not stop within 10s after Ctrl+C`, logs: lastLogs() } + await sleep(200) + } + } + sendKeys(exec, session, pane, 'Enter') + + if (!readyUrl) return { ok: true, message: `pane "${title}" restarted (no readiness URL in dev/dev.json, check its logs)` } + const start = now() + while (now() - start < READY_TIMEOUT) { + if (await ready(readyUrl)) return { ok: true, message: `pane "${title}" ready after ${((now() - start) / 1000).toFixed(1)}s` } + await sleep(500) + } + return { ok: false, message: `pane "${title}" not ready after 60s (${readyUrl}), report to the user instead of retrying`, logs: lastLogs() } +} diff --git a/packages/dev/setup.test.ts b/packages/dev/setup.test.ts new file mode 100644 index 0000000..747c552 --- /dev/null +++ b/packages/dev/setup.test.ts @@ -0,0 +1,51 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { mkdtempSync, mkdirSync, readFileSync, writeFileSync, existsSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' +import { setup } from './setup.js' + +const makeRepo = (scripts: Record, ui = true) => { + const root = mkdtempSync(path.join(tmpdir(), 'df-dev-')) + writeFileSync(path.join(root, 'package.json'), JSON.stringify({ scripts })) + if (ui) { + mkdirSync(path.join(root, 'ui')) + writeFileSync(path.join(root, 'ui/package.json'), '{}') + } + return root +} +const config = { ports: ['NGINX_PORT'], panes: {} } + +describe('setup', () => { + it('creates .env, runs build-types then builds the ui workspace', async () => { + const root = makeRepo({ 'build-types': 'df-build-types api/types' }) + const calls: string[] = [] + await setup(root, config, { run: (cmd, args) => { calls.push([cmd, ...args].join(' ')) }, isFree: async () => true }) + assert.ok(existsSync(path.join(root, '.env'))) + assert.deepEqual(calls, ['npm run build-types', 'npm -w ui run build']) + }) + + it('does not rebuild an existing UI build and skips build-types when the repo has none', async () => { + const root = makeRepo({}) + mkdirSync(path.join(root, 'ui/dist')) + writeFileSync(path.join(root, 'ui/dist/index.html'), 'real') + const calls: string[] = [] + await setup(root, config, { run: (cmd, args) => { calls.push(cmd) }, isFree: async () => true }) + assert.deepEqual(calls, []) + assert.equal(readFileSync(path.join(root, 'ui/dist/index.html'), 'utf8'), 'real') + }) + + it('builds no UI in a repo without a ui workspace', async () => { + const root = makeRepo({}, false) + const calls: string[] = [] + await setup(root, config, { run: (cmd, args) => { calls.push(cmd) }, isFree: async () => true }) + assert.deepEqual(calls, []) + }) + + it('runs the repo-specific setup commands before build-types', async () => { + const root = makeRepo({ 'build-types': 'df-build-types api/types' }) + const calls: string[] = [] + await setup(root, { ...config, setup: ['npm -w parquet-writer run build'] }, { run: (cmd, args) => { calls.push([cmd, ...args].join(' ')) }, isFree: async () => true }) + assert.deepEqual(calls, ['bash -c npm -w parquet-writer run build', 'npm run build-types', 'npm -w ui run build']) + }) +}) diff --git a/packages/dev/setup.ts b/packages/dev/setup.ts new file mode 100644 index 0000000..c6857d1 --- /dev/null +++ b/packages/dev/setup.ts @@ -0,0 +1,20 @@ +import { existsSync, readFileSync } from 'node:fs' +import path from 'node:path' +import type { DevConfig } from './config.js' +import { initEnv } from './env.js' +import { run as realRun } from './exec.js' + +export const setup = async (root: string, config: DevConfig, deps: { run?: typeof realRun, isFree?: (port: number) => Promise } = {}) => { + const run = deps.run ?? realRun + const env = await initEnv(root, config, { isFree: deps.isFree }) + console.log(env.written ? `created ${env.file}` : `kept existing ${env.file} (npx --no df-dev init-env --force to regenerate)`) + const pkg = JSON.parse(readFileSync(path.join(root, 'package.json'), 'utf8')) + // repo-specific steps (native modules…) that the types or the UI build depend on + for (const command of config.setup ?? []) run('bash', ['-c', command], root) + if (pkg.scripts?.['build-types']) run('npm', ['run', 'build-types'], root) + // the API's serve-spa reads ui/dist/index.html at startup, and the build generates the + // auto-import declarations check-types needs: build once, later runs keep the existing build + if (existsSync(path.join(root, 'ui/package.json')) && !existsSync(path.join(root, 'ui/dist/index.html'))) { + run('npm', ['-w', 'ui', 'run', 'build'], root) + } +} diff --git a/packages/dev/status-pane.sh b/packages/dev/status-pane.sh new file mode 100755 index 0000000..a30401c --- /dev/null +++ b/packages/dev/status-pane.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# Status pane of the zellij layout: containers (like `docker compose up --wait`) and one RAM line, +# refreshed every 3 s. Plain bash + awk on purpose: it runs all day in every session, a node +# process would hold ~55 MB for this. Agents use the one-shot `df-dev status` instead. +# Usage (from the repo root): df-status [--start-deps] [--once] +set -uo pipefail + +start_deps=false +once=false +for arg in "$@"; do + case "$arg" in + --start-deps) start_deps=true ;; + --once) once=true ;; + esac +done + +session=$(basename "$PWD") +up_status=$(mktemp) +# services the background `docker compose up` will start: shown as waiting until it returns +up_pending=$(mktemp) +trap 'rm -f "$up_status" "$up_pending"' EXIT + +# the status pane is the one that starts the containers: in the background, so the view shows them coming up +if [ "$start_deps" = true ]; then + docker compose config --services > "$up_pending" 2>/dev/null + ( docker compose up -d >/dev/null 2>"$up_status.err"; code=$?; [ "$code" -ne 0 ] && echo "docker compose up failed ($code): $(tail -1 "$up_status.err")" > "$up_status"; rm -f "$up_status.err"; : > "$up_pending" ) & +fi + +if [ -n "${NO_COLOR:-}" ]; then G=; Y=; R=; D=; N=; else G=$'\e[32m'; Y=$'\e[33m'; R=$'\e[31m'; D=$'\e[2m'; N=$'\e[0m'; fi + +# the compose file does not change while the session runs: parse it once, not every 3 s +services=$(docker compose --profile '*' config --services 2>/dev/null | sort) + +render() { + local ps names stats procs + ps=$(docker compose ps -a --format '{{.Service}}\t{{.State}}\t{{.Health}}\t{{.ExitCode}}\t{{.Name}}' 2>/dev/null) + names=$(printf '%s\n' "$ps" | awk -F'\t' '$2 == "running" { print $5 }') + stats="" + # shellcheck disable=SC2086 + [ -n "$names" ] && stats=$(docker stats --no-stream --format '{{.Name}}\t{{.MemUsage}}' $names 2>/dev/null) + procs=$(ps -u "$(id -u)" -o pid=,ppid=,rss=,args= 2>/dev/null) + + [ -s "$up_status" ] && printf '%s%s%s\n\n' "$R" "$(cat "$up_status")" "$N" + + # data goes through the environment: awk -v would interpret the backslashes of the ps arguments + S_SERVICES="$services" S_PS="$ps" S_STATS="$stats" S_PROCS="$procs" S_PENDING="$(cat "$up_pending")" \ + awk -F'\t' -v G="$G" -v Y="$Y" -v R="$R" -v D="$D" -v N="$N" -v session="$session" -v self="$$" ' + function kib(v, n, u) { + n = v + 0; u = v; sub(/^[0-9.]+ */, "", u) + if (u == "GiB") return n * 1048576; if (u == "MiB") return n * 1024; if (u == "KiB") return n + if (u == "GB") return n * 976562.5; if (u == "MB") return n * 976.5625; if (u == "kB") return n * 0.9765625 + return 0 + } + function human(k) { return k >= 1048576 ? sprintf("%.1fG", k / 1048576) : sprintf("%dM", k / 1024 + 0.5) } + BEGIN { + services = ENVIRON["S_SERVICES"]; ps = ENVIRON["S_PS"]; stats = ENVIRON["S_STATS"]; procs = ENVIRON["S_PROCS"] + nps = split(ps, rows, "\n") + for (i = 1; i <= nps; i++) if (split(rows[i], f, "\t") >= 5) { state[f[1]] = f[2]; health[f[1]] = f[3]; code[f[1]] = f[4]; cname[f[1]] = f[5]; if (f[2] == "running") running = 1 } + npe = split(ENVIRON["S_PENDING"], pe, "\n") + for (i = 1; i <= npe; i++) pending[pe[i]] = 1 + nst = split(stats, srows, "\n") + for (i = 1; i <= nst; i++) if (split(srows[i], f, "\t") >= 2) { split(f[2], m, " / "); mem[f[1]] = m[1] } + + nsv = split(services, svc, "\n") + for (i = 1; i <= nsv; i++) { + s = svc[i]; if (s == "") continue + if (!(s in state)) { printf "%s· %-18s not started%s\n", D, s, N; continue } + st = state[s]; h = health[s]; text = (h != "" ? h : st) + # a one-shot init container (s3-init) that completed, unless the whole stack is down: a stopped mongo exits with 0 too + if (st == "exited" && code[s] == "0" && running) { printf "%s✔%s %-18s %sdone%s\n", G, N, s, G, N; continue } + if (st == "exited" && code[s] == "0") { printf "%s· %-18s stopped%s\n", D, s, N; continue } + if (st != "running" && (s in pending)) { printf "%s⠿%s %-18s %swaiting%s\n", Y, N, s, Y, N; continue } + if (st == "exited") text = "exited " code[s] + if (text == "healthy" || (st == "running" && h == "")) { c = G; mark = "✔" } + else if (text == "starting" || st == "restarting" || st == "created") { c = Y; mark = "⠿" } + else { c = R; mark = "✘" } + k = (cname[s] in mem) ? kib(mem[cname[s]]) : 0 + containers += k + line = sprintf("%s%s%s %-18s %s%-11s%s %s", c, mark, N, s, c, text, N, (k ? human(k) : "")) + sub(/ +$/, "", line); print line + } + + # dev servers: descendants of this session zellij server, without this status script + npr = split(procs, prows, "\n") + for (i = 1; i <= npr; i++) { + if (split(prows[i], f, " ") < 4) continue + pid[i] = f[1]; ppid[i] = f[2]; rss[i] = f[3] + if (prows[i] ~ / --server / && prows[i] ~ ("/" session "$")) server = f[1] + } + if (server != "") { + insession[server] = 1; out[self] = 1 + do { + grew = 0 + for (i in pid) { + if (!(pid[i] in insession) && (ppid[i] in insession)) { insession[pid[i]] = 1; grew = 1 } + if (!(pid[i] in out) && (ppid[i] in out)) { out[pid[i]] = 1; grew = 1 } + } + } while (grew) + for (i in pid) if (pid[i] != server && (pid[i] in insession) && !(pid[i] in out)) devservers += rss[i] + } + printf "\nRAM containers %s · dev servers %s · total %s\n", human(containers), human(devservers), human(containers + devservers) + }' +} + +while true; do + view=$(render) + # also clear the scrollback, or the pane history grows by a screen every refresh + printf '\e[H\e[2J\e[3J%s\n' "$view" + [ "$once" = true ] && exit 0 + sleep 3 +done diff --git a/packages/dev/status-pane.test.ts b/packages/dev/status-pane.test.ts new file mode 100644 index 0000000..7454035 --- /dev/null +++ b/packages/dev/status-pane.test.ts @@ -0,0 +1,75 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { spawnSync } from 'node:child_process' +import { mkdtempSync, mkdirSync, writeFileSync, chmodSync, readFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' + +// fake docker and ps on the PATH: the status pane script only reads their output +const RUNNING_PS = 'mongo\\trunning\\thealthy\\t0\\tp-mongo-1\\nes\\trunning\\tstarting\\t0\\tp-es-1\\ns3-init\\texited\\t\\t0\\tp-s3-init-1\\nsd\\texited\\t\\t1\\tp-sd-1\\n' +const fakeBin = (session: string, psOut = RUNNING_PS, log = '/dev/null') => { + const bin = mkdtempSync(path.join(tmpdir(), 'fakebin-')) + const write = (name: string, body: string) => { writeFileSync(path.join(bin, name), '#!/usr/bin/env bash\n' + body); chmodSync(path.join(bin, name), 0o755) } + write('docker', `echo "$*" >> ${log} +case "$*" in + *"config --services"*) printf 'agents\\nmongo\\ns3-init\\nes\\nsd\\n' ;; + *"up -d"*) sleep 1 ;; + *"compose ps"*) printf '${psOut}' ;; + *stats*) printf 'p-mongo-1\\t512MiB / 31GiB\\np-es-1\\t1.5GiB / 31GiB\\n' ;; +esac`) + write('ps', `printf '%s\\n' \\ + ' 100 1 20000 /usr/bin/zellij --server /run/user/1000/zellij/0.45.1/${session}' \\ + ' 101 100 1048576 node --watch index.ts' \\ + ' 102 101 524288 node index.ts' \\ + ' 200 1 900000 claude'`) + return bin +} + +describe('status pane (bash)', () => { + it('renders containers like compose up --wait and one RAM line', () => { + const root = path.join(mkdtempSync(path.join(tmpdir(), 'repo-')), 'portals_x') + mkdirSync(root) + const res = spawnSync('bash', [path.join(import.meta.dirname, 'status-pane.sh'), '--once'], { cwd: root, encoding: 'utf8', env: { ...process.env, PATH: fakeBin('portals_x') + ':' + process.env.PATH, NO_COLOR: '1' } }) + assert.equal(res.status, 0, res.stderr) + // eslint-disable-next-line no-control-regex -- strips the ANSI codes of the pane output + const lines = res.stdout.replace(/\x1b\[[0-9;]*[A-Za-z]/g, '').split('\n').filter(Boolean) + assert.deepEqual(lines, [ + '· agents not started', + '⠿ es starting 1.5G', + '✔ mongo healthy 512M', + '✔ s3-init done', + '✘ sd exited 1', + 'RAM containers 2.0G · dev servers 1.5G · total 3.5G' + ]) + }) + + it('shows the containers the background compose up has not started yet as waiting, not failed', () => { + const root = path.join(mkdtempSync(path.join(tmpdir(), 'repo-')), 'portals_x') + mkdirSync(root) + const res = spawnSync('bash', [path.join(import.meta.dirname, 'status-pane.sh'), '--once', '--start-deps'], { cwd: root, encoding: 'utf8', env: { ...process.env, PATH: fakeBin('portals_x') + ':' + process.env.PATH, NO_COLOR: '1' } }) + assert.equal(res.status, 0, res.stderr) + assert.match(res.stdout, /⠿ sd\s+waiting/) + assert.match(res.stdout, /✔ s3-init\s+done/) + }) + + it('shows a stack stopped with exit code 0 as stopped, not done', () => { + const root = path.join(mkdtempSync(path.join(tmpdir(), 'repo-')), 'portals_x') + mkdirSync(root) + const stopped = 'mongo\\texited\\t\\t0\\tp-mongo-1\\ns3-init\\texited\\t\\t0\\tp-s3-init-1\\n' + const res = spawnSync('bash', [path.join(import.meta.dirname, 'status-pane.sh'), '--once'], { cwd: root, encoding: 'utf8', env: { ...process.env, PATH: fakeBin('portals_x', stopped) + ':' + process.env.PATH, NO_COLOR: '1' } }) + assert.equal(res.status, 0, res.stderr) + assert.match(res.stdout, /· mongo\s+stopped/) + assert.match(res.stdout, /· s3-init\s+stopped/) + assert.ok(!res.stdout.includes('done')) + }) + + it('parses the compose file once, not at every refresh', () => { + const root = path.join(mkdtempSync(path.join(tmpdir(), 'repo-')), 'portals_x') + mkdirSync(root) + const log = path.join(root, 'docker.log') + spawnSync('timeout', ['4', 'bash', path.join(import.meta.dirname, 'status-pane.sh')], { cwd: root, encoding: 'utf8', env: { ...process.env, PATH: fakeBin('portals_x', RUNNING_PS, log) + ':' + process.env.PATH, NO_COLOR: '1' } }) + const calls = readFileSync(log, 'utf8').split('\n') + assert.ok(calls.filter(c => c.includes('compose ps')).length >= 2, 'refreshed at least twice') + assert.equal(calls.filter(c => c.includes('config --services')).length, 1) + }) +}) diff --git a/packages/dev/status.test.ts b/packages/dev/status.test.ts new file mode 100644 index 0000000..60a3d42 --- /dev/null +++ b/packages/dev/status.test.ts @@ -0,0 +1,131 @@ +/* eslint-disable no-template-curly-in-string -- ${VAR} placeholders are the dev/dev.json syntax */ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { containerMark, renderStatus, sessionProcesses, collectStatus, checkReadiness } from './status.js' +import type { Exec } from './exec.js' + +describe('status', () => { + it('marks containers like docker compose up --wait', () => { + assert.equal(containerMark({ service: 'mongo', state: 'running', health: 'healthy' }).mark, '✔') + assert.equal(containerMark({ service: 'sd', state: 'running', health: '' }).mark, '✔') + assert.equal(containerMark({ service: 'es', state: 'running', health: 'starting' }).mark, '⠿') + assert.equal(containerMark({ service: 'df', state: 'exited', health: '' }).color, 'red') + assert.equal(containerMark({ service: 'agents', state: 'not started', health: '' }).color, 'dim') + assert.deepEqual(containerMark({ service: 's3-init', state: 'exited', health: '', exitCode: 0 }), { mark: '✔', color: 'green', text: 'done' }) + }) + + it('shows a stack stopped with exit code 0 as stopped, not done', () => { + const out = renderStatus({ + containers: [{ service: 'mongo', state: 'exited', health: '', exitCode: 0 }, { service: 'nginx', state: 'exited', health: '', exitCode: 0 }], + panes: [], + procs: [] + }, false) + assert.match(out, /· mongo\s+stopped/) + assert.match(out, /· nginx\s+stopped/) + assert.ok(!out.includes('done')) + }) + + it('renders containers, processes and the total in plain text', () => { + const out = renderStatus({ + containers: [{ service: 'mongo', state: 'running', health: 'healthy', memory: '115MiB' }], + panes: [], + procs: [{ label: 'api: node --watch index.ts', rssKb: 245760 }] + }, false) + assert.match(out, /✔ mongo\s+healthy\s+115MiB/) + assert.match(out, /api: node --watch index.ts\s+240M/) + assert.match(out, /processes total\s+240M/) + assert.ok(!out.includes('\x1b[')) + }) + + it('counts only the processes of the zellij session, with their full command line', () => { + const ps = [ + ' 100 1 20000 /usr/bin/zellij --server /run/user/1000/zellij/0.45.1/portals_feat-x', + ' 101 100 3000 /bin/bash -c npx --no df-dev wait-deps && node --watch index.ts', + ' 102 101 245760 node --watch index.ts', + ' 103 100 3000 bash -c vite', + ' 104 103 1048576 /home/u/.nvm/versions/node/v24.21.0/bin/node /x/portals_feat-x/node_modules/.bin/vite --port 21001', + ' 200 1 470000 claude', + ' 201 200 120000 npm exec chrome-devtools-mcp', + ' 300 1 20000 /usr/bin/zellij --server /run/user/1000/zellij/0.45.1/portals' + ].join('\n') + const cwds: Record = { 102: '/x/portals_feat-x/api', 104: '/x/portals_feat-x/ui' } + const procs = sessionProcesses(ps, 'portals_feat-x', pid => cwds[pid], '/x/portals_feat-x') + assert.deepEqual(procs, [ + { label: 'api: node --watch index.ts', rssKb: 245760 }, + { label: 'ui: vite --port 21001', rssKb: 1048576 } + ]) + }) + + it('finds children listed before their parent (pid wrap-around)', () => { + const ps = [ + ' 50 900 245760 node --watch index.ts', + ' 100 1 20000 /usr/bin/zellij --server /run/user/1000/zellij/0.45.1/portals', + ' 900 100 3000 bash -c node --watch index.ts' + ].join('\n') + assert.deepEqual(sessionProcesses(ps, 'portals', () => '/x/portals/api', '/x/portals').map(p => p.rssKb), [245760]) + }) + + it('shows exited panes with their exit code', () => { + const out = renderStatus({ + containers: [], + panes: [{ title: 'api', exited: true, exit_status: 1 }, { title: 'ui', exited: false, exit_status: null }], + procs: [] + }, false) + assert.match(out, /✘ api\s+exited \(1\)/) + assert.match(out, /✔ ui\s+running/) + }) + + it('still shows panes and processes when docker or ps fails, and says when the session is not running', () => { + const exec: Exec = (cmd, args) => { + if (cmd === 'docker') throw new Error('Cannot connect to the Docker daemon') + if (args[0] === 'list-sessions') throw new Error('No active zellij sessions found.') + if (cmd === 'ps') throw new Error('ps: command not found') + throw new Error('unexpected ' + cmd + ' ' + args.join(' ')) + } + const data = collectStatus('/x/portals', exec) + assert.deepEqual(data.procs, []) + assert.match(data.containersError ?? '', /Docker daemon/) + assert.match(data.panesError ?? '', /npm run dev-zellij/) + const out = renderStatus(data, false) + assert.match(out, /containers unavailable: .*Docker daemon/) + assert.match(out, /npm run dev-zellij/) + }) + + it('leaves out the status process itself and shortens executable paths', () => { + const ps = [ + ' 100 1 20000 /usr/bin/zellij --server /run/user/1000/zellij/0.45.1/portals', + ' 110 100 50000 node ./node_modules/.bin/df-dev status --watch', + ' 111 110 5000 ps -u 1000 -o pid=,ppid=,rss=,args=', + ' 120 100 90000 /home/u/.nvm/versions/node/v24.21.0/bin/node --watch index.ts' + ].join('\n') + const procs = sessionProcesses(ps, 'portals', () => '/x/portals/api', '/x/portals', 110) + assert.deepEqual(procs.map(p => p.label), ['api: node --watch index.ts']) + }) + + it('shows the dev URL first, in full so the terminal can open them', () => { + const out = renderStatus({ url: 'http://feat-x.localhost:28119/portals-manager/dev', containers: [], panes: [], procs: [] }, false) + assert.equal(out.split('\n')[0], 'http://feat-x.localhost:28119/portals-manager/dev') + }) + + it('leaves out the status pane and its ps children', () => { + const ps = [ + ' 100 1 20000 /usr/bin/zellij --server /run/user/1000/zellij/0.45.1/portals', + ' 130 100 60000 node /x/portals/node_modules/.bin/df-dev status --watch --start-deps', + ' 131 130 4000 ps -u 1000 -o pid=,ppid=,rss=,args=', + ' 140 100 3500 bash ./node_modules/.bin/df-status --start-deps', + ' 141 140 2000 sleep 3', + ' 120 100 90000 node --watch index.ts' + ].join('\n') + assert.deepEqual(sessionProcesses(ps, 'portals', () => '/x/portals/api', '/x/portals').map(p => p.label), ['api: node --watch index.ts']) + }) + + it('flags a running pane whose readiness URL does not answer (node --watch waiting after a crash)', async () => { + const panes = [{ title: 'api', exited: false, exit_status: null }, { title: 'ui', exited: false, exit_status: null }, { title: 'shell', exited: false, exit_status: null }] + const config = { ports: [], panes: { api: { ready: 'http://localhost:${P}/api/ping' }, ui: { ready: 'http://localhost:${P}/' } } } + const checked = await checkReadiness(panes, config, { P: '1' }, async url => !url.includes('api')) + assert.deepEqual(checked.map(p => p.ready), [false, true, undefined]) + const out = renderStatus({ containers: [], panes: checked, procs: [] }, false) + assert.match(out, /⠿ api\s+running, not ready \(http:\/\/localhost:1\/api\/ping\)/) + assert.match(out, /✔ ui\s+running/) + }) +}) diff --git a/packages/dev/status.ts b/packages/dev/status.ts new file mode 100644 index 0000000..8a12355 --- /dev/null +++ b/packages/dev/status.ts @@ -0,0 +1,132 @@ +import { readlinkSync } from 'node:fs' +import path from 'node:path' +import { realExec, jsonLines, type Exec } from './exec.js' +import { sessionName, sessionState, listPanes } from './zellij.js' +import { readDevConfig, readEnvFile, interpolate, type DevConfig } from './config.js' +import { checkReady } from './panes.js' + +export type Container = { service: string, state: string, health: string, memory?: string, exitCode?: number } +export type Proc = { label: string, rssKb: number } +export type PaneState = { title: string, exited: boolean, exit_status: number | null, ready?: boolean, readyUrl?: string } +export type StatusData = { url?: string, containers: Container[], containersError?: string, panes: PaneState[], panesError?: string, procs: Proc[] } + +const COLORS = { green: '32', yellow: '33', red: '31', dim: '2' } as const + +export const containerMark = (c: Container, stackRunning = true) => { + if (c.state === 'not started') return { mark: '·', color: 'dim' as const, text: 'not started' } + if (c.state === 'exited' && c.exitCode === 0) { + // a one-shot init container (s3-init) that completed, unless the whole stack is down: a stopped mongo exits with 0 too + return stackRunning ? { mark: '✔', color: 'green' as const, text: 'done' } : { mark: '·', color: 'dim' as const, text: 'stopped' } + } + const text = c.health || c.state + if (text === 'healthy' || (c.state === 'running' && !c.health)) return { mark: '✔', color: 'green' as const, text } + if (text === 'starting' || c.state === 'restarting' || c.state === 'created') return { mark: '⠿', color: 'yellow' as const, text } + return { mark: '✘', color: 'red' as const, text } +} + +const human = (kb: number) => kb >= 1048576 ? `${(kb / 1048576).toFixed(1)}G` : `${Math.round(kb / 1024)}M` + +export const renderStatus = (data: StatusData, color: boolean) => { + const paint = (c: keyof typeof COLORS, s: string) => color ? `\x1b[${COLORS[c]}m${s}\x1b[0m` : s + const lines = data.url ? [data.url, ''] : [] + const stackRunning = data.containers.some(c => c.state === 'running') + lines.push(...data.containersError + ? [paint('red', `containers unavailable: ${data.containersError}`)] + : data.containers.map(c => { + const m = containerMark(c, stackRunning) + return `${paint(m.color, m.mark)} ${c.service.padEnd(18)} ${paint(m.color, m.text.padEnd(11))} ${c.memory ?? ''}`.trimEnd() + })) + lines.push('') + if (data.panesError) lines.push(paint('red', data.panesError)) + for (const p of data.panes) { + if (p.exited) lines.push(`${paint('red', '✘')} ${p.title.padEnd(18)} ${paint('red', `exited (${p.exit_status ?? '?'})`)}`) + else if (p.ready === false) lines.push(`${paint('yellow', '⠿')} ${p.title.padEnd(18)} ${paint('yellow', `running, not ready (${p.readyUrl})`)}`) + else lines.push(`${paint('green', '✔')} ${p.title.padEnd(18)} ${paint('green', 'running')}`) + } + lines.push('') + for (const p of data.procs) lines.push(`${p.label.slice(0, 40).padEnd(40)} ${human(p.rssKb)}`) + lines.push(`${'processes total'.padEnd(40)} ${human(data.procs.reduce((sum, p) => sum + p.rssKb, 0))}`) + return lines.join('\n') +} + +const SHELLS = ['bash', 'sh', 'zsh'] + +/** Processes started by the panes of a zellij session: descendants of its server, shells excluded. */ +export const sessionProcesses = (psOut: string, session: string, cwdOf: (pid: number) => string | undefined, root: string, selfPid?: number): Proc[] => { + const rows = psOut.split('\n').map(line => line.trim().match(/^(\d+)\s+(\d+)\s+(\d+)\s+(.*)$/)).filter(m => !!m) + .map(m => ({ pid: Number(m[1]), ppid: Number(m[2]), rssKb: Number(m[3]), args: m[4] })) + const server = rows.find(r => / --server \S*\/zellij\/\S+\//.test(r.args) && r.args.endsWith('/' + session)) + if (!server) return [] + const inSession = new Set([server.pid]) + // the status commands (this one, the df-status pane) and their children (ps, sleep, docker) are tooling, not dev processes + const self = new Set([...(selfPid ? [selfPid] : []), ...rows.filter(r => / df-dev status\b|df-status\b/.test(' ' + r.args.replace(/^\S*node\s+\S*\//, ' '))).map(r => r.pid)]) + let grew = true + while (grew) { // ps order is not parents-first when pids wrap around + grew = false + for (const r of rows) { + if (!inSession.has(r.pid) && inSession.has(r.ppid)) { inSession.add(r.pid); grew = true } + if (!self.has(r.pid) && self.has(r.ppid)) { self.add(r.pid); grew = true } + } + } + const procs: Proc[] = [] + for (const r of rows) { + if (r.pid === server.pid || !inSession.has(r.pid) || self.has(r.pid) || SHELLS.includes(path.basename(r.args.split(' ')[0]))) continue + const cwd = cwdOf(r.pid) + const dir = cwd && (cwd === root || cwd.startsWith(root + path.sep)) ? (path.relative(root, cwd) || '.') : '?' + procs.push({ label: `${dir}: ${r.args.replace(/^(\S*\/)?node\s+(?=\S*node_modules\/\.bin\/)/, '').replace(/^\S*node_modules\/\.bin\//, '').replace(/^\S*\//, '')}`, rssKb: r.rssKb }) + } + return procs +} + +const collectContainers = (root: string, exec: Exec): Container[] => { + const ps = jsonLines(exec('docker', ['compose', '--project-directory', root, 'ps', '-a', '--format', 'json'])) + const all = exec('docker', ['compose', '--project-directory', root, '--profile', '*', 'config', '--services']).split('\n').filter(Boolean) + const memory: Record = {} + if (ps.length) { + for (const s of jsonLines(exec('docker', ['stats', '--no-stream', '--format', '{{json .}}', ...ps.map((c: any) => c.Name)]))) { + memory[s.Name] = String(s.MemUsage).split(' / ')[0] + } + } + return all.sort().map(service => { + const c = ps.find((p: any) => p.Service === service) + return c ? { service, state: c.State, health: c.Health ?? '', memory: memory[c.Name], exitCode: c.ExitCode } : { service, state: 'not started', health: '' } + }) +} + +// each part fails on its own: a docker error must not hide the panes, and the reverse +export const collectStatus = (root: string, exec: Exec = realExec): StatusData => { + const data: StatusData = { containers: [], panes: [], procs: [] } + try { + const env = readEnvFile(root) + const url = readDevConfig(root).url + if (url) data.url = interpolate(url, env) + } catch {} // the URL is a convenience, the rest of the status must still show + try { + data.containers = collectContainers(root, exec) + } catch (err: any) { + data.containersError = err.message.split('\n')[0] + } + const session = sessionName(root) + try { + const state = sessionState(exec, session) + if (state === 'running') data.panes = listPanes(exec, session).map(p => ({ title: p.title, exited: p.exited, exit_status: p.exit_status })) + else data.panesError = `zellij session "${session}" is not running (${state}): ask the user to start it with "npm run dev-zellij"` + } catch (err: any) { + data.panesError = `panes unavailable: ${err.message.split('\n')[0]}` + } + const cwdOf = (pid: number) => { try { return readlinkSync(`/proc/${pid}/cwd`) } catch { return undefined } } + try { + const user = process.getuid ? ['-u', String(process.getuid())] : ['-e'] + data.procs = sessionProcesses(exec('ps', [...user, '-o', 'pid=,ppid=,rss=,args=']), session, cwdOf, root, process.pid) + } catch {} // the process list is the least important part + return data +} + +/** A running pane can be broken: nodemon (or node --watch) stays alive "waiting for file changes" after a crash. */ +export const checkReadiness = async (panes: PaneState[], config: DevConfig, env: Record, ready = checkReady): Promise => + Promise.all(panes.map(async p => { + const template = config.panes[p.title]?.ready + if (p.exited || !template) return p + const readyUrl = interpolate(template, env) + return { ...p, ready: await ready(readyUrl), readyUrl } + })) diff --git a/packages/dev/worktree-delete.sh b/packages/dev/worktree-delete.sh new file mode 100755 index 0000000..e7c0bb0 --- /dev/null +++ b/packages/dev/worktree-delete.sh @@ -0,0 +1,123 @@ +#!/bin/bash +# Delete a worktree (and its docker containers/volumes/local images). +# +# Refuses to proceed if the worktree has uncommitted changes or commits not +# pushed to any origin ref. Pass -f / --force to bypass these checks. + +set -e + +FORCE=false +BRANCH_NAME="" + +while [ $# -gt 0 ]; do + case "$1" in + -f|--force) FORCE=true ;; + -h|--help) + echo "Usage: df-worktree-delete [-f|--force]" >&2 + echo " --force / -f skip uncommitted/unpushed safety checks" + exit 0 + ;; + -*) + echo "Error: unknown flag '$1'" >&2 + echo "Usage: df-worktree-delete [-f|--force]" >&2 + exit 1 + ;; + *) + if [ -z "$BRANCH_NAME" ]; then + BRANCH_NAME="$1" + else + echo "Error: too many positional arguments" >&2 + exit 1 + fi + ;; + esac + shift +done + +if [ -z "$BRANCH_NAME" ]; then + echo "Error: Please provide a branch name." >&2 + echo "Usage: df-worktree-delete [-f|--force]" >&2 + exit 1 +fi + +# the branch name becomes a directory, a zellij session and a DEV_HOST label +if ! [[ "$BRANCH_NAME" =~ ^[A-Za-z0-9._-]{1,60}$ ]]; then + echo "Error: invalid branch name \"$BRANCH_NAME\": letters, digits, '.', '_' and '-' only, no \"/\", at most 60 characters (e.g. feat-xyz)" >&2 + exit 1 +fi + +# the worktrees live next to the main checkout: refuse a subfolder or another worktree +TOPLEVEL=$(git rev-parse --show-toplevel 2>/dev/null) || { echo "Error: not inside a git repository." >&2; exit 1; } +if [ "$TOPLEVEL" != "$(pwd -P)" ]; then + echo "Error: run df-worktree-delete from the root of the repository: cd $TOPLEVEL" >&2 + exit 1 +fi +COMMON_DIR=$(git rev-parse --path-format=absolute --git-common-dir) +if [ "$(git rev-parse --absolute-git-dir)" != "$COMMON_DIR" ]; then + echo "Error: run df-worktree-delete from the main checkout, not from a worktree: cd $(dirname "$COMMON_DIR")" >&2 + exit 1 +fi + +REPO_NAME=$(basename "$PWD") +TARGET_DIR="../${REPO_NAME}_${BRANCH_NAME}" + +if [ ! -d "$TARGET_DIR" ]; then + echo "Error: Worktree directory $TARGET_DIR does not exist." + exit 1 +fi + +ORIGIN_DIR="$PWD" + +if [ "$FORCE" != true ]; then + echo "Checking $TARGET_DIR for uncommitted / unpushed work…" + cd "$TARGET_DIR" + + # Refresh remote refs so the "unpushed" check is accurate + git fetch origin --quiet 2>/dev/null || true + + DIRTY=$(git status --porcelain) + if [ -n "$DIRTY" ]; then + echo + echo "❌ Refusing to delete: $TARGET_DIR has uncommitted changes:" + echo "$DIRTY" | sed 's/^/ /' + echo + echo "Commit / push / stash them, or pass -f / --force to bypass." + exit 1 + fi + + # Commits reachable from HEAD but NOT reachable from any origin/* ref. + # Catches: never-pushed branches, branches ahead of upstream, detached HEAD with unique commits. + UNPUSHED_COUNT=$(git rev-list --count HEAD --not --remotes=origin 2>/dev/null || echo 0) + if [ "$UNPUSHED_COUNT" -gt 0 ]; then + echo + echo "❌ Refusing to delete: $TARGET_DIR has $UNPUSHED_COUNT commit(s) not present on any origin ref:" + git --no-pager log --oneline HEAD --not --remotes=origin | sed 's/^/ /' + echo + echo "Push them (e.g. 'git push -u origin $BRANCH_NAME'), or pass -f / --force to bypass." + exit 1 + fi + + cd "$ORIGIN_DIR" + echo "✓ working tree clean and all commits present on origin" +fi + +echo "Stopping & removing docker compose services in $TARGET_DIR (with volumes and orphans)" +cd "$TARGET_DIR" +# --profile '*' : catch every container the worktree may have started, whatever the repo names its profiles +# -v : drop named volumes (mongo, elasticsearch, s3mock, clamav…) — data is worktree-scoped +# --remove-orphans : sweep any container left over from a previous compose file +# --rmi local : remove images built locally for this worktree (keeps remote-pulled images) +docker compose --profile '*' down -v --remove-orphans --rmi local + +echo "Removing git worktree at $TARGET_DIR" +cd "$ORIGIN_DIR" +if [ "$FORCE" = true ]; then + git worktree remove --force "$TARGET_DIR" # untracked files (generated types, UI builds) would block a plain remove +else + git worktree remove "$TARGET_DIR" +fi +zellij delete-session "$(basename "$TARGET_DIR")" --force >/dev/null 2>&1 || true + +echo "-----------------------------------------------" +echo "✅ Worktree $BRANCH_NAME deleted!" +echo "-----------------------------------------------" diff --git a/packages/dev/worktree.sh b/packages/dev/worktree.sh new file mode 100755 index 0000000..3dd35e2 --- /dev/null +++ b/packages/dev/worktree.sh @@ -0,0 +1,164 @@ +#!/bin/bash +# Create a git worktree for the given branch with full dev setup. +# +# Handles four cases: +# 1. Branch is currently checked out HERE (with or without local changes): +# stash the changes, switch this checkout back to the main branch, move +# the requested branch into the worktree, then re-apply the stash inside +# the worktree. +# 2. Branch exists locally but is not active here: just check it out in the +# worktree (e.g. coming back to a branch after a previous worktree was +# deleted). +# 3. Branch exists only on origin: create a tracking local branch in the +# worktree. +# 4. Brand new branch name: create from the current branch (or main if +# current is detached). + +set -e + +BRANCH_NAME=$1 + +if [ -z "$BRANCH_NAME" ]; then + echo "Error: Please provide a branch name." >&2 + echo "Usage: df-worktree feat-xyz" >&2 + exit 1 +fi + +# the branch name becomes a directory, a zellij session and a DEV_HOST label +if ! [[ "$BRANCH_NAME" =~ ^[A-Za-z0-9._-]{1,60}$ ]]; then + echo "Error: invalid branch name \"$BRANCH_NAME\": letters, digits, '.', '_' and '-' only, no \"/\", at most 60 characters (e.g. feat-xyz)" >&2 + exit 1 +fi + +# the worktree goes next to the main checkout: refuse a subfolder or another worktree +TOPLEVEL=$(git rev-parse --show-toplevel 2>/dev/null) || { echo "Error: not inside a git repository." >&2; exit 1; } +if [ "$TOPLEVEL" != "$(pwd -P)" ]; then + echo "Error: run df-worktree from the root of the repository: cd $TOPLEVEL" >&2 + exit 1 +fi +COMMON_DIR=$(git rev-parse --path-format=absolute --git-common-dir) +if [ "$(git rev-parse --absolute-git-dir)" != "$COMMON_DIR" ]; then + echo "Error: run df-worktree from the main checkout, not from a worktree: cd $(dirname "$COMMON_DIR")" >&2 + exit 1 +fi + +REPO_NAME=$(basename "$PWD") +TARGET_DIR="../${REPO_NAME}_${BRANCH_NAME}" +ORIGIN_DIR="$PWD" + +# the zellij session is named after the worktree folder and its socket path is capped at 107 bytes +# (/run/user//zellij/contract_version_1/): 60 characters leave room for the prefix +SESSION_MAX=60 +if [ $(( ${#REPO_NAME} + 1 + ${#BRANCH_NAME} )) -gt $SESSION_MAX ]; then + echo "Error: branch name \"$BRANCH_NAME\" too long for this repository: the zellij session \"${REPO_NAME}_${BRANCH_NAME}\" must fit in $SESSION_MAX characters, so at most $(( SESSION_MAX - ${#REPO_NAME} - 1 )) characters for the branch" >&2 + exit 1 +fi + +if [ -e "$TARGET_DIR" ]; then + echo "Error: $TARGET_DIR already exists." + echo "Run npx --no df-worktree-delete $BRANCH_NAME first if you want to recreate it." + exit 1 +fi + +# Detect main branch (main vs master) — prefer origin/HEAD, fall back to local refs +MAIN_BRANCH=$(git symbolic-ref --short refs/remotes/origin/HEAD 2>/dev/null | sed 's|^origin/||' || true) +if [ -z "$MAIN_BRANCH" ]; then + if git show-ref --verify --quiet refs/heads/main; then + MAIN_BRANCH=main + elif git show-ref --verify --quiet refs/heads/master; then + MAIN_BRANCH=master + else + echo "Error: could not detect main branch (neither 'main' nor 'master' found)." + exit 1 + fi +fi + +echo "Fetching from origin" +git fetch origin --prune + +CURRENT_BRANCH=$(git branch --show-current || true) +LOCAL_EXISTS=no +REMOTE_EXISTS=no +git show-ref --verify --quiet "refs/heads/$BRANCH_NAME" && LOCAL_EXISTS=yes +git show-ref --verify --quiet "refs/remotes/origin/$BRANCH_NAME" && REMOTE_EXISTS=yes + +STASH_SHA="" +STASH_RESTORED=no +# a failure after the stash must not leave the user's changes buried in the shared stash stack +restore_stash() { + local code=$? + if [ "$code" -ne 0 ] && [ -n "$STASH_SHA" ] && [ "$STASH_RESTORED" != yes ]; then + if [ "$(git -C "$ORIGIN_DIR" branch --show-current)" = "$BRANCH_NAME" ] && git -C "$ORIGIN_DIR" stash apply -q "$STASH_SHA"; then + drop_stash + echo "Error: worktree setup failed, your local changes are back in $ORIGIN_DIR" >&2 + else + echo "Error: worktree setup failed, your local changes are saved in the stash $STASH_SHA: git stash apply $STASH_SHA" >&2 + fi + fi +} +drop_stash() { + local ref + ref=$(git -C "$ORIGIN_DIR" stash list --format='%gd %H' | awk -v sha="$STASH_SHA" '$2 == sha { print $1 }') + [ -n "$ref" ] && git -C "$ORIGIN_DIR" stash drop -q "$ref" + return 0 +} +trap restore_stash EXIT + +if [ "$LOCAL_EXISTS" = "yes" ] && [ "$CURRENT_BRANCH" = "$BRANCH_NAME" ]; then + echo "Branch '$BRANCH_NAME' is currently checked out in $PWD — migrating it to the worktree." + if [ -n "$(git status --porcelain)" ]; then + STASH_LABEL="worktree-migration-${BRANCH_NAME}-$(date +%s)" + echo "Stashing local changes (incl. untracked) as '$STASH_LABEL'" + git stash push --include-untracked --message "$STASH_LABEL" + # the stash stack is shared by every worktree: keep the sha, a stash@{n} index moves + STASH_SHA=$(git stash list --format='%H %gs' | grep -F "$STASH_LABEL" | head -1 | awk '{print $1}') + fi + echo "Switching this checkout to $MAIN_BRANCH" + git checkout "$MAIN_BRANCH" +fi + +if [ "$LOCAL_EXISTS" = "yes" ]; then + echo "Reusing existing local branch '$BRANCH_NAME' in $TARGET_DIR" + git worktree add "$TARGET_DIR" "$BRANCH_NAME" +elif [ "$REMOTE_EXISTS" = "yes" ]; then + echo "Creating local branch '$BRANCH_NAME' tracking origin/$BRANCH_NAME in $TARGET_DIR" + git worktree add -b "$BRANCH_NAME" --track "$TARGET_DIR" "origin/$BRANCH_NAME" +else + SOURCE_BRANCH="${CURRENT_BRANCH:-$MAIN_BRANCH}" + echo "Creating new branch '$BRANCH_NAME' from '$SOURCE_BRANCH' in $TARGET_DIR" + git worktree add -b "$BRANCH_NAME" "$TARGET_DIR" "$SOURCE_BRANCH" +fi + +if [ -n "$STASH_SHA" ]; then + echo "Re-applying stashed changes inside the worktree" + ( cd "$TARGET_DIR" && git stash apply "$STASH_SHA" ) + STASH_RESTORED=yes + drop_stash +fi + +cd "$TARGET_DIR" + +if [ -d "$OLDPWD/.claude" ]; then + echo "Copy local Claude settings" + mkdir -p .claude + for f in "$OLDPWD/.claude"/*.json; do + [ -f "$f" ] && cp "$f" .claude/ + done +fi + +echo "npm ci" +npm ci + +echo "df-dev setup (.env, types, UI build)" +./node_modules/.bin/df-dev setup # local bin only, never fetched from the registry + +echo "-----------------------------------------------" +echo "✅ Setup Complete!" +echo "Location: $TARGET_DIR" +echo "Branch: $BRANCH_NAME" +if [ -n "$STASH_SHA" ]; then + echo "(Local changes from your previous checkout were migrated.)" +fi +echo "-----------------------------------------------" +echo "Next step:" +echo " cd $TARGET_DIR && npm run dev-zellij" diff --git a/packages/dev/worktree.test.ts b/packages/dev/worktree.test.ts new file mode 100644 index 0000000..3881631 --- /dev/null +++ b/packages/dev/worktree.test.ts @@ -0,0 +1,98 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { spawnSync } from 'node:child_process' +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, realpathSync } from 'node:fs' +import { tmpdir } from 'node:os' +import path from 'node:path' + +const script = (name: string) => path.join(import.meta.dirname, name) +// git and the scripts must only see the throwaway repositories: a git hook (pre-push runs the tests) +// exports GIT_DIR and friends, which would point every command below at the real repository +const isolatedEnv = { + ...Object.fromEntries(Object.entries(process.env).filter(([key]) => !key.startsWith('GIT_'))), + GIT_AUTHOR_NAME: 't', + GIT_AUTHOR_EMAIL: 't@t', + GIT_COMMITTER_NAME: 't', + GIT_COMMITTER_EMAIL: 't@t', + GIT_CONFIG_GLOBAL: '/dev/null', + GIT_CONFIG_NOSYSTEM: '1', + GIT_CEILING_DIRECTORIES: tmpdir() +} +const run = (name: string, args: string[], cwd = mkdtempSync(path.join(tmpdir(), 'wt-'))) => spawnSync('bash', [script(name), ...args], { cwd, encoding: 'utf8', env: isolatedEnv }) + +const git = (cwd: string, ...args: string[]) => { + const res = spawnSync('git', args, { cwd, encoding: 'utf8', env: isolatedEnv }) + assert.equal(res.status, 0, `git ${args.join(' ')}: ${res.stderr}`) + return res.stdout.trim() +} +// a main checkout / cloned from a bare origin, on master with one commit +const checkout = (repo = 'portals') => { + const tmp = mkdtempSync(path.join(tmpdir(), 'wt-repo-')) + git(tmp, 'init', '-q', '--bare', '-b', 'master', 'origin.git') + git(tmp, 'clone', '-q', path.join(tmp, 'origin.git'), repo) + const root = path.join(tmp, repo) + // never commit anywhere but in the throwaway clone + assert.equal(git(root, 'rev-parse', '--show-toplevel'), realpathSync(root)) + writeFileSync(path.join(root, 'README.md'), 'x\n') + git(root, 'add', '.') + git(root, 'commit', '-q', '-m', 'init') + git(root, 'push', '-q', 'origin', 'master') + return { tmp, root } +} + +describe('worktree scripts', () => { + for (const name of ['worktree.sh', 'worktree-delete.sh']) { + it(`${name} rejects a branch name with a slash`, () => { + const res = run(name, ['feat/x']) + assert.equal(res.status, 1) + assert.match(res.stderr, /invalid branch name "feat\/x".*no "\/"/) + }) + + it(`${name} rejects a branch name longer than 60 characters`, () => { + const res = run(name, ['a'.repeat(61)]) + assert.equal(res.status, 1) + assert.match(res.stderr, /invalid branch name/) + }) + } + + it('worktree.sh refuses a branch that makes the zellij session name too long for this repository', () => { + const { root } = checkout('simple-directory') + const res = run('worktree.sh', ['a'.repeat(50)], root) + assert.equal(res.status, 1) + assert.match(res.stderr, /zellij session .* must fit in 60 characters, so at most 43 characters/) + }) + + for (const name of ['worktree.sh', 'worktree-delete.sh']) { + it(`${name} refuses to run from a subfolder of the repository`, () => { + const { root } = checkout() + mkdirSync(path.join(root, 'api')) + const res = run(name, ['feat-x'], path.join(root, 'api')) + assert.equal(res.status, 1) + assert.match(res.stderr, /from the root of the repository/) + }) + + it(`${name} refuses to run from a linked worktree`, () => { + const { tmp, root } = checkout() + git(root, 'worktree', 'add', '-q', '-b', 'feat-a', path.join(tmp, 'portals_feat-a')) + const res = run(name, ['feat-x'], path.join(tmp, 'portals_feat-a')) + assert.equal(res.status, 1) + assert.match(res.stderr, /from the main checkout, not from a worktree/) + }) + } + + it('worktree.sh puts the local changes back when the setup fails after stashing them', () => { + const { tmp, root } = checkout() + git(root, 'checkout', '-q', '-b', 'feat-x') + writeFileSync(path.join(root, 'README.md'), 'work in progress\n') + writeFileSync(path.join(root, 'notes.txt'), 'untracked\n') + // master is checked out by another worktree: switching the main checkout back to it fails + git(root, 'worktree', 'add', '-q', path.join(tmp, 'portals_other'), 'master') + const res = run('worktree.sh', ['feat-x'], root) + assert.notEqual(res.status, 0) + assert.match(res.stderr, /your local changes are back in/) + assert.equal(git(root, 'branch', '--show-current'), 'feat-x') + assert.equal(readFileSync(path.join(root, 'README.md'), 'utf8'), 'work in progress\n') + assert.equal(readFileSync(path.join(root, 'notes.txt'), 'utf8'), 'untracked\n') + assert.equal(git(root, 'stash', 'list'), '') + }) +}) diff --git a/packages/dev/zellij.test.ts b/packages/dev/zellij.test.ts new file mode 100644 index 0000000..509b65d --- /dev/null +++ b/packages/dev/zellij.test.ts @@ -0,0 +1,57 @@ +import { describe, it } from 'node:test' +import { strict as assert } from 'assert' +import { parseSessions, sessionState, requireSession, findPane, checkVersion, sessionName } from './zellij.js' +import type { Exec } from './exec.js' + +const panes = [ + { id: 0, title: 'status', exited: false, exit_status: null, is_plugin: false, terminal_command: 'npx --no df-dev status --watch' }, + { id: 1, title: 'api', exited: false, exit_status: null, is_plugin: false, terminal_command: 'bash -c ...' }, + { id: 0, title: 'tab-bar', exited: false, exit_status: null, is_plugin: true, terminal_command: null } +] + +const fakeExec = (outputs: Record): Exec => (cmd, args) => { + const key = [cmd, ...args].join(' ') + const match = Object.keys(outputs).find(k => key.includes(k)) + if (!match) throw new Error(`unexpected call: ${key}`) + const out = outputs[match] + if (out instanceof Error) throw out + return out +} + +describe('zellij sessions', () => { + it('names the session after the worktree dir', () => { + assert.equal(sessionName('/home/u/Documents/portals_feat-x'), 'portals_feat-x') + }) + + it('parses running and exited sessions', () => { + const out = 'portals [Created 2m ago] (current)\nportals_feat-x [Created 1h ago] (EXITED - attach to resurrect)\n' + assert.deepEqual(parseSessions(out), { portals: 'running', 'portals_feat-x': 'exited' }) + }) + + it('reports absent when zellij has no session at all', () => { + const exec = fakeExec({ 'list-sessions': new Error('No active zellij sessions found.') }) + assert.equal(sessionState(exec, 'portals'), 'absent') + }) + + it('tells the agent to ask the user when the session is not running', () => { + const exec = fakeExec({ '--version': 'zellij 0.45.1', 'list-sessions': 'portals [Created 1h ago] (EXITED - attach to resurrect)\n' }) + assert.throws(() => requireSession(exec, '/x/portals'), /not running.*npm run dev-zellij/) + }) + + it('refuses zellij older than 0.44', () => { + assert.throws(() => checkVersion(fakeExec({ '--version': 'zellij 0.40.1' })), /0\.44/) + checkVersion(fakeExec({ '--version': 'zellij 0.45.1' })) + }) +}) + +describe('zellij panes', () => { + it('finds a terminal pane by title', () => { + const exec = fakeExec({ 'list-panes': JSON.stringify(panes) }) + assert.equal(findPane(exec, 'portals', 'api').id, 1) + }) + + it('lists the available titles when a pane is not found', () => { + const exec = fakeExec({ 'list-panes': JSON.stringify(panes) }) + assert.throws(() => findPane(exec, 'portals', 'worker'), /no pane "worker".*status, api/) + }) +}) diff --git a/packages/dev/zellij.ts b/packages/dev/zellij.ts new file mode 100644 index 0000000..8dc0b01 --- /dev/null +++ b/packages/dev/zellij.ts @@ -0,0 +1,60 @@ +import path from 'node:path' +import type { Exec } from './exec.js' + +export type SessionState = 'running' | 'exited' | 'absent' +export type Pane = { id: number, title: string, exited: boolean, exit_status: number | null, is_plugin: boolean, terminal_command: string | null } + +export const sessionName = (root: string) => path.basename(root) + +export const parseSessions = (out: string) => { + const sessions: Record = {} + for (const line of out.split('\n')) { + const name = line.trim().split(/\s+/)[0] + if (name) sessions[name] = line.includes('EXITED') ? 'exited' : 'running' + } + return sessions +} + +export const sessionState = (exec: Exec, name: string): SessionState => { + let out: string + try { + out = exec('zellij', ['list-sessions', '--no-formatting']) + } catch { + return 'absent' // zellij exits 1 when there is no session at all + } + return parseSessions(out)[name] ?? 'absent' +} + +// list-panes --json, dump-screen -p and send-keys -p appeared in zellij 0.44 +export const checkVersion = (exec: Exec) => { + const version = exec('zellij', ['--version']).trim().split(' ')[1] ?? '0.0.0' + const [major, minor] = version.split('.').map(Number) + if (major === 0 && minor < 44) throw new Error(`zellij ${version} is too old, df-dev needs zellij >= 0.44`) +} + +export const requireSession = (exec: Exec, root: string) => { + checkVersion(exec) + const name = sessionName(root) + const state = sessionState(exec, name) + if (state !== 'running') throw new Error(`zellij session "${name}" is not running (${state}): ask the user to start it with "npm run dev-zellij"`) + return name +} + +export const listPanes = (exec: Exec, session: string): Pane[] => + (JSON.parse(exec('zellij', ['--session', session, 'action', 'list-panes', '--json'])) as Pane[]).filter(p => !p.is_plugin) + +export const findPane = (exec: Exec, session: string, title: string) => { + const all = listPanes(exec, session) + const pane = all.find(p => p.title === title) + if (!pane) throw new Error(`no pane "${title}" in session ${session}, available panes: ${all.map(p => p.title).join(', ')}`) + return pane +} + +const paneRef = (pane: Pane) => `terminal_${pane.id}` + +export const dumpScreen = (exec: Exec, session: string, pane: Pane) => + exec('zellij', ['--session', session, 'action', 'dump-screen', '-p', paneRef(pane), '--full']) + +export const sendKeys = (exec: Exec, session: string, pane: Pane, keys: string) => { + exec('zellij', ['--session', session, 'action', 'send-keys', '-p', paneRef(pane), keys]) +} diff --git a/skills/README.md b/skills/README.md index a2841aa..de92769 100644 --- a/skills/README.md +++ b/skills/README.md @@ -18,6 +18,7 @@ Skills for AI coding agents working in the data-fair stack. | [`portals-config`](./portals-config/SKILL.md) | Configuring a **portals v2** portal itself from the manager API: creation, draft/publish cycle, theme (assisted vs manual colours, theme CSS propagation), fonts, images (SVG kept as-is), navigation bar and menus, footer (legacy and rows shapes by version), catalog cards, dataset and application page sections, standard pages to create, sync with data-fair publication sites and simple-directory, and a verification recipe. Page content is `portals-pages`. | | [`data-fair-security`](./data-fair-security/SKILL.md) | Stack-wide security invariants for services and plugins: public vs private HTTP clients of lib-node (SSRF) and the clients they don't cover (fetch, SDKs, ftp/sftp, child processes), parsed-origin URL matching, credentials on redirects and pagination links, remote names in paths and commands, and why `reqIsInternal` alone never grants a privilege. Review checklist. | | [`upgrade-scripts`](./upgrade-scripts/SKILL.md) | Writing database migrations with `@data-fair/lib-node/upgrade-scripts`: which version goes in the folder name (the common gotcha), idempotency patterns, fresh-install handling, and debugging. | +| [`data-fair-service`](./data-fair-service/SKILL.md) | The dev environment of a service and the conventions services share: operating a running env as an agent (`df-dev status`, `logs`, `restart`, `up`), the rules that keep it light and crash-free (no npm wrappers in panes, status pane starts containers, readiness URLs, optional containers via compose profiles and `NODE_CONFIG`), migrating a repo to `@data-fair/lib-dev`, the shared tooling reference (eslint, commitlint, husky, tsconfig, Docker, CI), the dev docker-compose and nginx standards, and the settled and open architecture conventions (API prefix, branch names, ping, BUILD.json, admin routes…). | | [`deps-refresh`](./deps-refresh/SKILL.md) | Routine "up to date" maintenance pass on a service: measuring with npm audit + a trivy scan of the built image, security fixes first, then dead/phantom/replaceable deps, then majors proven behaviour-neutral against the repo's own fixtures, then a written blocker for each one declined. Manual invocation only. | | [`pr-ready`](./pr-ready/SKILL.md) | Pre-PR flight check. A macro pass that re-anchors on the original intent and reviews the branch against it for scope, completeness, and drift, flags risky or sensitive changes, and drafts a compact PR title (conventional-commit style) and description. Manual invocation only. | diff --git a/skills/data-fair-security/SKILL.md b/skills/data-fair-security/SKILL.md index 455adf9..a77e96e 100644 --- a/skills/data-fair-security/SKILL.md +++ b/skills/data-fair-security/SKILL.md @@ -22,7 +22,7 @@ Network rules (egress NetworkPolicies) exist in production but are a second laye The check runs when the connection is created, on the exact address used: IP literals, every address a DNS name resolves to (no rebinding), every redirect hop. -Operators tune it with `SSRF_PUBLIC_IPS` (IPs/CIDRs to accept, e.g. an intranet source of a self-hosted instance) and `SSRF_PRIVATE_IPS` (public IPs to refuse, e.g. database servers). Dev environments set `SSRF_PUBLIC_IPS=127.0.0.1,::1` (all dev services are on the loopback) — in each repo's `dev/init-env.sh`. When `HTTP_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` is set the protection is off (the agents connect to the proxy) and a warning is logged at startup. +Operators tune it with `SSRF_PUBLIC_IPS` (IPs/CIDRs to accept, e.g. an intranet source of a self-hosted instance) and `SSRF_PRIVATE_IPS` (public IPs to refuse, e.g. database servers). Dev environments set `SSRF_PUBLIC_IPS=127.0.0.1,::1` (all dev services are on the loopback): `df-dev init-env` (`@data-fair/lib-dev`) writes it to `.env`, a repo not migrated yet does it in its `dev/init-env.sh`. When `HTTP_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` is set the protection is off (the agents connect to the proxy) and a warning is logged at startup. Rules: @@ -78,4 +78,4 @@ Anything that does not go through node's `http.Agent` of lib-node escapes the ch - [ ] No remote or user-provided name joined into a path, no `exec` with interpolation. - [ ] Downloads and pagination loops are bounded. - [ ] No privilege granted on `reqIsInternal` alone. -- [ ] Dev env (`dev/init-env.sh`) sets `SSRF_PUBLIC_IPS=127.0.0.1,::1` if the repo uses lib-node >= 2.14. +- [ ] Dev env (`df-dev init-env`, or `dev/init-env.sh` before the migration) sets `SSRF_PUBLIC_IPS=127.0.0.1,::1` if the repo uses lib-node >= 2.14. diff --git a/skills/data-fair-service/SKILL.md b/skills/data-fair-service/SKILL.md new file mode 100644 index 0000000..f59feb1 --- /dev/null +++ b/skills/data-fair-service/SKILL.md @@ -0,0 +1,56 @@ +--- +name: data-fair-service +description: Use when working on the dev environment or the shared architecture of a data-fair service (data-fair, portals, catalogs, processings, events, simple-directory, registry, agents…) - migrating a repo to @data-fair/lib-dev / df-dev, editing .zellij.kdl, dev/dev.json, docker-compose dev services or profiles, init-env, worktree scripts, status or log panes, an agent needing a service's logs, status or a restart, a crash at dev startup, build-types restarting the API, RAM of dev stacks, aligning a service's tooling (eslint, commitlint, husky, tsconfig, .nvmrc, Dockerfile, CI workflows), its dev docker-compose (mongo / ES versions, service blocks) or dev nginx config, or a convention shared by services (ping route, BUILD.json, admin routes, /v1 prefix, vite.config, generated .d.ts, /dev pages). +--- + +# data-fair service: dev environment and shared conventions + +## Overview + +Every service runs the same dev environment: a zellij (>= 0.44) session per checkout, readable by humans, operated by agents only through `df-dev` (package `@data-fair/lib-dev`). The repo declares what is specific to it in `dev/dev.json` and `.zellij.kdl`; everything else (ports, setup, status, logs, restart, worktrees) comes from the package. **RAM is the scarce resource**: nothing in a pane may hold memory it does not need. + +Package reference: `packages/dev/README.md` in data-fair/lib. Full migration recipe with templates: [references/migrate-a-service.md](references/migrate-a-service.md). Architecture conventions, settled and open: [references/architecture-conventions.md](references/architecture-conventions.md). Tooling (eslint, commitlint, husky, tsconfig, CI, Docker, .gitignore): [references/tooling.md](references/tooling.md). Dev docker-compose (service blocks, mongo/ES versions, shared dev secrets, maildev): [references/docker-compose.md](references/docker-compose.md), the only place where those versions change. Dev nginx template (shared head, the 3 location shapes, per-repo deviations): [references/nginx.md](references/nginx.md). + +When aligning a service: one branch per service, a handful of commits: `chore(dev)` dev environment, `chore` shared tooling and conventions (lint, commitlint, tsconfig, gitignore, node, CI order, vite template), `build(docker)` images, then one `fix` per bug that could be cherry-picked alone; list the details in each commit body (lines ≤ 100 characters). + +## Operating a running dev env (agents) + +The user starts the session (`npm run dev-zellij`). An agent never starts or kills it. + +| Need | Command | +|---|---| +| What runs, what is broken | `npx --no df-dev status` (containers, panes running / exited / **not ready**, memory) | +| A process's output | `npx --no df-dev logs -n 100` | +| Restart one process | `npx --no df-dev restart `: Ctrl+C, rerun, waits for the pane's readiness URL | +| An optional container | `npx --no df-dev up ` / `down `, then `restart api` so the config sees it | + +- Read the logs before restarting: nodemon, vite and nuxt already reload on file changes. +- "not ready" from `restart` → report to the user with the logs, never retry in a loop. +- Never `docker compose restart`, `kill`, `zellij action`, nor `npm ci` / `npm install` while the env runs (watchers restart on half-installed modules). +- Always `npx --no`: a bare `npx df-dev` downloads an unrelated public package when the dependency is missing (dependency confusion). Scripts and CI call `./node_modules/.bin/df-dev`. + +## Rules for the env itself (and why) + +| Rule | Why | +|---|---| +| Panes run their command directly (`exec node …`, `exec ../node_modules/.bin/nodemon …`), never `npm run …` | Each idle npm parent holds ~67 MB; a session had 9 (~600 MB) | +| Node processes (api, worker, side scripts) restart with nodemon: `exec ../node_modules/.bin/nodemon -e js,ts,json index.ts` from the process's own folder, with a `nodemon.json` next to it (`"execMap": { "ts": "node" }`, `"ignoreRoot": []`, `"ignore": ["node_modules/", …generated or data folders]`), `nodemon` in the root devDependencies; never `node --watch` | `node --watch` follows every imported file and cannot narrow it (`--watch-path` has no exclusion and no extension filter); nodemon watches the folder it runs in, filters extensions, takes `--ignore` when a generated folder must not restart the process, and has `--exec` (data-fair runs `sh ../dev/run-api.sh`). Without the `execMap`, nodemon 3 runs `.ts` files with `ts-node` and the pane exits ("ts-node: not found"). One watcher everywhere, one set of docs. The team dropped nodemon for `node --watch` once and came back to it | +| No `tee`, no log files; the pane scrollback is the log | `df-dev logs` reads it; log files grew to 390 MB and lost colors | +| Only the status pane starts containers (`df-status --start-deps`); process panes start with `./node_modules/.bin/df-dev wait-deps &&` | Two panes running `compose up` at once conflict on container names; APIs crashed 2–3 times when started before healthy deps | +| Static panes are bash (`echo …; exec sleep infinity`), the status pane is `df-status` (bash) | A node process for a static line or a 3 s poll costs ~50 MB all day | +| vite and nuxt under `df-mem-cap` | A runaway dev server is killed alone at 4 GB instead of taking the terminal tab down | +| nuxt piped: `nuxt dev 2>&1 \| cat` | In a real terminal nuxt switches to an interactive screen | +| Every pane that should answer HTTP has a `ready` URL in `dev/dev.json`, **checked with curl on the running env** | `restart` and `status` rely on it; a wrong path reports "not ready" forever (portals' Nuxt has `/ping`, not `/`) | +| Core containers (what the service and its tests really use) have no compose profile; each optional one has a profile named after itself; CI starts the core only | À la carte RAM (clamav alone ~1 GB); CI tests what a developer runs; integration tests needing an optional service are reviewed, often a unit test suffices | +| An optional service the API is configured to call gets a `dev.json` `optional` entry whose value **passes the service's config schema** (`null` where the schema allows it, else `""` or `false`: check `api/config/type/schema.json`); every pane whose process reads one of those keys (api, worker…) exports `NODE_CONFIG=$(./node_modules/.bin/df-dev absent-config)` | Otherwise the API retries the absent service in a loop; NODE_CONFIG cannot overwrite a developer's `local-development` file; a value the schema rejects crashes the API at startup | +| mongo: `--wiredTigerCacheSizeGB 0.25` + `mongosh` healthcheck; ES heap `${ES_HEAP:-512m}` + `mem_limit`; versions from [references/docker-compose.md](references/docker-compose.md), never bumped by one service alone | Default cache ceiling is half the RAM minus 1 GB per stack (16 GB measured) | + +## Common mistakes + +- Copying `nvm use` wrappers into panes: zellij inherits the launching shell's PATH; start it from a shell on the `.nvmrc` version. +- Removing root `dev-*` npm scripts that CI still calls: inline the command in the workflow (with `./node_modules/.bin/…`) or keep the workspace `dev` script. +- Forgetting gitignored local config (e.g. `worker/config/local-development.mjs`) in a new worktree: copy it from the main checkout. +- `commit-msg` rejects scope `dev`: add `dev` to the repo's `scope-enum` and its commit doc. +- A fresh database exposes bugs hidden on old dev DBs (simple-directory's rate-limiter index): note them, do not patch around them in the service. +- Two instances of one API (registry's `api-upstream`): a second pane with its own env (`PORT=…`, `MONGO_URL=…`): environment variables mapped in `custom-environment-variables` win over `NODE_CONFIG` in node-config, so both panes can share the `absent-config` export. +- Inventing an architecture convention: only the settled ones in the reference are rules; ask for the open ones. diff --git a/skills/data-fair-service/references/architecture-conventions.md b/skills/data-fair-service/references/architecture-conventions.md new file mode 100644 index 0000000..b4d1594 --- /dev/null +++ b/skills/data-fair-service/references/architecture-conventions.md @@ -0,0 +1,51 @@ +# Architecture conventions shared by services + +Only "Settled" items are rules. "Open" items list today's variants so they are not mistaken for a convention: when a task touches one, follow the repo as it is and ask the maintainer instead of picking a variant. + +## Settled + +- **API version prefix.** No `/v1/`: routes live under `/api/...`. Only data-fair keeps `/api/v1/` for history. A service still on `/api/v1/...` (processings: `/api/v1/_ping`, `/api/v1/admin`) moves to `/api/...` and keeps the `/api/v1/...` path as a compatibility route. +- **Branch names.** Letters, digits, `.`, `_`, `-`, no `/`, and `_` at most 60 characters (enforced by `df-worktree`; the name becomes a directory, a zellij session whose socket path zellij caps at 107 bytes, and a `DEV_HOST` label). +- **Dev environment.** `@data-fair/lib-dev` + `dev/dev.json` + `.zellij.kdl` as described in SKILL.md and migrate-a-service.md. +- **Tooling.** [tooling.md](tooling.md). +- **test-env routers** (unauthenticated, destructive): mounted only when `process.env.NODE_ENV === 'development'`, with a dynamic import so production never loads the module. The dev environment is the test environment: CI starts the API with `NODE_ENV=development` too (with the mocks the dev config needs, e.g. portals' ingress-manager), never a separate `test` mode for the API. +- **BUILD.json.** Written by CI (`reuse-build.yml`) and copied by the Dockerfile (`COPY BUILD.json* ./`). The API reads it in production **without crashing when it is missing** (an image built outside CI must start): + +```ts +let info: Record = { version: process.env.NODE_ENV ?? 'development' } +if (process.env.NODE_ENV === 'production') { + try { + info = JSON.parse(await readFile(resolve(import.meta.dirname, '../../../BUILD.json'), 'utf8')) + } catch { + info = { version: 'unknown (built without CI)' } + } +} +``` + + No shared lib function: two lines per repo. +- **Generated `.d.ts`** (auto-imports, components, typed router): written under `ui/dts/` by the Vite plugins, gitignored (`ui/.gitignore`: `dts`), never committed; the UI build runs before `check-types` (locally `df-dev setup` builds the UI). +- **`/dev` pages of the UIs.** A repo with a single dev page keeps it at `/dev`; several dev pages go under `src/pages/dev/…`. They are excluded from the production build by the Vite command, not by `NODE_ENV` (which a build can inherit by mistake): + +```ts +export default defineConfig(({ command }) => ({ + plugins: [ + VueRouter({ dts: './dts/typed-router.d.ts', exclude: command === 'build' ? ['src/pages/dev.vue', 'src/pages/dev/**'] : [] }), + // … + ] +})) +``` + + Check after a build that no dev chunk is in `ui/dist/assets` (`ls ui/dist/assets | grep -i dev`). +- **`ui/vite.config.ts`.** Every choice below has a reason; keep the repo's plugin list (component dirs, polyfills, aliases legitimately differ). + - `base: '/'`, plus `experimental.renderBuiltUrl` and `html.cspNonce: '{CSP_NONCE}'` like the other services: the UI is served under a prefix and behind a CSP. + - `defineConfig(({ command, isPreview }) => …)`; a `server` block only when `command === 'serve' && !isPreview` (`vite preview` also runs as `serve` and must not require the dev ports). + - `server.port` and `server.hmr.port` from the `.env` variables declared in `dev/dev.json` (`DEV_UI_PORT`, `DEV_UI_HMR_PORT`), parsed with a check (no `NaN`), `strictPort: true`; never a hard-coded port (metrics and openapi-viewer both used 7200 and collided); no `proxy`: nginx routes. + - `optimizeDeps.include`: the dependencies Vite would discover. `optimizeDeps.noDiscovery: true` avoids the full-page reloads of first visits, but a forgotten CommonJS dependency then breaks the page instead of reloading it: enable it only during a live validation, after opening every page, and add each dependency the console names. + - Plugins: VueRouter (`dts: './dts/typed-router.d.ts'`, dev pages excluded at build), Vue, VueI18n, Vuetify (`settingsPath`), AutoImport (`dts: './dts/auto-imports.d.ts'`, `autoImports` from `@data-fair/lib-vuetify/vite.js`), Components (`dts: './dts/components.d.ts'`, `resolvers: [componentsResolver]`). + - No `build` block unless it differs from Vite's defaults. + +## Open (decide with the maintainer before changing) + +- **Health route.** Today: `app.get('/api/ping')` (portals, catalogs), `app.use('/api/ping')` (events, registry, agents), `/api/v1/ping` from a router (data-fair), `/api/v1/_ping` (processings), `/ping` (portals' Nuxt server). Out of the current overhaul. +- **Admin routes.** `/api/admin` vs `/api/v1/admin`, different endpoint sets. Out of the current overhaul. +- **Shared mongo / Elasticsearch** for all stacks: feasible with one database per worktree and an ES index prefix per worktree, after aligning on mongo 8 and ES 8. diff --git a/skills/data-fair-service/references/docker-compose.md b/skills/data-fair-service/references/docker-compose.md new file mode 100644 index 0000000..c3fefca --- /dev/null +++ b/skills/data-fair-service/references/docker-compose.md @@ -0,0 +1,273 @@ +# Dev docker-compose + +The canonical dev compose of the data-fair services: the **only** place where storage and tool +versions are decided. A version bump is made here first, then applied to every service in the same pass; a +service never bumps mongo, ES or a tool alone. Drift check from the directory holding the service checkouts: +`grep -H 'image: *\(mongo\|ghcr.io/data-fair/elasticsearch\|nginx\|maildev\|clamav\)' */docker-compose.y*ml`. +Applies to every service's `docker-compose.yml` (data-fair keeps its `.yaml` file name). Goal: the same service is declared the same way in every repo; a repo only adds what +it really uses. Reference: the deployments (infrastructure repo for production: mongo 8.0.17, ES 7.17.28 in 2026-10; +env-staging3: mongo 8.0.17, ES 8.19.9). Dev follows staging3, where master is deployed. + +## Rules + +- Service names: `nginx`, `simple-directory`, `data-fair`, `events`, `agents`, `registry`, `catalogs`, + `processings`, `portals`, `openapi-viewer`, `capture`, `maildev`, `clamav`, `elasticsearch`, `mongo`, `s3`, + `s3-init`. Volumes: `elasticsearch-data`, `mongo-data`, `s3-data`, `data-fair-data`. +- Order: `nginx`, `simple-directory`, `data-fair`, then the other sibling services in the order above, + then the tools (`openapi-viewer`, `capture`, `maildev`, `clamav`), then storage (`elasticsearch`, + `mongo`, `s3`, `s3-init`), then `volumes:`. Inside a service: `profiles`, `image`, `network_mode`, + `ports`, `depends_on`, `command`, `environment`, `volumes`, `healthcheck`, `mem_limit`. +- Images of sibling services: the development branch tag (`:master` or `:main`, whichever the repo + publishes), never a major tag. Storage: the major tag (`mongo:8`); ES has no floating tag in our + ghcr image (`ghcr.io/data-fair/elasticsearch` only publishes exact versions, checked 2026-10-14), so + `8.19.9`, as staging3. Tools (nginx, maildev, clamav): exact versions. +- maildev (optional, profile `maildev`) only where the repo has a mail flow of its own to develop or + test, and simple-directory is wired to it there: simple-directory (invitations…), events (notification + mails), portals (contact form), data-fair (API key expiration mails through sd `/api/mails`), + processings (the worker's own SMTP transport, plugins' `sendMail`). Not catalogs, agents, registry: + their notifications go through events, which tests the mail delivery. +- Optional services: a compose profile named after the service (`df-dev up `), nothing else in + that profile. No other profiles (no `test`, no `testImages`, no `dev`). +- Sibling services run with `network_mode: host` and talk to each other on `localhost:`; only the + public URLs (`PUBLIC_URL`, `DIRECTORY_URL`, `WS_PUBLIC_URL`) go through nginx + (`http://${DEV_HOST}:${NGINX_PORT}/`). Storage containers publish their port instead. +- Port variables: `NGINX_PORT` (plus `NGINX_PORT2` only where a second host is tested, data-fair), + `SD_PORT`, `DF_PORT`, `EVENTS_PORT`, `AGENTS_PORT`, `REGISTRY_PORT`, `CATALOGS_PORT`, + `PROCESSINGS_PORT`, `OAV_PORT`, `CAPTURE_PORT`, `MAILDEV_UI_PORT`, `MAILDEV_SMTP_PORT`, + `CLAMAV_PORT`, `ES_PORT`, `MONGO_PORT`, `S3_PORT`. The service's own processes: `DEV_API_PORT`, + `DEV_UI_PORT`, `DEV_UI_HMR_PORT`, …. +- Mongo databases: `simple-directory`, `data-fair`, `data-fair-events`, `data-fair-agents`, + `data-fair-registry`, `data-fair--development` for the repo's own (keep its current name). +- Every sibling service: `OBSERVER_ACTIVE=false`. No `DEBUG` in committed files (set it locally). +- Options a service does not read are removed (checked against its `custom-environment-variables`). +- Sibling images run with `NODE_ENV=production` on their `default` config only (their `development` + config never applies): every dev-relevant value is set here explicitly, see "Dev values" below. + +## Shared dev secrets + +Same value in every compose file and in every service's `development` config: + +| Key | Value | +|---|---| +| identities webhooks (`SECRET_IDENTITIES`, sd `IDENTITIES_WEBHOOKS` key) | `secret-identities` | +| events (`SECRET_EVENTS`) | `secret-events` | +| mails (`SECRET_SENDMAILS`) | `secret-sendmails` | +| sites (`SECRET_SITES`) | `secret-sites` | +| limits (`SECRET_LIMITS`) | `secret-limits` | +| catalogs (`SECRET_CATALOGS`) | `secret-catalogs` | +| registry (`SECRET_INTERNAL_SERVICES`) | `secret-registry-internal` | +| ingress (portals) | `secret-ingress` | +| `CIPHER_PASSWORD` (registry needs >= 32 chars) | `dev-cipher-password-0123456789abcdef` | +| `API_KEYS_SALT` (registry, >= 32 chars) | `dev-api-keys-salt-0123456789abcdef01` | + +Changing a cipher password makes the secrets already ciphered in a dev database unreadable: the pilots +reset their volumes with this change anyway (mongo 4.2 → 8). + +## Canonical blocks + +```yaml + nginx: + image: nginx:1.29.4-alpine + network_mode: host + environment: + - DEV_HOST=${DEV_HOST} + - NGINX_PORT=${NGINX_PORT} + # then every port the nginx template reads, nothing else + volumes: + - ./dev/resources/nginx.conf.template:/etc/nginx/templates/default.conf.template:ro + + simple-directory: + image: ghcr.io/data-fair/simple-directory:master + network_mode: host + depends_on: + mongo: + condition: service_healthy + environment: + - PORT=${SD_PORT} + - PUBLIC_URL=http://${DEV_HOST}:${NGINX_PORT}/simple-directory + - MONGO_URL=mongodb://localhost:${MONGO_PORT}/simple-directory + - OBSERVER_ACTIVE=false + - CIPHER_PASSWORD=dev-cipher-password-0123456789abcdef + - STORAGE_TYPE=file + - ADMINS=["test_superadmin@test.com","superadmin@test.com","alban.mouton@koumoul.com"] + - CONTACT=contact@test.com + - IDENTITIES_WEBHOOKS=[{"base":"http://localhost:${DEV_API_PORT}/api/identities","key":"secret-identities"}] + # tests log in through the API many times per minute + - AUTHRATELIMIT_ATTEMPTS=200 + - AUTHRATELIMIT_DURATION=1 + - MANAGE_DEPARTMENTS=true + - MANAGE_PARTNERS=true + - MANAGE_SITES=true + - SECRET_SITES=secret-sites + - SECRET_EVENTS=secret-events + - SECRET_SENDMAILS=secret-sendmails + # host network: sd's default SMTP localhost:1025 would reach another env's maildev + - MAILS_TRANSPORT={"host":"localhost","port":${MAILDEV_SMTP_PORT},"ignoreTLS":true} + # per recipient, stored in mongo: repeated test runs would stop the mails silently + - MAILSRATELIMIT_POINTS=100000 + # then only what this repo needs (data-fair: ACCEPT_UNKNOWN_SITE, LIST_*_MODE, ROLES_DEFAULTS…; + # portals: ANONYMOUS_CONTACT_FORM, maildev) + volumes: + - ./dev/resources/users.json:/app/data/users.json + - ./dev/resources/organizations.json:/app/data/organizations.json + healthcheck: + test: ["CMD", "nc", "-z", "localhost", "${SD_PORT}"] + interval: 2s + timeout: 2s + retries: 60 + + data-fair: + image: ghcr.io/data-fair/data-fair:master + # it can start before its dependencies answer + restart: on-failure:10 + network_mode: host + depends_on: + mongo: + condition: service_healthy + elasticsearch: + condition: service_healthy + environment: + - PORT=${DF_PORT} + - PUBLIC_URL=http://${DEV_HOST}:${NGINX_PORT}/data-fair + - WS_PUBLIC_URL=ws://${DEV_HOST}:${NGINX_PORT}/data-fair + - DIRECTORY_URL=http://${DEV_HOST}:${NGINX_PORT}/simple-directory + - PRIVATE_DIRECTORY_URL=http://localhost:${SD_PORT} + - PRIVATE_EVENTS_URL=http://localhost:${EVENTS_PORT} + - PRIVATE_AGENTS_URL=http://localhost:${AGENTS_PORT} + - MONGO_URL=mongodb://localhost:${MONGO_PORT}/data-fair + - ES_HOST=http://localhost:${ES_PORT} + # single node: with replicas the cluster stays yellow and /ping answers 500 + - ES_NB_REPLICAS=0 + - OBSERVER_ACTIVE=false + # default: 3 koumoul.com remote services fetched at boot, extensions would call production + - REMOTE_SERVICES=[] + - CACHE_PUBLIC_MAX_AGE=1 + # one IP for every test and the portal SSR: the per-IP / per-user limits answer 429 + - DEFAULT_LIMITS_API_RATE_ANONYMOUS_NB=100000 + - DEFAULT_LIMITS_API_RATE_USER_NB=100000 + - DEFAULT_LIMITS_API_RATE_ANONYMOUS_COMPUTE_MS=0 + - DEFAULT_LIMITS_API_RATE_USER_COMPUTE_MS=0 + - DEFAULT_LIMITS_API_RATE_ANONYMOUS_BANDWIDTH_DYNAMIC=100000000 + - DEFAULT_LIMITS_API_RATE_USER_BANDWIDTH_DYNAMIC=100000000 + - WORKER_BASE_CONCURRENCY=1 + # tile cache capped collection, 2 GB by default + - CACHE_MONGO_SIZE=200 + - SECRET_EVENTS=secret-events + # then this repo's integration: PRIVATE__URL=http://localhost:${DEV_API_PORT}, SECRET_ + + volumes: + # the image declares an anonymous /data: datasets files would vanish on recreate while mongo keeps them + - data-fair-data:/data + + events: + profiles: + - events + image: ghcr.io/data-fair/events:main + network_mode: host + depends_on: + - mongo + environment: + - PORT=${EVENTS_PORT} + - PRIVATE_DIRECTORY_URL=http://localhost:${SD_PORT} + - MONGO_URL=mongodb://localhost:${MONGO_PORT}/data-fair-events + - OBSERVER_ACTIVE=false + - SECRET_IDENTITIES=secret-identities + - SECRET_EVENTS=secret-events + - SECRET_SENDMAILS=secret-sendmails + + agents: + profiles: + - agents + image: ghcr.io/data-fair/agents:main + network_mode: host + depends_on: + - mongo + environment: + - PORT=${AGENTS_PORT} + - PRIVATE_DIRECTORY_URL=http://localhost:${SD_PORT} + - PRIVATE_EVENTS_URL=http://localhost:${EVENTS_PORT} + - MONGO_URL=mongodb://localhost:${MONGO_PORT}/data-fair-agents + - OBSERVER_ACTIVE=false + - CIPHER_PASSWORD=dev-cipher-password-0123456789abcdef + - SECRET_EVENTS=secret-events + # the image has no model by default: the mock provider of agents' development config + - PROVIDERS=[{"type":"mock","id":"global-mock","name":"Global Mock","enabled":true}] + - MODELS=[{"id":"mock-model","name":"Global Mock Model","provider":"global-mock","usage":["assistant","tools","summarizer","evaluator","moderator"],"inputPricePerMillion":0,"outputPricePerMillion":0}] + - DEFAULT_MODELS={"assistant":{"provider":"global-mock","id":"mock-model"}} + - DEFAULT_CREDITS=-1 + + registry: + image: ghcr.io/data-fair/registry:main + network_mode: host + depends_on: + - mongo + - simple-directory + environment: + - PORT=${REGISTRY_PORT} + - PRIVATE_DIRECTORY_URL=http://localhost:${SD_PORT} + - MONGO_URL=mongodb://localhost:${MONGO_PORT}/data-fair-registry + - OBSERVER_ACTIVE=false + - CIPHER_PASSWORD=dev-cipher-password-0123456789abcdef + - API_KEYS_SALT=dev-api-keys-salt-0123456789abcdef01 + - SECRET_INTERNAL_SERVICES=secret-registry-internal + - FILES_STORAGE=fs + - DATA_DIR=/tmp/registry-data + + elasticsearch: + image: ghcr.io/data-fair/elasticsearch:8.19.9 + ports: + - ${ES_PORT}:9200 + environment: + - discovery.type=single-node + - xpack.security.enabled=false + - ingest.geoip.downloader.enabled=false + # heap 512m; an agent that needs more sets ES_HEAP=1g in .env (df-dev never overwrites it) + - ES_JAVA_OPTS=-Xms${ES_HEAP:-512m} -Xmx${ES_HEAP:-512m} + volumes: + - elasticsearch-data:/usr/share/elasticsearch/data + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9200"] + # 2 s probes while starting (df-dev wait-deps gives up after 3 min), then one per minute + interval: 60s + timeout: 10s + retries: 3 + start_period: 3m + start_interval: 2s + mem_limit: 2g + # no swap: an idle ES got 1.7 GB swapped out + memswap_limit: 2g + + mongo: + image: mongo:8 + ports: + - ${MONGO_PORT}:27017 + # wiredTiger takes half the machine's RAM by default + command: ["--wiredTigerCacheSizeGB", "0.25"] + volumes: + - mongo-data:/data/db + healthcheck: + test: ["CMD", "mongosh", "--quiet", "--eval", "db.runCommand({ ping: 1 }).ok"] + # 2 s probes while starting, then one per minute: a mongosh probe costs ~1.3 s of CPU + interval: 60s + timeout: 10s + retries: 3 + start_period: 3m + start_interval: 2s + + maildev: + profiles: + - maildev + image: maildev/maildev:2.1.0 + ports: + - ${MAILDEV_UI_PORT}:1080 + - ${MAILDEV_SMTP_PORT}:1025 + healthcheck: # keep the one of portals' compose + + # and in simple-directory, where maildev exists: + - MAILDEV_ACTIVE=true + - MAILDEV_URL=http://localhost:${MAILDEV_UI_PORT} + - MAILS_TRANSPORT={"host":"localhost","port":${MAILDEV_SMTP_PORT},"ignoreTLS":true} +``` + +`registry` is optional (profile `registry`) where the repo does not need it at every start (data-fair); +core where plugins are resolved through it (catalogs, processings). diff --git a/skills/data-fair-service/references/migrate-a-service.md b/skills/data-fair-service/references/migrate-a-service.md new file mode 100644 index 0000000..9de6905 --- /dev/null +++ b/skills/data-fair-service/references/migrate-a-service.md @@ -0,0 +1,120 @@ +# Migrating a service to the df-dev convention + +Reference implementations: the portals, data-fair and catalogs migrations (branch `feat-dev-env` of each repo). Work in a worktree, never on the main checkout. Do not start the env yourself: prepare everything, then ask the user to run `npm run dev-zellij` and validate with them (last section). + +## 0. Inventory first (read-only) + +Collect before writing anything: + +- `.zellij.kdl`: panes, their commands, the footer URL; keep the user's layout (pane positions) unless asked. +- `dev/init-env.sh`: every port variable it writes → `ports` of `dev.json` (`df-dev init-env` also writes `DEV_HOST` and `SSRF_PUBLIC_IPS`). Repo-specific build steps in `dev/worktree.sh` (e.g. `npm -w parquet-writer run build`) → `setup`. +- `package.json` (root and workspaces): `dev-*` scripts, what CI calls (`.github/workflows/*.yml`). +- `docker-compose.yml`: services, profiles, mongo image version, healthchecks, `depends_on`. +- The API config of development (`api/config/development.*`): which URLs point to optional containers (`privateEventsUrl`, `privateAgentsUrl`, `privateRegistryUrl`, `clamav.active`, `privateProcessingsUrl`…). +- The API's health route and base path: where the router is mounted and whether nginx rewrites the prefix (`dev/resources/nginx.conf.template`). The readiness URL targets the process port directly. +- Gitignored local files a worktree needs (`*/config/local-*`). +- References to the old scripts: `grep -rn "init-env\|status.sh\|worktree.sh\|dev/logs\|dev-deps\|dev-api\|dev-ui\|tee " --exclude-dir=node_modules .` (AGENTS.md, CONTRIBUTING.md, tests, `.claude/skills/*`, docs/architecture, CI). + +## 1. Dependency and dev.json + +```bash +npm install -D @data-fair/lib-dev # file:../lib…/packages/dev while unpublished; never ship a file: dependency +``` + +Also require `@data-fair/lib-node` >= 2.16.1 (its events queue goes inactive by itself without the events service, which the `optional` events override relies on) and `@data-fair/lib-types-builder` >= 1.14.5 (atomic writes: build-types no longer restarts a running API). + +`dev/dev.json` (data-fair's, trimmed): + +```json +{ + "ports": ["NGINX_PORT", "NGINX_PORT2", "DEV_API_PORT", "DEV_UI_PORT", "MONGO_PORT", "EVENTS_PORT", "MOCK_PORT"], + "panes": { + "api": { "ready": "http://localhost:${DEV_API_PORT}/api/v1/ping" }, + "ui": { "ready": "http://localhost:${DEV_UI_PORT}/data-fair/" }, + "mock": { "ready": "http://localhost:${MOCK_PORT}/" } + }, + "url": "http://${DEV_HOST}:${NGINX_PORT}/data-fair/", + "optional": { + "events": { "privateEventsUrl": null }, + "clamav": { "clamav": { "active": false } } + }, + "setup": ["npm -w parquet-writer run build"] +} +``` + +Pane titles in `panes` must equal the `name` of the panes in `.zellij.kdl`; other panes (the free shell, keep the repo's name for it if it has one) need no entry. A pane without an HTTP endpoint (worker, mock without route) gets `{}`. `${VAR}` in `ready` and `url` resolves any `.env` variable. `optional` values must pass the service's config schema: `null` only where the schema allows it (registry types `privateEventsUrl` as `string`: use `""`). Then `git rm dev/init-env.sh` and run `./node_modules/.bin/df-dev setup` (writes `.env`, runs `setup`, build-types, builds `ui` when `ui/dist/index.html` is missing). + +## 2. docker-compose.yml + +- Core services (needed for the service to work): remove their `profiles`. +- Optional services: `profiles: []`, nothing else (no `test` profile). +- CI starts the **core** services only (`docker compose up -d --wait`; `--profile '*'` only for `logs` / `down`), like a developer's default env, and its API / worker get `NODE_CONFIG=$(./node_modules/.bin/df-dev absent-config)`. The core holds what the tests really use: before moving a service to a profile, grep `tests/` for it. A test that needs an optional service is reviewed rather than kept by default: checking that a service emits its events is a unit test of the emission, not an api/e2e test that starts `events`. +- `wait-deps` waits for every container of the active profiles: `healthy` when it has a healthcheck, `running` otherwise, one-shot containers exited 0. +- Service blocks (mongo 8, ES 8.19.9, simple-directory, nginx…), their healthchecks, memory caps and `depends_on`: copy them from [docker-compose.md](docker-compose.md), the only place they change; the nginx template from [nginx.md](nginx.md). +- One-shot init containers (e.g. `s3-init`) are fine: `wait-deps` and `df-status` treat "exited 0" as done. + +## 3. .zellij.kdl (catalogs, without its DEBUG flags) + +```kdl +// Dev environment of catalogs, see @data-fair/lib-dev. Agents: npx --no df-dev status | logs | restart +layout { + pane split_direction="vertical" { + pane name="shell" borderless=true + pane name="status" command="./node_modules/.bin/df-status" { + args "--start-deps" + } + pane name="bridge" command="./node_modules/.bin/df-agents-bridge" { + start_suspended true + } + } + pane split_direction="vertical" { + pane name="ui" command="bash" { + args "-c" "cd ui && FORCE_COLOR=1 NODE_CONFIG_DIR=../api/config/ exec ../node_modules/.bin/df-mem-cap vite --port $DEV_UI_PORT" + } + pane name="api" command="bash" { + args "-c" "./node_modules/.bin/df-dev wait-deps && export NODE_CONFIG=$(./node_modules/.bin/df-dev absent-config) && cd api && FORCE_COLOR=1 NODE_ENV=development exec ../node_modules/.bin/nodemon -e js,ts,json index.ts" + } + pane name="worker" command="bash" { + args "-c" "./node_modules/.bin/df-dev wait-deps && export NODE_CONFIG=$(./node_modules/.bin/df-dev absent-config) && cd worker && FORCE_COLOR=1 NODE_ENV=development exec ../node_modules/.bin/nodemon -e js,ts,json index.ts" + } + } + pane name="url" size=1 borderless=true command="bash" { + args "-c" "echo -n -e \"Dev server available at \\e[1;96mhttp://$DEV_HOST:$NGINX_PORT/catalogs/dev\\e[0m\"; exec sleep infinity" + } +} +``` + +- `df-dev zellij` exports `.env` (and `COMPOSE_PROFILES` from `--services`) into the session: panes read `$DEV_HOST`, `$NGINX_PORT`… directly. +- `export NODE_CONFIG=…` in **every** pane whose process reads an `optional` key (catalogs' and processings' workers read `privateEventsUrl` too), and only there. A second instance of the same API (registry's `api-upstream`) is another pane with its own env overrides (`PORT=$DEV_UPSTREAM_API_PORT MONGO_URL=… DATA_DIR=…`): env variables mapped in `custom-environment-variables` win over `NODE_CONFIG`. +- Keep the repo's vite invocation: pass `--port $DEV_UI_PORT` only if the old script did (registry's `vite.config.ts` reads `.env` itself). +- Keep the repo's env vars and DEBUG flags from the old `dev` scripts; drop `mkdir -p dev/logs`, `| tee …`, `| ts …`, `nvm use`. +- Nuxt: `… ../node_modules/.bin/df-mem-cap nuxt dev 2>&1 | cat` (no `exec`). +- A data-fair API keeps its `nodemon --exec 'sh ../dev/run-api.sh'` (no restart feedback loop). +- A repo on `node --watch` moves to nodemon (rule in SKILL.md): add `nodemon` to the root devDependencies; a script outside a workspace watches its own folder, e.g. portals' `nodemon -e ts --watch dev --exec "node --disable-warning=ExperimentalWarning" dev/ingress-manager.ts` (run from the root, nodemon would otherwise watch the whole repo). +- Parse-check the layout without starting anything: copy it with every `command="…"` replaced by `command="true"`, `zellij --layout attach -b check`, `zellij -s check action list-panes --json`, then `kill-session` + `delete-session`. With an isolated zellij 0.45, set `ZELLIJ_CONFIG_DIR` to a copy of the config dir: its first run migrates `~/.config/zellij/config.kdl` to a format 0.40 cannot read. + +## 4. Scripts, CI, tests, docs + +- Root `package.json`: `"dev-zellij": "df-dev zellij"`, `"stop-dev-deps": "docker compose --profile '*' stop"`; remove `dev-deps`, `dev-api`, `dev-ui`, `dev-worker`, `dev-bridge`, `dev-mock` (keep workspace `dev` scripts only if CI uses them). +- Delete `dev/init-env.sh`, `dev/status.sh`, `dev/worktree.sh`, `dev/delete-worktree.sh`, `dev/scripts/log-tee.sh`, `dev/scripts/dev-deps.sh`, `dev/with-memory-cap.sh` (replaced by `df-mem-cap`). +- CI: `./node_modules/.bin/df-dev init-env` instead of `./dev/init-env.sh`; processes started with local binaries (`./node_modules/.bin/dotenv -- npm -w api run dev &`), never a bare `npx`. +- `tests/state-setup.ts`: drop the `tail -f dev/logs/…` spawn, and the `state-teardown` project and file that killed it; the "API unavailable" message tells agents `npx --no df-dev logs api` / `npx --no df-dev restart api`. +- Config error messages mentioning `dev/init-env.sh` → `npx --no df-dev setup`. +- AGENTS.md "Dev environment": the command table and the four rules of SKILL.md "Operating a running dev env", the list of optional containers, worktrees via `npx --no df-worktree ` / `npx --no df-worktree-delete ` (branch names: letters, digits, `.`, `_`, `-`, no `/`, and `_` at most 60 characters). +- CONTRIBUTING.md install: `npm ci`, `npx --no df-dev setup`, `npm run dev-zellij [-- --services a,b]`. +- `.claude/skills/agents-sim/SKILL.md` and docs/architecture: `status.sh` → `df-dev status`; the bridge pane is suspended until the user presses Enter. +- Commit scope `dev` (add it to `scope-enum` where commitlint enforces one). Run `npm run lint` and `check-types` (or the repo's type ratchet). + +## 5. Live validation with the user + +Ask the user to start the env, then (agent): + +| Check | Expected | +|---|---| +| `npx --no df-dev status` | core containers ✔, optional ones `not started`, every pane running, no "not ready" | +| `npx --no df-dev logs api -n 5` | the API's "listening" line, no crash before it | +| `npx --no df-dev restart api` (and worker) | `ready after …s` | +| `npx --no df-dev up ` then `restart api` | container ✔, config plugged in (no retry loop in the logs) | +| `npm run build-types` while running | 0 restart / `ERR_MODULE_NOT_FOUND` in `logs api` | +| The repo's API tests | same results as on the main checkout | +| RAM (`df-dev status`, `docker stats`) vs the main checkout's session | no npm wrapper, mongo cache max 0.25 GB | diff --git a/skills/data-fair-service/references/nginx.md b/skills/data-fair-service/references/nginx.md new file mode 100644 index 0000000..def395b --- /dev/null +++ b/skills/data-fair-service/references/nginx.md @@ -0,0 +1,133 @@ +# Dev nginx template + +`dev/resources/nginx.conf.template`, mounted by the compose `nginx` service (`nginx:1.29.4-alpine`, host network) +at `/etc/nginx/templates/default.conf.template`; the image entrypoint substitutes the env vars and includes +the result in the `http` block. One shape in every repo: the shared head below **verbatim** (same comments, same +order), then the repo's `server` blocks. Every `${VAR}` the template uses must be in the nginx service +`environment` of the compose, or nginx refuses to start. + +It mirrors production minus two things: no shared proxy cache and no gzip in dev (a cache keeps what must not +be cached while developing). + +## Shared head + +```nginx +# Dev reverse proxy of the data-fair stack. This head is identical in every service repo +# (data-fair-service skill, references/nginx.md) and mirrors the production ingress-nginx config, +# minus the shared cache and gzip. The nginx image entrypoint substitutes the env vars and +# includes the result in the http block. + +# websocket support (realtime channels, Vite HMR): upgrade only the requests that ask for it +map $http_upgrade $connection_upgrade { + default upgrade; + '' close; +} +proxy_http_version 1.1; +proxy_set_header Upgrade $http_upgrade; +proxy_set_header Connection $connection_upgrade; + +# Transmit host, protocol and user ip, we use it for routing, rate limiting, etc. +proxy_set_header Host $http_host; +proxy_set_header X-Forwarded-Host $http_host; +proxy_set_header X-Forwarded-Proto $scheme; +proxy_set_header X-Forwarded-For $remote_addr; +proxy_set_header X-Real-IP $remote_addr; +proxy_set_header X-Client-IP $remote_addr; +# the relay secret proves a forwarded client IP to the ingress; services never see it +proxy_set_header X-Trusted-Forwarder ""; +proxy_set_header X-Forwarded-Path $request_uri; +# Headers data-fair uses for better private caching +proxy_set_header X-Private-If-Modified-Since $http_if_modified_since; +proxy_set_header X-Private-If-None-Match $http_if_none_match; +# range requests support (pmtiles) +proxy_set_header Range $http_range; + +# accept some occasional long queries +proxy_read_timeout 600s; +# body size limits are implemented by services themselves +client_max_body_size 0; +# direct streaming of uploads +proxy_request_buffering off; +# direct streaming of downloads, enabled on-demand by services using x-accel-buffering header +proxy_buffering off; +# allow large response headers (setCookies with large session token) +proxy_buffer_size 32k; +proxy_buffers 4 32k; + +# No server or location below sets proxy_set_header: it would drop every header above. +``` + +## Server block + +`listen ${NGINX_PORT}; server_name ${DEV_HOST};`, then one trailing-slash redirect listing every service prefix +the server routes (prod does it in its server-snippet and in portals-ingress-manager): + +```nginx +rewrite ^(/catalogs|/simple-directory|/data-fair|/events|/agents|/registry)$ $1/ redirect; +``` + +`location = /` redirects to the repo's UI. Route only the services the repo's compose runs. + +## The 3 location shapes + +1. **Prefix kept** (services built with `createSiteMiddleware`): simple-directory, events, catalogs, processings, + registry, agents, portals-manager, openapi-viewer (prefixOptional, staging3 and portal hosts keep the prefix). + No URI on `proxy_pass`, so the encoded URI goes through as-is (`%2F` in registry artefact ids). + ```nginx + location /events/ { + proxy_pass http://localhost:${EVENTS_PORT}; + } + ``` +2. **Prefix stripped**: data-fair's API, like prod (`/data-fair/(.*)` with `rewrite-target: /$1`). + ```nginx + location /data-fair/ { + rewrite ^/data-fair/(.*) /$1 break; + proxy_pass http://localhost:${DF_PORT}; + } + ``` +3. **The repo's own service**: API and ui-config to the API server, the rest to the Vite dev server with a long + timeout for the HMR websocket (the only place the 600 s timeout is raised). + ```nginx + location = /catalogs/ { + return 302 /catalogs/dev; + } + location /catalogs/api/ { + proxy_pass http://localhost:${DEV_API_PORT}; + } + location ~ /catalogs/(.*)-ui-config.js { + proxy_pass http://localhost:${DEV_API_PORT}; + } + location /catalogs/ { + proxy_read_timeout 86400s; + proxy_pass http://localhost:${DEV_UI_PORT}; + } + ``` + +Never `proxy_set_header` in a server or location: nginx would drop every header inherited from the head. + +## Per-repo deviations + +- **data-fair** + - Keeps a shared cache between the head and its servers (`proxy_cache_*` with prod's key + `"$request_method $scheme$host$request_uri"`, `X-Cache-Status`, bypass by `x-cache-bypass` header or + `cache_bypass` cookie): `tests/features/datasets/query/cache-headers.api.spec.ts` checks MISS/HIT/REVALIDATED. + - `proxy_pass_header X-Accel-Buffering`, read by tests. + - Its own API is served stripped, so shape 3 uses `rewrite` (API paths are regexes so `/data-fair/api-doc` and + `/data-fair/applications` stay on the UI). `location = /data-fair/` sends the websocket upgrade to the API, the + page to Vite. + - `/capture/` stripped (capture has prefixOptional; prod does not expose it). + - Second server on `${NGINX_PORT2}` simulating a publication site: API only under `/data-fair/`, + openapi-viewer as `/api-doc/` (stripped), simple-directory, capture, events. +- **portals**: `server_name ${DEV_HOST} *.${DEV_HOST}`. `/` on DEV_HOST redirects to `/portals-manager/dev`, any + other path or portal subdomain goes to the portal Nuxt dev server (long timeout, like shape 3). + `/openapi-viewer/` is a text stub (the service is not in the compose; the portal's api-doc pages iframe it). + No `/catalogs`, `/registry`, `/processings`: those services are not in its compose. +- **catalogs**, **processings**: no deviation. + +## Production references + +- infrastructure `helmconfigs/production/ingress.yaml`: controller config (buffering off, body size 0, 600 s + read timeout, 32k buffers, X-Private-If-*, X-Forwarded-Path, X-Client-IP, Range, the cache and gzip left out here). +- infrastructure `manifests/env-production/ingress.yaml` and env-staging3 `manifests/data-fair/ingress.yaml`: + routes, data-fair stripped, the others kept (staging3 `main-ingress-complement`), trailing-slash redirects. +- portals-ingress-manager `src/service.ts`: portal hosts, the same prefixes kept, data-fair stripped. diff --git a/skills/data-fair-service/references/tooling.md b/skills/data-fair-service/references/tooling.md new file mode 100644 index 0000000..e93fcf2 --- /dev/null +++ b/skills/data-fair-service/references/tooling.md @@ -0,0 +1,70 @@ +# Tooling reference for a data-fair service + +The shared values every service converges to. Apply them in the service branch as one `chore` tooling commit, its body listing each item (see SKILL.md for the commit layout). When a repo differs, check "Legitimate differences" before flattening it, and leave every item of "Not decided" as it is. + +## Reference values + +| Item | Value | Why | +|---|---|---| +| eslint | latest major (`eslint ^10`), `neostandard ^0.14` called as `neostandard({ ts: true })` | neostandard 0.14 is the first release accepting eslint 10; portals already runs it | +| Shared rules | `...dfLibRecommended` from `@data-fair/lib-utils/eslint/recommended.js`, `@data-fair/lib-utils` in the root devDependencies | the only eslint config lib provides | +| Root eslint config | `[{ ignores: ['ui/', '**/.type/', 'data/', …generated files] }, ...dfLibRecommended, ...neostandard({ ts: true })]` | the 8-line shape most repos already have; `node_modules` is ignored by default | +| ui eslint config | `dfLibRecommended` + `pluginVue.configs['flat/recommended']` + `pluginVuetify.configs['flat/recommended-v4']` + a `**/*.vue` block with `parser: '@typescript-eslint/parser'` + `neostandard({ ts: true })`; ignores `['dist/*', 'dts/*', 'src/components/vjsf/*']` | every UI runs Vuetify 4 | +| Lint dependencies | `eslint`, `neostandard`, `eslint-plugin-vue`, `eslint-plugin-vuetify`, `@data-fair/lib-utils`, `typescript` in the **root** devDependencies | one copy of each plugin, no "cannot redefine plugin" | +| commitlint | `@commitlint/cli` + `@commitlint/config-conventional` (same major), config `commitlint.config.ts`: `export default { extends: ['@commitlint/config-conventional'] }` | config-conventional everywhere; **only portals keeps its `scope-enum`** (its release notes are generated from scopes) | +| `.husky/commit-msg` | `npx --no -- commitlint --edit "$1"` | `--edit ""` lints the stale `COMMIT_EDITMSG` on merges instead of `MERGE_MSG` | +| `.husky/pre-commit` / `pre-push` | `npm run lint` / `bash dev/pre-push.sh` (type check, unit tests, the `@smoke` set, the changed spec files) | the full suite runs in CI on pull requests, releases and the nightly (data-fair's model, #623) | +| Nightly workflow | only data-fair, portals and simple-directory (biggest, most sensitive) | the others' PR runs are enough | +| CI test jobs | split (by Playwright project, then `--shard`) only when a job goes over ~10 min | each job pays npm ci, the build artifact and the compose stack again | +| husky | `husky ^9`, `"prepare": "husky \|\| true"` | | +| tsconfig (root) | the common block + `"erasableSyntaxOnly": true` | services run `.ts` through Node's type stripping, which fails at runtime on enums and parameter properties; 0 new errors measured on 6 repos | +| tsconfig `"types"` | `"types": ["node"]` only in a tsconfig that type-checks code using Node globals or `node:` modules, directly or through imports: the root (API) one, or a `ui/tsconfig.json` whose imports reach API code (catalogs' UI imports reach `api/src/config.ts`, which uses `node:path` and `node:os`); not in a UI tsconfig that only sees browser code | TS 6 no longer loads every `@types/*` package by default, so these fail with TS2591 ("add 'node' to the types field"); keeping Node types out of browser-only code catches `process`/`Buffer` used by mistake in the UI | +| `check-types` | `tsc && npm -w ui run check-types` (+ other front workspaces, e.g. portals' `portal`); `vue-tsc` in ui | 5 repos never type-checked their UI; **fix every UI type error in the branch** (no ratchet) | +| CI order | `npm ci` → `lint` → `build-types` → `npm -w ui run build` → `check-types` → env init (`./node_modules/.bin/df-dev init-env`) → deps → tests → `npm audit --omit=dev --audit-level=critical` | vue-tsc needs the `.d.ts` files the UI build generates | +| `.nvmrc` | `24`, unless the repo needs a feature of a later 24.x: then the exact version **and a comment in CONTRIBUTING.md saying which feature** (`.nvmrc` cannot express `^24.21`) | verify each repo that pins one before changing it | +| `engines` | `"node": "v24"` | | +| Docker base | `node:24.21.0-alpine3.24` (or the newest 24 used by the other repos), `apk upgrade --no-cache` in `base` | | +| Dockerfile runtime | `ENV NODE_ENV=production`, npm/npx/corepack/yarn removed; a minimal `.dockerignore` (`**/node_modules`, `**/dist`, `ui/dts`, `.env`, `data`, `.git`) when the Dockerfile copies whole folders (`ADD ui ui`): otherwise a local build ships the developer's node_modules / stale dist | see "Legitimate differences" for npm and `USER node` | +| `.gitignore` (root) | the ordered template below; already tracked `docs/superpowers` plans are removed with `git rm --cached`; drop entries nothing produces any more (check before removing) | today several are ignored only by the maintainer's global git ignore | +| `ui/.gitignore` | the Vite template + `dts` | generated `.d.ts` are never committed (see architecture-conventions.md) | +| Scripts | `"lint": "eslint . && npm -w ui run lint"`, `"lint-fix"` mirrored, `"quality"` in the CI order | | + +## Root .gitignore template + +```gitignore +# dependencies and generated code +node_modules/ +.type/ + +# local environment and data +.env +**/config/local-* +data/ +dev/logs/ + +# test output +test-results/ +playwright-report/ + +# agent working files +docs/superpowers/ +.superpowers/ +.playwright-mcp/ + +# + +``` + +## Legitimate differences (do not flatten) + +- Extra workspaces in `lint` / `check-types`: portals' `portal` (Nuxt), data-fair's `parquet-writer`. +- data-fair keeps `check-types-ratchet` in `quality` (1800+ legacy API errors) until its baseline reaches 0. +- processings keeps npm in both runtime images, api and worker (lib-node-registry runs `npm rebuild` on plugins); it only drops corepack and yarn. +- `USER node` stays commented where volumes are mounted as root (data-fair, processings, openapi-viewer, customers, metrics daemon). +- metrics tests with `node --test`; openapi-viewer's `quality` runs only `test-unit`. +- data-fair's base image is alpine3.23 (check its libcrypto / gdal pins before bumping). +- customers runs on GitLab CI. + +## Not decided (leave as is) + +TypeScript major (6), GitHub Actions majors, `USER node`, CI calling `npm run quality` instead of explicit steps, sharing more eslint config through lib, Playwright lint rules from test-suites, `optimizeDeps.noDiscovery` (enabled per service during live validation only, see architecture-conventions.md).