feat(dev): add @data-fair/lib-dev and the data-fair-service skill

CONTRIBUTING.md

@@ -25,6 +25,7 @@ Workspaces are declared in the root `package.json` (`workspaces`). Each lives in
 | `packages/vue`             | `@data-fair/lib-vue`             | Composables and utilities for Vue 3 applications (e.g. `useSession`, ws). |
 | `packages/vuetify`         | `@data-fair/lib-vuetify`         | UI components built on Vuetify. |
 | `packages/processing-dev`  | `@data-fair/lib-processing-dev`  | Utilities for the development environment of a processing. |
+| `packages/dev`             | `@data-fair/lib-dev`             | Dev environment tooling of the services (`df-dev`, `df-worktree`, `df-mem-cap`, `df-status`). |
 
 ## Prerequisites
 

README.md

@@ -15,6 +15,7 @@ This is an npm-workspaces monorepo. Each package is published independently in t
 | [`@data-fair/lib-vue`](./packages/vue) | Composables and utilities for Vue 3 applications: session, fetch, reactive search params, websockets, notifications… |
 | [`@data-fair/lib-vuetify`](./packages/vuetify) | Vuetify components shared by the stack UIs: personal menu, owner pickers, theme and language switchers… |
 | [`@data-fair/lib-processing-dev`](./packages/processing-dev) | Development and test utilities for processing plugins. |
+| [`@data-fair/lib-dev`](./packages/dev) | The dev environment of the services, for humans and agents: `df-dev` (setup, zellij session, status, logs, restart), `df-worktree`, `df-mem-cap`. |
 
 ## Coding agent skills
 

package.json

@@ -59,6 +59,7 @@
   },
   "workspaces": [
     "packages/common-types",
+    "packages/dev",
     "packages/express",
     "packages/node",
     "packages/processing-dev",

packages/dev/README.md

@@ -0,0 +1,39 @@
+# @data-fair/lib-dev
+
+Dev environment of a data-fair service, for humans and agents. Needs zellij >= 0.44 and docker compose v2.
+
+## In a service
+
+`dev/dev.json` declares what is specific to the repo:
+
+```json
+{
+  "ports": ["NGINX_PORT", "DEV_API_PORT", "DEV_UI_PORT", "MONGO_PORT"],
+  "panes": { "api": { "ready": "http://localhost:${DEV_API_PORT}/api/ping" } },
+  "url": "http://${DEV_HOST}:${NGINX_PORT}/portals-manager/dev",
+  "optional": { "events": { "privateEventsUrl": "" } }
+}
+```
+
+- `ports`: names of the port variables written to `.env`, each a free port drawn in 20000-29999, so worktrees never collide. `.env` also gets `DEV_HOST` (`<branch>.localhost`) and `SSRF_PUBLIC_IPS`; `init-env --force` draws them again and keeps the variables added by hand.
+- `panes`: zellij pane titles; `ready` is polled by `restart`.
+- `url`: the page to open, the UI's `/dev` page when it has one (its dev navigation bar), printed by `df-dev status` so an agent knows where to point a browser (the footer pane of the layout prints it with a plain `echo`, no node process).
+- `setup`: repo-specific commands `df-dev setup` runs before build-types (e.g. `npm -w parquet-writer run build`).
+- `optional`: one key per optional compose service (a service with a profile), its value the config overrides to apply **while that service does not run**, e.g. `{ "events": { "privateEventsUrl": "" }, "clamav": { "clamav": { "active": false } } }`: without events the API does not call it. Values must pass the config schema. `df-dev absent-config` merges the overrides of every stopped service; the api pane does `export NODE_CONFIG=$(df-dev absent-config)` (node-config merges it over the config files, no local file to write).
+- No key for the UI: `setup` builds the `ui` workspace when `ui/dist/index.html` is missing (the API's serve-spa reads it at startup, and the build generates the auto-import declarations `check-types` needs).
+
+## Commands
+
+| Command | |
+|---|---|
+| `npx --no df-worktree <branch>` / `npx --no df-worktree-delete <branch> [-f]` | from the main checkout: create `../<repo>_<branch>` (npm ci + setup) / delete it with its containers, volumes and zellij session; branch names: letters, digits, `.`, `_`, `-`, no `/`, and `<repo>_<branch>` (the zellij session) at most 60 chars |
+| `npx --no df-dev setup` | first install / new worktree: `.env` if missing, `setup` commands, build-types, the `ui` build when `ui/dist/index.html` is missing |
+| `npx --no df-dev init-env [--force]` | `.env` only |
+| `npm run dev-zellij -- --services agents,events` | attach or create the session of this worktree (`df-dev zellij`) |
+| `npx --no df-dev status` | for agents: containers, panes (running / exited / not ready), dev processes, memory |
+| `df-status [--start-deps]` | the status pane: containers like `compose up --wait` and one RAM line, every 3 s, in bash (~4 MB); `--start-deps` starts the containers first |
+| `npx --no df-dev logs <pane> [-n 100]` | last lines of a pane |
+| `npx --no df-dev restart <pane>` | Ctrl+C, rerun, wait for readiness |
+| `npx --no df-dev up <service…>` / `down <service…>` | optional compose services |
+| `npx --no df-dev wait-deps` | first command of the panes that need the containers |
+| `df-mem-cap <cmd>` | run a dev server under a 4G cgroup cap (`DEV_MEMORY_MAX`) |

packages/dev/cli.test.ts

@@ -0,0 +1,46 @@
+import { describe, it } from 'node:test'
+import { strict as assert } from 'assert'
+import { execFileSync, spawnSync } from 'node:child_process'
+import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, chmodSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import path from 'node:path'
+
+const cli = path.join(import.meta.dirname, 'cli.js')
+
+describe('df-dev cli', () => {
+  it('lists its commands', () => {
+    const out = execFileSync('node', [cli, '--help'], { encoding: 'utf8' })
+    for (const cmd of ['setup', 'init-env', 'zellij', 'status', 'logs', 'restart', 'up', 'down', 'wait-deps']) assert.match(out, new RegExp(cmd))
+  })
+
+  it('runs init-env from a sub-directory of a repo', () => {
+    const root = mkdtempSync(path.join(tmpdir(), 'df-dev-'))
+    mkdirSync(path.join(root, 'dev'))
+    mkdirSync(path.join(root, 'api'))
+    writeFileSync(path.join(root, 'dev/dev.json'), JSON.stringify({ ports: ['NGINX_PORT'] }))
+    const res = spawnSync('node', [cli, 'init-env'], { cwd: path.join(root, 'api'), encoding: 'utf8' })
+    assert.match(readFileSync(path.join(root, '.env'), 'utf8'), /NGINX_PORT=2\d{4}/)
+    assert.equal(res.stderr, '', 'outside a git repo, git errors must not leak')
+  })
+
+  it('fails with a readable message, not a stack trace', () => {
+    try {
+      execFileSync('node', [cli, 'status'], { cwd: tmpdir(), encoding: 'utf8', stdio: 'pipe' })
+      assert.fail('should have failed')
+    } catch (err: any) {
+      assert.match(err.stderr, /dev\/dev\.json not found/)
+      assert.ok(!err.stderr.includes('    at '))
+    }
+  })
+
+  it('absent-config prints the overrides of the optional services that do not run', () => {
+    const root = mkdtempSync(path.join(tmpdir(), 'df-dev-'))
+    mkdirSync(path.join(root, 'dev'))
+    writeFileSync(path.join(root, 'dev/dev.json'), JSON.stringify({ optional: { events: { privateEventsUrl: null }, clamav: { clamav: { active: false } } } }))
+    const bin = mkdtempSync(path.join(tmpdir(), 'fakebin-'))
+    writeFileSync(path.join(bin, 'docker'), '#!/usr/bin/env bash\nprintf "mongodb\\nevents\\n"\n')
+    chmodSync(path.join(bin, 'docker'), 0o755)
+    const res = spawnSync('node', [cli, 'absent-config'], { cwd: root, encoding: 'utf8', env: { ...process.env, PATH: bin + ':' + process.env.PATH } })
+    assert.deepEqual(JSON.parse(res.stdout), { clamav: { active: false } })
+  })
+})

packages/dev/cli.ts

@@ -0,0 +1,73 @@
+#!/usr/bin/env node
+import { program } from 'commander'
+import { findRoot, readDevConfig, readEnvFile } from './config.js'
+import { initEnv } from './env.js'
+import { setup } from './setup.js'
+import { launch } from './launch.js'
+import { logs, restart } from './panes.js'
+import { composeUp, composeDown, waitDeps } from './compose.js'
+import { collectStatus, renderStatus, checkReadiness } from './status.js'
+import { absentConfig } from './optional.js'
+import { realExec } from './exec.js'
+
+const root = () => findRoot(process.cwd())
+
+program.name('df-dev').description('Dev environment of a data-fair service: setup, zellij session, status, logs and restart.')
+
+program.command('setup').description('create .env if missing, build types, build the ui workspace if missing (first install and new worktrees)')
+  .action(async () => { const r = root(); await setup(r, readDevConfig(r)) })
+
+program.command('init-env').description('write .env with free ports if missing').option('--force', 'draw the ports and DEV_HOST again, keep the variables added by hand')
+  .action(async (opts) => {
+    const r = root()
+    const res = await initEnv(r, readDevConfig(r), { force: opts.force })
+    console.log(res.written ? `wrote ${res.file}` : `${res.file} exists, use --force to regenerate it`)
+  })
+
+program.command('zellij').description('attach to the session of this worktree, or create it').option('--services <list>', 'optional compose services to start, comma separated', '')
+  .action((opts) => { process.exitCode = launch(root(), opts.services.split(',').filter(Boolean)) })
+
+program.command('status').description('containers, panes (with readiness) and dev processes memory, for agents (the status pane is df-status)')
+  .action(async () => {
+    const r = root()
+    const data = collectStatus(r)
+    try {
+      data.panes = await checkReadiness(data.panes, readDevConfig(r), readEnvFile(r))
+    } catch (err: any) {
+      data.panesError = `readiness not checked: ${err.message}`
+    }
+    console.log(renderStatus(data, false))
+  })
+
+program.command('absent-config').description('JSON config overrides (dev.json "optional") for the optional services that do not run, for NODE_CONFIG')
+  .action(() => {
+    const r = root()
+    const running = realExec('docker', ['compose', '--project-directory', r, 'ps', '--services', '--status', 'running']).split('\n').filter(Boolean)
+    console.log(JSON.stringify(absentConfig(readDevConfig(r).optional ?? {}, running)))
+  })
+
+program.command('logs <pane>').description('last lines of a zellij pane, plain text').option('-n, --lines <n>', 'number of lines', '100')
+  .action((pane, opts) => { console.log(logs(root(), pane, Number(opts.lines))) })
+
+program.command('restart <pane>').description('stop and rerun a zellij pane, then wait for its readiness URL')
+  .action(async (pane) => {
+    const r = root()
+    const res = await restart(r, pane, readDevConfig(r))
+    console.log(res.message)
+    if (res.logs) console.log('--- last logs ---\n' + res.logs)
+    if (!res.ok) process.exitCode = 1
+  })
+
+program.command('up <services...>').description('start optional compose services in the running env')
+  .action((services) => { composeUp(root(), services) })
+
+program.command('down <services...>').description('stop optional compose services')
+  .action((services) => { composeDown(root(), services) })
+
+program.command('wait-deps').description('wait until the containers of the active profiles are healthy (first command of process panes; the status pane starts them)')
+  .action(async () => { await waitDeps(root()) })
+
+program.parseAsync().catch((err) => {
+  console.error(`df-dev: ${err.message}`)
+  process.exitCode = 1
+})

packages/dev/compose.test.ts

@@ -0,0 +1,77 @@
+import { describe, it } from 'node:test'
+import { strict as assert } from 'assert'
+import { composeUp, composeDown, waitDeps, pendingDeps } from './compose.js'
+import type { Exec } from './exec.js'
+
+const recorder = () => {
+  const calls: string[] = []
+  return { calls, run: (cmd: string, args: string[]) => { calls.push([cmd, ...args].join(' ')) } }
+}
+
+describe('compose', () => {
+  it('starts named services and waits for them', () => {
+    const r = recorder()
+    composeUp('/x', ['agents', 'events'], r.run)
+    assert.deepEqual(r.calls, ['docker compose up -d --wait agents events'])
+  })
+
+  it('refuses to stop the whole stack', () => {
+    assert.throws(() => composeDown('/x', [], recorder().run), /name the services/)
+  })
+
+  it('stops only the named services', () => {
+    const r = recorder()
+    composeDown('/x', ['agents'], r.run)
+    assert.deepEqual(r.calls, ['docker compose stop agents'])
+  })
+
+  it('lists the services of the active profiles that are not ready yet', () => {
+    const rows = [
+      { Service: 'mongo', State: 'running', Health: 'healthy' },
+      { Service: 'elasticsearch', State: 'running', Health: 'starting' },
+      { Service: 'nginx', State: 'running', Health: '' }
+    ]
+    assert.deepEqual(pendingDeps(['elasticsearch', 'mongo', 'nginx', 'simple-directory'], rows), ['elasticsearch (starting)', 'simple-directory (not created)'])
+  })
+
+  it('counts a one-shot container that exited with 0 as done (s3-init)', () => {
+    assert.deepEqual(pendingDeps(['s3-init', 'job'], [{ Service: 's3-init', State: 'exited', ExitCode: 0 }, { Service: 'job', State: 'exited', ExitCode: 1 }]), ['job (exited 1)'])
+  })
+
+  it('waits for the containers without ever starting them (no race between panes)', async () => {
+    let t = 0
+    const calls: string[] = []
+    const exec: Exec = (cmd, args) => {
+      calls.push(args.join(' '))
+      if (args.includes('config')) return 'mongo\n'
+      return t < 3000 ? '' : JSON.stringify({ Service: 'mongo', State: 'running', Health: 'healthy' }) + '\n'
+    }
+    await waitDeps('/x', { exec, sleep: async (ms) => { t += ms }, now: () => t, log: () => {} })
+    assert.ok(t >= 3000)
+    assert.ok(!calls.some(c => c.includes(' up')), calls.join(' | '))
+  })
+
+  it('fails at once when a dependency stays exited with an error code', async () => {
+    let t = 0
+    const exec: Exec = (cmd, args) => args.includes('config') ? 'mongo\n' : JSON.stringify({ Service: 'mongo', State: 'exited', ExitCode: 1 }) + '\n'
+    await assert.rejects(waitDeps('/x', { exec, sleep: async (ms) => { t += ms }, now: () => t, log: () => {} }), /containers failed: mongo \(exited 1\).*docker compose logs mongo/)
+    assert.ok(t < 10_000, `waited ${t} ms`)
+  })
+
+  it('keeps waiting for a container that restarts after a failed attempt', async () => {
+    let t = 0
+    const states = [{ State: 'exited', ExitCode: 1 }, { State: 'restarting' }, { State: 'running', Health: 'healthy' }]
+    const exec: Exec = (cmd, args) => {
+      if (args.includes('config')) return 'data-fair\n'
+      return JSON.stringify({ Service: 'data-fair', ...states[Math.min(Math.floor(t / 1000), states.length - 1)] }) + '\n'
+    }
+    await waitDeps('/x', { exec, sleep: async (ms) => { t += ms }, now: () => t, log: () => {} })
+    assert.equal(t, 2000)
+  })
+
+  it('gives up after 3 minutes and names what is not ready', async () => {
+    let t = 0
+    const exec: Exec = (cmd, args) => args.includes('config') ? 'mongo\n' : ''
+    await assert.rejects(waitDeps('/x', { exec, sleep: async (ms) => { t += ms }, now: () => t, log: () => {} }), /not ready after 180s: mongo \(not created\).*status pane/)
+  })
+})

packages/dev/compose.ts

@@ -0,0 +1,61 @@
+import { run as realRun, realExec, jsonLines, type Exec } from './exec.js'
+
+// optional services live in a compose profile named after them; naming a service starts it whatever its profile
+export const composeUp = (root: string, services: string[], runFn = realRun) => {
+  runFn('docker', ['compose', 'up', '-d', '--wait', ...services], root)
+}
+
+export const composeDown = (root: string, services: string[], runFn = realRun) => {
+  if (!services.length) throw new Error('name the services to stop, the whole stack is stopped by the user when closing zellij')
+  runFn('docker', ['compose', 'stop', ...services], root)
+}
+
+type PsRow = { Service: string, State: string, Health?: string, ExitCode?: number }
+
+export const pendingDeps = (services: string[], rows: PsRow[]) => services.flatMap(service => {
+  const row = rows.find(r => r.Service === service)
+  if (!row) return [`${service} (not created)`]
+  if (row.State === 'running' && (!row.Health || row.Health === 'healthy')) return []
+  if (row.State === 'exited' && row.ExitCode === 0) return [] // one-shot init container (s3-init…) that completed
+  return [`${service} (${row.Health || row.State}${row.State === 'exited' ? ` ${row.ExitCode}` : ''})`]
+})
+
+/** Dependencies that exited with an error code: they will not become ready by waiting. */
+export const failedDeps = (services: string[], rows: PsRow[]) => rows
+  .filter(r => services.includes(r.Service) && r.State === 'exited' && r.ExitCode !== 0)
+  .map(r => `${r.Service} (exited ${r.ExitCode})`)
+
+const WAIT_TIMEOUT = 180_000
+// a container with a restart policy is briefly "exited" between two attempts: fail on a lasting exit only
+const FAILED_POLLS = 3
+
+/**
+ * First command of the panes that need the containers. It only waits: the status pane starts
+ * the containers, and two panes running "compose up" at the same time conflict on container names.
+ * The services are those of the active profiles (COMPOSE_PROFILES comes from the zellij session env).
+ */
+export const waitDeps = async (root: string, deps: { exec?: Exec, sleep?: (ms: number) => Promise<void>, now?: () => number, log?: (msg: string) => void } = {}) => {
+  const exec = deps.exec ?? realExec
+  const sleep = deps.sleep ?? ((ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms)))
+  const now = deps.now ?? Date.now
+  const log = deps.log ?? console.log
+  const compose = ['compose', '--project-directory', root]
+  const services = exec('docker', [...compose, 'config', '--services']).split('\n').filter(Boolean).sort()
+  const start = now()
+  let lastLog = -Infinity
+  let failedPolls = 0
+  while (true) {
+    const rows: PsRow[] = jsonLines(exec('docker', [...compose, 'ps', '-a', '--format', 'json']))
+    const pending = pendingDeps(services, rows)
+    if (!pending.length) return
+    const failed = failedDeps(services, rows)
+    failedPolls = failed.length ? failedPolls + 1 : 0
+    if (failedPolls >= FAILED_POLLS) throw new Error(`containers failed: ${failed.join(', ')} (see "docker compose logs ${failed.map(f => f.split(' ')[0]).join(' ')}")`)
+    if (now() - start >= WAIT_TIMEOUT) throw new Error(`containers not ready after 180s: ${pending.join(', ')} (see the status pane)`)
+    if (now() - lastLog >= 10_000) {
+      log(`waiting for containers: ${pending.join(', ')}`)
+      lastLog = now()
+    }
+    await sleep(1000)
+  }
+}