Repository navigation
Conversation
- dev/dev.json for df-dev: port variables, panes, pages to open, optional services
- core containers without profile, mongo cache capped
- events and openapi-viewer optional, unplugged through NODE_CONFIG when absent
- zellij layout on df-dev, old dev scripts removed
- agents, contributors, tests and CI use df-dev
- CI starts only the core compose services
Applies the dev docker-compose reference of the data-fair-service skill (lib), the only place where
storage versions change:
- service names simple-directory, data-fair, events, agents, registry, elasticsearch, mongo;
canonical order; sibling services on their branch tag; one compose profile per optional service,
named after it
- mongo:8 with a mongosh healthcheck, ES ghcr.io/data-fair/elasticsearch:8.19.9 (as staging3) with a
${ES_HEAP:-512m} heap and mem_limit 2g; mongo 4.2 / ES 7 volumes must be reset (docker compose
down -v)
- mongo and ES healthchecks probe every 2 s while starting, then once a minute (a mongosh probe
costs ~1.3 s of CPU)
- simple-directory: MONGO_URL, nc healthcheck, auth rate limit 200 per second for the API logins of
the tests, the shared secret-* keys
- one dev cipher password (registry needs >= 32 chars): data ciphered with the old one is unreadable
- simple-directory and data-fair on :master instead of :8 / :6
- optional maildev, wired to sd and to the worker's own SMTP transport (plugins' sendMail)
- NGINX_PORT1 renamed NGINX_PORT (NGINX_PORT2 was unused): existing worktrees rename the key in
their .env and add MAILDEV_UI_PORT / MAILDEV_SMTP_PORT (df-dev init-env never rewrites a .env)
- sibling images run with NODE_ENV=production on their default config, so the dev values are
explicit: data-fair (REMOTE_SERVICES=[], CACHE_PUBLIC_MAX_AGE=1, API rate limits, worker
concurrency 1, CACHE_MONGO_SIZE=200, named /data volume), sd (MAILS_TRANSPORT on the env's own
maildev port, mail rate limit), agents (mock models, unlimited credits), ES without swap
- nginx template on the shared head of the skill (prod ingress settings: 600 s read timeout,
buffering off, 32k buffers, X-Forwarded-For overwritten, websocket map, trailing-slash redirects;
no gzip)
- nginx: /events keeps its prefix (stripped, every external call got a 404), /data-fair stripped as
in prod
- eslint 10, neostandard 0.14, vue recommended and vuetify recommended-v4 rules
- @commitlint/cli 21, commit-msg hook lints the file git passes ("$1")
- tsconfig erasableSyntaxOnly: Node type stripping rejects non-erasable syntax
- check-types also type-checks the UI with vue-tsc
- gitignore in the shared ordered layout, agent working files ignored
- CONTRIBUTING requires node 24 as in .nvmrc
- CI: static job (lint, build-types, UI build, check-types, audit) passes the generated
types and the UI build as an artifact to one test job (unit, api, e2e; no shards);
Playwright browser cache keyed on its version; TEST_RETRIES input (nightly 0);
cancel-in-progress on pull requests only; nightly full gate, skipped when master has
not moved since the last green nightly; node-version-file .nvmrc
- vite config per the service template: dev ports from .env checked, strictPort,
dev page excluded at build
…rignore The dev compose no longer builds and runs the main and worker images (testImages profile, npm run test-images): unused, and the status pane listed them as not started.
The api and the worker only started the events queue when both privateEventsUrl and secretKeys.events were set, and the per-call guards did not match that condition. With events absent in dev (privateEventsUrl: ""), every create, patch and run pushed into an uninitialized queue: lib-node throws "events queue was not initialized", so the API answered 500. - api and worker always start the queue with privateEventsUrl and secretKeys.events; the queue goes inactive by itself without the events service (pushEvent is then a no-op) and reports an internal error when the url is set but the secret is missing (fix in @data-fair/lib-node events-queue, needs the lib-node release that ships it) - drop the per-service guards in the api router and the worker runs
…er test pyramid Playwright config and fixtures: - no --max-failures in test, test-unit, test-api, test-e2e; test-api-fast keeps it to iterate - timeout 30 s per test (was 60 s), navigationTimeout 10 s; specs that run plugins through the worker call test.slow() (lifecycle, branch-plugin run) - trace retain-on-failure, screenshot only-on-failure, json report to dev/logs/test-timings.json - CI retries from TEST_RETRIES (nightly passes 0) - e2e login through the simple-directory API (one login per user and worker, renewed after 10 min) instead of the login form and its cookie cache; a future sd-keepalive localStorage timestamp turns off lib-vue's keepalive, so no page rotates the shared exchange token (the old fixture hit "session was expired or deleted server side" -> 401 on the edit page) - clean() fails with the list of runs still triggered/running after 10 s - warmup timestamps as annotations instead of console output - vite optimizeDeps.noDiscovery: a dependency discovered while browsing reloaded the page; the commonjs dependencies are listed in include with their exact import specifier (cronstrue/locales/en, ajv-i18n/localize/en/index.js...); validated on a cold vite cache: warmup + 8 e2e in 39 s (1.8 min with discovery) - vite.config: import.meta.dirname instead of __dirname (vite's native config loader warning) Quality gates: - dev/pre-push.sh (husky pre-push): check-types, unit, @smoke api/e2e, and the api/e2e spec files changed since origin/master; the full suite runs in CI (PR, release, nightly) - @smoke: lifecycle create+run, permissions across roles, broken plugin 404 on config patch, list empty state (e2e), edit form save/cancel (e2e) Test pyramid: - memory-oom.e2e -> memory-oom.api: no browser in it, it no longer waits for the e2e warmup - translateRegistryError moved to api/src/processings/operations.ts (pure), unit-tested for its 403/404/502 messages - removed available-plugin.e2e: the save-button e2e renders the same edit form of an available plugin, it now also asserts the broken-plugin banner is absent - removed layout "list renders a card": the leave-guard e2e clicks that card, broken-plugin e2e checks the list - removed branch-plugin "creates a processing": the run test creates it too, and now asserts processing.plugin; removed "re-upload bumps dataUpdatedAt": registry behaviour, covered by registry tests/artefacts.api.spec.ts
….14.5 - lib-node 2.16.1: the events queue goes inactive by itself without the events service (the events queue fix commit of this branch relies on it) - lib-types-builder 1.14.5: generated files replaced atomically, unchanged ones skipped - lib-express 1.31.0: scoped sessions guard, cookie sessions unchanged - lib-node and lib-express ranges move from ~ to ^, like the other services
…ion timeout The api, worker and ui started in CI logged nowhere, a failing run showed only the compose logs. The warmup's first cold goto ran under the 10 s navigation timeout; 60 s as in the other services. Node types in the root tsconfig (TS 6 loads no @types package by default).
ES pre-touches its initial heap: a fixed 512m kept ~360m resident per idle dev stack for nothing (measured 1001 -> 640 MiB idle, 1079 -> 722 MiB under load).
…bled An uncapped ES 8 boot peaks at 5-6 cores. ES sizes its thread pools, GC and JIT threads to the quota, so cpus: 2 halves the boot CPU (104 -> 45 s cold, 96 -> 46 s restart) with no slower time to green. ML, watcher, stack templates and deprecation indexing are unused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move processings' dev environment and tooling to the convention shared by the data-fair services (processings is one of the 4 pilots, with data-fair, portals and catalogs).
Why: one dev environment and one tooling standard across services, cheap enough in RAM to run several worktrees side by side, and a test suite that reports its own failures clearly.
df-dev(@data-fair/lib-dev) and zellij replace the per-repo scripts;dev/dev.jsondeclares ports, panes and optional services, each optional container behind its own compose profile.cpus: 2, unused ES features disabled; mongo cache capped..gitignoreconventions (hence the vuetify v4 class renames inui/src), deps bumped (lib-node 2.16.1, lib-express 1.31.0, lib-types-builder 1.14.5, lib-dev 0.1.0).@smokeset;translateRegistryErrormoved toapi/src/processings/operations.tsand unit-tested; duplicated e2e cases removed (each one's coverage named in the commit message). CI now runs the e2e tests too, which it did not before.Heads-up — changes reaching production code:
BUILD.json(an image built outside CI), reporting its version as unknown.docker compose down -v).