mail (1/4) SMTP manager backend by cmyers-mieweb · Pull Request #502 · mieweb/opensource-server

cmyers-mieweb · 2026-10-28T23:16:16Z

SMTP PR (1/4)

This pull request introduces the foundational database schema and model changes necessary to support a full-featured mail hosting stack, including per-domain mail enablement, DKIM key management, per-user mail accounts, unsubscribe handling, and agent mail service reporting. The changes are primarily new Sequelize migrations and model updates, establishing all core tables, relationships, and constraints for the mail system.

Database schema for mail hosting:

Added columns to ExternalDomains for mail enablement, DNS verification states, and last check results, enabling per-domain mail send/receive control and status tracking. (create-a-container/migrations/20261008000000-add-external-domain-mail-columns.js, create-a-container/models/external-domain.js) [1] [2]
Created the DkimKeys table and model for per-domain DKIM key pairs, including status and selector, with associations to ExternalDomain. (create-a-container/migrations/20261008000001-create-dkim-keys.js, create-a-container/models/dkim-key.js, create-a-container/models/external-domain.js) [1] [2] [3]
Added the MailAccounts table for per-user mailboxes, with references to both Users and ExternalDomains, including quota, password hash, and unsubscribe header settings. (create-a-container/migrations/20261008000002-create-mail-accounts.js)
Introduced the MailSuppressions and MailUnsubscribeKeys tables to support one-click unsubscribe, with suppression tracking per account and AES key management for secure tokens. (create-a-container/migrations/20261008000004-create-mail-suppressions.js, create-a-container/migrations/20261008000005-create-mail-unsubscribe-keys.js) [1] [2]
Created SQL views and migrations for the mail stack (Dovecot/Postfix) to read live mailbox and suppression data, and established read-only DB roles (mail_dovecot, mail_postfix) with appropriate grants. (create-a-container/migrations/20261008000006-create-mail-views.js, create-a-container/migrations/20261012000000-create-mail-db-roles.js) [1] [2]
Added a migration to backfill DKIM keys for existing domains. (create-a-container/migrations/20261008000007-backfill-dkim-keys.js)

Agent and application integration:

Extended the Agents table and model with fields for enabled services, missing binaries, mail-host claim, locality, and API key pinning, supporting mail service reporting and the single-mail-host invariant. (create-a-container/migrations/20261008000003-add-agent-mail-fields.js, create-a-container/models/agent.js) [1] [2]
Registered a new /u route for public one-click unsubscribe, mounted before CSRF protection for compatibility with mail clients. (create-a-container/app.js)

These changes lay the groundwork for robust mail hosting, security, and management features, and are designed for compatibility with both local and remote mail hosts.

…ibe (#67) Split 1/4 of #495. Adds the manager-side mail foundation: - DB migrations, models, SQL views, and mail DB roles - /api/v1/mail-accounts resource with suppressions - Per-domain mail enablement, DKIM keys, and Check DNS - Mail-host claim and agent snapshot mail section - One-click unsubscribe (keys, tokens, public /u route) - OpenAPI spec for mail endpoints Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add AES-256-GCM token creation and verification for #67, concealing account and recipient data and authenticating the key ID. Allow the VS Code SQLite dependency's install scripts.

Copilot

🟡 Changes recommended

Unresolved critical issues can expose mail secrets over plaintext HTTP, authorize renamed domains using stale DNS verification, accept invalid SPF configurations, and erase mail settings when saving the existing Settings page.

6 open issues

What changed in this PR

This PR adds the manager-side foundation for mail hosting, connecting database models, mail-account APIs, DNS verification, and agent configuration.

Changes:

Adds mail tables, domain verification fields, SQL views, database roles, and DKIM backfilling.
Implements mailbox management, credential rotation, DNS checks, and public unsubscribe handling.
Extends agent check-ins with mail-host claims and configuration snapshots, with tests and API/schema documentation.

File	Description
`mie-opensource-landing/docs/developers/database-schema.md`	Documents mail entities, relationships, and SQL views.
`mail-helper/src/unsubscribe-token.js`	Adds encrypted unsubscribe-token encoding and verification.
`create-a-container/utils/unsubscribe-token.js`	Adds the manager’s unsubscribe-token codec.
`create-a-container/utils/mail-views.js`	Defines dialect-aware mail lookup views.
`create-a-container/utils/mail-host.js`	Manages mail-host claims and mail IP resolution.
`create-a-container/utils/mail-dns.js`	Builds and verifies mail DNS records.
`create-a-container/utils/mail-account.js`	Generates credentials and defines reserved addresses.
`create-a-container/utils/dkim.js`	Generates DKIM keys and DNS values.
`create-a-container/utils/agent-config.js`	Builds mail configuration snapshots for agents.
`create-a-container/utils/__tests__/unsubscribe-token.test.js`	Tests token roundtrips and tamper rejection.
`create-a-container/utils/__tests__/mail-dns.test.js`	Tests DNS records and verification gates.
`create-a-container/routers/unsubscribe.js`	Adds public unsubscribe confirmation and submission.
`create-a-container/routers/api/v1/settings.js`	Exposes mail configuration settings.
`create-a-container/routers/api/v1/mail.js`	Adds mail-host administration and PTR lookup.
`create-a-container/routers/api/v1/index.js`	Registers mail API routes.
`create-a-container/routers/api/v1/external-domains.js`	Adds domain mail controls and DKIM export.
`create-a-container/routers/api/v1/agents.js`	Adds mail claims, key pinning, and snapshot delivery.
`create-a-container/resources/mailaccounts/validator.js`	Validates mailbox requests and identifiers.
`create-a-container/resources/mailaccounts/service.js`	Implements mailbox permissions and lifecycle operations.
`create-a-container/resources/mailaccounts/serializer.js`	Serializes accounts without password hashes.
`create-a-container/resources/mailaccounts/router.js`	Wires authenticated mailbox endpoints.
`create-a-container/resources/mailaccounts/repository.js`	Encapsulates mailbox database access.
`create-a-container/resources/mailaccounts/controller.js`	Handles mailbox requests and one-time credentials.
`create-a-container/resources/mailaccounts/__tests__/service.test.js`	Tests credentials, permissions, and account controls.
`create-a-container/package.json`	Adds a SQLite dependency script allowance.
`create-a-container/openapi.v1.yaml`	Documents mail APIs, settings, and agent responses.
`create-a-container/models/mail-unsubscribe-key.js`	Models unsubscribe keys and verification retention.
`create-a-container/models/mail-suppression.js`	Models per-account recipient suppressions.
`create-a-container/models/mail-account.js`	Defines mailbox fields and associations.
`create-a-container/models/external-domain.js`	Adds mail verification fields and associations.
`create-a-container/models/dkim-key.js`	Models per-domain DKIM key pairs.
`create-a-container/models/agent.js`	Adds mail-service reporting and claim fields.
`create-a-container/migrations/20261012000000-create-mail-db-roles.js`	Creates read-only mail database users and grants.
`create-a-container/migrations/20261008000007-backfill-dkim-keys.js`	Generates DKIM keys for existing domains.
`create-a-container/migrations/20261008000006-create-mail-views.js`	Creates mail lookup views and grants.
`create-a-container/migrations/20261008000005-create-mail-unsubscribe-keys.js`	Creates unsubscribe-key storage.
`create-a-container/migrations/20261008000004-create-mail-suppressions.js`	Creates recipient suppression storage.
`create-a-container/migrations/20261008000003-add-agent-mail-fields.js`	Adds agent mail fields and claim constraints.
`create-a-container/migrations/20261008000002-create-mail-accounts.js`	Creates mailbox storage and indexes.
`create-a-container/migrations/20261008000001-create-dkim-keys.js`	Creates DKIM key storage.
`create-a-container/migrations/20261008000000-add-external-domain-mail-columns.js`	Adds domain mail enablement and DNS state.
`create-a-container/app.js`	Mounts public unsubscribe routes before CSRF protection.

_{Review effort: Balanced}

+    mailDnsVerified: {
+      type: DataTypes.BOOLEAN,
+      allowNull: false,
+      defaultValue: false,
+      comment: 'SPF/DKIM/DMARC check passed. mailEnabled && mailDnsVerified => domain can send'


+        await queryInterface.sequelize.query('DROP ROLE IF EXISTS mail_dovecot');
+        await queryInterface.sequelize.query('DROP ROLE IF EXISTS mail_postfix');


+    if (!holder) {
+      await agent.update({ mailHostSince: new Date() }, { transaction });
+      return { status: 'holder' };


runleveldev · 2026-10-29T10:54:58Z

+router.use(apiAuth);
+
+// `/domains` must precede `/:id`.
+router.get('/domains', ctrl.listDomains);


Seems to me like this should be /api/v1/external-domains?mail-enabled=true rather than a dedicated route.

I Kept the dedicated route /external-domains admin-only, while any authenticated user needs the sendable-domain list (id + name only) to create a mail account. Folding it in would either expose admin data or break non-admin account creation.

- Stateless unsubscribe keys: drop the MailUnsubscribeKeys table/model/ migration; AES keys are now HKDF-derived from the server's session secrets (newest mints, older ones keep old links verifying) - Fold /mail/enable + /mail/disable into PATCH /external-domains/:id with mailEnabled (enable keeps the Check DNS gate) - Fold /api/v1/mail into the agents resource: GET/DELETE /agents/mail-host and GET /agents/mail-host/ptr (per-route csrfGuard — the agents router mounts before the app-level guard) - Move the unsubscribe pages into views/unsubscribe/*.ejs (ejs dependency) - Replace hand-rolled CIDR math with Node's net.BlockList; TODO points at mailauth for full RFC 7208 SPF evaluation - Inline the mail view SQL into migration 20261008000006; delete utils/mail-views.js - Drop the password 'warning' string from create/rotate responses (UI copy) - Comment the isLocal agent field; TODO for folding smtp_url/ smtp_noreply_address into the mail system - Remove stray mail-helper/src/unsubscribe-token.js and the accidental package.json allowScripts entry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Mail-host endpoints moved under agents: GET/DELETE /api/v1/agents/mail-host, GET /api/v1/agents/mail-host/ptr - Enable/disable mail is now PATCH /external-domains/:id { mailEnabled } - One-time password warning copy is owned by the UI, not the API response Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Unresolved critical flaws allow mail authorization without DNS verification, retain suspended users’ mail access, bypass recipient suppressions through account recreation, and split mailbox data during automatic host failover.

9 open issues

1 previously missed issue

In code that hasn't changed since last review

Rejects compliant multipart unsubscribe requests

create-a-container/routers/unsubscribe.js:31

RFC 8058 §3.2 permits—and recommends—multipart/form-data unsubscribe POSTs, but express.urlencoded() does not parse that encoding. Such a request reaches the body check without List-Unsubscribe and returns 400 even with a valid token, so compliant mail clients cannot unsubscribe. Add a bounded multipart field parser for this endpoint alongside the URL-encoded parser.

1 resolved since last review

Prevent spoofed forwarded HTTPS headers from exposing secrets

1 file not reviewed

create-a-container/package-lock.json: Generated file

_{Review effort: Balanced}

…f sensitive information' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Unresolved critical defects can erase mail configuration during settings saves, enable renamed domains without DNS verification, and allow admin sessions to bypass pinned-agent API-key checks.

9 open issues

3 previously missed issues

In code that hasn't changed since last review

Remote check-ins bypass pinned agent API-key validation

create-a-container/routers/api/v1/agents.js:82

A remote admin session bypasses the agent's API-key pin because apiAuth accepts the session without setting req.apiKey, and this condition then skips the check. A different admin API key can obtain such a session through another authenticated endpoint and use it to impersonate the pinned agent. Require a matching API key for remote check-ins to pinned agents, rejecting an absent key as well as a mismatched one.

Packaged installs omit templates required by unsubscribe pages

create-a-container/routers/unsubscribe.js:36

The production package does not include these templates: create-a-container/Makefile copies only APP_DIRS, which does not contain views. Consequently, every unsubscribe page render—including the invalid-link response—fails with ENOENT and returns 500 on packaged installations, even though it works from the checkout. Add views to the package's installation inputs so the public endpoint works after deployment.

Multipart one-click unsubscribe requests are not parsed

create-a-container/routers/unsubscribe.js:31

RFC 8058 receivers may send the one-click POST as multipart/form-data, but neither this parser nor the app-level JSON/URL-encoded parsers handle that content type. Such a request leaves req.body empty and receives a 400 even with a valid token and a List-Unsubscribe=One-Click form field. Add bounded multipart field parsing alongside URL-encoded parsing so these unsubscribe requests succeed.

1 file not reviewed

create-a-container/package-lock.json: Generated file

_{Review effort: Balanced}

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Update logic to retain degraded holder's claim for Maildirs. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Domain updates can retain or reuse stale DNS verification flags, allowing an unchecked domain to be trusted for sending and receiving mail.

7 open issues

2 resolved since last review

1 file not reviewed

create-a-container/package-lock.json: Generated file

_{Review effort: Balanced}

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Saving unrelated settings can erase mail configuration, while stale domain verification and incorrect SPF validation can enable sending for unverified or misconfigured domains.

6 open issues

1 resolved since last review

Reject domains with multiple SPF records

1 file not reviewed

create-a-container/package-lock.json: Generated file

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

Owners can bypass admin-only suppression removal by deleting and recreating a mail account, which erases its unsubscribe records.

6 open issues

1 file not reviewed

create-a-container/package-lock.json: Generated file

_{Review effort: Balanced}

- Reset DNS verification (flags + check metadata) whenever a domain is renamed, in both PUT and PATCH, and run the mailEnabled=true Check DNS against the name being saved so a rename cannot enable mail using the old name's DNS result. - Preserve omitted mail settings on PUT /settings: mail keys are written only when present in the request (explicit empty string still clears), and the relayhost/SPF-include rule validates the effective saved pair. - Soft-delete mail accounts (paranoid) so MailSuppressions survive delete + recreate of the same address: recreation restores the same row/id, keeping its opt-outs; the account SQL views exclude soft-deleted rows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Deleted-account restoration can grant another user access to the previous owner’s retained mailbox and leave old credentials active if replacement fails.

5 open issues

3 resolved since last review

1 file not reviewed

create-a-container/package-lock.json: Generated file

_{Review effort: Balanced}

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Refactor restore function to use a transaction for restoring and updating the account. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Concurrent initial PostgreSQL claims can fail with a 500 instead of returning a conflict snapshot, and migration rollback silently leaves both login roles and their grants behind.

3 open issues

2 resolved since last review

1 file not reviewed

create-a-container/package-lock.json: Generated file

_{Review effort: Balanced}

cmyers-mieweb and others added 2 commits October 28, 2026 15:51

Add encrypted one-click unsubscribe tokens

ed5983e

Add AES-256-GCM token creation and verification for #67, concealing account and recipient data and authenticating the key ID. Allow the VS Code SQLite dependency's install scripts.

cmyers-mieweb requested review from runleveldev and a balanced review from Copilot October 28, 2026 23:16

Copilot started reviewing on behalf of cmyers-mieweb October 28, 2026 23:16

View session

This was referenced Oct 28, 2026

mail 2/4: agent mail service group — Postfix/Dovecot/OpenDKIM #503

Merged

mail 3/4: opensource-mail-helper milter, packaging, and dev stack #504

Merged

Copilot AI reviewed Oct 28, 2026

View reviewed changes

runleveldev requested changes Oct 29, 2026

View reviewed changes

cmyers-mieweb mentioned this pull request Oct 29, 2026

mail 4/4: email accounts UI, domain mail, mail-host, settings, docs #505

Merged

cmyers-mieweb changed the title ~~Cmyers mail 1 manager backend~~ mail (1/4) SMTP manager backend Oct 29, 2026

Copilot AI balanced review requested due to automatic review settings October 29, 2026 17:20

Copilot started reviewing on behalf of cmyers-mieweb October 29, 2026 17:21

View session

github-advanced-security Bot found potential problems Oct 29, 2026

View reviewed changes

Comment thread create-a-container/routers/api/v1/external-domains.js Fixed

Copilot AI reviewed Oct 29, 2026

View reviewed changes

Potential fix for pull request finding 'CodeQL / Clear-text logging o…

e395fe2

…f sensitive information' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 30, 2026 14:39

Copilot started reviewing on behalf of cmyers-mieweb October 30, 2026 14:39

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Add join with Users table in mail view creation

2780253

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 30, 2026 14:46

Copilot started reviewing on behalf of cmyers-mieweb October 30, 2026 14:47

View session

Modify mail host claim handling for degraded holders

cd5cf93

Update logic to retain degraded holder's claim for Maildirs. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 30, 2026 14:56

Copilot started reviewing on behalf of cmyers-mieweb October 30, 2026 14:56

View session

Refactor SPF record handling to support multiple records

fd87b96

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 30, 2026 15:05

Copilot started reviewing on behalf of cmyers-mieweb October 30, 2026 15:05

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings November 2, 2026 18:11

Copilot started reviewing on behalf of cmyers-mieweb November 2, 2026 18:12

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

Comment thread create-a-container/resources/mailaccounts/repository.js Resolved

Comment thread create-a-container/resources/mailaccounts/service.js Resolved

Enhance address existence check for non-admin users

5094a3c

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings November 2, 2026 18:46

Use transaction in restore function for account

42e2413

Refactor restore function to use a transaction for restoring and updating the account. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot started reviewing on behalf of cmyers-mieweb November 2, 2026 18:46

View session

Copilot AI reviewed Nov 2, 2026

View reviewed changes

runleveldev mentioned this pull request Nov 3, 2026

Email Service Hosted on the Manager #495

Closed

runleveldev approved these changes Nov 3, 2026

View reviewed changes

cmyers-mieweb merged commit 9dde47f into main Nov 3, 2026
8 of 11 checks passed

cmyers-mieweb deleted the cmyers_mail-1-manager-backend branch November 3, 2026 16:31

		await queryInterface.sequelize.query('DROP ROLE IF EXISTS mail_dovecot');
		await queryInterface.sequelize.query('DROP ROLE IF EXISTS mail_postfix');

Conversation

cmyers-mieweb commented Oct 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

runleveldev Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

cmyers-mieweb Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants