Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,35 @@

## Unreleased

- Select a wake-up adapter automatically. `wakeUp` now takes a name or an
adapter, as `config.cache_store` does in Rails, and defaults to
`"automatic"`. Selection prefers `SOLID_OBJECTS_REDIS_URL`, then PostgreSQL
notifications, then polling. `"in_process"` opts out, and an unknown name
throws rather than polls quietly.
- PostgreSQL applications that configure nothing now use notifications. They
gain cross-process wake-up, one dedicated listening client, and one `NOTIFY`
per commit. Set `wakeUp: "in_process"` to keep polling.
- Prove the PostgreSQL notification path before selecting it, because `LISTEN`
does not survive a transaction pooler such as PgBouncer. Selection listens on
a probe channel, sends one `NOTIFY` from a second connection, and waits up to
two seconds for it to arrive. A probe that does not deliver falls back to
polling and warns once.
- Report the resolved choice. `runtime.wakeUpCapability()` names the adapter,
whether it crosses processes, its measured floor, and why it was chosen. The
doctor reports it as a `wakeUp` check, and the polling-only warning now fires
on what was installed rather than on whether a setting was set.
- Keep the capability a configured adapter reports about itself. A configured
`InProcessWakeUpAdapter` now reports `in_process` and warns, rather than
claim that it crosses processes.
- Poll rather than pretend when a requested adapter cannot be built.
`wakeUp: "postgresql"` on a database with no notification channel,
`wakeUp: "redis"` without `SOLID_OBJECTS_REDIS_URL`, and a Redis URL without
the `redis` package each log `solid_objects.wake_up.unavailable` once and
report the reason in the capability, so the doctor warns rather than claim a
cross-process wake-up that cannot happen.
- Select once per runtime. `runtime.wakeUpAdapter()` memoises the selection, so
callers that race for the first use share one probe rather than run one each.

- Add reminder reading. `reminder()` returns one armed alarm as a
`ScheduledReminder`, and `reminders()` lists every key of one operation. Both
apply the intents staged so far in the turn, so a read agrees with what the
Expand Down
19 changes: 19 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -664,6 +664,25 @@ wait. `WakeUpWatch.wait()` returns `true` for a notification and `false` for a
timeout or cancellation. A legacy `void` result remains accepted and preserves
the fast polling cadence.

`configuration.wakeUp` takes a name or an adapter. `WakeUpSetting` is that
union, `WakeUpName` is one of `WAKE_UP_NAMES`, and the default is
`"automatic"`. Selection prefers `SOLID_OBJECTS_REDIS_URL`, then PostgreSQL
notifications, then polling. An unknown name throws rather than polls.

`selectWakeUp(options)` runs that choice and returns a `SelectedWakeUp`, which
pairs the adapter with a `WakeUpCapability`. `WakeUpSelectionOptions` names the
inputs: the setting, the database, the idle polling interval, a logger, and an
optional Redis URL and probe timeout. `WakeUpCapability` reports
`WakeUpAdapterName`, whether the adapter crosses processes, the measured floor
in milliseconds, and the reason. `runtime.wakeUpAdapter()` and
`runtime.wakeUpCapability()` return the choice, which is made once per runtime.

An adapter may declare its own `defaultCapability`, which is what it reports
about itself. Selection keeps what a configured adapter declares rather than
assume, and `runtime.wakeUpCapability()` reports what selection installed. `NotificationWakeUpAdapter` adds
`channelFor(role)`, which a database that offers a notification channel
provides through `database.wakeUp(options)`.

### Errors

The root exports `SolidObjectsError` and its supported subclasses:
Expand Down
51 changes: 38 additions & 13 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,19 +49,19 @@ as an instruction to divide the actor. See

## Runtime roles and supervision

| Option | Default | Contract |
| ------------------------------------------- | -----------------------: | -------------------------------------------------------- |
| `workerCount` | `1` | Non-negative actor workers. |
| `effectWorkerCount` | `1` | Non-negative effect workers. |
| `broadcastWorkerCount` | `1` | Non-negative broadcast workers when realtime is enabled. |
| `reminderSchedulerCount` | `1` | Non-negative reminder schedulers. |
| `processHeartbeatIntervalMilliseconds` | `15_000` | Positive persisted heartbeat cadence. |
| `processAliveThresholdMilliseconds` | `60_000` | Positive age after which an owner is stale. |
| `shutdownTimeoutMilliseconds` | `15_000` | Positive shared graceful-shutdown budget. |
| `supervisorRestartDelayMilliseconds` | `100` | Positive initial failed-role replacement delay. |
| `supervisorMaximumRestartDelayMilliseconds` | `10_000` | Positive cap no smaller than the initial delay. |
| `wakeUp` | `InProcessWakeUpAdapter` | Adapter implementing `watch`, `notify`, and `close`. |
| `logger` | console methods | Structured `debug`, `info`, `warn`, and `error` sink. |
| Option | Default | Contract |
| ------------------------------------------- | --------------: | ----------------------------------------------------------------- |
| `workerCount` | `1` | Non-negative actor workers. |
| `effectWorkerCount` | `1` | Non-negative effect workers. |
| `broadcastWorkerCount` | `1` | Non-negative broadcast workers when realtime is enabled. |
| `reminderSchedulerCount` | `1` | Non-negative reminder schedulers. |
| `processHeartbeatIntervalMilliseconds` | `15_000` | Positive persisted heartbeat cadence. |
| `processAliveThresholdMilliseconds` | `60_000` | Positive age after which an owner is stale. |
| `shutdownTimeoutMilliseconds` | `15_000` | Positive shared graceful-shutdown budget. |
| `supervisorRestartDelayMilliseconds` | `100` | Positive initial failed-role replacement delay. |
| `supervisorMaximumRestartDelayMilliseconds` | `10_000` | Positive cap no smaller than the initial delay. |
| `wakeUp` | `"automatic"` | A name or an adapter implementing `watch`, `notify`, and `close`. |
| `logger` | console methods | Structured `debug`, `info`, `warn`, and `error` sink. |

Counts may be zero, but the complete configuration must leave at least one
runtime role enabled. Broadcast workers are started only when `broadcast` or
Expand Down Expand Up @@ -158,6 +158,31 @@ short capped backoff only when no synchronous deadline is active.
| `applicationName` | `"solid-objects"` |
| `onPoolError` | structured `console.error` |

`wakeUp` takes a name or an adapter, as `config.cache_store` does in Rails.
`"automatic"` is the default. It prefers `SOLID_OBJECTS_REDIS_URL`, then
PostgreSQL notifications, then polling. `"in_process"` opts out, `"postgresql"`
and `"redis"` force one, and an unknown name throws rather than polls quietly.

A requested adapter that the environment cannot provide polls instead and says
so. `"postgresql"` on a database with no notification channel, `"redis"` without
`SOLID_OBJECTS_REDIS_URL`, and a Redis URL without the `redis` package each log
`solid_objects.wake_up.unavailable` once and report the reason in the
capability. Only a name that does not exist is refused, because a typo cannot be
honoured at all.

Selection runs once per runtime, on first use. `runtime.wakeUpCapability()`
reports what it chose, whether that choice crosses processes, its measured
floor in milliseconds, and why. The doctor reports the same record, and the
polling-only warning fires on what was installed rather than on whether a
setting was set. An adapter that declares its own `defaultCapability` keeps it, so a
configured `InProcessWakeUpAdapter` still warns.

Automatic selection proves the PostgreSQL path before it chooses it. It listens
on a probe channel, sends one `NOTIFY` from a second connection, and waits up to
two seconds for it to arrive. A probe that does not deliver falls back to
polling and warns once, because `LISTEN` does not survive a transaction pooler
such as PgBouncer.

`database.wakeUp(options)` creates a dedicated notification adapter using the
same connection string. Its options are `channelPrefix = "solid_objects"`,
`applicationName = "solid-objects-wake-up"`, and `onListenerError`. Use a
Expand Down
23 changes: 16 additions & 7 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,25 @@ work and wake-up notifications reset the role to the fast interval. Actor
workers clamp the ceiling to `leaseRenewalIntervalMilliseconds` while they may
hold cached activations.

The default generation-based wake-up adapter interrupts waits for new actor
The runtime selects a wake-up adapter on first use. It prefers
`SOLID_OBJECTS_REDIS_URL`, then PostgreSQL notifications, then polling.
The generation-based in-process adapter interrupts waits for new actor
messages, effects, reminders, and broadcasts in the same Node process. It does
not cross a process boundary. When live processes share the database without a
configured adapter, the runtime logs
`solid_objects.polling_only_cross_process_wake_up` once. Use PostgreSQL
notifications or optional Redis Pub/Sub when separate processes need prompt
delivery. Without one of them, newly committed work can wait for the current
idle polling interval. The runtime isolates notification errors and logs them by
not cross a process boundary. When live processes share the database and the
installed adapter does not cross processes, the runtime logs
`solid_objects.polling_only_cross_process_wake_up` once. Without a
cross-process adapter, newly committed work can wait for the current idle
polling interval. The runtime isolates notification errors and logs them by
role and error class. The committed work does not fail.

`runtime.wakeUpCapability()` and the `wakeUp` doctor check report the adapter
that is installed, whether it crosses processes, its measured floor, and why it
was chosen. On PostgreSQL, selection first proves the path: it listens on a
probe channel, notifies it from a second connection, and waits for the
notification. A probe that does not arrive logs
`solid_objects.wake_up.pooled_session` once and falls back to polling, because
`LISTEN` does not survive a transaction pooler such as PgBouncer.

The warning excludes process rows with the current hostname and host process ID.
It can therefore appear during a rolling deployment or restart overlap when an
older and newer process briefly share the same database. A process that stopped
Expand Down
27 changes: 22 additions & 5 deletions docs/parity.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,28 @@ Ruby field names; this does not change runtime delivery semantics.
| PostgreSQL wake-up | Native | `database.wakeUp()` uses one dedicated event-driven client, role-specific `LISTEN/NOTIFY`, generation fencing, reconnectable listeners, and durable polling fallback. |
| Redis wake-up | Native | An optional `redis` peer provides role-specific Pub/Sub over separate lazy publisher/subscriber connections, with bounded failures and durable polling fallback. |

Every wake-up adapter above is opt-in. Neither runtime selects one
automatically. An application that configures nothing keeps polling. Each
runtime warns once when live processes share a database without a configured
cross-process adapter. This limit is intentional in both runtimes. It is not a
gap between them.
Both runtimes select a wake-up adapter automatically. `wakeUp` takes a name or
an adapter and defaults to `"automatic"`, which prefers a configured Redis URL,
then PostgreSQL notifications, then polling. Selection proves the PostgreSQL
path with a probe notification, because `LISTEN` does not survive a transaction
pooler. Each runtime reports what it installed, whether that crosses processes,
its measured floor, and why, and warns once when live processes share a
database and the installed adapter cannot reach them. MySQL still polls in both
runtimes, because MySQL has no notification channel.

A requested adapter that the environment cannot provide polls instead and says
so in both runtimes, rather than claim a cross-process wake-up that cannot
happen. Only a name that does not exist is refused.

Three details differ, and each follows from the language rather than from the
feature. The pooled-session warning is emitted once per runtime here and once
per process in Ruby, because this runtime supports several runtimes in one
process. A configured adapter must implement `watch`, `notify`, and `close`
here, while Ruby requires `signal` and `wait` and treats `watch` and `stop` as
optional, which is each runtime's own adapter contract. A selection that a later
edit of the settings makes impossible is reported once here and raised in Ruby;
both runtimes refuse an unknown name when the configuration is built, so this
only reaches code that changes the setting afterwards.

## Realtime and browser behavior

Expand Down
2 changes: 1 addition & 1 deletion src/broadcast-worker.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ export class BroadcastWorker {
await this.ensureRegistered()
await this.runtime.warnIfPollingIsOnlyCrossProcessWakeUp()
while (!signal.aborted && !this.stopping) {
const wakeUp = await this.runtime.settings.wakeUp.watch("broadcasts")
const wakeUp = await this.runtime.watchWakeUp("broadcasts")
const processed = await this.runOnce()
if (processed > 0) {
this.pollingBackoff.reset("work")
Expand Down
33 changes: 23 additions & 10 deletions src/configuration.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
import { InvalidActor } from "./errors.js"
import type { Database } from "./database/types.js"
import type { DeepReadonly, JsonObject, JsonValue, Logger, LongRunningComponent } from "./types.js"
import { InProcessWakeUpAdapter, type WakeUpAdapter } from "./wake-up.js"
import { WAKE_UP_NAMES, type WakeUpAdapter, type WakeUpSetting } from "./wake-up.js"

const wakeUpNames: readonly string[] = WAKE_UP_NAMES

export interface AuthorizationInput {
actorType: string
Expand Down Expand Up @@ -79,7 +81,7 @@ export interface SolidObjectsConfiguration {
authorizeSubscription?: (input: SubscriptionAuthorizationInput) => boolean | Promise<boolean>
instrumentation?: (event: InstrumentationEvent) => void
broadcast?: (event: BroadcastEvent) => Promise<void>
wakeUp?: WakeUpAdapter
wakeUp?: WakeUpSetting
}

export interface BroadcastEvent {
Expand All @@ -97,8 +99,7 @@ export interface RuntimeSettings extends Required<
logger: Logger
broadcast?: (event: BroadcastEvent) => Promise<void>
instrumentation?: (event: InstrumentationEvent) => void
wakeUp: WakeUpAdapter
wakeUpConfigured: boolean
wakeUp: WakeUpSetting
authorizationPoliciesConfigured: Readonly<Record<string, boolean>>
}

Expand Down Expand Up @@ -156,8 +157,7 @@ export function buildSettings(configuration: SolidObjectsConfiguration): Runtime
processRetentionMilliseconds: configuration.processRetentionMilliseconds ?? 7 * 86_400_000,
pruneBatchSize: configuration.pruneBatchSize ?? 1_000,
logger: configuration.logger ?? consoleLogger,
wakeUp: configuration.wakeUp ?? new InProcessWakeUpAdapter(),
wakeUpConfigured: configuration.wakeUp !== undefined,
wakeUp: configuration.wakeUp ?? "automatic",
authorizeMessage: configuration.authorizeMessage ?? (() => false),
authorizeQuery: configuration.authorizeQuery ?? (() => false),
authorizeDestroy: configuration.authorizeDestroy ?? (() => false),
Expand Down Expand Up @@ -193,12 +193,25 @@ export function validateComponent(component: LongRunningComponent): void {
}
}

function validateSettings(settings: RuntimeSettings): void {
function validateWakeUp(setting: WakeUpSetting): void {
if (typeof setting !== "string") return validateWakeUpAdapter(setting)
if (wakeUpNames.includes(setting)) return

throw new TypeError(
`unknown wakeUp ${JSON.stringify(setting)}, expected one of ${WAKE_UP_NAMES.join(", ")} or an adapter`,
)
}
Comment thread
cardmagic marked this conversation as resolved.

function validateWakeUpAdapter(adapter: WakeUpAdapter): void {
for (const name of ["watch", "notify", "close"] as const) {
if (typeof settings.wakeUp[name] !== "function") {
throw new TypeError(`wakeUp must implement ${name}`)
}
if (typeof adapter[name] === "function") continue

throw new TypeError(`wakeUp must implement ${name}`)
}
}

function validateSettings(settings: RuntimeSettings): void {
validateWakeUp(settings.wakeUp)
if (!/^[a-z][a-z0-9_]*$/.test(settings.tableNamePrefix)) {
throw new TypeError("tableNamePrefix must contain lowercase letters, digits, and underscores")
}
Expand Down
3 changes: 3 additions & 0 deletions src/database/types.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import type { NotificationWakeUpAdapter } from "../wake-up.js"

export type DatabaseFamily = "sqlite" | "postgresql" | "mysql"

export interface RunResult {
Expand All @@ -20,6 +22,7 @@ export interface Database {
readonly family: DatabaseFamily
readonly schemaIdentity: string
transactionActive?(): boolean
wakeUp?(options?: { channelPrefix?: string }): NotificationWakeUpAdapter
connection<Result>(callback: (connection: DatabaseConnection) => Promise<Result>): Promise<Result>
transaction<Result>(
callback: (connection: DatabaseConnection) => Promise<Result>,
Expand Down
21 changes: 21 additions & 0 deletions src/doctor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ export class Doctor {
schema,
await this.checkAuthorization(),
await this.checkDatabase(),
await this.checkWakeUp(),
]
checks.push(
schema.status === "fail"
Expand Down Expand Up @@ -145,6 +146,26 @@ export class Doctor {
})
}

private async checkWakeUp(): Promise<DoctorCheck> {
try {
const capability = await this.runtime.wakeUpCapability()
const floor = capability.measuredFloorMilliseconds
const summary = `${capability.adapter}: ${capability.reason}${
floor === undefined ? "" : `, floor ${floor} ms`
}`
if (capability.crossesProcesses) {
return check({ name: "wakeUp", status: "pass", message: summary })
}
return check({
name: "wakeUp",
status: "warn",
message: `${summary}; a commit in one process cannot wake another`,
})
} catch (error) {
return failedCheck("wakeUp", error)
}
}

private checkConfiguration(): DoctorCheck {
return check({
name: "configuration",
Expand Down
Loading
Loading