Skip to content
33 changes: 33 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,39 @@

## Unreleased

- Select a wake-up adapter automatically. `config.wake_up_adapter` now takes a
name or an adapter, as `config.cache_store` and
`config.active_job.queue_adapter` do, and defaults to `:automatic`. Selection
prefers a configured Redis URL, then PostgreSQL notifications, then polling.
`:in_process` opts out, and an unknown name raises rather than quietly
polling.
- PostgreSQL deployments that configure nothing now use notifications. They gain
cross-process wake-up, a connection per waiting thread outside the pool, and
one `NOTIFY` per enqueue after the commit. Set
`config.wake_up_adapter = :in_process` to keep polling.
- Prove the PostgreSQL notification path before selecting it, because `LISTEN`
does not survive a transaction pooler such as PgBouncer. Selection listens on
a probe channel, sends one `NOTIFY` from a second connection, and waits up to
two seconds for it to arrive. A probe that does not deliver falls back to
polling and warns once.
- Select the wake-up adapter once per process. `SolidObjects.wake_up` memoised
without a lock, so threads that raced for the first use each ran a full
selection.
- Keep the capability that a configured adapter reports about itself. A
configured `SolidObjects::WakeUp` now reports `:in_process` and warns, rather
than claim that it crosses processes.
- Poll rather than pretend when a requested adapter cannot be built.
`wake_up_adapter = :postgresql` on a database with no notification channel,
`:redis` without `SOLID_OBJECTS_REDIS_URL`, and a Redis URL without the redis
gem each log `solid_objects.wake_up.unavailable` once and record the reason in
the capability, so the doctor warns rather than claim a cross-process wake-up
that cannot happen.
- Validate `wake_up_adapter` in `configure`. An unknown name raised at the first
wake-up, which is after a commit, rather than at boot.
- Report the resolved choice. `SolidObjects.wake_up.capability` names the
adapter, whether it crosses processes, its measured floor, and why it was
chosen. The doctor reports it, and the polling-only warning now fires on what
was installed rather than on whether a setting was set.
- Read the durable row rather than the query cache in `MessageReference#status`,
`MessageReference#result`, and an actor snapshot. A caller that polls holds one
query cache for the whole poll, and the worker that finishes the message is
Expand Down
12 changes: 11 additions & 1 deletion docs/adr/0011-wake-up-strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,14 @@ The interface supports:

MySQL uses polling or optional Redis. SQLite uses polling plus the in-process signal; multi-host SQLite is outside its supported operating model.

Selection is automatic. `wake_up_adapter` takes a name or an adapter and
defaults to `:automatic`, which prefers a configured Redis URL, then PostgreSQL
notifications, then polling. PostgreSQL is chosen only after a probe
notification arrives, because `LISTEN` does not survive a transaction pooler. A
requested adapter that the environment cannot provide polls and records why,
rather than claim a wake-up it cannot deliver. `SolidObjects.wake_up.capability`
reports the choice, and the doctor reports the same record.

The synchronous caller first attempts to claim and execute the actor locally,
so the normal path has no worker polling leg. When another process owns the
activation, coordination overhead from completion commit until the caller's
Expand All @@ -45,5 +53,7 @@ Timeout does not cancel durable work.
- Redis loss only increases latency and never loses durable work.
- Every adapter retains periodic polling to close startup, reconnect, and missed-message races.
- A process that returns `false` from a timed wait participates in backoff; an older custom adapter that returns `nil` keeps the fast cadence.
- A multi-process deployment without an adapter trades idle database load for up to the current idle polling interval of notification latency and logs that topology once.
- A multi-process deployment whose installed adapter cannot cross processes trades idle database load for up to the current idle polling interval of notification latency and logs that topology once.
- A PostgreSQL deployment that configures nothing now pays one `NOTIFY` per enqueue after the commit and one listening connection per waiting thread, outside the pool.
- Selection runs once per process, under a lock, because the probe opens connections and waits.
- Notification payloads never contain actor arguments or results.
28 changes: 21 additions & 7 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,13 +245,27 @@ to `idle_polling_interval`, which defaults to one second. Actor workers clamp
the ceiling to `lease_renewal_interval` while they may hold cached activations.
Set the fast and idle values equal for a fixed cadence.

The default wake-up interrupts waits only in the current Ruby process. When a
live process record shows that the database is shared across processes and no
adapter is configured, the runtime logs
`solid_objects.polling_only_cross_process_wake_up` once. Configure
`WakeUpAdapters::Postgresql` or `WakeUpAdapters::Redis` when separate processes
need prompt delivery. Without one, newly committed work can wait up to the
current idle polling interval.
Solid Objects selects a wake-up adapter on first use. `wake_up_adapter` defaults
to `:automatic`, which prefers `SOLID_OBJECTS_REDIS_URL`, then PostgreSQL
notifications, then polling. `SolidObjects.wake_up.capability` and the
`wake_up` doctor check report what was installed, whether it crosses processes,
its measured floor, and why.

The in-process signal interrupts waits only in the current Ruby process. When a
live process record shows that the database is shared across processes and the
installed adapter does not cross them, the runtime logs
`solid_objects.polling_only_cross_process_wake_up` once. Newly committed work
can then wait up to the current idle polling interval.

On PostgreSQL, selection proves the path first: it listens on a probe channel,
notifies it from a second connection, and waits for the notification. A probe
that does not arrive logs `solid_objects.wake_up.pooled_session` once and falls
back to polling, because `LISTEN` does not survive a transaction pooler such as
PgBouncer. A requested adapter that the environment cannot provide, such as
`:postgresql` on MySQL or `:redis` without `SOLID_OBJECTS_REDIS_URL`, logs
`solid_objects.wake_up.unavailable` once and polls rather than claim a
cross-process wake-up that cannot happen. Only an unknown name is refused, and
`configure` refuses it at boot.

The warning excludes process rows with the current hostname and PID. It can
therefore appear during a rolling deployment or restart overlap when an older
Expand Down
58 changes: 32 additions & 26 deletions docs/realtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,46 +140,52 @@ explicitly serve the module. Turbo's normal morph rules still apply; use

## Cross-process wake-up

Runtime roles poll for work and are woken early by an in-process signal. That
signal cannot cross process boundaries, so a commit in a Puma process does not
wake a broadcast executor in a worker process, and delivery waits out
Runtime roles poll for work and are woken early by a signal. An in-process
signal cannot cross process boundaries, so a commit in a Puma process would not
wake a broadcast executor in a worker process, and delivery would wait out
`polling_interval`, 100 ms by default.

On PostgreSQL, install the notification adapter to remove that delay:
Solid Objects selects the adapter for you. `wake_up_adapter` defaults to
`:automatic`, which prefers a configured Redis URL, then PostgreSQL
notifications, then polling:

```ruby
# config/initializers/solid_objects.rb
configuration.wake_up_adapter = SolidObjects::WakeUpAdapters.for
configuration.wake_up_adapter = :automatic # the default
configuration.wake_up_adapter = :in_process # opt out
configuration.wake_up_adapter = :postgresql # force one
configuration.wake_up_adapter = MyAdapter.new # your own
```

`WakeUpAdapters.for` returns notifications on PostgreSQL and the in-process
default on SQLite and MySQL, so the same line is safe across adapters. Name
`SolidObjects::WakeUpAdapters::Postgresql.new` directly to require it.

MySQL has no notification primitive. MySQL applications either keep polling and
tune `polling_interval`, or configure the Redis adapter:

```ruby
configuration.wake_up_adapter = SolidObjects::WakeUpAdapters::Redis.new(
url: ENV["REDIS_URL"]
)
```

Measured latency for a cross-process wake-up drops from 103.8 ms to 5.7 ms at
p50. The `redis` gem is not a dependency of this gem, so applications add it
themselves. One background subscription per process fans out to every waiting
role in memory, rather than one connection per thread, and `WakeUpAdapters.for`
does not select it: Redis is infrastructure this gem otherwise does not require,
so choosing it is explicit.
`SolidObjects.wake_up.capability` reports what was installed, whether it crosses
processes, its measured floor, and why. `bin/rails solid_objects:doctor` reports
the same record.

On PostgreSQL, selection proves the path before it chooses it. It listens on a
probe channel, sends one `NOTIFY` from a second connection, and waits for it to
arrive, because `LISTEN` does not survive a transaction pooler such as
PgBouncer. A probe that does not deliver falls back to polling and warns once.
Measured latency for a cross-process wake-up drops from 103.7 ms to 2.9 ms at
p50. The adapter keeps `polling_interval` as the upper bound: a missed or failed
notification costs latency, never correctness, and signalling never raises into
the caller that committed. `LISTEN` needs its own connection, so the adapter
opens one outside the pool and releases it on `stop`.

Applications on SQLite or MySQL, or that do not configure the adapter, keep the
existing polling behaviour.
MySQL has no notification primitive, so MySQL applications either keep polling
and tune `polling_interval`, or set `SOLID_OBJECTS_REDIS_URL`, which selects
Redis on any database:

```bash
SOLID_OBJECTS_REDIS_URL=redis://localhost:6379/0
```

Measured latency for a cross-process wake-up drops from 103.8 ms to 5.7 ms at
p50. The `redis` gem is not a dependency of this gem, so applications add it
themselves, and selection polls and says so when the gem is missing. One
background subscription per process fans out to every waiting role in memory,
rather than one connection per thread. Name
`SolidObjects::WakeUpAdapters::Redis.new(url:)` directly for a URL that does not
come from the environment.

## Batched component refreshes

Expand Down
32 changes: 17 additions & 15 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,21 +122,23 @@

- Wake-up strategy: in-process signaling, durable polling, injection, and
cross-process adapters for PostgreSQL and Redis are implemented and tested.
What is not done is making any of them automatic. In-process signaling cannot
cross process boundaries, so by default a commit in a web process does not
wake a broadcast executor in a worker process and that delivery waits up to
the current adaptive polling interval, up to the one-second
`idle_polling_interval` default. The runtime warns once when it observes this
topology without an adapter. An adapter removes that floor, measured before
adaptive polling at 103.7 ms to 2.9 ms at p50 on PostgreSQL and 103.8 ms to
5.7 ms on Redis, but each stays opt-in for a reason: the PostgreSQL adapter
opens a connection per waiting thread outside the pool and `LISTEN` does not
survive a transaction-pooling proxy such as PgBouncer, and Redis is not a
dependency of this gem.
`WakeUpAdapters.for` selects notifications on PostgreSQL and the in-process
default elsewhere; it never selects Redis. An application that configures
nothing keeps polling, and MySQL applications keep polling unless they
configure Redis explicitly.
Selection is automatic. `config.wake_up_adapter` defaults to `:automatic` and
prefers a configured Redis URL, then PostgreSQL notifications, then polling,
so an application that configures nothing no longer polls on PostgreSQL. An
adapter removes the one-second floor, measured before adaptive polling at
103.7 ms to 2.9 ms at p50 on PostgreSQL and 103.8 ms to 5.7 ms on Redis. Each
carries a cost that selection now states rather than hides: the PostgreSQL
adapter opens a connection per waiting thread outside the pool and adds one
`NOTIFY` per enqueue, and Redis is not a dependency of this gem.
`LISTEN` does not survive a transaction-pooling proxy such as PgBouncer, so
selection listens, sends one `NOTIFY` from a second connection, and waits for
it to arrive. A probe that does not deliver falls back to polling and warns
once, as does a requested adapter the environment cannot provide, so a
downgrade is recorded rather than hidden.
`SolidObjects.wake_up.capability` reports the adapter, whether it crosses
processes, its floor, and why, and the doctor shows the same record.
MySQL still polls. It has no notification channel, and no MySQL notifier has
been measured against polling on the same hardware, so none is shipped.
- Realtime: scalar and dependency-driven keyed ERB component replacement or
morphing, personalized refresh authorization, revision fencing, coalescing,
reconnect convergence, batched refreshes, and personalized state payloads are
Expand Down
36 changes: 33 additions & 3 deletions lib/solid_objects.rb
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
require "solid_objects/actor_channel"
require "solid_objects/action_cable_broadcast_adapter"
require "solid_objects/database_adapter"
require "solid_objects/wake_up_capability"
require "solid_objects/wake_up"
require "solid_objects/wake_up_adapters/postgresql"
require "solid_objects/wake_up_adapters/redis"
Expand Down Expand Up @@ -78,6 +79,8 @@
module SolidObjects
extend Instrumentation

@wake_up_mutex = Thread::Mutex.new

class << self
# @rbs () -> Configuration
def configuration
Expand Down Expand Up @@ -180,11 +183,11 @@ def mutable_copy(value)
# @rbs () -> void
def reset!
ProcessRegistry.reset_polling_warning! if defined?(ProcessRegistry)
reset_wake_up!
@configuration = Configuration.new
@registry = ActorRegistry.new
@client = nil
@database_adapter = nil
@wake_up = nil
@caller_process = nil
@effect_registry = EffectRegistry.new
@commit_action_registry = CommitActionRegistry.new
Expand All @@ -198,9 +201,36 @@ def database_adapter
@database_adapter ||= DatabaseAdapter.for(SolidObjects::Record.connection)
end

# @rbs () -> WakeUp
# @rbs () -> untyped
def wake_up
@wake_up ||= configuration.wake_up_adapter || WakeUp.new
@wake_up || @wake_up_mutex.synchronize { @wake_up ||= resolve_wake_up }
end

# @rbs () -> void
def reset_wake_up!
@wake_up_mutex.synchronize { @wake_up = nil }
WakeUpAdapters.reset_pooled_warning!
end

# @rbs () -> untyped
def resolve_wake_up
WakeUpAdapters.build(configuration.wake_up_adapter)
rescue ArgumentError
raise
rescue => error
unreachable_wake_up(error)
end

# @rbs (Exception) -> untyped
def unreachable_wake_up(error)
adapter = WakeUp.new
adapter.capability = WakeUpCapability.new(
adapter: :in_process,
crosses_processes: false,
measured_floor_ms: nil,
reason: "the database could not be reached to select an adapter: #{error.class}"
)
adapter
end
end
end
23 changes: 22 additions & 1 deletion lib/solid_objects/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ def initialize
@connects_to = nil
@stream_signing_secret = nil
@broadcast_adapter = nil
@wake_up_adapter = nil
@wake_up_adapter = :automatic
@component_path_resolver = nil
@component_authorization_context = ->(controller:) { controller }
@payload_authorization_context = ->(connection:) { connection }
Expand Down Expand Up @@ -237,6 +237,7 @@ def validate!
raise ArgumentError, "actor type cannot be empty" if actor_type.to_s.empty?
raise ArgumentError, "instance retention must be positive" unless retention.positive?
end
validate_wake_up_adapter!
unless component_path_resolver.nil? || component_path_resolver.respond_to?(:call)
raise ArgumentError, "component_path_resolver must respond to call"
end
Expand All @@ -252,6 +253,26 @@ def validate!

private

# @rbs () -> void
def validate_wake_up_adapter!
return if wake_up_adapter.nil?
return validate_wake_up_object! unless wake_up_adapter.is_a?(Symbol)
return if WakeUpAdapters::NAMES.include?(wake_up_adapter)

raise ArgumentError,
"unknown wake_up_adapter #{wake_up_adapter.inspect}, " \
"expected one of #{WakeUpAdapters::NAMES.join(", ")} or an adapter"
end

# @rbs () -> void
def validate_wake_up_object!
%i[signal wait].each do |method_name|
next if wake_up_adapter.respond_to?(method_name)

raise ArgumentError, "wake_up_adapter must respond to #{method_name}"
end
end

# @rbs () -> Hash[Symbol, Numeric]
def positive_values
{
Expand Down
14 changes: 14 additions & 0 deletions lib/solid_objects/doctor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ def call
schema_check,
check_authorization,
check_database_server,
check_wake_up,
schema_check.failed? ? skipped_runtime : check_runtime,
ready_for_round_trip?(configuration_check, schema_check) ?
check_sync_round_trip :
Expand Down Expand Up @@ -208,6 +209,19 @@ def check_database_server
warn_check(:database_server, "#{error.class}: #{error.message}")
end

# @rbs () -> Check
def check_wake_up
capability = SolidObjects.wake_up.capability
floor = capability.measured_floor_ms
summary = "#{capability.adapter}: #{capability.reason}"
summary += ", floor #{floor} ms" if floor
return pass(:wake_up, summary) if capability.crosses_processes

warn_check(:wake_up, "#{summary}; a commit in one process cannot wake another")
rescue => error
warn_check(:wake_up, "#{error.class}: #{error.message}")
end

# @rbs () -> Check
def check_runtime
cutoff = SolidObjects.database_adapter.database_now -
Expand Down
4 changes: 3 additions & 1 deletion lib/solid_objects/process_registry.rb
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@ def deregister(process_record, now: SolidObjects.database_adapter.database_now)

# @rbs () -> void
def warn_if_polling_is_only_cross_process_wake_up
return if SolidObjects.configuration.wake_up_adapter
wake_up = SolidObjects.wake_up
return unless wake_up.respond_to?(:capability)
return if wake_up.capability.crosses_processes

polling_warning_mutex.synchronize do
return if polling_warning_emitted?
Expand Down
12 changes: 12 additions & 0 deletions lib/solid_objects/wake_up.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

module SolidObjects
class WakeUp
include ReportsWakeUpCapability

class Watch
# @rbs @wake_up: WakeUp
# @rbs @generation: Integer
Expand Down Expand Up @@ -29,6 +31,16 @@ def initialize
@generation = 0
end

# @rbs () -> WakeUpCapability
def default_capability
WakeUpCapability.new(
adapter: :in_process,
crosses_processes: false,
measured_floor_ms: nil,
reason: "in-process signalling, which a commit in another process cannot reach"
)
end

# @rbs () -> void
def signal
mutex.synchronize do
Expand Down
Loading
Loading