Define control-connection recovery after finite reconnection exhaustion

[dkropachev]

Problem

PR #1024 makes heartbeat returns stop re-arming a finite control-connection reconnection schedule once that schedule is exhausted for the same defunct connection.

python-driver/cassandra/cluster.py

Lines 4119 to 4124 in 72db15c

	if (exhausted and not reconnect_requested and
	self._failed_connection is not None and
	control_connection._connection is
	self._failed_connection):
	control_connection._reconnection_exhausted_connection = \
	self._failed_connection

The guard remains until a successful connection is installed. This is consistent with the public ReconnectionPolicy contract: a finite iterable means reconnection attempts cease after exhaustion.

python-driver/cassandra/policies.py

Lines 873 to 878 in 72db15c

	def new_schedule(self):
	"""
	This should return a finite or infinite iterable of delays (each as a
	floating point number of seconds) in-between each failed reconnection
	attempt. Note that if the iterable is finite, reconnection attempts
	will cease once the iterable is exhausted.

The operational result is terminal automatic recovery for that failed control connection after finite exhaustion. Later heartbeat returns cannot start another recovery attempt.

python-driver/cassandra/cluster.py

Lines 5358 to 5365 in 72db15c

	def return_connection(self, connection):
	if connection is self._connection and (connection.is_defunct or connection.is_closed):
	with self._reconnection_lock:
	if connection is self._reconnection_exhausted_connection:
	log.debug("[control connection] Reconnection schedule is "
	"exhausted for the defunct connection")
	return
	self.reconnect()

Scope

This differs from #1029. That issue covers recovery latency while an infinite or still-active handler is backing off. This issue covers the state after a finite handler has stopped.

Design constraints

• Silently restarting after a cooldown turns max_attempts=N into unlimited batches of N attempts.
• A custom policy exposes an arbitrary iterable, so the driver cannot derive a universal maximum policy delay.
• Re-enabling every idle heartbeat risks restoring the reconnect storm tracked by After a node restarts, driver reconnects multiple times, causing queries to fail #295.
• The default infinite reconnection policy is unaffected by finite exhaustion.

Expected outcome

Define the control-connection recovery contract after finite schedule exhaustion. If automatic recovery should resume, expose that behavior explicitly through policy or configuration semantics rather than implicitly redefining every finite schedule.

Acceptance criteria

• Terminal versus periodic recovery is chosen and documented.
• Built-in and custom finite policies have deterministic behavior.
• Periodic recovery, if supported, does not reset active backoff or create one attempt per heartbeat.
• Unit tests cover exhaustion, repeated heartbeat returns, successful recovery, and shutdown.
• Operator-facing documentation explains how to select terminal or continuing recovery.

Related

• control-connection: reconnect when down handling is skipped #1024 introduced and documents the current guard.
• Nothing re-arms a backing-off control connection: recovery latency grows from idle_heartbeat_interval to max_reconnection_delay #1029 tracks active-handler recovery latency.
• After a node restarts, driver reconnects multiple times, causing queries to fail #295 tracks reconnect storms caused by repeated schedule resets.