Skip to content

scheduler: complete queued work when executor rejects submission #1045

Description

@dkropachev

Problem

After #392, normal explicit and interpreter shutdown stop the scheduler before executor teardown. An externally shut-down or broken executor can still reject _Scheduler.run() after the task has been removed from the queue.

def run(self):
while True:
if self.is_shutdown:
return
try:
while True:
run_at, i, task = self._queue.get(block=True, timeout=None)
if self.is_shutdown:
if task:
log.debug("Not executing scheduled task due to Scheduler shutdown")
return
if run_at <= time.time():
self._scheduled_tasks.discard(task)
fn, args, kwargs = task
kwargs = dict(kwargs)
future = self._executor.submit(fn, *args, **kwargs)
future.add_done_callback(self._log_if_failed)

The scheduler thread then exits without completing the claimed task or draining shutdown callbacks, which can leave a waiting future pending. Runtimes without the early threading hook have the same fallback gap.

Expected behavior

Executor rejection makes the scheduler terminal and completes callbacks for the claimed and queued tasks exactly once.

Acceptance criteria

  • The claimed task is not lost when submit() rejects it.
  • Queued shutdown callbacks are drained once.
  • Concurrent and reentrant shutdown remain deadlock-free.
  • Unexpected executor failure is logged.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions