Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The new fromlist probe can invoke module-level __getattr__ twice and alter import behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Optimizes sandbox imports to bypass module locks for fully loaded modules, addressing intermittent Python 3.10 workflow validation failures.
Changes:
- Adds a fast path for loaded modules.
- Adds regression coverage and a changelog entry.
File summaries
| File | Description |
|---|---|
temporalio/worker/workflow_sandbox/_importer.py |
Implements the loaded-module fast path. |
tests/worker/workflow_sandbox/test_importer.py |
Tests repeated imports bypassing importlib. |
CHANGELOG.md |
Documents the fix. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
DABH
marked this pull request as ready for review
September 16, 2026 06:17
The sandbox importer routed every import, including re-imports of modules already present in the sandbox's sys.modules, through the pure-Python importlib.__import__. On Python 3.10 that path always acquires the per-module lock in importlib._bootstrap._find_and_load, and _ModuleLock.acquire is not re-entrant there: it stores the current thread in the single-slot _blocking_on dict and deletes it in a finally block (fixed in 3.12 by python/cpython#91351). A cyclic GC pass can run inside that window while a workflow module is being loaded. Finalizing a never-awaited coroutine (or showing a warning with a source object) makes the C runtime call PyImport_Import("warnings"), which goes through the sandbox's builtins.__import__ and so re-entered _ModuleLock.acquire on the same thread. The nested call removed the _blocking_on entry and the outer acquire failed with KeyError(<thread id>), surfacing as "RuntimeError: Failed validating workflow" when a worker started. CPython's C import never takes the lock for an initialized module, so plain Python does not hit this for already-imported modules. Mirror that: when the target module (and, for from-imports of a package, every requested attribute) is already fully imported, return it directly and only fall back to importlib.__import__ for real loads. Module identity, passthrough handling and restriction wrapping are unchanged, and imports executed inside workflow code get cheaper on every Python version.
The fast path used hasattr for fromlist names, which runs a package's module-level __getattr__ before importlib runs it again on the fallback, so a missing name was probed twice. Look only at the module dict; dynamic attributes keep going through importlib exactly as before.
DABH
force-pushed
the
flake/sandbox-importer-race
branch
from
September 16, 2026 06:17
5e39e12 to
d20dd2c
Compare
Rebuilds importlib._bootstrap._ModuleLock.acquire from the interpreter's
own _bootstrap.py with a nested builtins.__import__("warnings") placed
inside the _blocking_on window while the sandbox loads a non-passthrough
module: the exact re-entry a GC finalizer causes on Python 3.10/3.11.
Fails on main with KeyError(<thread id>); passes with the loaded-module
fast path; skipped from 3.12, where the lock is re-entrant.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Python 3.10/3.11, loading a workflow module in the sandbox could intermittently fail with
RuntimeError: Failed validating workflow(root cause:KeyError: <thread id>inimportlib._bootstrap._ModuleLock.acquire). h/t #585 for original report.Seems the issue is that the sandbox importer routed every import, including re-imports of already-loaded modules, through pure-Python
importlib.__import__, which takes the non-re-entrant module lock. A GC finalizer importingwarningsinside that lock re-entered it.Already-loaded modules are now served straight from the sandbox's
sys.modules, as CPython's C import path does.Background
CPython bug 91351: before 3.12 the per-module import lock is not re-entrant, and on 3.10/3.11 the pure-Python import path acquires it even for loaded modules. Plain Python avoids this for loaded modules because the C import path skips the lock; the sandbox did not, so a never-awaited-coroutine finalizer (which imports
warnings) inside a workflow-module load re-entered the lock on the same thread.Residual and inherent to CPython: a nested import that must load a module for the first time in the process (e.g.
tracemallocon the first displayed coroutine warning) still takes the lock and can still trip 91351 on 3.10/3.11. No sandbox-layer change can avoid a first-time load's lock; pre-importing such modules at worker start would be the mitigation if it ever surfaces.Testing
mainfails 3/3 withKeyError(<thread id>); this branch passes 3/3.mainunder 3.10, passes here, skipped on 3.12+.poe lint;poe test -s tests/worker/workflow_sandbox(3.14);pytest tests/worker/workflow_sandbox/test_importer.py(3.10).