Skip to content

[kernel][thread] wake the pinned cpu when a sleeping thread wakes up - #547

Merged
zhangxp1998 merged 1 commit into
littlekernel:masterfrom
zhangxp1998:sleep-pinned-wakeup
Oct 29, 2026
Merged

zhangxp1998 merged 1 commit into
littlekernel:masterfrom
zhangxp1998:sleep-pinned-wakeup

Conversation

@zhangxp1998

@zhangxp1998 zhangxp1998 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

A thread can be pinned to another CPU while it sleeps; for a sleeping thread thread_set_pinned_cpu() only records the CPU. Its sleep timer still fires on the CPU it fell asleep on, and thread_sleep_handler() only requeues it there. Unlike thread_unblock() and thread_resume(), it sends no reschedule IPI, so if the new CPU is idle nothing ever runs the thread.

  • When the thread is pinned to another CPU, thread_sleep_handler() now calls wakeup_cpu_for_thread() and returns INT_NO_RESCHEDULE, since a local reschedule can't pick the thread up. A thread pinned to this CPU, or unpinned, is handled as before.
  • test_thread_pin_sleeping starts a thread on another CPU, waits for it to fall asleep, repins it to the test's own CPU and waits in thread_join(), which leaves that CPU idle. Without the fix the join times out; with it the thread runs on its new CPU. If the sleeper wakes before it can be repinned, the test tries again (up to 3 times).
  • docs/threading_and_scheduler.md and the thread_set_pinned_cpu() comment in thread.h now cover the sleeping case.

Tested with scripts/run-qemu-boot-tests.py, and by running ut thread_tests five times per boot on arm64 and x86-64:

  • without the fix, the new test failed every time on both (thread_join() returned ERR_TIMED_OUT)
  • with the fix, x86-64, x86, arm64, arm and arm-m3 pass, and the new test passed 5/5 on arm64 and x86-64

Copilot AI balanced review requested due to automatic review settings October 27, 2026 07:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused fix follows existing scheduler wakeup behavior, includes regression coverage, and has no identified blocking issues.

No open issues 🎉

Review effort: Balanced

@travisg travisg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks right to me. On #544 I'd wanted re-pin-while-sleeping left for the per-cpu scheduler rework, but this is reachable under the contract that already landed, not only that deferred case: a thread running on A gets pinned to B by someone else (which only records the pin), then calls thread_sleep() before A reschedules. The timer is armed on A, the handler requeues it there, A can't run it and nothing pokes B. So it's a real hang today.

It also isn't the eager-wakeup pattern I pushed back on: mp_reschedule() drops the local cpu, so a thread pinned where it slept costs a call and no IPI, and unpinned sleepers are untouched. It brings the sleep path in line with resume/unblock/wait-queue wakes, and the scheduler rework on the preempt branch does the same thing in its sleep handler, so this converges with where that's going. When that branch rebases this hunk will fold into its version, but the test should come along, like the #544 one.

A few small things:

  • docs/threading_and_scheduler.md, CPU Pinning: "kernel/test/thread_tests.c covers the self-migration and repinning a queued thread" should mention the sleeping case now.
  • kernel/include/kernel/thread.h, the comment above thread_set_pinned_cpu(): "Blocked or suspended threads only record the new affinity" leaves out sleeping, which the docs bullet includes.

The test is nicely built: without the fix it fails via the join timeout rather than hanging, and the retry covers the sleeper waking before it can be repinned. On a platform with a periodic tick it would pass either way, which is fine.

Comment thread kernel/thread.c Resolved

@travisg travisg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Ran the boot tests locally (gcc): arm64, x86-64 and riscv64 pass with the change. With just the thread.c hunk reverted, arm64 and x86-64 each fail only test_thread_pin_sleeping (join returns ERR_TIMED_OUT) and the run still completes, so the test does catch it. The nits from the earlier review are fine as a follow-up or folded in here, your call.

A thread can be pinned to another cpu while it sleeps. Its sleep timer
still fires on the cpu it fell asleep on, and thread_sleep_handler() only
requeued it there, so if the new cpu was idle nothing ever ran the thread.
When the thread is pinned to another cpu, send that cpu the reschedule IPI
that thread_unblock() and thread_resume() send, and skip the local
reschedule, which cannot pick the thread up. Add a test that repins a
sleeping thread to an idle cpu, and mention the sleeping case in the
scheduler docs and the thread_set_pinned_cpu() comment.
@zhangxp1998

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Updated:

  • thread_sleep_handler() returns INT_NO_RESCHEDULE when the thread is pinned to another CPU.
  • CPU Pinning docs: the test sentence now mentions repinning a sleeping thread.
  • thread_set_pinned_cpu() comment: "Blocked, sleeping or suspended threads only record the new affinity."

Re-ran ut all on x86-64, x86, arm64, arm and arm-m3, and ut thread_tests five times per boot on arm64 and x86-64.

@zhangxp1998
zhangxp1998 merged commit d3d8ddf into littlekernel:master Oct 29, 2026
118 checks passed
@zhangxp1998
zhangxp1998 deleted the sleep-pinned-wakeup branch October 29, 2026 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants