Skip to content

Shut down vm on unrecoverable errors. - #2755

Open
jounathaen wants to merge 2 commits into
mainfrom
sched_abort
Open

jounathaen wants to merge 2 commits into
mainfrom
sched_abort

Conversation

@jounathaen

@jounathaen jounathaen commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

I encountered the hypervisor indefinitely hanging on unrecoverable errors. This PR shuts down the VM in these cases.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark Results

Details
Benchmark Current: bd82420 Previous: 2e23902 Performance Ratio
startup_benchmark Build Time 84.02 s 80.34 s 1.05 ❗
startup_benchmark File Size 0.85 MB 0.80 MB 1.07 ❗
Startup Time - 1 core 0.73 s (±0.03 s) 0.75 s (±0.02 s) 0.98
Startup Time - 2 cores 0.72 s (±0.02 s) 0.74 s (±0.02 s) 0.98
Startup Time - 4 cores 0.73 s (±0.02 s) 0.74 s (±0.02 s) 0.98
multithreaded_benchmark Build Time 84.39 s 82.11 s 1.03 ❗
multithreaded_benchmark File Size 0.95 MB 0.86 MB 1.10 ❗
Multithreaded Pi Efficiency - 2 Threads 90.27 % (±9.01 %) 85.89 % (±6.61 %) 1.05
Multithreaded Pi Efficiency - 4 Threads 44.66 % (±3.06 %) 43.43 % (±2.56 %) 1.03
Multithreaded Pi Efficiency - 8 Threads 26.36 % (±1.51 %) 25.76 % (±1.53 %) 1.02
micro_benchmarks Build Time 82.01 s 80.40 s 1.02 ❗
micro_benchmarks File Size 0.95 MB 0.86 MB 1.10 ❗
Scheduling time - 1 thread 65.31 ticks (±2.62 ticks) 62.65 ticks (±4.06 ticks) 1.04
Scheduling time - 2 threads 35.04 ticks (±4.04 ticks) 34.08 ticks (±4.10 ticks) 1.03
Micro - Time for syscall (getpid) 3.78 ticks (±0.61 ticks) 3.45 ticks (±0.58 ticks) 1.10
Memcpy speed - (built_in) block size 4096 80434.96 MByte/s (±56046.84 MByte/s) 82448.38 MByte/s (±56997.13 MByte/s) 0.98
Memcpy speed - (built_in) block size 1048576 30515.62 MByte/s (±24501.74 MByte/s) 30585.98 MByte/s (±24707.84 MByte/s) 1.00
Memcpy speed - (built_in) block size 16777216 29030.03 MByte/s (±23898.40 MByte/s) 26340.06 MByte/s (±21720.96 MByte/s) 1.10
Memset speed - (built_in) block size 4096 79945.25 MByte/s (±55700.64 MByte/s) 82292.76 MByte/s (±56891.50 MByte/s) 0.97
Memset speed - (built_in) block size 1048576 31245.65 MByte/s (±24923.30 MByte/s) 31323.85 MByte/s (±25145.86 MByte/s) 1.00
Memset speed - (built_in) block size 16777216 29803.51 MByte/s (±24350.99 MByte/s) 27104.68 MByte/s (±22209.94 MByte/s) 1.10
Memcpy speed - (rust) block size 4096 74844.25 MByte/s (±52240.71 MByte/s) 74097.96 MByte/s (±51811.44 MByte/s) 1.01
Memcpy speed - (rust) block size 1048576 30178.07 MByte/s (±24413.29 MByte/s) 30361.60 MByte/s (±24602.37 MByte/s) 0.99
Memcpy speed - (rust) block size 16777216 29000.67 MByte/s (±23864.30 MByte/s) 27625.34 MByte/s (±22806.88 MByte/s) 1.05
Memset speed - (rust) block size 4096 74998.46 MByte/s (±52344.15 MByte/s) 74373.47 MByte/s (±51976.48 MByte/s) 1.01
Memset speed - (rust) block size 1048576 30900.70 MByte/s (±24830.46 MByte/s) 31110.89 MByte/s (±25033.24 MByte/s) 0.99
Memset speed - (rust) block size 16777216 29770.22 MByte/s (±24313.61 MByte/s) 28386.93 MByte/s (±23265.03 MByte/s) 1.05
alloc_benchmarks Build Time 82.34 s 74.76 s 1.10 ❗
alloc_benchmarks File Size 0.93 MB 0.87 MB 1.07 ❗
Allocations - Allocation success 91.37 % 91.31 % 1.00 ❗
Allocations - Deallocation success 100.00 % 100.00 % 1
Allocations - Pre-fail Allocations 61.61 % 61.44 % 1.00 ❗
Allocations - Average Allocation time 6316.00 Ticks (±98.57 Ticks) 5860.58 Ticks (±98.43 Ticks) 1.08 ❗
Allocations - Average Allocation time (no fail) 6886.21 Ticks (±112.63 Ticks) 6554.81 Ticks (±92.86 Ticks) 1.05 ❗
Allocations - Average Deallocation time 1273.03 Ticks (±286.43 Ticks) 1805.01 Ticks (±250.35 Ticks) 0.71 ❗
mutex_benchmark Build Time 82.97 s 79.82 s 1.04 ❗
mutex_benchmark File Size 0.95 MB 0.86 MB 1.10 ❗
Mutex Stress Test Average Time per Iteration - 1 Threads 12.26 ns (±0.52 ns) 12.10 ns (±0.41 ns) 1.01
Mutex Stress Test Average Time per Iteration - 2 Threads 68.16 ns (±3.66 ns) 40.26 ns (±1.68 ns) 1.69 ❗

This comment was automatically generated by workflow using github-action-benchmark.

Comment thread src/scheduler/mod.rs
Comment on lines +214 to +217
if remaining_tasks == 0 {
warn!("Last task exited with exit code {exit_code}, shutting down");
shutdown(-1);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could even do a panic here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that would be another option. I don't have a direct preference.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The edge case here would be the last task exiting with sys_exit. That is probably not supposed to happen, but it's also not a panic, is it?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The all tasks exit with sys_exit: https://github.com/rust-lang/rust/blob/ce37c4ff837c5c5dc45cddf9e1e3704e42ca67ea/library/std/src/sys/pal/hermit/mod.rs#L72-L74

But sys_exit shuts down the system, not only the task.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, I meant sys_thread_exit.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rust-world does not expose sys_thread_exit at all. They make use of the fact that the thread function is allowed to return. The kernel calls scheduler exit internally, then. sys_thread_exit is used in pthread-embedded, even though it does not have to be.

I don't know how such a situation is possible. The main thread shuts down the whole process and kills all threads, so this situation cannot be caused by an application. It might be a kernel-internal situation, which might warrant panicking, I believe.

Comment thread src/scheduler/mod.rs
Comment on lines +892 to +893
error!("Aborting kernel execution");
shutdown(-1);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed when we already have the previous commit?

Note that sys_abort currently shuts down Hermit, but scheduler::abort only kills the affected task deliberately, so other tasks are not affected.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the motivation for this behaviour on "abort"?
For example, if we have two tasks running, and the other task is e.g., blocking, we'd end up in a frozen state.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's how things always work, no? It's the application's responsibility to shut down the system when a thread or process aborting is an unrecoverable failure for the application.

When a second process segfaults, the first process should not be affected. For threads, we might want to kill the whole process since we don't have signals, but other recovery mechanisms would also work.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, but we are not talking about processes here, but only threads, right? So this change implements the "kill the whole process since we don't have signals" aspect.

Or is your point about the masos case?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants