slurm: job-scoped epilog cleanup, default on, with a user escape hatch (#1407)

docs/slurm-cluster/README.md

@@ -195,6 +195,8 @@ Read through the [slurm usage guide](slurm-usage.md) for more information.
 The default Slurm deployment includes a collection of prolog and epilog scripts that should be modified to suit a particular system.
 For more information, see the [prolog/epilog documentation](slurm-prolog-epilog/README.md).
 
+From release 26.11 the cleanup that runs when a user's job ends is job-scoped by default and no longer touches the user's other running jobs on the node; `slurm_epilog_cleanup_scope: user` restores the previous behaviour for one release. See [Job-scoped epilog cleanup](epilog-job-scoped-cleanup.md).
+
 ### Lua plugins
 
 Slurm's Lua plugins (for example `JobSubmitPlugins=lua` or `CliFilterPlugins=lua`) only load when Slurm was built with Lua support. DeepOps does not request Lua support or install its development package by default; Slurm may still detect a preinstalled Lua library. To enable it, set the following in `config/group_vars/slurm-cluster.yml` and run the Slurm playbook:

docs/slurm-cluster/epilog-job-scoped-cleanup.md

@@ -0,0 +1,132 @@
+# Job-scoped epilog cleanup
+
+Starting with release 26.11, the Slurm epilog cleans up after a finished job
+without touching other jobs of the same user that are running on the same node.
+This page explains what changed, why, and how to go back to the old behaviour
+for one release.
+
+## What changed
+
+When a job ends, the epilog (`/etc/slurm/epilog.d/*-lastuserjob-*`) removes what
+the user left behind on the node: stray processes, files in `/tmp` and
+`/dev/shm`, enroot containers, and the user's SSH access entry.
+
+Before 26.11 this was decided from the controller's view alone: the epilog asked
+`squeue` whether the user had another job on the node, and if not it killed
+every process the user owned (`killall -9 -u`) and deleted everything they owned
+in `/tmp` and `/dev/shm`. Two things could go wrong:
+
+- A new job of the same user could be placed on the node just after `squeue`
+  answered, and have its processes killed a moment later.
+- The controller can lag behind the node, and it cannot see processes that left
+  the job's cgroup.
+
+With `slurm_epilog_cleanup_scope: job` (the new default) the epilog also looks
+at the node itself. Every process of a Slurm job lives in a `job_<id>` cgroup, so
+the epilog reads `/proc/<pid>/cgroup` (cgroup v1 and v2 are both understood) and
+decides per process:
+
+| Process is in… | Epilog does |
+|---|---|
+| the ending job's cgroup | kills it (Slurm does too; this is a backstop) |
+| another job's cgroup | never touches it, whatever `squeue` said |
+| no job cgroup (an orphan, for example a leftover daemon) | kills it only when the user has no other job on the node |
+| a cgroup entry it cannot read or recognise | leaves it alone and logs it |
+
+The file cleanup (`/tmp`, `/dev/shm`), enroot directory removal, and SSH access
+removal still happen per user, because files carry a user id and not a job id.
+They now run only when neither `squeue` nor the node's cgroups show another job
+of that user. If that check cannot be completed, the cleanup is skipped and
+logged, then happens when the user's next job ends. A skipped cleanup is a delay,
+not a loss.
+
+In short: **less is deleted, never more.** Nothing that was spared before is
+removed now.
+
+### What you may notice
+
+- If a user's job is still running when another of their jobs ends, that job's
+  processes, `/tmp` files and enroot containers now survive. Previously they
+  could be removed.
+- After the user's last job ends, leftovers are still removed as before.
+- `/tmp` cleanup can be delayed on a node where the cgroup data cannot be read.
+  The epilog logs a `[WARN]` line (`slurm` tag in the journal, also in
+  `/var/log/slurm/prolog-epilog`) when it defers.
+- Any extra `*-lastuserjob-*` script you added under `/etc/slurm/epilog.d`
+  follows the same, stricter guard and therefore runs less often.
+- Accounts listed in `/etc/slurm/localusers.backup`, and root, are exempt as
+  before.
+- Requires cgroup-based process tracking (`ProctrackType=proctrack/cgroup`, the
+  DeepOps default). With a different process tracker the epilog cannot tell
+  which job a process belongs to, so it treats every process as unassigned and
+  falls back to asking `squeue`: if the user has another job on the node nothing
+  is cleaned up, otherwise all of the user's processes are removed. It does not
+  defer. Use `user` (below) or fix the tracker if that is not what you want.
+- Processes started outside Slurm while a job is finishing, for example over
+  SSH, are not protected: they are not part of any job, so they count as
+  unassigned and can be removed when the user's last job ends.
+
+### Related fix: SSH access entry
+
+The script that removes a user's SSH access entry (`41-lastuserjob-ssh`) matched
+names as patterns rather than whole lines: ending a job for user `al` could also
+remove the entries for `alice` or `halley`, and a dot in a name matched any
+character. It now matches the exact line. This applies in both scopes.
+
+## Going back to the 26.09 behaviour
+
+If you need the old behaviour, set this in `config/group_vars/slurm-cluster.yml`
+and re-run the Slurm playbook (`ansible-playbook playbooks/slurm-cluster.yml`) so the epilog scripts are re-rendered on the compute nodes:
+
+```yaml
+slurm_epilog_cleanup_scope: user
+```
+
+The scripts for `user` are the 26.09 scripts byte-for-byte (apart from the SSH
+match fix above). The playbook prints a deprecation notice when `user` is set.
+
+**`user` is deprecated and will be removed in release 27.01.** It exists for one
+release so a site with an unusual setup, for example a custom epilog, a process
+tracker other than cgroup, or user daemons that live outside any job cgroup, can
+keep the old behaviour while it reports the gap. If you need `user`, please open
+an issue at https://github.com/NVIDIA/deepops/issues saying why.
+
+## Per-job `/tmp` and `/dev/shm`: `job_container/tmpfs`
+
+The complete answer for per-job temporary files is Slurm's own
+`job_container/tmpfs` plugin, which gives each job a private `/tmp` and
+`/dev/shm` that disappear when the job ends. DeepOps does not configure it for
+you, but if you enable it in `slurm.conf` the epilog notices
+(`JobContainerType=job_container/tmpfs`) and skips its own `/tmp` and `/dev/shm`
+file cleanup, since the plugin already handles it:
+
+```
+JobContainerType=job_container/tmpfs
+```
+
+and add a `job_container.conf` next to `slurm.conf`. See the Slurm documentation
+for `job_container.conf` (the `BasePath` must exist and be writable by slurmd on
+each compute node). Wiring this into the role is tracked separately.
+
+## Enroot and Pyxis
+
+The enroot cleanup removes the user's runtime and data directories, which are
+shared by all of that user's jobs on the node. Per-job enroot directories were
+considered and not adopted, because people rely on containers persisting between
+jobs. If you want containers that disappear with the job, Pyxis has a
+container scope setting for job-lifetime containers; see the Pyxis documentation
+for your version.
+
+## Checking what the epilog did
+
+Each epilog logs through `logger -t slurm-epilog` and `-t slurm`. Typical lines:
+
+```
+slurm-epilog: Job-scoped process cleanup for alice job 4711: killed 3, spared 12, unclassified 0
+slurm: Another job of alice has processes on this node (found by cgroup); not running the last-user-job scripts.
+```
+
+## Testing
+
+The unprivileged unit tests are in `tests/slurm-epilog/run-tests.sh` and run in CI.
+They use a fake `/proc` and cover both cgroup layouts.

roles/slurm/defaults/main.yml

@@ -91,6 +91,19 @@ slurm_enable_prolog_epilog: true
 # Useful to delete old files (e.g. from previous versions of deepops)
 slurm_clear_old_prolog_epilog: false
 
+# How the epilog's "last job of the user" cleanup (scripts 40-lastuserjob-*
+# through 50-lastuserjob-*) decides what to remove when a job ends.
+#   job:  (default) job-scoped. The cleanup also checks the node's own cgroups
+#         and never touches processes of another job of the same user; files and
+#         enroot directories are removed only when the user has no other job on
+#         the node. Requires cgroup process tracking (ProctrackType=proctrack/cgroup,
+#         the default in this role); cgroup v1 and v2 are both understood.
+#   user: DEPRECATED. The 26.09 behaviour, rendered byte-for-byte: the cleanup
+#         is decided by squeue alone and kills every process the user owns.
+#         Kept for one release train as an escape hatch; removed in 27.01.
+# See docs/slurm-cluster/epilog-job-scoped-cleanup.md
+slurm_epilog_cleanup_scope: job
+
 # Fileglob patterns used to copy prolog and epilog files
 prolog_fileglob:
   - "templates/etc/slurm/prolog.d/*"

roles/slurm/tasks/compute.yml

@@ -82,6 +82,17 @@
   notify: restart slurmd
   tags: config
 
+- name: validate slurm_epilog_cleanup_scope
+  assert:
+    that:
+      - slurm_epilog_cleanup_scope in ['job', 'user']
+    fail_msg: >-
+      slurm_epilog_cleanup_scope must be 'job' or 'user'
+      (got '{{ slurm_epilog_cleanup_scope }}').
+  tags:
+    - prolog
+    - epilog
+
 - name: copy shared scripts
   template:
     src: "{{ item }}"
@@ -93,6 +104,47 @@
     - prolog
     - epilog
 
+- name: warn that slurm_epilog_cleanup_scope user is deprecated
+  debug:
+    msg: >-
+      slurm_epilog_cleanup_scope: user restores the 26.09 epilog cleanup and is
+      deprecated. It will be removed in 27.01; see
+      docs/slurm-cluster/epilog-job-scoped-cleanup.md.
+  when: slurm_epilog_cleanup_scope == 'user'
+  tags:
+    - prolog
+    - epilog
+
+- name: create shared library directory for job-scoped epilog cleanup
+  file:
+    path: "{{ slurm_config_dir }}/shared/lib"
+    state: directory
+    owner: slurm
+    mode: 0755
+  when: slurm_epilog_cleanup_scope == 'job'
+  tags:
+    - prolog
+    - epilog
+
+- name: copy job-scoped epilog helpers
+  template:
+    src: etc/slurm/shared/lib/epilog-job-scope.sh
+    dest: "{{ slurm_config_dir }}/shared/lib/epilog-job-scope.sh"
+    mode: 0644
+  when: slurm_epilog_cleanup_scope == 'job'
+  tags:
+    - prolog
+    - epilog
+
+- name: remove job-scoped epilog helpers when the scope is user
+  file:
+    path: "{{ slurm_config_dir }}/shared/lib/epilog-job-scope.sh"
+    state: absent
+  when: slurm_epilog_cleanup_scope == 'user'
+  tags:
+    - prolog
+    - epilog
+
 - name: include prolog/epilog
   include_tasks: prolog_epilog.yml
   when: slurm_enable_prolog_epilog

roles/slurm/templates/etc/slurm/epilog.d/40-lastuserjob-processes

@@ -36,8 +36,104 @@ case "$lookup_rc" in
         ;;
 esac
 
+{% if slurm_epilog_cleanup_scope == 'job' %}
+# Job-scoped cleanup (slurm_epilog_cleanup_scope: job).
+#
+# `killall -9 -u` reaps every process the user owns on the node, whichever job
+# it belongs to. Here each process is classified by the job cgroup it sits in
+# (/proc/<pid>/cgroup, see shared/lib/epilog-job-scope.sh) and only the ones
+# that are this job's, or that belong to no job and the user has no job left,
+# are killed:
+#
+#   in the ending job's cgroup   kill (Slurm does this too; belt and braces)
+#   in any other job's cgroup    never kill, whatever squeue said
+#   in no job cgroup (orphan)    kill only when the dispatcher found no other
+#                                job of this user, by squeue and by cgroup
+#   cgroup line not understood   skip and log
+#
+# Anything that cannot be established is a skip, never a kill, and the script
+# never fails the epilog: a failing epilog drains the node.
+kill_bin=kill
+epilog_proc_root=/proc
+job_scope_lib='{{ slurm_config_dir }}/shared/lib/epilog-job-scope.sh'
+
+if [ "$SLURM_JOB_USER" = root ]; then
+    exit 0
+fi
+
+if [ ! -r "$job_scope_lib" ] || ! . "$job_scope_lib"; then
+    logger -s -t slurm-epilog \
+        "cannot load ${job_scope_lib}; skipping process cleanup"
+    exit 0
+fi
+
+if ! epilog_resolve_uid || [ "$epilog_uid" = 0 ]; then
+    logger -s -t slurm-epilog \
+        "no usable uid for ${SLURM_JOB_USER}; skipping process cleanup"
+    exit 0
+fi
+
+if ! pids=$(epilog_user_pids "$epilog_uid"); then
+    logger -s -t slurm-epilog \
+        "cannot list processes under ${epilog_proc_root}; skipping process cleanup"
+    exit 0
+fi
+
+# The dispatcher exports 1 only when neither squeue nor the cgroup scan found
+# another job of this user. Run on its own the script treats orphans as live.
+last_user_job="${EPILOG_LAST_USER_JOB:-0}"
+
+killed=0
+spared=0
+unsure=0
+for pid in $pids; do
+    [ "$pid" != "$$" ] || continue
+
+    rc=0
+    jobs=$(epilog_pid_jobs "$pid") || rc=$?
+    case "$rc" in
+        0) ;;
+        1) continue ;;  # exited since the listing
+        *)
+            logger -s -t slurm-epilog \
+                "cannot classify pid ${pid} (unreadable or unrecognised cgroup data); leaving it alone"
+            unsure=$((unsure + 1))
+            continue
+            ;;
+    esac
+
+    in_ending=0
+    in_other=0
+    for job in $jobs; do
+        if [ "$job" = "$SLURM_JOB_ID" ]; then
+            in_ending=1
+        else
+            in_other=1
+        fi
+    done
+
+    if [ "$in_other" = 1 ]; then
+        spared=$((spared + 1))
+        continue
+    fi
+
+    if [ "$in_ending" = 0 ] && [ "$last_user_job" != 1 ]; then
+        # An orphan, but the user may still be running here.
+        spared=$((spared + 1))
+        continue
+    fi
+
+    if "$kill_bin" -9 -- "$pid" 2>/dev/null; then
+        killed=$((killed + 1))
+    fi
+done
+
+logger -s -t slurm-epilog \
+    "Job-scoped process cleanup for ${SLURM_JOB_USER} job ${SLURM_JOB_ID}: killed ${killed}, spared ${spared}, unclassified ${unsure}"
+{% else %}
 if [ "$SLURM_JOB_USER" != root ]; then
     if killall -9 -u "$SLURM_JOB_USER" ; then
         logger -s -t slurm-epilog 'Killed residual user processes'
     fi
 fi
+{% endif %}

roles/slurm/templates/etc/slurm/epilog.d/41-lastuserjob-ssh

@@ -1,8 +1,19 @@
 #!/usr/bin/env bash
 set -ex
 
-if grep -q -w "$SLURM_JOB_USER" /etc/slurm/localusers.backup ; then
+# Both lookups below act on whole lines. `grep -w` and an unanchored
+# `sed "/${user}/d"` match substrings and regular expressions instead: with a
+# job user `al`, the sed deleted the SSH allow entry of `alice`, `halley` and
+# every other account containing "al", and a dot in a name (`je.kim`) matched
+# any character. Match the exact line, literally, in both places.
+localusers_backup=/etc/slurm/localusers.backup
+localusers_live=/etc/localusers
+
+if grep -qxF -- "$SLURM_JOB_USER" "$localusers_backup" ; then
     exit 0  # don't revoke access for these users
 fi
 
-sed -i "/${SLURM_JOB_USER}/d" /etc/localusers
+# Escape the characters that are special in a sed regular expression (and the
+# `/` delimiter) so the account name is matched literally.
+user_pattern=$(printf '%s' "$SLURM_JOB_USER" | sed -e 's/[][\.*^$/]/\\&/g')
+sed -i "/^${user_pattern}\$/d" "$localusers_live"

roles/slurm/templates/etc/slurm/epilog.d/42-lastuserjob-cleanup

@@ -33,6 +33,23 @@ case "$lookup_rc" in
         ;;
 esac
 
+{% if slurm_epilog_cleanup_scope == 'job' %}
+# Job-scoped cleanup (slurm_epilog_cleanup_scope: job).
+#
+# Files carry a uid, never a job id, so deleting "this job's files" from
+# ownership alone is not possible: this script stays per user, and the
+# dispatcher only runs it when neither squeue nor the node's cgroups show
+# another job of the user. When Slurm's job_container/tmpfs plugin is active,
+# /tmp and /dev/shm are private to each job and vanish with it, so this
+# cleanup has nothing to add and would only reach into the host's copies.
+slurm_conf='{{ slurm_config_dir }}/slurm.conf'
+if grep -qE '^[[:space:]]*JobContainerType[[:space:]]*=[[:space:]]*job_container/tmpfs' "$slurm_conf" 2>/dev/null; then
+    logger -s -t slurm-epilog \
+        "job_container/tmpfs is configured; per-job /tmp and /dev/shm are removed with the job, skipping file cleanup"
+    exit 0
+fi
+
+{% endif %}
 if [[ -n "$SLURM_JOB_USER" && "$SLURM_JOB_USER" != "root" ]]; then
     # `find -user` needs the account to resolve. When the directory service is
     # unavailable find exits non-zero without printing anything, and the old