Module install hangs on Debian 13 after a module removal

[stephdl]

A new module can hang forever on Debian 13. It happens when the module is added right after another module was removed from the same node.

The removed module frees its Linux user ID (UID). The new module gets the same UID. The node agent then enables "linger" for the new user, but systemd never starts the user manager (user@<uid>.service). The module agent never starts, so the add-module task never completes.

Steps to reproduce

• Take a node running Debian 13 (systemd 257).
• Add a rootless module, for example api-cli run add-module --data '{"image":"ghcr.io/nethserver/imapsync:1.3.3","node":3}'. It gets imapsync2 with UID 1004.
• Remove it: api-cli run remove-module --data '{"module_id":"imapsync2","preserve_data":false}'.
• Add it again at once. The image is already in cache, so this is fast.

Expected behavior

The second add-module completes. imapsync3 runs.

Actual behavior

imapsync3 gets UID 1004 again. add-module never completes. user@1004.service stays inactive.

12:04:19 userdel: delete user 'imapsync2'
12:04:24 useradd: new user: name=imapsync3, UID=1004
12:04:26 agent@node: loginctl enable-linger imapsync3
12:04:26 systemd: Finished user-runtime-dir@1004.service
12:04:29 systemd: Stopped user-runtime-dir@1004.service

Starting the user manager by hand unblocks it: systemctl start user@1004.service. The agent starts and create-module completes.

It failed on the first try, on a real node. The same thing happens in the core test suite on Debian 13: the openldap suite removes openldap1, then samba1 gets its UID and hangs.

Rocky Linux 9 (systemd 252) does not show it in the same test suite.

Components

• NethServer core 3.22.0
• Debian GNU/Linux 13 (trixie), systemd 257.13-1~deb13u1

Possible fix

In core/imageroot/var/lib/nethserver/node/actions/add-module/50update, start user@<uid>.service explicitly after loginctl enable-linger. It does nothing when the service already runs.

[stephdl]

This is the script I used to reproduce it. It runs from a workstation with SSH access to the leader node and to the Debian 13 node. Set LEADER, NODE and NODE_ID for your cluster.

It adds a module once so the image is in cache. Then it removes and adds it again in a loop. It stops as soon as add-module hangs for more than 150 seconds, and prints the user and the systemd journal for the reused UID.

#!/bin/bash
# Reproduce the add-module hang after a remove-module on Debian 13.
# GAP: seconds to wait between remove and add (default 0).
# ITER: number of remove/add rounds (default 5).
LEADER=root@leader.example.org
NODE=root@node.example.org
NODE_ID=3
IMG=ghcr.io/nethserver/imapsync:1.3.3

L() { ssh -o BatchMode=yes "$LEADER" "$@"; }
N() { ssh -o BatchMode=yes "$NODE" "$@"; }
add() { timeout 150 ssh -o BatchMode=yes "$LEADER" "api-cli run add-module --data '{\"image\":\"$IMG\",\"node\":$NODE_ID}'" | jq -r .module_id; }
rm_() { [ -n "$1" ] && L "api-cli run remove-module --data '{\"module_id\":\"$1\",\"preserve_data\":false}'" >/dev/null; }

mid=$(add); [ -n "$mid" ] && [ "$mid" != null ] || { echo "warmup failed"; exit 1; }
echo "warmup $mid uid=$(N id -u $mid)"
for i in $(seq 1 ${ITER:-5}); do
  uid=$(N id -u "$mid"); rm_ "$mid"; sleep ${GAP:-0}; t0=$(date +%s)
  mid=$(add)
  if [ -z "$mid" ] || [ "$mid" = null ]; then
    echo "iter $i: HANG, add-module stuck after removing uid $uid ($(( $(date +%s)-t0 ))s)"
    N "awk -F: '\$3==$uid' /etc/passwd; systemctl is-active user@$uid.service; journalctl -S -4min --no-pager | grep -E 'useradd|userdel|linger|user@$uid|user-runtime-dir@$uid' | tail -12"
    exit 2
  fi
  nuid=$(N id -u $mid)
  echo "iter $i: $mid uid=$nuid (old $uid) user@=$(N systemctl is-active user@$nuid.service) $(( $(date +%s)-t0 ))s"
done
echo "last mid=$mid"

Results on a Debian 13 node (systemd 257):

• GAP=0: hang on the first round, UID 1004 reused.
• GAP=30 ITER=3: 3 rounds out of 3 pass, UID 1004 reused each time.

So waiting before the next install avoids it. It points to a timing issue, not to the UID reuse alone.

When it hangs, systemctl start user@<uid>.service unblocks the task. The script leaves the last module installed: remove it by hand afterwards.