[chart/redis-ha][BUG] Masterless deadlock: sentinel keeps naming a master whose actual role is replica; split-brain container never heals it #412

Describe the bug

After the Redis master pod is lost (spot node reclaim in our case) and its replacement is rescheduled behind the same announce Service ClusterIP, the set can wedge into a state with zero actual masters that nothing in the chart ever heals:

• The replacement pod's init.sh can't get a usable answer at boot and falls back to replicating announce-0.
• Sentinel keeps advertising the old master's announce IP as master — the IP is still up (it's a stable ClusterIP, now answering from the replacement pod), so sentinel's down detection never triggers a failover.
• The other replicas keep replicating the sentinel-named "master", which is itself a replica: they loop on Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master once per second, indefinitely.
• The split-brain-fix container never intervenes: it only reinitializes when a pod's local master_host disagrees with sentinel (or when sentinel returns no master → MAX_QUORUM_FAILURES reset). Here every member agrees with sentinel about the stale master, so both conditions pass. The one pod that does get reinit'd (the one owning the stale announce IP, since sentinel names it master while its role is replica) just re-runs init.sh, whose fallback makes it a replica again — a ~4-minute reinit loop that never converges.

Observed on chart 4.39.0 (Redis 7.4, 5 replicas, quorum: 3, haproxy enabled, no PVC): the set stayed masterless — accepting zero writes — for 7.5 hours until manual intervention (restarting two members and rescheduling the stale-IP pod finally re-seated the topology). Logs from the two ends of the cycle:

server-0 (agreed with sentinel that the dead master's announce IP was master):

16:13:31 * Connection with master lost.
16:13:31 * Reconnecting to MASTER 172.20.242.95:6379
16:13:31 # Error condition on socket for SYNC: Connection refused
...then hours of: * Trying a partial resynchronization (request ...)

server-4 (the replacement pod behind that announce IP, running as replica of announce-0):

16:16:00 * Connecting to MASTER 172.20.156.64:6379
16:16:00 * Master is currently unable to PSYNC but should be in the future: -NOMASTERLINK Can't SYNC while not connected with my master
(repeats once per second for 7.5h)

To Reproduce

1. Deploy redis-ha (≥3 replicas, sentinel defaults, no persistence).
2. Kill the node hosting the current master pod (uncleanly — e.g. spot interruption) so the pod is rescheduled while its announce ClusterIP stays constant.
3. If the replacement boots before sentinel completes a failover (or init.sh hits its announce-0 fallback), the set wedges: INFO replication shows role:slave on every member, sentinel get-master-addr-by-name still returns the old master's announce IP, and no failover ever fires.

Timing-dependent — it needs the replacement pod to come back on the same announce IP before sentinel declares odown — but node-loss + reschedule makes it reachable in practice.

Expected behavior

The split-brain container (or sentinel config shipped by the chart) should detect that the sentinel-named master does not actually hold role:master and force a failover, instead of leaving the set masterless forever.

Additional context

PR incoming: adds a stale-master check to the split-brain loop's agreement path — if the sentinel-named master reports a non-master role for MAX_STALE_MASTER_FAILURES consecutive checks (default 3), issue SENTINEL FAILOVER <group> so sentinel promotes one of the known replicas. Unreachable masters are still left to sentinel's own down detection.