Repository navigation
Conversation
--redis-write accepts any role, so after a failover that demotes its node, rtpengine reconnects to the same address, accepts the replica, and every later write is refused with READONLY. --redis-require-master (off by default) makes the connections that write call data require the master role at connect, and drop the connection on a write refused with READONLY or MASTERDOWN, so that the next operation reconnects and checks the role. Other refusals keep the connection. Tested by auto-daemon-tests-redis-require-master.pl.
Behind Redis Sentinel, rtpengine stays on the address it was given: after a failover its writes go to the old master, which refuses them until it is demoted, then serves them as a replica. New options --redis-sentinel, --redis-sentinel-master and --redis-sentinel-interval; the keyword "sentinel" replaces IP:PORT in --redis, --redis-write or --redis-subscribe. A "redis sentinel" thread asks every Sentinel for the master, takes the highest config epoch (never an older one than in use), checks the node with ROLE and publishes it with a generation number. It also runs at once when a connection meets a replica. redis_check_conn() compares the generation and reconnects on a change; these connections require the master role. Without "sentinel", none of this runs. Tested by auto-daemon-tests-redis-sentinel.pl and -sentinel-start.pl.
A subscription made through "sentinel" stayed on the old master after a failover, and missed the notifications of the new one. With Sentinel, the notification event loop gets a one-second timer: when the master changed, it closes the subscription, and redis_notify_loop() subscribes again on the new master. The subscription's context also gets a connect callback, since hiredis frees a context that fails to connect without calling the disconnect callback. Notifications missed during the switch are not replayed. Tested by auto-daemon-tests-redis-sentinel-subscribe.pl.
After a failover the new master refuses writes with NOREPLICAS until it has a replica. Those writes are counted and logged, but never made again: a call created then is missing at the next restore, and a call deleted then comes back. With --redis-repair-writes (off by default), an own call whose write was not stored (an error reply, a connection lost before the reply, a skipped write) is marked in a table keyed by call ID: the call itself, held, while it is in the call table; its ID and database once it is deleted, as a deleted call keeps its memory until its last reference goes. A stored write clears the mark. A marked SET supersedes a marked DEL of the same ID; if its call was never written, it takes that key over, so that its own deletion deletes it. A "redis repair" thread writes the marked calls again, one at a time, through redis_write_call(), now shared with redis_update_onekey(): a live call gets its current state, the key of a deleted call a DEL, unless a new call uses its ID. Signalling writes waiting for r->lock go first, for 100 ms at most. A failed write is retried after a second; at most 100000 calls are kept. New statistic: redisrepairedwrites. A repaired DEL assumes one writer per database. Tested by auto-daemon-tests-redis-repair.pl and auto-daemon-tests-redis-repair-ids.pl.
Replication is asynchronous, so a master that is replaced takes with it the writes it acknowledged but had not replicated: a call created then is missing on the new master, and a call deleted then comes back. --redis-resync-on-master-change (off by default) keeps the master_replid of the INFO that redis_connect() already sends. When the write connection reconnects to another ID, every own call stored at least once, and the own DELs of the last --redis-resync-delete-window (at most --redis-resync-delete-max), are queued for the repair thread of --redis-repair-writes. The write connection must require the master role. No key is scanned. New statistics: redisresyncs, redisresynccalls and redisresyncdeletes. Tested by auto-daemon-tests-redis-resync.pl.
There is no way to reach a Redis server that only accepts TLS (managed services, or tls-port with port 0); asked on the mailing list in 2022: https://groups.google.com/g/rtpengine/c/gO2hWMKxCNA/m/DNH37Z54BQAJ With --redis-cafile or --redis-capath, every Redis connection uses TLS, and --redis-certfile with --redis-keyfile adds a client certificate. As for MQTT, the options exist only in a build with hiredis_ssl. hiredis checks only the chain, so the name is checked here, in the subjectAltName: the host as configured, the IP address, or --redis-tls-server-name. The handshake is bounded by the command timeout, or by the connect timeout when there is none. Tested by auto-daemon-tests-redis-tls.pl (libio-socket-ssl-perl added to Build-Depends).
With the TLS options, the Sentinel thread still spoke plain TCP to the Sentinels and for the role check of the master they name, so a Sentinel set with tls-port and port 0 could not be used. sentinel_connect() now starts TLS with the same context and options. A Sentinel's certificate must match its host as given in --redis-sentinel; a master's must match the address the Sentinels gave, or --redis-tls-server-name. The handshake happens in the thread, never under r->lock. Tested by auto-daemon-tests-redis-tls.pl.
kbinetruy
force-pushed
the
redis-tls
branch
4 times, most recently
from
October 23, 2026 12:40
355339a to
739148c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Redis server that only accepts TLS (a managed service, or
tls-portwithport 0) cannot be used today. This was asked on the mailing list in 2022: https://groups.google.com/g/rtpengine/c/gO2hWMKxCNA/m/DNH37Z54BQAJThis PR depends on #2212, which depends on #2211. The branch has 13 commits; only the last two are new:
MT#55283 redis: optional TLS for the Redis connections(12 files, +550 −5)MT#55283 redis: TLS for the Redis Sentinel connections(4 files, +87 −8)The two commits on their own: kbinetruy/rtpengine@redis-sentinel...redis-tls
1. With
--redis-cafileor--redis-capath, as for the MQTT options, every Redis connection uses TLS:--redis,--redis-write,--redis-subscribe, the restore copies, and the async connections.--redis-certfileand--redis-keyfileadd a client certificate. OneSSL_CTXis made at startup. The handshake follows the command timeout. On a reconnect it runs in the thread that makes the attempt, withoutr->lock. A refused certificate or a handshake timeout is a failed connect, logged with the reason.Released hiredis (1.2.0 to 1.4.1) checks only the chain, so the name is checked here, in the subjectAltName only. An address needs an IP entry, and no SNI is sent. A host name is checked with
X509_CHECK_FLAG_NO_PARTIAL_WILDCARDS, and sent as SNI.--redis-tls-server-namereplaces the expected name of every data node.2. The Sentinel thread uses the same context: a Sentinel's certificate must match its host as given in
--redis-sentinel. TheROLEcheck of a master must match the address that the Sentinels gave, or--redis-tls-server-name.--redis-cafile=FILE,--redis-capath=PATH--redis-certfile=FILE,--redis-keyfile=FILE--redis-tls-server-name=NAMETLS 1.2 is the minimum. Files are PEM, and password-protected keys are not supported, as for MQTT.
utils/gen-common-flagsdefinesHAVE_HIREDIS_SSLwhenpkg-configfindshiredis_ssl. On Debian trixie and Ubuntu noble,libhiredis-devships it. The TLS options exist only in a build withhiredis_ssl. On other builds (Debian bookworm) they are unknown command-line options, as the MQTT ones, and aredis-cafilekey in the configuration file is ignored.Lab. Everything runs over TLS, with Sentinel, repair and resync and client certificates: failover ×3 and frozen master ×3. Each run had 0 lost, 0 deleted back, and 156 calls restored by a takeover. A reconnect to the new master took 3.4–4.6 ms over TLS, against 4.3 ms over plain TCP.
Testing.
t/auto-daemon-tests-redis-tls.pl(126 checks, about 7 s) runs fake Redis and fake Sentinels overIO::Socket::SSL, with certificates made byopensslat test time (libio-socket-ssl-perladded to Build-Depends). It covers every connection type, every refusal (another CA, another address or name, partial wildcard, CN only), SNI, a required client certificate, a silent peer and the startup checks. It fails on the parent commit, and catches mutants of the checks (one equivalent mutant survives).make asan-checkon master3865f896: exit 0 on Debian trixie and Ubuntu noble, 9666ok, 0not ok. The patches apply withgit amon3865f896.Questions:
--redis-cafile(as for MQTT), or by an explicit--redis-tlsswitch?tls-*directives), or have their own?Lab method, per-run figures and test details: https://github.com/kbinetruy/rtpengine/blob/redis-notes/redis-sentinel-notes.md