Skip to content

redis: optional TLS for the Redis and Redis Sentinel connections - #2213

Open
kbinetruy wants to merge 7 commits into
sipwise:masterfrom
kbinetruy:redis-tls
Open

kbinetruy wants to merge 7 commits into
sipwise:masterfrom
kbinetruy:redis-tls

Conversation

@kbinetruy

@kbinetruy kbinetruy commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

A Redis server that only accepts TLS (a managed service, or tls-port with port 0) cannot be used today. This was asked on the mailing list in 2022: https://groups.google.com/g/rtpengine/c/gO2hWMKxCNA/m/DNH37Z54BQAJ

This PR depends on #2212, which depends on #2211. The branch has 13 commits; only the last two are new:

  1. MT#55283 redis: optional TLS for the Redis connections (12 files, +550 −5)
  2. MT#55283 redis: TLS for the Redis Sentinel connections (4 files, +87 −8)

The two commits on their own: kbinetruy/rtpengine@redis-sentinel...redis-tls

1. With --redis-cafile or --redis-capath, as for the MQTT options, every Redis connection uses TLS: --redis, --redis-write, --redis-subscribe, the restore copies, and the async connections. --redis-certfile and --redis-keyfile add a client certificate. One SSL_CTX is made at startup. The handshake follows the command timeout. On a reconnect it runs in the thread that makes the attempt, without r->lock. A refused certificate or a handshake timeout is a failed connect, logged with the reason.

Released hiredis (1.2.0 to 1.4.1) checks only the chain, so the name is checked here, in the subjectAltName only. An address needs an IP entry, and no SNI is sent. A host name is checked with X509_CHECK_FLAG_NO_PARTIAL_WILDCARDS, and sent as SNI. --redis-tls-server-name replaces the expected name of every data node.

2. The Sentinel thread uses the same context: a Sentinel's certificate must match its host as given in --redis-sentinel. The ROLE check of a master must match the address that the Sentinels gave, or --redis-tls-server-name.

Option Default Meaning
--redis-cafile=FILE, --redis-capath=PATH unset Either one turns TLS on for every Redis connection.
--redis-certfile=FILE, --redis-keyfile=FILE unset Client certificate. Both or neither.
--redis-tls-server-name=NAME unset Name expected from every data node (and sent as SNI).

TLS 1.2 is the minimum. Files are PEM, and password-protected keys are not supported, as for MQTT. utils/gen-common-flags defines HAVE_HIREDIS_SSL when pkg-config finds hiredis_ssl. On Debian trixie and Ubuntu noble, libhiredis-dev ships it. The TLS options exist only in a build with hiredis_ssl. On other builds (Debian bookworm) they are unknown command-line options, as the MQTT ones, and a redis-cafile key in the configuration file is ignored.

Lab. Everything runs over TLS, with Sentinel, repair and resync and client certificates: failover ×3 and frozen master ×3. Each run had 0 lost, 0 deleted back, and 156 calls restored by a takeover. A reconnect to the new master took 3.4–4.6 ms over TLS, against 4.3 ms over plain TCP.

Testing. t/auto-daemon-tests-redis-tls.pl (126 checks, about 7 s) runs fake Redis and fake Sentinels over IO::Socket::SSL, with certificates made by openssl at test time (libio-socket-ssl-perl added to Build-Depends). It covers every connection type, every refusal (another CA, another address or name, partial wildcard, CN only), SNI, a required client certificate, a silent peer and the startup checks. It fails on the parent commit, and catches mutants of the checks (one equivalent mutant survives). make asan-check on master 3865f896: exit 0 on Debian trixie and Ubuntu noble, 9666 ok, 0 not ok. The patches apply with git am on 3865f896.

Questions:

  1. Should TLS be turned on by --redis-cafile (as for MQTT), or by an explicit --redis-tls switch?
  2. Should the Sentinels share the data nodes' TLS options (as Redis configures both with the same tls-* directives), or have their own?

Lab method, per-run figures and test details: https://github.com/kbinetruy/rtpengine/blob/redis-notes/redis-sentinel-notes.md

--redis-write accepts any role, so after a failover that demotes its
node, rtpengine reconnects to the same address, accepts the replica,
and every later write is refused with READONLY.

--redis-require-master (off by default) makes the connections that
write call data require the master role at connect, and drop the
connection on a write refused with READONLY or MASTERDOWN, so that the
next operation reconnects and checks the role. Other refusals keep the
connection.

Tested by auto-daemon-tests-redis-require-master.pl.
Behind Redis Sentinel, rtpengine stays on the address it was given:
after a failover its writes go to the old master, which refuses them
until it is demoted, then serves them as a replica.

New options --redis-sentinel, --redis-sentinel-master and
--redis-sentinel-interval; the keyword "sentinel" replaces IP:PORT in
--redis, --redis-write or --redis-subscribe. A "redis sentinel" thread
asks every Sentinel for the master, takes the highest config epoch
(never an older one than in use), checks the node with ROLE and
publishes it with a generation number. It also runs at once when a
connection meets a replica. redis_check_conn() compares the generation
and reconnects on a change; these connections require the master role.
Without "sentinel", none of this runs.

Tested by auto-daemon-tests-redis-sentinel.pl and -sentinel-start.pl.
A subscription made through "sentinel" stayed on the old master after a
failover, and missed the notifications of the new one.

With Sentinel, the notification event loop gets a one-second timer:
when the master changed, it closes the subscription, and
redis_notify_loop() subscribes again on the new master. The
subscription's context also gets a connect callback, since hiredis
frees a context that fails to connect without calling the disconnect
callback. Notifications missed during the switch are not replayed.

Tested by auto-daemon-tests-redis-sentinel-subscribe.pl.
After a failover the new master refuses writes with NOREPLICAS until it
has a replica. Those writes are counted and logged, but never made
again: a call created then is missing at the next restore, and a call
deleted then comes back.

With --redis-repair-writes (off by default), an own call whose write was
not stored (an error reply, a connection lost before the reply, a
skipped write) is marked in a table keyed by call ID: the call itself,
held, while it is in the call table; its ID and database once it is
deleted, as a deleted call keeps its memory until its last reference
goes. A stored write clears the mark. A marked SET supersedes a marked
DEL of the same ID; if its call was never written, it takes that key
over, so that its own deletion deletes it. A "redis repair" thread
writes the marked calls again, one at a time, through
redis_write_call(), now shared with redis_update_onekey(): a live call
gets its current state, the key of a deleted call a DEL, unless a new
call uses its ID. Signalling writes waiting for r->lock go first, for
100 ms at most. A failed write is retried after a second; at most 100000
calls are kept. New statistic: redisrepairedwrites. A repaired DEL
assumes one writer per database.

Tested by auto-daemon-tests-redis-repair.pl and
auto-daemon-tests-redis-repair-ids.pl.
Replication is asynchronous, so a master that is replaced takes with it
the writes it acknowledged but had not replicated: a call created then
is missing on the new master, and a call deleted then comes back.

--redis-resync-on-master-change (off by default) keeps the
master_replid of the INFO that redis_connect() already sends. When the
write connection reconnects to another ID, every own call stored at
least once, and the own DELs of the last --redis-resync-delete-window
(at most --redis-resync-delete-max), are queued for the repair thread
of --redis-repair-writes. The write connection must require the master
role. No key is scanned. New statistics: redisresyncs, redisresynccalls
and redisresyncdeletes.

Tested by auto-daemon-tests-redis-resync.pl.
There is no way to reach a Redis server that only accepts TLS (managed
services, or tls-port with port 0); asked on the mailing list in 2022:
https://groups.google.com/g/rtpengine/c/gO2hWMKxCNA/m/DNH37Z54BQAJ

With --redis-cafile or --redis-capath, every Redis connection uses TLS,
and --redis-certfile with --redis-keyfile adds a client certificate. As
for MQTT, the options exist only in a build with hiredis_ssl. hiredis
checks only the chain, so the name is checked here, in the
subjectAltName: the host as configured, the IP address, or
--redis-tls-server-name. The handshake is bounded by the command
timeout, or by the connect timeout when there is none.

Tested by auto-daemon-tests-redis-tls.pl (libio-socket-ssl-perl added to
Build-Depends).
With the TLS options, the Sentinel thread still spoke plain TCP to the
Sentinels and for the role check of the master they name, so a Sentinel
set with tls-port and port 0 could not be used.

sentinel_connect() now starts TLS with the same context and options. A
Sentinel's certificate must match its host as given in
--redis-sentinel; a master's must match the address the Sentinels gave,
or --redis-tls-server-name. The handshake happens in the thread, never
under r->lock.

Tested by auto-daemon-tests-redis-tls.pl.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant