Add replicated citizenlab asnmeta updaters

[aagbsn]

Combined #185 and #186 updaters (citizenlab and asnmeta tables) and shares the same testing framework used by both; applies the requested changes by @hellais from #186

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.99%. Comparing base (07e13bd) to head (ad1a982).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #190      +/-   ##
==========================================
- Coverage   83.07%   81.99%   -1.08%     
==========================================
  Files          91       93       +2     
  Lines        6411     6560     +149     
==========================================
+ Hits         5326     5379      +53     
- Misses       1085     1181      +96     

Flag	Coverage Δ
oonidata	77.87% <ø> (ø)
oonipipeline	83.72% <ø> (-1.61%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[aagbsn]

Migration: citizenlab and asnmeta to one replicated table on all nodes

Target state: on data1, data2 and data3:

Table	Engine	Keeper path
ooni.citizenlab	ReplicatedReplacingMergeTree	/clickhouse/oonidata_cluster/tables/ooni/citizenlab
ooni.asnmeta	ReplicatedMergeTree	/clickhouse/oonidata_cluster/tables/ooni/asnmeta

These are the paths in the merged updaters. There is no /{shard} suffix any more, so every existing replicated copy (.../citizenlab/01, .../citizenlab_flip/01, .../asnmeta/01) has to be recreated. This includes the citizenlab tables on data2 and data3.

The new paths are the parents of the old /01 paths. That means the old replicas must be dropped and their leftover Keeper nodes removed before the new tables can be created. Until then, each node serves its data from a local copy.

State on 2026-09-29:

Table	data1	data2	data3
ooni.citizenlab	⚠️ bound to .../citizenlab_flip/01 (swapped by the old updater's local EXCHANGE)	.../citizenlab/01	.../citizenlab/01
ooni.asnmeta	MergeTree, 329,093 rows	MergeTree, 0 rows	ReplicatedMergeTree at .../asnmeta/01, 0 rows

Every ON CLUSTER statement below needs all three nodes up. Otherwise it hangs and fails with Code 159.

---

0. Pause the updaters

• Pause halfhour_updaters (citizenlab) and weekly_updaters (asnmeta) in Airflow.

Keep them paused until step 10. The old updaters would undo steps 2–8: the next weekly run is 2026-10-05 00:00 UTC, and citizenlab runs every 30 minutes.

1. Preflight checks

• Check that the macros are defined on every node. cluster should be oonidata_cluster everywhere, and each node should have its own replica:

SELECT hostName(), macro, substitution
FROM clusterAllReplicas('oonidata_cluster', system.macros)
ORDER BY 1, 2;

• Record the current tables and Keeper paths. The old updater swaps the citizenlab and citizenlab_flip names on data1 on every run, so the table above may be out of date:

SELECT hostName() AS host, name, engine, total_rows
FROM clusterAllReplicas('oonidata_cluster', system.tables)
WHERE database = 'ooni'
  AND name IN ('citizenlab', 'citizenlab_flip', 'asnmeta', 'asnmeta_tmp')
ORDER BY name, host;

SELECT hostName(), `table`, zookeeper_path
FROM clusterAllReplicas('oonidata_cluster', system.replicas)
WHERE database = 'ooni' AND `table` IN ('citizenlab', 'citizenlab_flip', 'asnmeta')
ORDER BY `table`, 1;

• Check that SHOW CREATE TABLE ooni.citizenlab and SHOW CREATE TABLE ooni.asnmeta on data1 have the same columns, in the same order, as the definitions below. Steps 2 and 7 use SELECT *.

2. citizenlab: replace each node's replicated table with a local copy (on each node, not ON CLUSTER)

• On data1, data2 and data3 in turn:

CREATE TABLE ooni.citizenlab_local (
    `domain` String, `url` String, `cc` FixedString(32), `category_code` String
) ENGINE = ReplacingMergeTree
ORDER BY (domain, url, cc, category_code)
SETTINGS index_granularity = 4;

INSERT INTO ooni.citizenlab_local SELECT * FROM ooni.citizenlab;

EXCHANGE TABLES ooni.citizenlab AND ooni.citizenlab_local;

After this, citizenlab is a local ReplacingMergeTree on every node. The old replicated table is now named citizenlab_local.

3. citizenlab: check the copies and drop the old replicated tables

• All three nodes show the same count and hash, and engine is ReplacingMergeTree:

SELECT hostName(), count(), groupBitXor(cityHash64(domain, url, cc, category_code)) AS h
FROM clusterAllReplicas('oonidata_cluster', ooni.citizenlab)
GROUP BY 1;

SELECT hostName(), engine
FROM clusterAllReplicas('oonidata_cluster', system.tables)
WHERE database = 'ooni' AND name = 'citizenlab';

• Drop the old replicas. This drops both /01 paths: the new updater no longer uses citizenlab_flip.

DROP TABLE ooni.citizenlab_local ON CLUSTER oonidata_cluster SYNC;
DROP TABLE IF EXISTS ooni.citizenlab_flip ON CLUSTER oonidata_cluster SYNC;

4. asnmeta: clear data3's empty replicated copy (on data3 only)

• Drop the stale tables and create an empty local placeholder:

DROP TABLE ooni.asnmeta SYNC;
DROP TABLE IF EXISTS ooni.asnmeta_tmp SYNC;
CREATE TABLE ooni.asnmeta (
    asn UInt32, org_name String, cc String, changed Date, aut_name String, source String
) ENGINE = MergeTree ORDER BY (asn, changed);

SYNC is required, both here and in step 3. Without it, the dropped replicas stay registered in Keeper for about 8 minutes, and step 6 fails with Code 253.

5. Clean up the old Keeper paths

• Nothing should be bound to the old paths any more. This should return no rows:

SELECT hostName(), `table`, zookeeper_path
FROM clusterAllReplicas('oonidata_cluster', system.replicas)
WHERE database = 'ooni';

(If other replicated tables exist in ooni, filter on zookeeper_path LIKE '%/citizenlab%' OR zookeeper_path LIKE '%/asnmeta%'.)

• Look at what's left under the tables node:

SELECT name, numChildren
FROM system.zookeeper
WHERE path = '/clickhouse/oonidata_cluster/tables/ooni'
  AND name IN ('citizenlab', 'citizenlab_flip', 'asnmeta');

Dropping the last replica removes .../citizenlab/01, but it can leave an empty .../citizenlab node behind. The same applies to asnmeta. An existing node at the new path makes the CREATE in step 6 fail: it retries and then errors with "Cannot create table, because it is created concurrently every time or because of wrong zookeeper_path".

• If numChildren is 0, remove the node. rm refuses to delete a node that has children, so this is safe:

clickhouse-keeper-client  "rm '/clickhouse/oonidata_cluster/tables/ooni/citizenlab'"
clickhouse-keeper-client  "rm '/clickhouse/oonidata_cluster/tables/ooni/asnmeta'"
clickhouse-keeper-client  "rm '/clickhouse/oonidata_cluster/tables/ooni/citizenlab_flip'"

• If numChildren is > 0, stop. Something is still registered, so recheck the system.replicas query above before going on.

6. Create the replicated tables at the new paths

CREATE TABLE ooni.citizenlab_v2 ON CLUSTER oonidata_cluster (
    `domain` String, `url` String, `cc` FixedString(32), `category_code` String
) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/{database}/citizenlab', '{replica}')
ORDER BY (domain, url, cc, category_code)
SETTINGS index_granularity = 4;

CREATE TABLE ooni.asnmeta_v2 ON CLUSTER oonidata_cluster (
    asn UInt32, org_name String, cc String, changed Date, aut_name String, source String
) ENGINE = ReplicatedMergeTree('/clickhouse/{cluster}/tables/{database}/asnmeta', '{replica}')
ORDER BY (asn, changed);

The Keeper path contains the literal citizenlab / asnmeta, not {table}. So after the EXCHANGE in step 8, the names match what the updaters create.

7. Copy the data (on data1)

• Copy the rows:

INSERT INTO ooni.citizenlab_v2 SELECT * FROM ooni.citizenlab;
INSERT INTO ooni.asnmeta_v2 SELECT * FROM ooni.asnmeta;

8. Verify replication, swap in the new tables and clean up

• citizenlab_v2 has the same count and hash on every node as data1's citizenlab in step 3:

SELECT hostName(), count(), groupBitXor(cityHash64(domain, url, cc, category_code)) AS h
FROM clusterAllReplicas('oonidata_cluster', ooni.citizenlab_v2)
GROUP BY 1;

• asnmeta_v2 has the same count on every node as data1's asnmeta (329,093 on 2026-09-29):

SELECT hostName(), count()
FROM clusterAllReplicas('oonidata_cluster', ooni.asnmeta_v2)
GROUP BY 1;

• Swap and drop the local copies:

EXCHANGE TABLES ooni.citizenlab AND ooni.citizenlab_v2 ON CLUSTER oonidata_cluster;
EXCHANGE TABLES ooni.asnmeta AND ooni.asnmeta_v2 ON CLUSTER oonidata_cluster;

DROP TABLE ooni.citizenlab_v2 ON CLUSTER oonidata_cluster SYNC;
DROP TABLE ooni.asnmeta_v2 ON CLUSTER oonidata_cluster SYNC;
DROP TABLE IF EXISTS ooni.asnmeta_tmp ON CLUSTER oonidata_cluster SYNC;

9. Verify both tables

• Each node has exactly one citizenlab on /clickhouse/oonidata_cluster/tables/ooni/citizenlab, and one asnmeta on /clickhouse/oonidata_cluster/tables/ooni/asnmeta. There is no citizenlab_flip, and no path ends in /01:

SELECT hostName(), `table`, zookeeper_path
FROM clusterAllReplicas('oonidata_cluster', system.replicas)
WHERE database = 'ooni' AND `table` IN ('citizenlab', 'citizenlab_flip', 'asnmeta')
ORDER BY `table`, 1;

• asnmeta shows 329093 rows and max(changed) = 2026-09-01 on all three nodes:

SELECT hostName(), count(), max(changed)
FROM clusterAllReplicas('oonidata_cluster', ooni.asnmeta)
GROUP BY 1;

10. Deploy the new updaters

• Deploy Add replicated citizenlab asnmeta updaters #190:

./play -i inventory -l data1.htz-fsn.prod.ooni.nu deploy-airflow.yml -t oonipipeline --diff

• Unpause halfhour_updaters and weekly_updaters. Trigger each once with all three nodes up: alter_sync = 2 waits for every replica.
• Check that the updater logs show REPLACE PARTITION succeeding. CREATE TABLE IF NOT EXISTS ... ON CLUSTER should do nothing, because the tables already exist at the right paths.
• Check that citizenlab matches on all nodes (same count and hash) with the query from step 8, run against ooni.citizenlab.
• Rerun the asnmeta count query from step 9. All nodes should match.

11. Load balancer

• Apply the add load balancer rules for private API routes devops#488 private API routes. With both tables replicated, all routes can move to oonimeasurements at once. There's no need to hold back asnmeta, networks, domains and domain_metadata.

[aagbsn]

drop the {shard} key for citizenlab and asnmeta; reduce the verbosity of comments to the salient points

[hellais]

LGTM