fix(server): wait for the stores at startup instead of exiting on a cold start

[SebastianGruza]

Purpose of the PR

• close [Bug][HStore] Server exits 1 on cold start when Stores miss the 38s partition-lookup retry ceiling #3203

With usePD=true the first hstore graph opened at startup (a local conf/graphs graph, or the system graph created on the first boot) needs pd.initial-store-count active stores, and the store client gives up after a fixed 10 retries (about 38 s). Stores that register later than that make the server exit 1 on a cold start; under Kubernetes the container is restarted and usually succeeds the second time, without an orchestrator the server stays down.

Before → after

Scenario	Before	After
First boot, Stores register late	The first HStore graph exhausts the Store client's fixed retry window and Server exits.	Before opening any graph, wait for PD to report Cluster_OK, bounded by pd.stores_wait_timeout.
Restart of an initialized cluster	Open existing graphs.	Existing partitions bypass the readiness wait, including when a Store is down.
PD cannot be reached	Client RPC timeouts can delay startup.	Each probe has a deadline bounded by the remaining wait budget; timeout reports PD's last message.

Main Changes

• Add pd.stores_wait_timeout in seconds: default 300; 0 disables the gate and preserves the previous behavior. Applies when usePD=true.
• Run the gate in GraphManager before local graphs and PD metadata are loaded. An existing partition lets startup continue; an empty cluster waits for PD's own Cluster_OK, without deriving an active-Store threshold from replica count.
• Use short-lived gRPC channels with the existing PD credentials and inbound message size setting. The two RPCs in a probe share one deadline of min(remaining budget, 5 s); channels close after the wait and no watchers are created.
• Keep the generic NodeTxExecutor retry behavior unchanged. This is a bounded bootstrap wait, not a guarantee that all later graph operations succeed.

Verifying these changes

• Follow-up validation on JDK 11: GraphManagerStoresWaitTest 5/5 passed, including a real gRPC partition response over 4 MiB. The new regression test fails with RESOURCE_EXHAUSTED before the inbound-limit fix and passes afterward.

• mvn editorconfig:format, git diff --check, and mvn clean compile -Dmaven.javadoc.skip=true passed (all 38 reactor modules).

• Original unit/core/GraphManagerStoresWaitTest coverage (added to UnitTestSuite), 4/4 on JDK 11: an initialised cluster is not waited for (one probe, zero wait); a cold start waits until PD reports OK, with one unreachable answer retried; a black-holed probe gets deadlines that shrink with the budget and the total stays under timeout + one poll; the timeout message keeps PD's last message and names the option.

• Author-reported reproduction on 3 VMs (1 PD + 3 Store + 1 Server from a tarball, usePD=true; same setup as in my comment on [Bug][HStore] Server exits 1 on cold start when Stores miss the 38s partition-lookup retry ceiling #3203, script cluster/repro_coldstart.sh, logs in results/issue-3203/fix/ of https://github.com/SebastianGruza/hugegraph-validation):

run	result
master 1a15e762, cold start, stores +5 s / +71 s after PD, first boot	18 × error code = 105, backoff 1,1,1,2,3,4,5,6,7,8, upper limit : 10, exit 1 after 38 s
this PR, same cold start	14 progress lines Waiting for the PD cluster: Cluster_Not_Ready: The number of active stores is 1, less than pd.initial-store-count:3, then PD cluster ready after 70s: PD reports Cluster_OK, 0 × 105, exit 0 after 81 s, REST 200
this PR, restart with the node2 store killed (kill -9, port closed), cluster initialised	PD cluster ready after 0s: cluster already has 24 partition(s), exit 0 after 8 s, REST 200 (PD still listed the store as Up 200 s later, so the skip must not depend on PD's store view)
this PR, pd.peers=192.168.80.250:8686 (black hole), pd.stores_wait_timeout=20	Waiting for the PD cluster: 192.168.80.250:8686: UNAVAILABLE (16s left … 10s … 4s), Timed out after 20s waiting for the PD cluster to be ready (192.168.80.250:8686: UNAVAILABLE); start the stores first or raise pd.stores_wait_timeout, exit 1 after 31 s (10 s JVM boot + 20 s budget)

Notes

• The wait sits in the GraphManager constructor, not in createSysGraphIfNeed(): with a local conf/graphs hstore graph that local graph is opened before loadMetaFromPD() and already trips the retry ceiling (a first version that waited only before the system graph did nothing on the VMs).
• Cluster_OK alone would not do for restarts: StoreNodeService.checkStoreStatus() reports Cluster_Not_Ready whenever fewer than pd.initial-store-count stores are active, so with three stores and initial-store-count: 3 a restart with one store down would wait too. Hence "partitions exist" as the first criterion and Cluster_OK only for an empty cluster.
• The [wait-storage] gate in the image entrypoint from feat(helm): add HStore deployment chart #3132 stays the right thing for containers; this change covers the tarball and any start without an orchestrator.

The terminal-log forwarding suggestion in #3203 is outside this change; container logging and orchestration gates remain separate concerns.

[codecov]

Codecov Report

❌ Patch coverage is 61.00000% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 38.03%. Comparing base (1a15e76) to head (21af9de).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
...n/java/org/apache/hugegraph/core/GraphManager.java	59.79%	27 Missing and 12 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3210      +/-   ##
============================================
+ Coverage     37.90%   38.03%   +0.13%     
- Complexity     6600     6637      +37     
============================================
  Files           800      802       +2     
  Lines         69035    69266     +231     
  Branches       9186     9222      +36     
============================================
+ Hits          26167    26348     +181     
- Misses        39795    39840      +45     
- Partials       3073     3078       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[imbajin]

Blocking: no. Summary: The startup wait needs a hard deadline for each PD RPC and a lifecycle-safe client. Evidence: current-head static tracing shows the new calls use PDClient's default 60-second gRPC deadline, and the temporary client starts watchers when its first blocking stub is created.

[imbajin]

⚠️ Blocking: no. PDClient.create() is not a one-shot client: the first getPDConfig() call enters newBlockingStub(), which invokes startWatch() and opens the watch streams. This local client is discarded after the wait, while PDClient exposes no shutdown (closeStub() even notes that the managed channel is not closed), so every usePD startup leaves an unused watcher set for the process lifetime. Please use a closable one-shot stub or retain and close this client from the GraphManager lifecycle.

[SebastianGruza]

Done in 3d64daa. Instead of a PDClient there is now a PdReadinessProbe: one plaintext gRPC channel per PD peer, a raw PDGrpc.newBlockingStub with the same Authentication interceptor that AbstractClient.setBlockingParams() adds (credentials from PDConfig.setAuthority(), so without PD authentication no interceptor is attached), no watchers, closed with shutdownNow() in a try-with-resources once the wait is over. PD forwards requests to its leader on the server side, so any peer answers; the probe rotates through the peers so one dead PD does not eat every poll.

[imbajin]

⚠️ Blocking: no. pd.stores_wait_timeout is not an upper bound for this startup wait. PDConfig.of(this.pdPeers) keeps the PD client's default 60-second gRPC deadline, so getPDConfig() here and each getActiveStores() call inside waitForStores() can block for up to 60s; the supplier is invoked before the deadline check as well. With pd.stores_wait_timeout=20 and a black-holed PD, startup can stall well past 20s. Please propagate a per-call deadline bounded by the remaining wait budget (or otherwise time-bound the RPC) and add a delayed/unreachable-PD test.

[SebastianGruza]

Done in 3d64daa. Every probe call gets withDeadlineAfter(min(remaining budget, 5 s poll)), and the budget is checked before the call, not after. testBlackholedPdStaysWithinTheBudget: a probe that hangs for its whole deadline on every call receives shrinking deadlines (all ≤ 1 s at a 1 s poll) and the whole wait ends below timeout + one poll. On the lab with pd.peers=192.168.80.250:8686 (a black hole) and pd.stores_wait_timeout=20: Waiting for the PD cluster: 192.168.80.250:8686: UNAVAILABLE (16s left … 10s … 4s), then Timed out after 20s waiting for the PD cluster to be ready (192.168.80.250:8686: UNAVAILABLE); start the stores first or raise pd.stores_wait_timeout, exit 1 after 31 s (10 s of JVM boot plus the 20 s budget). Log: results/issue-3203/fix/after-v2-unreachable-pd-timeout20.log in https://github.com/SebastianGruza/hugegraph-validation.

[bitflicker64]

Blocking: no. Summary: the wait requires shard_count active stores on every start with usePD=true, not only on a cold start, so restarting the server while one of three stores is down now blocks for pd.stores_wait_timeout and exits, although PD reports the cluster as usable. Evidence: static read of GraphManager.waitForActiveStores at 0084e77, PD ConfigService (served PDConfig has no min_store_count), StoreNodeService.allocShards/checkStoreStatus. Latest-head CI is green.

[bitflicker64]

⚠️ Important. With the PDConfig PD serves, min_store_count is always 0 (ConfigService only sets partition and shard count), so required is shard_count on every cluster, and this runs on every start, not just the first boot.

That turns a normal restart into a failure: 3 stores, default-shard-count: 3, one store down for maintenance. Each raft group still has 2 of 3 replicas, PD's checkStoreStatus() reports Cluster_OK (it compares against pd.initial-store-count and a per-group majority), and before this PR the server opened its graphs. Now getActiveStores() returns 2, the loop waits 300 s and throws. The count is also not what PD enforces on first boot: allocShards() checks pd.initial-store-count and caps shards at min(shard_count, stores), so a cluster with fewer stores than shard_count never starts, and one with initial-store-count above shard_count still hits the original retry ceiling.

Requested change: gate the wait on PD's own readiness instead of a derived count, for example poll pdClient.getClusterStats() until the state is Cluster_OK (which already encodes initial-store-count and the majority check), or at least skip the wait once shard groups exist. Add a test for the restart-with-a-store-down case.

[SebastianGruza]

Done in 3d64daa, thanks, that case was a real regression. The probe first asks queryPartitions with an empty query: if the cluster has any partition it is ready and nothing is waited for, whatever PD thinks of the stores at that moment. Only a cluster without partitions (first boot) waits, and it waits for getClusterStats() == Cluster_OK, which is exactly what allocShards() needs, with no count derived on the server side. One correction to your description: checkStoreStatus() also sets Cluster_Not_Ready when fewer than pd.initial-store-count stores are active (StoreNodeService.java:831-835), so with three stores and initial-store-count: 3 Cluster_OK on its own would block a restart with one store down as well; hence "partitions exist" as the first criterion and Cluster_OK only for an empty cluster.

Measured on the lab (1 PD + 3 Store + 1 Server from a tarball, usePD=true): a cold start with stores at +5 s / +71 s → 14 progress lines carrying PD's message (The number of active stores is 1, less than pd.initial-store-count:3), PD cluster ready after 70s: PD reports Cluster_OK, 0 × 105, exit 0 after 81 s, REST 200; a restart with the node2 store killed with kill -9 (port 8500 closed) → PD cluster ready after 0s: cluster already has 24 partition(s), exit 0 after 8 s, REST 200; PD still listed that store as Up 200 s later, which only confirms that the skip must not depend on PD's view of the stores. testInitialisedClusterIsNotWaitedFor covers the case (one probe, zero wait). Logs: results/issue-3203/fix/after-v2-*.log.

[SebastianGruza]

Round 1 in 3d64daa: a one-shot gRPC probe (a channel per PD peer, no watchers, closed after the wait) replaces the temporary PDClient, every call carries a deadline bounded by the remaining budget, a cluster that already has partitions is not waited for at all, and a cluster without partitions waits for PD's own Cluster_OK instead of a store count derived on the server. Four unit tests (initialised cluster, cold start, black-holed PD within budget, timeout keeping PD's last message), 4/4 on JDK 11. Three lab runs in the table in the description: cold start with 70 s of waiting and exit 0, restart with a killed store in 8 s and exit 0, unreachable PD exiting 1 after the 20 s budget. Description updated.

[imbajin]

+1, the current scope is appropriate for #3203. The bounded wait before opening any graph addresses the cold-start failure while preserving existing-cluster startup and the generic Store retry behavior. The earlier review concerns and the probe response-size regression are addressed.

A non-blocking follow-up would be a lightweight PD bootstrap-status query, avoiding the full partition-list response, with probing encapsulated in a closable PD client helper. That broader API/client change does not need to expand this fix.

Validation: JDK 11 full reactor compilation and 5/5 focused tests passed, including a real gRPC response over 4 MiB that failed before the fix. CI for this head is still running; merge after it passes.