perf(docker): reuse runtime layers and scope Maven builds

[lokidundun]

Visual summary

Purpose of the PR

• close perf(docker): track follow-up image build optimizations #3179

Main Changes

• Move runtime package installation before COPY --from=build in all four Dockerfiles, allowing dependency layers to survive application source changes.
• Keep configuration edits that depend on copied files after the corresponding COPY.
• Build the Server, PD, and Store distribution modules with -pl and -am, reducing the reactor from 38 to 27 modules while retaining required dependencies.
• Preserve existing runtime packages, configurable Maven arguments, and identical shared Maven build stages across all four Dockerfiles.

The existing single-job Bake flow and QEMU-based ARM64 build remain in place.

Verifying these changes

• Trivial rework / code cleanup without any test coverage. (No Need)
• Already covered by existing checks: multi-platform image inspection, Compose integration, Gremlin CRUD, and Standalone smoke tests.
• Additional verification: before/after benchmarks and distribution/image content comparisons.

Runtime dependency layer benchmark

Both variants used the same GitHub-hosted Ubuntu runner class and separate GHCR registry caches. After seeding each cache, identical configuration comments were added to trigger distribution changes and a full Maven rebuild.

Scenario / metric	Before	After	Reduction
Initial cache seed: image build	552 s	510 s	42 s (7.6%)
Initial cache seed: complete job	704 s	669 s	35 s (5.0%)
Source change: image build	601 s	461 s	140 s (23.3%)
Source change: complete job	755 s	602 s	153 s (20.3%)

Before the change, the four ARM64 runtime package installation steps ran under QEMU and took 150.6–160.5 seconds each. After the change, all eight runtime dependency layers across amd64 and arm64 were restored from registry cache in 2.0–9.3 seconds per layer, without rerunning package installation.

Cache export was measured separately:

Cache export vertex	Before	After
Shared build cache	205.2 s	165.7 s
PD	87.8 s	5.5 s
Store	56.1 s	4.9 s
HStore Server	83.3 s	6.6 s
Standalone Server	40.3 s	8.1 s

These BuildKit vertices overlap; their durations must not be added together or treated as independent end-to-end savings.

Maven reactor benchmark

This separate experiment compared the full reactor with the scoped reactor using the same source and existing runtime layer optimization.

Metric	Full reactor	Scoped reactor	Reduction
Reactor modules	38	27	11 modules
Maven-reported build time	270 s	176 s	94 s (34.8%)
Distribution build, cache and local export	424 s	269 s	155 s (36.6%)
Complete distribution job, including artifact upload	492 s	351 s	141 s (28.7%)

The distribution-job timings above are not complete image publication timings.

Correctness checks

• Compared 391 distribution files, including permissions and normalized nested JAR contents.
• Compared all four images on both amd64 and arm64: 780 records per architecture covering application files, installed packages, and image configuration, with no missing, added, or changed records.
• JAR comparisons normalized ZIP timestamps, compression, entry order, and specified build-time metadata; this verifies effective content equivalence rather than byte-identical archives.
• The runtime-layer CI built and published all four images with both architectures, passing local platform inspection, Compose, Gremlin CRUD, Standalone smoke tests, and published manifest checks. Functional checks in this flow ran on amd64.
• A separate native ARM experiment passed functional checks on both architectures and final multi-platform publication. That experimental workflow is not included in this PR.

Successful CI runs

Validation	Result	CI run
Runtime layer benchmark, multi-platform build and publication	All 4 jobs passed	33950351952
Full/scoped distribution benchmark, content comparison, and native architecture validation	All 6 jobs passed	33953324289
Full/scoped image content comparison on amd64 and arm64	Both jobs passed	33953049148

These are fork validation runs for the implementation approach. The Maven experiments enabled the same module selection through MAVEN_ARGS; this PR places that selection directly in the Dockerfiles. These runs do not replace CI on the final PR commit.

Measurements are single-run observations using GHCR, not the official Docker Hub publication environment. Savings from separate experiments are not additive.

Commands for local verification

Run from the repository root using Bash.

Build the selected distributions:

mvn install \
  -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \
  -am -e -B -ntp \
  -Dmaven.test.skip=true \
  -Dmaven.javadoc.skip=true

Inspect the shared Bake configuration:

docker buildx bake -f docker/bake.hcl --print

Build all four images through the shared Bake flow:

docker buildx bake -f docker/bake.hcl --progress=plain

The default Bake targets include amd64 and arm64. Local multi-platform loading requires a compatible builder and the containerd image store; building ARM64 on an amd64 host also requires emulation.

Check a direct Dockerfile build:

docker buildx build \
  --platform linux/amd64 \
  -f hugegraph-server/Dockerfile \
  -t hugegraph-standalone:pr-check \
  --load .

The commands above are build checks; the linked CI runs provide the integration, content-comparison, and publication validation.

Does this PR potentially affect the following parts?

• Dependencies
• Modify configurations
• The public API
• Other affects: Docker build layer caching and Maven reactor selection.
• Nope

Documentation Status

• Doc - TODO
• Doc - Done
• Doc - No Need

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.14%. Comparing base (2e2c02c) to head (7d0b388).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3194      +/-   ##
============================================
- Coverage     41.15%   41.14%   -0.02%     
+ Complexity     7218     7212       -6     
============================================
  Files           802      802              
  Lines         69393    69393              
  Branches       9237     9237              
============================================
- Hits          28562    28549      -13     
- Misses        37570    37582      +12     
- Partials       3261     3262       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bitflicker64]

Blocking: no. Summary: Both mechanisms hold up. The four build stages stay byte-identical, so the shared Bake cache and the existing docker-bake-check guard still work, and the -pl ... -am scoping is content-safe because all three assembly descriptors are dependency-driven rather than module-driven. Four non-blocking points: the runtime apt layer now refreshes only when the base image digest changes, the reactor scope is no longer settable through MAVEN_ARGS, .dockerignore was not narrowed to match the new scope, and Dockerfile-hstore gained an avoidable layer. Evidence: git diff origin/master...304bea1; lines 20-33 of all four Dockerfiles hash identically; the server, pd and store assembly descriptors contain no <moduleSet>, so -am builds exactly the closure they consume; .github/workflows/cluster-test-ci.yml still runs an unfiltered full-reactor mvn clean package on every pull request, so compile coverage of the 11 dropped modules is retained.

[bitflicker64]

Blocking: no. Summary: All four points from the review on 304bea1 are handled at this head: the epoch ARG, the MAVEN_PROJECTS build arg, the merged hstore layer, and the .dockerignore follow-up note in the README. The build stages are still byte-identical and the runtime reordering keeps image behaviour the same. One small point is left. The default module list now lives in docker/bake.hcl as well as in the four Dockerfiles, and CI only checks the Dockerfile copies. Evidence: git diff 36811483...230580d and git diff 304bea1 230580d. The docker-bake-check job in .github/workflows/docker-build-ci.yml diffs only the AS build stages, and its jq assertion on bake --print never reads args. The docker-build job runs plain docker build with only SOURCE_REVISION. The red hstore check (VertexCoreTest.testAddVertexWithTtlAndTtlStartTime) is the TTL timing flake already noted on #3187. That job runs Java core tests and does not use the files changed here, but it needs a rerun.

[imbajin]

No blocking findings at 6bac8f274 after two independent global reviews and one adversarial review. Review score: 9/10. Bake defaults and explicit overrides pass locally; the scoped Maven reactor validates, all four shared build stages match, and all four Docker image builds pass in CI. The five earlier review threads are addressed and resolved. No additional code changes are needed.

The remaining CI failure was Maven Central returning HTTP 502 while CodeQL downloaded maven-jar-plugin:3.2.0; I reran the failed job. Please wait for that rerun before merging. A local image rebuild was unavailable because the Docker daemon was not running; image-build evidence comes from CI.

[imbajin]

+1. The CodeQL rerun has passed and all current checks are green at 6bac8f2. The independent review remains clean (9/10); ready to merge.