[3/4] Tier and shard complete integration testing

[jlnav]

• Shards manual complete CI into core, external, slow, MPI smoke, ProxyStore, Open MPI, and macOS jobs.
• Adds semantic test metadata, stable runner sharding, and dependency-specific service setup.
• Adds a genuine Python 3.12 Open MPI Pixi environment and verifies mpi4py against Open MPI 5.0.10.
• Stack: parent ci/fast-feedback-topology; child [4/4] Reduce slow integration test workloads #1781.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 76.19%. Comparing base (cccf689) to head (b3ed213).

Files with missing lines	Patch %	Lines
libensemble/gen_funcs/aposmm_localopt_support.py	0.00%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (cccf689) and HEAD (b3ed213). Click for more details.

HEAD has 8 uploads less than BASE

Flag	BASE (cccf689)	HEAD (b3ed213)
	9	1

Additional details and impacted files

@@                      Coverage Diff                       @@
##           ci/fast-feedback-topology    #1782       +/-   ##
==============================================================
- Coverage                      88.98%   76.19%   -12.79%     
==============================================================
  Files                             84       72       -12     
  Lines                           9093     8381      -712     
  Branches                        1402     1318       -84     
==============================================================
- Hits                            8091     6386     -1705     
- Misses                           755     1698      +943     
- Partials                         247      297       +50     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jlnav]

Diagnosed the long-running core job: test_persistent_aposmm_ibcdfo_pounders.py was incorrectly treated as a core test because it lacked TESTSUITE_EXTRA. It is an external IBCDFO/POUNDERS integration with sim_max=500 and no wall-clock bound.

Fixed in 34fdfc1f0 (also propagated to the final stack branch):

• Marked both IBCDFO POUNDERS tests as TESTSUITE_EXTRA: true, TESTSUITE_TIER: external, and tagged them with ibcdfo pounders.
• Added wallclock_max=300 to both runs as a safety bound.
• Verified the runner excludes the test from --tier core and selects it under --tier external -e.

[jlnav]

The new run confirmed the previous wallclock_max change was insufficient for this hang. wallclock_max is checked by the libEnsemble manager, but APOSMM/IBCDFO can block inside the persistent generator before the manager regains control; therefore it cannot enforce a hard timeout here.

The external shard is now bounded in commit 985a9ce04: timeout terminates the test command after 8 minutes, with a 30-second kill-after grace period, and the GitHub job has a 10-minute timeout-minutes backstop. The affected run 35143116947 was cancelled.

The external (0) shard also failed quickly in that run; that is separate from the still-running external (1) hang and should be investigated from the completed logs once available.

[jlnav]

Root cause identified and fixed in d6126c8e4: the Mypy lockfile refresh advanced unpinned IBCDFO from 538a7ad to 26fbe4c. The newer IBCDFO requires MINQ ea8b980, but CI installs MINQ 7749b83; its compatibility check calls sys.exit(). APOSMM only caught Exception, not SystemExit, so the child exited without signalling the parent and the test blocked forever.

The fix pins IBCDFO to the previously locked 538a7ad revision compatible with CI MINQ and changes the APOSMM optimizer wrapper to return all startup failures, including SystemExit, to the parent. The focused POUNDERS test now completes locally with both optimizer cases.