spark_dataproc: update expected events for OpenLineage 1.37–1.54 changes

[tnazarew]

Summary

Four spark_dataproc scenarios fail on recent OpenLineage releases because the producer's output changed intentionally, not because of regressions. Each commit adds a version-gated expected event for the new behaviour and caps the old one. The old expectations are still checked for older releases.

Commit	Scenario	Behaviour change	Change in this PR
e5b907d	bigquery	From 1.44.0 the parent facet's root job is writing_to_big_query	parent_test_1.44.0; parent_test_1.38.0 capped at 1.43.0
84d396b	spanner	From 1.37.0 the indirect GROUP_BY dependency is in dataset-level lineage (columnLineage.dataset)	columnLineage_test_1.37.0; old test capped at 1.36.0
2cf3989	cloudsql	1.34–1.45 emitted 4 input fields (2 spurious); ≥1.46 emits 2 again, like ≤1.33	columnLineage_test_1.34.0 capped at 1.45.0; columnLineage_test_1.46.0 reuses events/columnLineage.json
bec13f8	bigquery_to_iceberg	1.54.0 no longer emits the nested append_data job; lineage is on atomic_replace_table_as_select	_1.54.0 variants of all 12 tests; old tests capped at 1.53.0

Verification

The repo's semantic validator (OLSemanticValidator with the configs from this branch) was run on events downloaded from CI run 38380529545 (2026-10-13, Spark 3.5.1):

Scenario	OL 1.54.0	OL 1.32.0
bigquery	10/10	10/10
bigquery_to_iceberg	12/12	11/11
cloudsql	12/12	12/12
spanner	12/12	12/12

scripts/check_configs.sh passes.

Notes for reviewers

• cloudsql: one CI run from June 2026 doesn't fit the 1.34–1.45 / ≥1.46 split, and I couldn't explain it. Current 1.54 and 1.32 events match the new expectations.
• bigquery_to_iceberg: storage_1.54.0 does not assert fileFormat. It is empty on atomic_replace_table_as_select in every version, which may be an upstream gap worth reporting separately.
• The Hive flakiness is handled separately in Add hive_insert scenario and gate hive to OpenLineage 1.55.0 #335.

🤖 Generated with Claude Code