Add hive_insert scenario and gate hive to OpenLineage 1.55.0

[tnazarew]

Problem

The spark_dataproc/hive scenario fails intermittently: every test tied to the execute_insert_into_hive_table.warehouse_t2 (Spark 3.5.1) or execute_create_hive_table_as_select_command.default_t2 (Spark ≤3.3.2) COMPLETE event reports "not found". These failures account for many of the automated failure issues.

The test is correct. The event is lost by the OpenLineage GcsTransport:

• It names each object <eventTime epochMillis>.json and throws already exists if that name is taken (GcsTransport.java, emitEvent/uploadObject).
• CREATE TABLE ... AS SELECT runs a nested insert, and its COMPLETE event often has the same millisecond eventTime as the CTAS COMPLETE. One of the two is dropped, and the exception is only logged.
• Spark listener timestamps are in milliseconds, so finer-grained file names would not help. The fix belongs upstream: unique object names, for example <epochMillis>-<UUID>.json.

Evidence from CI (Dataproc job 6e106620-f4d0-47f5-bbee-4b64531f2fa5, 2026-10-13):

ERROR EventEmitter: Could not emit lineage w/ exception
io.openlineage.client.OpenLineageClientException: File with given name gs://open-lineage-e2e/events/1791886589000698053-22/1791886632115.json, already exists!
	at io.openlineage.client.transports.gcs.GcsTransport.uploadObject(GcsTransport.java:95)
	...
	at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:161)

Change

• New scenario hive_insert. It covers the same t1 -> t2 lineage (schema, column lineage, dataSource, symlinks and the GCP facets), but uses CREATE TABLE t2 ... followed by INSERT INTO t2 SELECT * FROM t1 instead of CTAS. There is no nested execution, so no two events share a timestamp. Without CTAS, Spark 3.3.2 and 3.5.1 emit the same insert job, so one set of expected events covers all Spark versions.
• hive gated to OpenLineage ≥ 1.55.0 (scenario-level openlineage_versions.min). It comes back once the transport is fixed upstream. If the fix misses 1.55.0, the gate should be raised.

Verification (local Spark with the OpenLineage listener)

Check	hive (CTAS)	hive_insert
File transport: runs with two events in the same millisecond	14–16 / 20	0 / 50 (Spark 3.5.1 × OL 1.54 / 1.32, Spark 3.3.2 × OL 1.54)
Real GCS transport (OL 1.32.0)	COMPLETE lost in 6/6 runs	5/5 runs, 0 already exists errors
Semantic tests (warehouse path mapped to /user/hive/warehouse)	10/10	10/10 on Spark 3.3.2 and 3.5.1, OL 1.54 and 1.32

scripts/check_configs.sh passes. get_valid_test_scenarios.sh selects only hive_insert for 1.32 and 1.54, and both scenarios from 1.55.0.

Not verified locally: Dataproc-only facet values (gcp_dataproc.projectId, gcp_lineage) and Spark 3.1.3. The CI run on this PR should cover them.

🤖 Generated with Claude Code