dataplex: fix link validation (search, indexing delay, hidden diffs)

[tnazarew]

Problem

The Dataplex consumer's link checks were unreliable in three ways, and a fourth bug hid every difference on matched entities:

1. Wrong search. SearchLinksRequest takes source or target, which form a protobuf oneof. The validator set both, so only target was used and links were looked up by target only.
2. GCS granularity. Dataplex stores GCS link endpoints at bucket level (gcs:mock-bucket). Searching by an object FQN (gcs:mock-bucket.`result.csv` ) returns nothing, so object-level expected links could never match.
3. Indexing delay. Links show up in search 45–120s or more after the events are ingested. The validator searched about 5s after sending.
4. Hidden diffs. compare_by_start_time had an inverted condition ([] if any(e for e in d if len(e) > 0) else next(...)). It returned no diffs whenever a candidate differed, and an empty diff otherwise, so mismatches on matched lineage events and links were never reported.

Change

• validator.py:
  • Collect every entity FQN from the lineage events, plus its bucket-level form for GCS objects, and search links by source for each one.
  • Poll validation every 15s for up to 240s (LINK_INDEXING_TIMEOUT_SECONDS). Dump mode waits 120s before dumping.
  • Fix the diff condition: a match succeeds if any candidate has no diff; otherwise the first candidate's diff is reported.
• scenarios/airflow/validation/links/links.json: expected links rewritten at bucket level (5 distinct links). The gcs:mock-bucket -> gcs:mock-bucket self-link is dropped because its start time varies between runs.

⚠️ Validation still fails after this PR

With the diff bug fixed, the full run against projects/gcp-open-lineage-testing/locations/us reports real mismatches in fixtures that were never actually checked. All 3 scenarios fail:

Scenario	Failure	Likely fix
airflow	lineage_event expected names are pinned to process 1275f3986dc31d314011311e70aacf4a; Dataplex now creates different process IDs	Use a regex for process/run IDs in the expected names
airflow	Links at 13:05:39, 13:05:29 and 13:05:34 missing after 240s, although a dump taken about 120s after ingestion contained them	Not explained yet: indexing variance, or the search FQNs for those links
simple_run_event	Link at 2023-04-04T13:21:16.098Z missing after 240s (present in the dump)	Same as above
spark_dataproc_simple_producer_test	Expects dbfs: FQNs, Dataplex returns hdfs:; expected 1 link, Dataplex returns 2	Regenerate the fixture from a dump

The previous green results came from the hidden-diff bug, not from matching fixtures. Merging this as-is will turn the Dataplex consumer red until the fixtures are regenerated. I'll follow up in this PR or a separate one.

Note

clean_up() (unchanged) deletes all processes in the target location before and after each scenario, so validation and dumps must not run while CI is using the same location.

🤖 Generated with Claude Code