feat(bigquery): accelerate row-based query() with Arrow wire format - #14405
jinseopkim0 wants to merge 18 commits into
Conversation
… for hasMorePages
…ssingSerializedSchema
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for executing fast-path queries with Arrow-formatted wire responses in BigQueryImpl, deserializing Arrow IPC schemas and record batches, and handling multi-page results using ArrowQueryPageFetcher. It also adds comprehensive unit tests covering various Arrow query scenarios. The feedback suggests optimizing the fallback path in queryRpcArrow when a query is complete but the Arrow schema is missing; instead of calling getJob and triggering redundant RPCs, the method can directly construct and return a TableResult using the standard JSON format already present in the response.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request adds support for the Arrow query results format on the fast-path query execution in BigQueryImpl, introducing deserialization of Arrow IPC schemas and record batches, pagination handling via ArrowQueryPageFetcher, and corresponding unit tests. Feedback on the changes suggests extracting the duplicated logic for calculating numRows from QueryResponse in both createTableResultFromJsonResponse and queryRpcArrow into a single private helper method to improve maintainability.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces support for executing fast-path queries expecting Arrow-formatted wire responses in BigQueryImpl. It adds the queryRpcArrow method to handle deserialization of Arrow IPC schemas and record batches, configures pagination via ArrowQueryPageFetcher, and updates query and queryWithTimeout to route Arrow-formatted queries appropriately. Comprehensive unit tests are added in BigQueryImplTest to validate these changes. The review feedback highlights potential resource leaks in the newly added tests where BigIntVector instances are not safely closed if an exception occurs during allocation or initialization, recommending the use of try-with-resources to ensure proper cleanup.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request implements support for Arrow-formatted query results in the fast-path query execution path (queryRpcArrow). It refactors existing JSON response parsing into helper methods, updates queryWithTimeout to route Arrow-format queries to the new queryRpcArrow method, and adds comprehensive unit tests to verify Arrow query execution, pagination, and error handling. Feedback was provided to improve the fallback mechanism when the Arrow schema is missing or incomplete by checking if a standard JSON schema is available before throwing an exception.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request adds support for the Arrow query results format in fast-path query execution by implementing queryRpcArrow, which deserializes Arrow IPC schemas and record batches, and handles pagination using ArrowQueryPageFetcher. It also updates the query method to support this fast-path execution while throwing an UnsupportedOperationException for the unsupported slow-path Arrow execution, accompanied by comprehensive unit tests. The review feedback suggests throwing an UnsupportedOperationException instead of returning an incomplete Job when jobComplete is false during Arrow query execution, ensuring consistency since the slow query path is not yet supported.
…pportedOperationException
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request adds support for executing fast-path queries expecting Arrow-formatted wire responses in BigQueryImpl. It introduces the queryRpcArrow method to handle deserialization of Arrow IPC schemas and record batches into standard TableResult representations, including pagination support via ArrowQueryPageFetcher. It also updates queryWithTimeout to route Arrow-format queries appropriately and adds comprehensive unit tests to validate fast-path Arrow queries, multi-page results, serialization, and error handling. There are no review comments, so I have no feedback to provide.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request adds support for executing fast-path query RPC requests expecting Arrow-formatted wire responses in BigQueryImpl. It introduces the queryRpcArrow method to deserialize Arrow IPC schemas and record batches, handles pagination using ArrowQueryPageFetcher, and integrates this flow into the main query execution path. Additionally, comprehensive unit tests have been added to BigQueryImplTest to validate various Arrow query scenarios, including fast-path execution, multi-page results, serialization, and error handling. There are no review comments to evaluate, and the implementation appears complete and well-tested.
1 participant
Enables Apache Arrow wire acceleration for the traditional
BigQuery.query()API returning row-basedTableResult.Part of the BigQuery Apache Arrow support stack. Based on #14404 (page fetcher).