Convert FixedSizeBinary scalars to fixed-size lists, not Binary

[yiquanzhou]

Problem

scalar_from_df folds ScalarValue::FixedSizeBinary into the same match arm as the variable-width binary variants, binding the width to _:

ScalarValue::Binary(b) | ScalarValue::BinaryView(b) | ScalarValue::LargeBinary(b)
| ScalarValue::FixedSizeBinary(_, b) => b
    .as_ref()
    .map(|b| Scalar::binary(ByteBuffer::from(b.clone()), Nullability::Nullable))

The result is a DType::Binary scalar. Compared against a column whose storage dtype is FixedSizeList(u8, N) — notably any UUID-typed column, whose Arrow form is FixedSizeBinary(16) — the literal reaches the compare kernel with a dtype that cannot match:

Cannot compare different DTypes fixed_size_list(u8)[16]? and binary?

This is not a planning problem. DataFusion builds a correctly typed literal — arrow_typeof(arrow_cast(X'…','FixedSizeBinary(16)')) returns FixedSizeBinary(16) — and the type is discarded in conversion.

We hit this filtering an OpenTelemetry trace ID stored as a UUID-tagged FixedSizeBinary(16). Every natural spelling fails:

WHERE trace_id = X'fa10832799d45f3de907df151c47fed9'
WHERE trace_id = arrow_cast(X'fa10832799d45f3de907df151c47fed9', 'FixedSizeBinary(16)')
WHERE arrow_cast(trace_id, 'Binary') = X'fa10832799d45f3de907df151c47fed9'

Two probes against the same table isolate it to the literal path — the kernel and the column type are both fine:

-- ✓ 934480 rows: fixed-size list compares against itself, pushed down
WHERE trace_id = trace_id
-- ✓ 4 rows: a runtime value of the column's own type works
WHERE trace_id = (SELECT trace_id FROM logs WHERE span_id = 17071181531735081315 LIMIT 1)

The only workaround we found is encode(arrow_cast(trace_id, 'Binary'), 'hex') = '…', which works precisely because encode is not translatable, so the predicate never reaches Vortex: the whole column is decoded and filtered above the scan, giving up pruning.

Fix

Give ScalarValue::FixedSizeBinary its own arm, building a fixed-size list of non-nullable u8.

Two details worth calling out:

• The element dtype must be non-nullable u8. That is what Uuid::validate_dtype requires of its storage, and DType equality compares list element dtypes down to their own nullability, so a nullable element would still fail to match.
• No extension scalar is constructed. compare_arrays already unwraps a lone extension side against its raw storage, so a plain fixed-size list literal matches an Extension(vortex.uuid) column as well as a bare FixedSizeList one. This also keeps the conversion free of any session/registry lookup.

Scalar::fixed_size_list derives the list size from the children it is handed, so the value length is checked against the declared width rather than silently producing a differently typed scalar.

Tests

test_binary_variants asserted the previous behaviour (FixedSizeBinary(5, …) → as_binary()), so that case is removed and replaced by three tests: the new dtype, the null case preserving its width, and agreement with the dtype from_arrow_field produces for a UUID-tagged FixedSizeBinary(16) field.

cargo test -p vortex-datafusion passes (313 + 14). cargo fmt clean.

Open questions

• Scope. from_arrow_data_type deliberately rejects bare FixedSizeBinary as a column type, with a regression test for DType::from_arrow panics with unimplemented! for Duration, Interval and FixedSizeBinary instead of returning an error #8346. This PR only changes how a scalar literal converts, on the grounds that the sole sensible Vortex representation of a fixed-width binary value is the fixed-size list that UUID already stores — but if you'd rather see FixedSizeBinary handled uniformly (or still rejected, loudly, instead of silently becoming Binary), say so and I'll rework it.
• Reverse direction. try_to_df still bails on DType::FixedSizeList, so a FixedSizeBinary scalar does not round-trip. Happy to add the inverse arm here or in a follow-up — left out to keep this focused on the bug.

Opened as a draft per the contributing guide, since there's no issue for it yet. Glad to file one if you'd prefer that first.

AI assistance

Disclosed per the AI policy: this change was investigated and drafted with Claude (Claude Code), working from the failing queries above. The diagnosis, the code, and the tests were reviewed by me before submitting, and the behaviour was verified against a real Vortex-backed table.