feat: low-precision float types bf16/float16/float8/float4 and vecf8/vecf4 vector columns (#20567)

[cpegeric]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #20567

What this PR does / why we need it:

Low-precision float types: scalar bf16/float16/float8/float4, and block-scaled vector columns vecf8(N) (MXFP8) / vecf4(N) (NVFP4).

Draft: the vector_matmul / vector_matmul_merge batch search (CPU, then cuBLASLt GPU) continues on this branch.

Scalar bf16 / float16 / float8 (E4M3) / float4 (E2M1)

Design: docs/design/20260929-low-precision-scalar-floats.md

• Type ids, parser, container, casts via a float32 bridge, arithmetic/comparison/aggregates, ORDER BY, TAE zonemap and value codec, MySQL protocol, LOAD.
• Float8/Float4 codecs pinned bit-exactly to CUDA cuda_fp8.h/cuda_fp4.h by a generated golden test.

Vector columns vecf8(N) / vecf4(N)

Design: docs/design/20260930-low-precision-vector-storage.md

• Cell: 12-byte header (version, format, dim, fp32 global scale g), then block scales (E8M0 per 32 for vecf8; UE4M3 per 16 for vecf4), then elements. Value = g × block scale × element; the layout matches cuBLASLt's block-scaled matmul inputs.
• DDL: column types in CREATE/ALTER; rejected in primary key, partition key, and every index.
• Casts: text ↔ vecf8/vecf4, vecf32 ↔ vecf8/vecf4.
• Arithmetic + - * / promotes to vecf32.
• Distances: inner_product, l2_distance, l2_distance_sq, l1_distance, cosine_distance, cosine_similarity between vecf8/vecf4/vecf32 in any combination; a text literal binds as vecf32.
• Also vector_dims, normalize_l2, ANY_VALUE, COUNT, GROUP_CONCAT, ORDER BY / GROUP BY / window.
• Output, export, CDC/ISCP replication, and LOAD (CSV, Parquet LIST<FLOAT/DOUBLE> and text columns).
• Distance kernels (pkg/vectorindex/metric/distance_func_vecblock*.go): scalar Go, no SIMD. They are generated (mkvecblock.go), one per operand pair and metric, fully unrolled over 16-element units, with table-lookup decode. 768-d, one row: 200–311 ns for dot/L2, 303–437 ns for L1/cosine.

perf(metric): clear AVX upper state after archsimd kernels

The archsimd distance kernels returned with dirty upper zmm state. The legacy-SSE scalar float code that follows them then carries a false dependency on those bits.

• Fix: archsimd.ClearAVXUpperBits() after the reduction in 40 kernels. The 4 bf16 AVX2 kernels are excluded (+17% there, no gain after them).
• Effect, 768-d, Ryzen AI 9 365:
  • vecf8 dequantize ×2 + f32 dot product: 1702 → 364 ns.
  • Scalar loop after an AVX-512 kernel: −7.6%.
  • f16 AVX2 l2sq / l1: −20% / −11%.

Tests

• Unit tests: codecs (incl. CUDA golden), parse/validation, decode paths, kernels against a float64 reference at tail dims and in both argument orders, SQL functions, casts, frontend/LOAD/CDC paths, zonemap/sort/compare.
• BVT: dtype/vecblock.sql (65 statements, including flush, prepared statements and rejections), plus the scalar low-precision cases.
• Regression (mo-tester -m run -g -n), all 0 failed:

Directory	Passed	Ignored
dtype	7729/7761	32
array	1501/1501	0
vector	2604/2606	2

🤖 Generated with Claude Code

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[XuPeng-SH]

Decision: REQUEST_CHANGES — value contracts are still not closed

Reviewed d4822f2e307cfc6c76e9bf3792f5714f5af24e4b against base/merge-base 43c37847b16d37cfa8b9eef4235ba4388b5e25ce. The architecture has useful reuse: codecs belong in types, top-k partial/merge/spill use the existing aggregate framework, and GPU execution stays behind the engine boundary. The three inline findings are concrete correctness defects in the new value contracts, not requests for additional compatibility machinery.

Confirmed current-head failures

1. P1: NVFP4 replication/reload changes already-stored values. A 17-element vector with only elements 0 and 16 nonzero (8.7649145, 5.7432985) stores the latter as 6.2606535. Production CDC emits the decoded text; reloading it stores 6.8867183, approximately 10% additional drift. Both the production CDC extraction/SQL formatter and a fresh single-CN MySQL-protocol insert/select/reinsert reproduce this. This is additional loss during transport, not the permitted initial quantization.
2. P2: equality disagrees with grouping/distinct. Real SQL says CAST('[1.2031566]' AS vecf4(1)) = CAST('[1.2031565]' AS vecf4(1)) is true, but storing those values produces 2 GROUP BY groups and 2 DISTINCT values. MXFP8 also fails: [447] and [449] both decode to [448], compare equal, and still produce 2 groups. A same-encoding vecf8 neighbour returns 1 group as expected. Deterministic encoding of one input does not establish a canonical encoding for all equal decoded values.
3. P2: ARM64 fused decoding breaks distance invariants. With normal Go 1.26.4 compilation, a valid vecf4(16) containing sixteen 9.9999994e29 values gives VecBlockL2DistanceSq(v,v) = +Inf; real SQL l2_distance_sq(v,v) returns error 20101. Casting that same stored vector to vecf32 first returns 0. Smaller values also have nonzero self-distance. The new TestVecBlockKernelsSymmetric fails deterministically (3/3); package-local -d=fmahash=n makes both symmetry and the self-distance probe pass. This is a production decode/rounding boundary defect, not merely an overstrict test.

Resolve these at the existing owners: preserve the stored cell/value across transport; extend the shared equality-key contract rather than adding an alternate grouping path; make the kernel generator's decode rounding match the stored-value contract rather than adding self-distance special cases or globally disabling FMA.

Historical review disposition

• My three old inline threads are now resolved/outdated. The current encoder/parser rejects decoded infinity; host staging/tile admission uses the aggregate account; device selection filters the declared compute-capability baseline. Current finiteness and fake-engine admission/cleanup tests pass.
• iamlinjunhong's select-list finding is fixed; TestNarrowCastHonorsSelectList passes.
• The previous NVFP4 K=48/no-algorithm concern is addressed in source by padding K to 32 and preflighting tile algorithms, together with an explicit eligible GPU => execute or fail decision. That decision can be reasonable; it is different from a CPU-fallback promise. The design still says fallback on memory denial at lines 269 and 524, contradicting its final no-fallback decision and implementation. Reconcile those paragraphs and the stale draft/merge references in the PR body. No real GPU execution is claimed here.
• fengttt's bitwise-format question has an author answer distinguishing CUDA-compatible element/nibble encoding from the custom block/global quantizer. These are not the same assertion; keep that distinction in the final contract. The current CPU types suite, including checked-in CUDA golden verification, passes.

QA, test quality and scope

Fresh CPU native build succeeded. Relevant types/CDC/aggregate/function tests passed; explicit skipped-row CAST passed; the aggregate owning package passed once under -race, and TestVectorMatmulConfigShared passed 100/100 race repetitions. Full types/compare/sort suites passed. The metric suite did not pass, for the symmetry failure above. Reviewer-only overlays and logs were kept outside the clean review worktree; no production code was changed and no subagent was used.

The existing text round-trip check only proves that reparsing succeeds, not that the stored value is preserved. CDC/ISCP fixtures use exact small values, so they miss scale-boundary drift. Add small orthogonal oracles for transport identity, equality-versus-key agreement in both formats, and decoded self-distance with FMA enabled. Do not fix this by deleting the failing symmetry case, widening every tolerance, or piling up repetitions of the exact-grid fixtures.

The 188-file change was mapped by responsibility, with implementation, tests, documentation and generated code considered separately. The mandatory design gate remains blocked by these counterexamples: this is not a blanket acceptance of every remaining hunk, unhappy path, GPU performance claim or device configuration. GPU benchmarks/real-device validation were not rerun on this CPU-only host. Stop-the-cluster upgrades are the scope; mixed-version rolling compatibility is not a blocker, but persisted-value and replication correctness remain required.

[cpegeric]

fixed

[cpegeric]

Thanks for the review. All three are fixed, each with a test that fails without the fix.

P1: NVFP4 replication drift. Fixed by carrying the stored cell instead of its decoded values.

• New exact text form, vecblock_json(v): {"g":g,"s":[block scales],"v":[element values]} (904b890, shape finalized in
  c31f5d0).
  • Every number is a code value of its format: E8M0/UE4M3 scales, E4M3/E2M1 elements.
  • Parsing it rebuilds the same cell bytes with no quantization, and anything that isn't an exact code value is rejected, never
    rounded.
  • CAST, INSERT and LOAD all accept it.
• Transport now uses it. CDC and ISCP replication SQL, data branch merge, INTO OUTFILE / external writes (CSV and JSONL), MODUMP
  QUERY_RESULT and the checkpointtool dump all emit the exact form (904b890, b03c4e3, 6794455). Your 17-element vector
  replays bit for bit, and element 16 stays at 6.2606535.
• Exact binary form too, vecblock_binary(v) (836a8ca): a BLOB of a valid cell casts back to the same bytes. Parquet binary
  columns load cells exactly (4ef9ad3), and vector_matmul accepts exact query cells via "query_format":"vecblock"
  (10550e2).
• Query output still shows the decoded values.
• Tests: CDC/ISCP/data-branch identity tests with your vector, TestBlockScaledJSONIdentity, and BVT export→LOAD and Parquet round
  trips.

P2: equality vs GROUP BY / DISTINCT. Fixed: every equality consumer now agrees with =.

• Hash keys (d0064a5): the vecf8/vecf4 key in keycodec is now the decoded float32 values with signed zero unified.
  • This covers GROUP BY, DISTINCT, hash joins, set operations, IN (subquery), COUNT(DISTINCT) and approx_count_distinct.
  • So cells that encode the same values with different scales are one key: vecf8 [447] and [449] both store 448.
  • SAMPLE on vecf8 is fixed as well.
• Peer groups (5f5fb57): ORDER BY ties and window PARTITION BY / ORDER BY now compare decoded values instead of raw bytes.
• Tests: TestCanonicalVecBlockKeyMatchesEquality (keys agree with CompareBlockScaledFromBytes), TestBlockScaledPartition, and
  vecblock BVT cases for GROUP BY, DISTINCT, joins, UNION, IN, rank() OVER and PARTITION BY on byte-different, value-equal cells.

P3: arm64 FMA decode. Fixed in 76771e7.

• Each decoded element is now rounded once to float32 with an explicit conversion (float32(code * scale)), which Go never fuses.
  So the kernels decode exactly as At() and the equality key do, on every architecture.
• Kernel generator updated and the kernels regenerated.
• Proof: cross-compiling for arm64, the fused multiply-adds in the decode paths are gone; FMADDS/FMSUBS fell from 560 to 400
  across the kernels. The ones left are in the accumulation, where fusing is the same as for vecf32.
• Test: TestVecBlockSelfDistance checks that a stored vector is at distance 0 from itself, including values near the float32
  maximum (9.9999994e29).

A self-review on top of these also fixed:

• Cosine: a float32 norm overflow or underflow is now recomputed in float64, matching vecf32 (8a1ebd1).
• bf16/float16/float8/float4 text and comparisons:
  • text and float64 values round once (81b54d1);
  • numeral text literals and prepared IN lists round to these types (5c37d1a).
• User variables keep vecf8/vecf4 cells (7083782).