[core] Support scalar residual filter on primary-key vector search

[JunRuiLee]

Purpose

Part of #514.

Add scalar WHERE residual-filter support to the primary-key vector search read path, mirroring Java PrimaryKeyVectorRead's residual support. A caller can attach a data-column predicate via VectorSearchBuilder::with_filter; vector recall is then narrowed so only rows satisfying the predicate are returned, while best-first ordering and Top-K still hold.

The predicate is used two ways:

• Pushed into the scan (PkVectorScan), where it prunes whole data files by their column stats (with_scan_all_files() is retained so level-0 files still reach the exact fallback).
• Applied per row as a residual over the surviving files: per-file allowed physical positions are computed once and threaded into the bucket search as an allow-list. The ANN path intersects the allow-list into the reader's include-row-ids and re-verifies each returned hit against it; the exact fallback excludes positions outside the allow-list and skips files with no allowed rows without opening a reader.

To answer correctly rather than silently mis-answer, the following all fail loud: a filter off the primary-key vector path, a filter without deletion-vectors-enabled + merge-on-read-disabled, an ANN hit resolving outside the pre-filter, and a residual position past a source file's row count.

Brief change log

• feat(spec): add CoreOptions::deletion_vectors_merge_on_read accessor (default false), used by the residual guard.
• feat(table): residual filter support across the PK-vector read path — bucket-search allow-list (ANN + exact fallback), per-file residual position computation, PkVectorScan filter pushdown for file-level stats pruning, map_ann_results post-verification of ANN hits against the allow-list, out-of-range residual position fails loud, and skipping exact-reader preload for files with an empty residual set.
• test(table): end-to-end coverage of the residual filter on the baseline PK-vector fixture.

Tests

• Unit: build_live_row_ids (residual∩active∩¬DV intersection, cross-file offsets, out-of-range fail-loud), map_ann_results (rejects a hit outside the residual allow-list), bucket_search exact residual (allow-list, absent/empty-entry skip, DV intersection), should_preload_exact_reader, residual_positions_by_file (file-local positions, empty/all/none, non-active skip, missing first_row_id), PkVectorScan::plan file-level pruning on real tables (a primary-key-column predicate without deletion vectors, and a non-primary-key-column predicate under deletion vectors + no-merge-on-read), and the fail-loud guards.
• Integration: pk_vector_baseline_test — unfiltered vs residual searches differ and every residual hit satisfies the predicate.
• cargo test -p paimon green (1617 lib + integration, 0 failed); cargo clippy -p paimon --lib --tests -- -D warnings and cargo fmt --check clean.

API and Format

• No on-disk format change.
• Public API: VectorSearchBuilder::with_filter(Predicate) now consumed by the primary-key vector path.
• Note: file-level stats pruning reads a data file's value_stats. The Rust primary-key writer currently stores column stats in key_stats and leaves value_stats empty, so file-level pruning fires on Java-written tables (the read target of Support reading primary-key vector (bucket-local ANN) search in paimon-rust #514) but not yet on Rust-written primary-key tables. Populating writer-side value_stats is a separate, pre-existing gap outside this change; correctness is unaffected either way because the per-row residual still applies over surviving files.

Documentation

• Doc comments on the affected functions; no user-facing docs change.