Skip to content

[core] Create primary-key vector exact-fallback readers on demand#535

Draft
JunRuiLee wants to merge 4 commits into
apache:mainfrom
JunRuiLee:feat/pk-vector-lazy-exact-reader
Draft

[core] Create primary-key vector exact-fallback readers on demand#535
JunRuiLee wants to merge 4 commits into
apache:mainfrom
JunRuiLee:feat/pk-vector-lazy-exact-reader

Conversation

@JunRuiLee

Copy link
Copy Markdown
Contributor

Purpose

Part of #514.

Make primary-key vector search create exact-fallback readers on demand instead of eagerly preloading them, mirroring the on-demand reader creation on the Java read path (PrimaryKeyVectorBucketSearch).

Previously the read path preloaded an exact-fallback reader for every ANN-uncovered active file before search ran, holding each file's whole vector column in memory even when recall never reached the exact fallback. The preload was a workaround: the reader factory's create is async, while the bucket-search kernel was synchronous, so the readers had to be awaited up front.

This change makes the exact-reader factory async (returning a boxed future) through bucket_search and the orchestrator, so a reader is built on demand only for a file the fallback actually searches — after the covered / residual / FAST skips have already applied. That removes the builder's eager preload loop and the duplicated "which files need an exact reader" decision, converging it into the one place (the kernel) that already made it.

This is a pure IO/memory-peak optimization: output is unchanged (best-first order, Top-K, _PKEY_VECTOR_SCORE, projected columns, and the fail-loud guards all hold; the baseline fixture is byte-identical).

Brief change log

  • refactor(table): bucket_search becomes async fn; its exact-reader factory returns a boxed future (ExactReaderFuture) instead of a synchronous Result. The factory keeps a Send bound so the resulting search future stays Send for the DataFusion async scan path.
  • refactor(table): thread the async factory through PkVectorOrchestrator::search_candidates.
  • perf(table): delete the builder's eager exact-reader preload loop and the should_preload_exact_reader helper; build readers on demand inside the factory closure (owning all captured data before .await).

Tests

  • Existing kernel/orchestrator tests preserved unchanged as the behavior regression net; the laziness guard asserts the factory is invoked only for ANN-uncovered files (never for covered or residual-empty files).
  • cargo test -p paimon green (1616 lib + integration, 0 failed), including the PK-vector baseline fixture; cargo build -p paimon-datafusion builds (Send path); cargo clippy -p paimon --lib --tests -- -D warnings and cargo fmt --check clean.

API and Format

  • No on-disk format change. No public API change (internal factory seam only).

Documentation

  • Doc comments on the affected functions; no user-facing docs change.

Add "vector recall + scalar WHERE residual filter" to the primary-key
vector read path, mirroring Java PrimaryKeyVectorRead's residual support.

The predicate is pushed into PkVectorScan so the scan prunes whole data
files by their column stats, and is applied per row as a residual over the
surviving files: per-file allowed physical positions are computed once and
threaded into the bucket search as an allow-list. The ANN path intersects
the allow-list into the reader's include-row-ids and re-verifies each
returned hit against it; the exact fallback excludes positions outside the
allow-list and skips files with no allowed rows without opening a reader.

Fail loud rather than answer incorrectly: a filter off the primary-key
vector path, a filter without deletion-vectors/merge-on-read guarantees, an
ANN hit outside the pre-filter, and a residual position past a source
file's row count are all rejected.
The primary-key vector read path eagerly preloaded an exact-fallback
reader for every ANN-uncovered active file before search ran, holding
each file's whole vector column in memory even when recall never reached
the exact fallback. Preloading was a workaround: the reader factory's
`create` is async while the bucket search kernel was synchronous.

Make the exact-reader factory async (returning a boxed future) through
`bucket_search` and the orchestrator, so a reader is built on demand only
for a file the fallback actually searches, after the covered / residual /
FAST skips already applied. This removes the builder's eager preload loop
and the duplicate preload decision, converging it into the kernel and
mirroring the on-demand reader creation on the Java read path. Output is
unchanged: best-first order, Top-K, scores, and the fail-loud guards all
hold, and the baseline fixture is byte-identical.

The factory closure keeps a Send bound so the resulting search future
stays Send for the DataFusion async scan path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant