[core] Create primary-key vector exact-fallback readers on demand#535
Draft
JunRuiLee wants to merge 4 commits into
Draft
[core] Create primary-key vector exact-fallback readers on demand#535JunRuiLee wants to merge 4 commits into
JunRuiLee wants to merge 4 commits into
Conversation
Add "vector recall + scalar WHERE residual filter" to the primary-key vector read path, mirroring Java PrimaryKeyVectorRead's residual support. The predicate is pushed into PkVectorScan so the scan prunes whole data files by their column stats, and is applied per row as a residual over the surviving files: per-file allowed physical positions are computed once and threaded into the bucket search as an allow-list. The ANN path intersects the allow-list into the reader's include-row-ids and re-verifies each returned hit against it; the exact fallback excludes positions outside the allow-list and skips files with no allowed rows without opening a reader. Fail loud rather than answer incorrectly: a filter off the primary-key vector path, a filter without deletion-vectors/merge-on-read guarantees, an ANN hit outside the pre-filter, and a residual position past a source file's row count are all rejected.
The primary-key vector read path eagerly preloaded an exact-fallback reader for every ANN-uncovered active file before search ran, holding each file's whole vector column in memory even when recall never reached the exact fallback. Preloading was a workaround: the reader factory's `create` is async while the bucket search kernel was synchronous. Make the exact-reader factory async (returning a boxed future) through `bucket_search` and the orchestrator, so a reader is built on demand only for a file the fallback actually searches, after the covered / residual / FAST skips already applied. This removes the builder's eager preload loop and the duplicate preload decision, converging it into the kernel and mirroring the on-demand reader creation on the Java read path. Output is unchanged: best-first order, Top-K, scores, and the fail-loud guards all hold, and the baseline fixture is byte-identical. The factory closure keeps a Send bound so the resulting search future stays Send for the DataFusion async scan path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Part of #514.
Make primary-key vector search create exact-fallback readers on demand instead of eagerly preloading them, mirroring the on-demand reader creation on the Java read path (
PrimaryKeyVectorBucketSearch).Previously the read path preloaded an exact-fallback reader for every ANN-uncovered active file before search ran, holding each file's whole vector column in memory even when recall never reached the exact fallback. The preload was a workaround: the reader factory's
createisasync, while the bucket-search kernel was synchronous, so the readers had to beawaited up front.This change makes the exact-reader factory async (returning a boxed future) through
bucket_searchand the orchestrator, so a reader is built on demand only for a file the fallback actually searches — after the covered / residual / FAST skips have already applied. That removes the builder's eager preload loop and the duplicated "which files need an exact reader" decision, converging it into the one place (the kernel) that already made it.This is a pure IO/memory-peak optimization: output is unchanged (best-first order, Top-K,
_PKEY_VECTOR_SCORE, projected columns, and the fail-loud guards all hold; the baseline fixture is byte-identical).Brief change log
refactor(table):bucket_searchbecomesasync fn; its exact-reader factory returns a boxed future (ExactReaderFuture) instead of a synchronousResult. The factory keeps aSendbound so the resulting search future staysSendfor the DataFusion async scan path.refactor(table): thread the async factory throughPkVectorOrchestrator::search_candidates.perf(table): delete the builder's eager exact-reader preload loop and theshould_preload_exact_readerhelper; build readers on demand inside the factory closure (owning all captured data before.await).Tests
cargo test -p paimongreen (1616 lib + integration, 0 failed), including the PK-vector baseline fixture;cargo build -p paimon-datafusionbuilds (Send path);cargo clippy -p paimon --lib --tests -- -D warningsandcargo fmt --checkclean.API and Format
Documentation