Skip to content

[core] Align primary-key vector source metadata with Java#531

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JunRuiLee:feat/pk-vector-source-meta-drift
Jul 17, 2026
Merged

[core] Align primary-key vector source metadata with Java#531
JingsongLi merged 1 commit into
apache:mainfrom
JunRuiLee:feat/pk-vector-source-meta-drift

Conversation

@JunRuiLee

@JunRuiLee JunRuiLee commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • parse Java's current primary-key vector source metadata frame with dataLevel
  • filter current ANN segments by exact level source coverage, matching Java PkVectorBucketIndexState
  • update PK-vector source-meta fixtures and tests

Motivation

This follows the Java-side PK-vector source metadata changes in apache/paimon#8672.

Java now writes dataLevel into GlobalIndexMeta.sourceMeta between the frame version and source-file count. Without this update, Rust reads dataLevel as the source-file count and cannot correctly consume Java-written PK-vector metadata.

Java also selects current ANN payloads by exact level coverage: active compacted files are grouped by data level, each segment's sourceMeta.dataLevel() must match one level, and the segment's source files must exactly equal that level's active files. Mirroring that rule keeps Rust from searching stale or partially-overlapping ANN segments after compaction rewrites files.

Tests

  • cargo fmt --check
  • cargo test -p paimon pk_vector_source --lib
  • cargo test -p paimon pk_vector_scan --lib
  • cargo test -p paimon vindex::pkvector::bucket --lib
  • cargo test -p paimon vindex::pkvector::ann --lib
  • cargo test -p paimon pk_vector_orchestrator --lib
  • cargo test -p paimon --test pk_vector_baseline_test
  • cargo clippy -p paimon --all-targets -- -D warnings

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 22a13b6 into apache:main Jul 17, 2026
12 checks passed
@JunRuiLee
JunRuiLee deleted the feat/pk-vector-source-meta-drift branch July 17, 2026 03:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants