Skip to content

perf(blob): parallelize descriptor range reads#534

Merged
JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/blob-descriptor-bounded-async-read
Jul 17, 2026
Merged

perf(blob): parallelize descriptor range reads#534
JingsongLi merged 4 commits into
apache:mainfrom
JingsongLi:codex/blob-descriptor-bounded-async-read

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

What changed

  • Resolve BlobDescriptor columns and URI range groups concurrently.
  • Share a reader-lifetime limiter across metadata and range I/O.
  • Cap active I/O at 8 requests and range admission at 64 MiB.
  • Preserve row and column order, with deterministic concurrency tests.

Why

External BlobDescriptor materialization awaited descriptor columns, URI groups, and merged ranges serially. On high-latency object storage, those round trips accumulated across a batch. This change adds bounded asynchronous concurrency while retaining the existing per-URI range coalescing.

Impact

This improves latency and throughput for batches containing multiple non-coalescible external descriptors while bounding request pressure and admitted range bytes. The ordinary .blob reader and public configuration remain unchanged.

Validation

  • cargo test -p paimon
  • cargo clippy -p paimon --all-targets -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

@JingsongLi
JingsongLi marked this pull request as ready for review July 17, 2026 06:06

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest update resolves the Clippy failure. I rechecked the bounded parallel descriptor-read implementation and the new concurrency coverage, and found no correctness issues. The targeted overlap test and formatting checks pass locally.

@JingsongLi
JingsongLi merged commit ae53bbf into apache:main Jul 17, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants