perf(btreemap): add insert_many for batched insertion#440
Open
hpeebles wants to merge 9 commits into
Open
Conversation
`insert` writes the target node to stable memory on every call, and when
a node splits it writes the new sibling and the parent too. Inserting a
run of keys therefore rewrites the same nodes repeatedly: building a
10k-entry map one key at a time performs 15,976 node writes across just
1,997 distinct nodes.
`insert_many` keeps the current root-to-leaf path in memory and writes a
node out only once the input has moved past it, bringing those same 10k
keys down to 4,032 writes. The header is likewise written once per call
rather than once per key.
map.insert_many((0..1_000_000).map(|i| (i, compute(i))));
The iterator is consumed lazily and never collected, so the memory cost
is one tree path however many pairs are supplied.
## Ordering
Coalescing pays off when consecutive keys land in the same node, so keys
should be supplied in sorted order. Direction is irrelevant: a
descending run unwinds the path leftwards instead of rightwards and
produces an identical write count.
Out-of-order keys stay correct rather than corrupting the tree. Each
level of the held path records the key range its node owns at both ends,
so a key arriving early unwinds to an ancestor that covers it, as far as
the root if necessary. Tracking the lower bound as well as the upper one
costs a comparison per key.
## Measurements
canbench, 10k u64 pairs each:
ascending, empty map 334.19M -> 76.91M instructions (4.3x)
descending, empty map 386.65M -> 96.43M instructions (4.0x)
unordered, caller sorts 356.43M -> 86.45M instructions (4.1x)
unordered, fed as-is 356.43M -> 341.07M instructions (1.0x)
ascending run into 100k 294.07M -> 49.06M instructions (6.0x)
The sort in the third row is inside the measured section. Unordered
input fed straight in forfeits the coalescing, but is not slower than
the insert loop it replaces.
## Implementation notes
Holding the whole path rather than only the leaf is what makes this
work. Two thirds of the writes in a split-heavy workload are the sibling
and the parent, and a held parent absorbs many child splits before it is
written; holding only the leaf coalesces 2x rather than 4x.
Node splits are handled in place while the path is held, except in two
cases that fall back to the ordinary insert path: a full root leaf, and
a parent with no room for a promoted median. Those fire 277 times per
10k ascending keys, about one insert in 36. A fallback flushes the path,
so nodes it had buffered are written again when the batch re-descends
into them; that accounts for the whole gap between the 4,032 writes
above and the 1,997 that writing each dirty node exactly once would
take. Handling the split cascade in place would close it, at the cost of
considerably more intricate code in the riskiest part of the tree, so it
is left as a possible follow-up.
The batch machinery lives in `src/btreemap/bulk_insert.rs`, leaving
`BTreeMap::insert_many` in `btreemap.rs` as a thin wrapper over it.
Dropping `BulkInsert` writes back whatever it still holds, which is how
a batch is completed: the drop is load-bearing on every call.
`insert` is unchanged; its body moves into `insert_serialized` so the
fallback can reuse it with an already-serialized value. All 252
pre-existing benchmarks are unchanged to within noise.
## Testing
Differential proptests and unit tests build the same map with repeated
`insert` and with `insert_many`, and require the results to be
indistinguishable: across node cache sizes 0/1/16, the V1,
V1-migrated-to-V2 and V2 layouts, unbounded values large enough to spill
onto overflow pages, duplicate and overwriting batches, and batch sizes
chosen to hit both fallbacks. Ordering is covered by ascending,
descending, zigzag, sawtooth and shuffled inputs. Allocator chunk counts
return to zero after draining a map built by `insert_many`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
|
|
|
A leaf split promotes a median into its parent, so a full parent used to end the batch: the path was flushed and the ordinary insert path grew the tree instead. Every node the path had buffered was then written again on the way back down, which cost roughly half of the writes `insert_many` was saving. The split now cascades. `split_leaf` walks up to the topmost full level, grows a new root if the whole path is full, then splits top-down: each split leaves its node half empty, so the level below always has room for the median it promotes. Writing each dirty node exactly once is now reached rather than approached — 10k ascending keys perform 1,997 writes, matching the number of distinct nodes in the resulting tree, down from 4,032. An `insert` loop over the same keys performs 15,976. ascending, empty map 77.17M -> 35.88M instructions (4.3x -> 9.3x) unordered, caller sorts 86.71M -> 45.42M instructions (4.1x -> 7.8x) descending, empty map 96.58M -> 52.53M instructions (4.0x -> 7.4x) Nothing else moves: 262 benchmarks, 0 regressed, 3 improved. The workloads that barely split — inserting into gaps in an existing map, pure overwrites, unordered input — are unchanged, which is where the saving was expected to come from. The levels below a split keep the bounds they already hold: the separators either side of them survive, landing in one half or the other. Only their index among the parent's children moves, by the number of children that stayed with the left half. Removing the fallback also removes the last call from `BulkInsert` back into `BTreeMap::insert`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Splits a batch into four canbench scopes behind the existing
`bench_scope` feature, matching what `node/v1.rs` and `node/v2.rs`
already do for node loads and saves:
bulk_insert_release a node leaving the path, written or cached
bulk_insert_descend taking one child out of the cache
bulk_insert_leaf placing the entry, including any split
bulk_insert_split the split cascade, nested in the previous
The feature is off in the benchmark build, so the default wasm is
unchanged: re-running the `insert_many` benchmarks without it reports
`change: max 0` against the committed results.
The immediate use is attributing the gap between ascending and
descending input, which is otherwise only inferable. Over 10k keys the
whole 16.4M difference sits in `bulk_insert_leaf`, while writes and path
management are identical:
asc desc delta
bulk_insert_leaf 44.51M 60.93M +16.42M
of which split 25.60M 28.35M +2.75M
placing the entry 18.91M 32.58M +13.67M
bulk_insert_release 0.07M 0.07M +0.00M
node_save_v2 11.99M 12.00M +0.01M
Ascending keys append to a node's entry vector; descending keys insert
at index 0 and shift everything after it. The 2.75M charged to the split
is the same shift promoting a median into a parent. Nothing else moves —
`node_save_v2` confirms both directions write identically, as the node
counts already showed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`insert_into_leaf` searched the leaf for the insertion point, then searched it a second time before inserting. The second search exists because a split invalidates the first answer, but it ran on every insert, including the majority where no split happens and nothing has touched the leaf in between. Reuse the first answer when the leaf was not full, and re-search only after a split. ascending, empty map 35.88M -> 34.23M instructions (-4.6%) ascending run into 100k 49.33M -> 47.37M instructions (-4.0%) unordered, caller sorts 45.42M -> 43.76M instructions (-3.7%) descending, empty map 52.53M -> 50.88M instructions (-3.2%) 262 benchmarks, 0 regressed. The overwrite and unsorted batches are unchanged, as expected: overwriting returns on the first search without ever reaching the second, and unsorted input is dominated by re-descents rather than by anything in the leaf. The `bench_scope` scopes added in the previous commit are what made this visible — `bulk_insert_leaf` minus `bulk_insert_split` isolates the cost of placing an entry, and it was larger than a single binary search over at most eleven keys could account for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`split_level` re-indexes the level below when it travels into the right half, by subtracting the number of children the left half kept. That the subtraction cannot underflow rests on a several-step argument — the level below covers the key, the key is above the median, so its child index was at least the left half's child count — which is now stated where it is relied upon rather than left to be re-derived. Behaviour is unchanged; the assertion is compiled out of release builds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The existing tests check behaviour — iteration matches `std`, lookups agree, draining returns every allocator chunk — so they notice a broken tree only through its consequences. `insert_many` manipulates the shape itself when it splits a node and cascades the split upwards, which is worth checking directly. `assert_btree_invariants` walks the tree out of memory and asserts that every node is within its occupancy bounds, that keys are sorted and fall strictly between the separators either side of them in the parent, that an internal node has one more child than it has entries, that every leaf sits at the same depth, and that the entries counted match the length in the header. The driver interleaves batches of varying shape and ordering with single inserts and removes, walking the tree after every step. The value is in where a failure surfaces. Injecting a wrong child index after a split, a bound left too wide, a cascade run in the wrong direction, or a dropped ordering check all fail here at the node that broke, rather than downstream as a key that came back missing. `B` and `CAPACITY` become `pub(super)` so the occupancy assertions use the real bounds instead of a copy of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
insertwrites the target node to stable memory on every call, and when a node splits it writes the new sibling and the parent too. Inserting a run of keys therefore rewrites the same nodes repeatedly: building a 10k-entry map one key at a time performs 15,976 node writes across just 1,997 distinct nodes.insert_manykeeps the current root-to-leaf path in memory and saves a modified node to stable memory only once the input has moved past it. The same 10k keys come down to 1,997 writes — exactly the number of distinct nodes, so every node is written once and none is rewritten. The header is likewise written once per call rather than once per key.The iterator is consumed lazily and never collected, so the memory cost is one tree path however many pairs are supplied.
BTreeSet::insert_manydelegates to it.Ordering
Coalescing pays off when consecutive keys land in the same node, so keys should be supplied in sorted order. Either direction works: a descending run unwinds the path leftwards instead of rightwards and performs an identical number of writes.
Descending does cost more CPU, as the table below shows — not from any extra I/O, but because an ascending key appends to a node's entry vector while a descending one inserts at index 0 and shifts everything after it. The
bench_scopebuild attributes the whole difference to that shift.Out-of-order keys stay correct rather than corrupting the tree. Each level of the held path records the key range its node owns at both ends, so a key arriving early unwinds to an ancestor that covers it, as far as the root if necessary. Tracking the lower bound as well as the upper one costs a comparison per key.
Those recorded ranges are also what lets the fast path skip re-searching the ancestors it is still holding. A key stored in an ancestor is one of that ancestor's separators, and the separators are exactly the exclusive bounds handed down to the children, so a key that any held node covers cannot be sitting in a node above it.
Measurements
canbench. The first five rows insert 10k
u64pairs; the last overwrites 2k existing keys whose 1 KiB values live on V2 overflow pages.insertloopinsert_manyThe sort in the third row is inside the measured section. Unordered input fed straight in forfeits the coalescing, but is not slower than the insert loop it replaces.
The overwrite row gains from a second effect.
inserthas to read the displaced value back in order to return it, which for a value on overflow pages means reading it out of stable memory;insert_manyreturns nothing, so the newNode::set_valuereplaces the value without loading the old one. Measured separately, coalescing alone accounts for 74.29M -> 14.64M and skipping the discarded reads takes it the rest of the way to 10.08M.Implementation notes
Holding the whole path rather than only the leaf is what makes this work. Two thirds of the writes in a split-heavy workload are the sibling and the parent, and a held parent absorbs many child splits before it is written; holding only the leaf coalesces 2x rather than 9x.
Splits are handled entirely in place. A leaf split promotes a median into its parent, which may itself be full, so the split cascades:
split_leafwalks up to the topmost full level, grows a new root if the whole path is full, then splits top-down — each split leaves its node half empty, so the level below always has room for the median it promotes. The levels below a split keep the bounds they already hold, since the separators either side of them survive and simply land in one half or the other; only their index among the parent's children moves, by the number of children that stayed with the left half.The batch machinery lives in
src/btreemap/bulk_insert.rs, leavingBTreeMap::insert_manyinbtreemap.rsas a thin wrapper over it. DroppingBulkInsertwrites back whatever it still holds, which is how a batch is completed: the drop is load-bearing on every call.A batch is split into four scopes —
bulk_insert_release,bulk_insert_descend,bulk_insert_leafandbulk_insert_split— behind the existingbench_scopefeature, asnode/v1.rsandnode/v2.rsalready do for node loads and saves. The feature is off in the benchmark build, so the default wasm is unchanged and the committed results re-run atchange: max 0.The scopes have already paid for themselves:
bulk_insert_leafminusbulk_insert_splitisolates the cost of placing an entry, and it was far larger than a binary search over at most eleven keys could explain — which turned out to be a second, redundant search of the leaf on every insert that did not split.insertis untouched — the diff tobtreemap.rsis purely additive.BulkInsertcarries the value unserialized and encodes it at the point of use. The new benchmarks name their plain-loop baselinesinsert_loop_*so they read as the counterparts ofinsert_many_*. All 252 pre-existing benchmarks are unchanged to within noise.Public API added
BTreeMap::insert_manyBTreeSet::insert_manyTesting
Differential proptests and unit tests build the same map with repeated
insertand withinsert_many, and require the results to be indistinguishable: across node cache sizes 0/1/16, the V1, V1-migrated-to-V2 and V2 layouts, unbounded values large enough to spill onto overflow pages, duplicate and overwriting batches, and batch sizes chosen to hit the cascade at the node capacity boundaries. Ordering is covered by ascending, descending, zigzag, sawtooth and shuffled inputs. Batches large enough to grow the root repeatedly are drained back to empty afterwards, so a mislinked child or a stale index would show up as chunks the allocator never gets back.Those all check behaviour, and so would notice a broken tree only through its consequences.
assert_btree_invariantschecks the shape itself: it walks the tree out of memory and asserts that every node is within its occupancy bounds, that keys are sorted and fall strictly between the separators either side of them in the parent, that an internal node has one more child than it has entries, that every leaf sits at the same depth, and that the entries counted match the length in the header. Its driver interleaves batches of varying shape and ordering with single inserts and removes, walking the tree after every step, so a split that linked a child wrongly fails at that node rather than wherever it happens to surface.🤖 Generated with Claude Code