Close LIST_ALL_QUERY (null-prefix) and S3-handler empty-prefix coverage gaps - #3270
Open
zichengl wants to merge 1 commit into
Open
Close LIST_ALL_QUERY (null-prefix) and S3-handler empty-prefix coverage gaps#3270zichengl wants to merge 1 commit into
zichengl wants to merge 1 commit into
Conversation
…ge gaps After linkedin#3265 fixed the empty-prefix LIST regression on option 4, two test-coverage gaps remained that linkedin#3265 didn't close: 1. MySqlNamedBlobDbListOperationIntegrationTest exercised list() only with non-null prefixes, so LIST_ALL_QUERY (and the new option-4 LIST_ALL_SQL window-function variant) had no parametrized integration coverage. The TMC failure that motivated linkedin#3265 specifically traversed this path (S3 SDK sends prefix= empty -> parseS3 collapses to null -> list() with null prefix -> LIST_ALL under option 4), and the int-test matrix would not have caught a regression in that branch. 2. S3IntegrationTest had no end-to-end test that explicitly emits `?prefix=` (parameter present, value empty) -- the exact request shape the AWS S3 SDK uses when ListObjectsRequest is built with an empty or unset prefix. Without it, a future refactor of S3ListHandler or NamedBlobPath.parseS3 that drops the empty->null collapse would slip past CI. This commit adds both: (a) ambry-named-mysql/src/integration-test: - testListNamedBlobsWithNullPrefix -- option-agnostic basic test that PUTs 5 blobs and asserts list(account, container, null, null, null) returns all 5 in blob_name order. Runs against options 2/3/4 via the existing @parameterized matrix. - testListAllNullPrefixHidesDeletedLatestUnderOption4 -- option-4 only via Assume.assumeTrue; mirrors the invariant testListHidesBlobWhenLatestVersionIsExpired added in linkedin#3260 but for the null-prefix path. Verifies that under option 4 a blob whose latest version is TTL-expired is hidden entirely from LIST_ALL_SQL (the deleted_ts-on-outer-SELECT contract). Options 2/3 have the opposite legacy semantic by design and are excluded. (b) ambry-frontend/src/integration-test: - s3ListEmptyPrefixTest -- exercises GET /s3/{account}/{container}?prefix= end-to-end through HTTP -> Netty -> S3ListHandler -> NamedBlobPath.parseS3 -> NamedBlobListHandler -> NamedBlobDb#list. Asserts 200 OK for both v1 (no list-type) and v2 (list-type=2) shapes. The named-blob DB in this integration test is InMemNamedBlobDbFactory, so this catches regressions in the S3-handler routing layer (empty-prefix collapse, parseS3 logic) but not SQL-side regressions -- those are covered by (a) against a real MySQL backend in :ambry-named-mysql:intTest. Full stitched coverage (S3 handler -> real MySQL with option 4 on a real container) is left to a follow-up that refactors S3IntegrationTest's buildFrontendVProps to accept a configurable NamedBlobDbFactory, allowing a sibling test class to swap in the MySQL-backed factory. That refactor wasn't bundled here to keep this diff small and review-focused on the specific gaps. Testing Done ------------ - ./gradlew :ambry-named-mysql:compileIntTestJava -> BUILD SUCCESSFUL. - ./gradlew :ambry-frontend:compileIntTestJava -> BUILD SUCCESSFUL. - ./gradlew :ambry-named-mysql:intTest --tests '*MySqlNamedBlobDbListOperationIntegrationTest.testListNamedBlobs WithNullPrefix*' --tests '*MySqlNamedBlobDbListOperationIntegrationTest.testListAllNullPrefix HidesDeletedLatestUnderOption4*' -> 12 runs (6 per method x 2 methods), 8 passed, 4 skipped. The 4 skipped are the option-4-only invariant on options 2/3 via Assume.assumeTrue, as intended. - ./gradlew :ambry-frontend:intTest --tests '*S3IntegrationTest.s3ListEmptyPrefixTest' -> 1 passed. - Full :ambry-named-mysql:intTest matrix runs against a local MySQL 8.0 container; passes 30/30 baseline + 12 new = 42 total with 8 added passes + 4 added Assume-skips. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3270 +/- ##
=============================================
- Coverage 64.24% 50.94% -13.30%
+ Complexity 10398 8699 -1699
=============================================
Files 840 937 +97
Lines 71755 80199 +8444
Branches 8611 9633 +1022
=============================================
- Hits 46099 40859 -5240
- Misses 23004 35956 +12952
- Partials 2652 3384 +732 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two integration tests that close coverage gaps remaining after #3265:
MySqlNamedBlobDbListOperationIntegrationTest— the existing parametrized matrix never invokednamedBlobDb.list(...)with anullprefix, so theLIST_ALL_QUERYpath (and the new option-4LIST_ALL_SQLwindow-function variant introduced in Fix option-4 empty-prefix LIST regression; add window-function LIST_ALL #3265) had no parametrized integration coverage. The TMC failure that motivated Fix option-4 empty-prefix LIST regression; add window-function LIST_ALL #3265 traversed exactly this path (prefix=""→parseS3collapses tonull→list()withnull→LIST_ALL_QUERYunder option 4). This PR adds two test methods exercising it across options 2/3/4 × hard-delete on/off.S3IntegrationTest— no end-to-end test sent the request shape?prefix=(parameter present, value empty) that the AWS S3 SDK emits whenListObjectsRequestis built with an unset/empty prefix. Addeds3ListEmptyPrefixTestcovering HTTP → Netty →S3ListHandler→NamedBlobPath.parseS3→NamedBlobListHandler→NamedBlobDb#list.What this catches
LIST_ALL_QUERYreturns wrong rows on null prefixtestListNamedBlobsWithNullPrefix(across all 6 parameter combinations)LIST_ALL_SQLsemantic bug — deleted_ts-placement footgun on the null-prefix pathtestListAllNullPrefixHidesDeletedLatestUnderOption4(option 4 only viaAssume, mirrors the with-prefix invariant test in #3260)NamedBlobPath.parseS3empty-prefix-to-null collapses3ListEmptyPrefixTest(end-to-end through HTTP)S3ListHandlerempty-prefix routings3ListEmptyPrefixTestWhat's left for a follow-up
Full stitched coverage of S3 handler → real MySQL with option 4 on a realistic container is not in this PR.
s3ListEmptyPrefixTestruns againstInMemNamedBlobDbFactory(the default backing inS3IntegrationTest), so it catches handler-layer regressions but not SQL-side regressions on the stitched path. Adding a MySQL-backed sibling ofS3IntegrationTestwould require refactoringbuildFrontendVPropsto accept a configurableNamedBlobDbFactoryplus a port-collision-aware sibling class. Reserving that for a follow-up to keep this PR's diff small and review-focused.The SQL-side coverage gap is fully closed by (1) —
MySqlNamedBlobDbListOperationIntegrationTestruns against a real MySQL container (matching the OSS CI setup) on every PR.Diff size
Testing Done
./gradlew :ambry-named-mysql:compileIntTestJava→ BUILD SUCCESSFUL../gradlew :ambry-frontend:compileIntTestJava→ BUILD SUCCESSFUL../gradlew :ambry-named-mysql:intTest --tests '*MySqlNamedBlobDbListOperationIntegrationTest.testListNamedBlobsWithNullPrefix*' --tests '*MySqlNamedBlobDbListOperationIntegrationTest.testListAllNullPrefixHidesDeletedLatestUnderOption4*'→ 12 runs, 8 passed, 0 failed, 4 skipped. The 4 skipped are the option-4-only invariant test on options 2/3 (viaAssume.assumeTrue)../gradlew :ambry-frontend:intTest --tests '*S3IntegrationTest.s3ListEmptyPrefixTest'→ 1 passed.:ambry-named-mysql:intTestmatrix (MySqlNamedBlobDbListOperationIntegrationTest*) against a local MySQL 8.0 container → 42 total, 38 passed, 4 skipped, 0 failed (30 baseline + 12 new, with the 4 Assume-skips for the option-4-only invariant).Durability risk
Test-only addition. No production code path touched, no schema, no config defaults. Zero durability risk.
AI Usage
Used Claude Code (Opus 4.7) to identify the coverage gap (both
MySqlNamedBlobDbListOperationIntegrationTest's 9list()invocations passing non-null prefix, andS3IntegrationTest's reliance on the helper method that drops empty prefix values), draft the test methods, run them against a local Docker MySQL 8.0 container, and author this PR description. Author reviewed.