Skip to content

Close LIST_ALL_QUERY (null-prefix) and S3-handler empty-prefix coverage gaps - #3270

Open
zichengl wants to merge 1 commit into
linkedin:masterfrom
zichengl:zichengl/list-null-prefix-coverage
Open

Close LIST_ALL_QUERY (null-prefix) and S3-handler empty-prefix coverage gaps#3270
zichengl wants to merge 1 commit into
linkedin:masterfrom
zichengl:zichengl/list-null-prefix-coverage

Conversation

@zichengl

Copy link
Copy Markdown
Contributor

Summary

Adds two integration tests that close coverage gaps remaining after #3265:

  1. MySqlNamedBlobDbListOperationIntegrationTest — the existing parametrized matrix never invoked namedBlobDb.list(...) with a null prefix, so the LIST_ALL_QUERY path (and the new option-4 LIST_ALL_SQL window-function variant introduced in Fix option-4 empty-prefix LIST regression; add window-function LIST_ALL #3265) had no parametrized integration coverage. The TMC failure that motivated Fix option-4 empty-prefix LIST regression; add window-function LIST_ALL #3265 traversed exactly this path (prefix=""parseS3 collapses to nulllist() with nullLIST_ALL_QUERY under option 4). This PR adds two test methods exercising it across options 2/3/4 × hard-delete on/off.

  2. S3IntegrationTest — no end-to-end test sent the request shape ?prefix= (parameter present, value empty) that the AWS S3 SDK emits when ListObjectsRequest is built with an unset/empty prefix. Added s3ListEmptyPrefixTest covering HTTP → Netty → S3ListHandlerNamedBlobPath.parseS3NamedBlobListHandlerNamedBlobDb#list.

What this catches

Failure class Caught by
Option-2/3/4 LIST_ALL_QUERY returns wrong rows on null prefix testListNamedBlobsWithNullPrefix (across all 6 parameter combinations)
Option-4 LIST_ALL_SQL semantic bug — deleted_ts-placement footgun on the null-prefix path testListAllNullPrefixHidesDeletedLatestUnderOption4 (option 4 only via Assume, mirrors the with-prefix invariant test in #3260)
Regression in NamedBlobPath.parseS3 empty-prefix-to-null collapse s3ListEmptyPrefixTest (end-to-end through HTTP)
Regression in S3ListHandler empty-prefix routing s3ListEmptyPrefixTest

What's left for a follow-up

Full stitched coverage of S3 handler → real MySQL with option 4 on a realistic container is not in this PR. s3ListEmptyPrefixTest runs against InMemNamedBlobDbFactory (the default backing in S3IntegrationTest), so it catches handler-layer regressions but not SQL-side regressions on the stitched path. Adding a MySQL-backed sibling of S3IntegrationTest would require refactoring buildFrontendVProps to accept a configurable NamedBlobDbFactory plus a port-collision-aware sibling class. Reserving that for a follow-up to keep this PR's diff small and review-focused.

The SQL-side coverage gap is fully closed by (1) — MySqlNamedBlobDbListOperationIntegrationTest runs against a real MySQL container (matching the OSS CI setup) on every PR.

Diff size

ambry-frontend/src/integration-test/.../S3IntegrationTest.java                                  | +59
ambry-named-mysql/src/integration-test/.../MySqlNamedBlobDbListOperationIntegrationTest.java   | +96
2 files changed, +155 / -0

Testing Done

  • ./gradlew :ambry-named-mysql:compileIntTestJavaBUILD SUCCESSFUL.
  • ./gradlew :ambry-frontend:compileIntTestJavaBUILD SUCCESSFUL.
  • ./gradlew :ambry-named-mysql:intTest --tests '*MySqlNamedBlobDbListOperationIntegrationTest.testListNamedBlobsWithNullPrefix*' --tests '*MySqlNamedBlobDbListOperationIntegrationTest.testListAllNullPrefixHidesDeletedLatestUnderOption4*'12 runs, 8 passed, 0 failed, 4 skipped. The 4 skipped are the option-4-only invariant test on options 2/3 (via Assume.assumeTrue).
  • ./gradlew :ambry-frontend:intTest --tests '*S3IntegrationTest.s3ListEmptyPrefixTest'1 passed.
  • Full :ambry-named-mysql:intTest matrix (MySqlNamedBlobDbListOperationIntegrationTest*) against a local MySQL 8.0 container → 42 total, 38 passed, 4 skipped, 0 failed (30 baseline + 12 new, with the 4 Assume-skips for the option-4-only invariant).

Durability risk

Test-only addition. No production code path touched, no schema, no config defaults. Zero durability risk.

AI Usage

Used Claude Code (Opus 4.7) to identify the coverage gap (both MySqlNamedBlobDbListOperationIntegrationTest's 9 list() invocations passing non-null prefix, and S3IntegrationTest's reliance on the helper method that drops empty prefix values), draft the test methods, run them against a local Docker MySQL 8.0 container, and author this PR description. Author reviewed.

…ge gaps

After linkedin#3265 fixed the empty-prefix LIST regression on option 4, two
test-coverage gaps remained that linkedin#3265 didn't close:

1. MySqlNamedBlobDbListOperationIntegrationTest exercised list() only
   with non-null prefixes, so LIST_ALL_QUERY (and the new option-4
   LIST_ALL_SQL window-function variant) had no parametrized
   integration coverage. The TMC failure that motivated linkedin#3265
   specifically traversed this path (S3 SDK sends prefix= empty ->
   parseS3 collapses to null -> list() with null prefix -> LIST_ALL
   under option 4), and the int-test matrix would not have caught a
   regression in that branch.

2. S3IntegrationTest had no end-to-end test that explicitly emits
   `?prefix=` (parameter present, value empty) -- the exact request
   shape the AWS S3 SDK uses when ListObjectsRequest is built with an
   empty or unset prefix. Without it, a future refactor of
   S3ListHandler or NamedBlobPath.parseS3 that drops the empty->null
   collapse would slip past CI.

This commit adds both:

(a) ambry-named-mysql/src/integration-test:
    - testListNamedBlobsWithNullPrefix -- option-agnostic basic test
      that PUTs 5 blobs and asserts list(account, container, null,
      null, null) returns all 5 in blob_name order. Runs against
      options 2/3/4 via the existing @parameterized matrix.
    - testListAllNullPrefixHidesDeletedLatestUnderOption4 -- option-4
      only via Assume.assumeTrue; mirrors the invariant
      testListHidesBlobWhenLatestVersionIsExpired added in linkedin#3260 but
      for the null-prefix path. Verifies that under option 4 a blob
      whose latest version is TTL-expired is hidden entirely from
      LIST_ALL_SQL (the deleted_ts-on-outer-SELECT contract). Options
      2/3 have the opposite legacy semantic by design and are
      excluded.

(b) ambry-frontend/src/integration-test:
    - s3ListEmptyPrefixTest -- exercises GET /s3/{account}/{container}?prefix=
      end-to-end through HTTP -> Netty -> S3ListHandler ->
      NamedBlobPath.parseS3 -> NamedBlobListHandler -> NamedBlobDb#list.
      Asserts 200 OK for both v1 (no list-type) and v2 (list-type=2)
      shapes. The named-blob DB in this integration test is
      InMemNamedBlobDbFactory, so this catches regressions in the
      S3-handler routing layer (empty-prefix collapse, parseS3
      logic) but not SQL-side regressions -- those are covered by
      (a) against a real MySQL backend in :ambry-named-mysql:intTest.

Full stitched coverage (S3 handler -> real MySQL with option 4 on a
real container) is left to a follow-up that refactors
S3IntegrationTest's buildFrontendVProps to accept a configurable
NamedBlobDbFactory, allowing a sibling test class to swap in the
MySQL-backed factory. That refactor wasn't bundled here to keep this
diff small and review-focused on the specific gaps.

Testing Done
------------
- ./gradlew :ambry-named-mysql:compileIntTestJava -> BUILD SUCCESSFUL.
- ./gradlew :ambry-frontend:compileIntTestJava -> BUILD SUCCESSFUL.
- ./gradlew :ambry-named-mysql:intTest --tests
  '*MySqlNamedBlobDbListOperationIntegrationTest.testListNamedBlobs
  WithNullPrefix*' --tests
  '*MySqlNamedBlobDbListOperationIntegrationTest.testListAllNullPrefix
  HidesDeletedLatestUnderOption4*'
  -> 12 runs (6 per method x 2 methods), 8 passed, 4 skipped.
  The 4 skipped are the option-4-only invariant on options 2/3
  via Assume.assumeTrue, as intended.
- ./gradlew :ambry-frontend:intTest --tests
  '*S3IntegrationTest.s3ListEmptyPrefixTest' -> 1 passed.
- Full :ambry-named-mysql:intTest matrix runs against a local
  MySQL 8.0 container; passes 30/30 baseline + 12 new = 42 total
  with 8 added passes + 4 added Assume-skips.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.94%. Comparing base (52ba813) to head (2572752).
⚠️ Report is 396 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3270       +/-   ##
=============================================
- Coverage     64.24%   50.94%   -13.30%     
+ Complexity    10398     8699     -1699     
=============================================
  Files           840      937       +97     
  Lines         71755    80199     +8444     
  Branches       8611     9633     +1022     
=============================================
- Hits          46099    40859     -5240     
- Misses        23004    35956    +12952     
- Partials       2652     3384      +732     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants