knowledge: doc-conformance check controls + object-storage key persistence (5 insights to 2 pages)

INDEX.md

@@ -10,10 +10,10 @@ follow the cross-pointers in their index or take the next matching seeded domain
 | Domain | Status | Route here when |
 |--------|--------|-----------------|
 | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior |
-| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
+| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
 | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility |
 | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) |
-| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |
+| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests, checks whose subject is a document rather than code (release-process quality → qa) |
 | [qa](wiki/qa/index.md) | **seeded** | Release-quality process: release gates, regression scoping, bug reports, severity/priority triage, exploratory testing (writing automated test code → testing) |
 | [debugging](wiki/debugging/index.md) | **seeded** | Diagnosing a failure — finding what is wrong and why: reproducing, bisection, hypothesis testing, traces/logs, intermittent failures (fixing the diagnosed fault → its owning domain) |
 | [security](wiki/security/index.md) | **seeded** | Trust-boundary decisions: input validation, session-vs-token auth choice, per-resource authorization (IDOR), secrets hygiene, dependency trust, PII handling (XSS rendering → frontend; CI secrets → infrastructure; JWT implementation → backend/frontend auth) |

log.md

@@ -34,3 +34,5 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
 ## [2026-07-13] ingest | databases +1 (query-optimization): streaming-large-result-sets — memory-bounded export of a huge single-query result. Client-side cursor pulls the whole set to libpq on execute (fetchmany caps only the Python-list explosion); only a server-side/named cursor truly streams but needs a transaction, so it fails under autocommit or a proxy that blocks BEGIN → fall back to client-side fetchmany + disk spool + openpyxl write_only (measured 300k rows 838MB→38MB). Derived from RNR-3440 (potential-listing weekly extract memory peak); QueryPie BEGIN-block generalized to "read-only access proxy", field-tested. Sources: psycopg2 usage/cursor docs (named-cursor WITHOUT HOLD + autocommit exception), openpyxl optimized-modes (write-only near-constant memory, lxml=speed-not-memory).
 ## [2026-07-23] ingest | databases +2: schema-design/online-schema-changes (ACCESS EXCLUSIVE lock avoidance — non-volatile default fast path, ADD CONSTRAINT NOT VALID + VALIDATE at SHARE UPDATE EXCLUSIVE, CHECK-NOT-NULL trick, CREATE INDEX CONCURRENTLY, expand-and-contract to decouple DB migration from app deploy, lock_timeout for lock-queue pile-up) + operations/autovacuum-and-wraparound (NEW category operations: per-table scale_factor/cost_limit tuning for hot tables, age(datfrozenxid)/relfrozenxid + n_dead_tup monitoring, wraparound read-only cliff and superuser VACUUM recovery, VACUUM FULL vs pg_repack). Derived from the Hatchet "Postgres survival guide"; both cross-checked against PostgreSQL official docs (sql-altertable, routine-vacuuming).
+## [2026-07-30] ingest | testing +1 (NEW category docs-as-spec): document-conformance-checks — positive+negative control per check for gates whose subject is a document; validate the pattern on a conforming sibling when the target file does not exist yet (absence makes every pattern red); four-axis split (structure / normative modality / enum row completeness / cross-reference) each with a mutation that breaks only what it owns; coverage vs value-validity as separate checks; GFM unescaped-pipe row splitting. Folded from 4 queued insights (grep-gate falsifiability, spec-gate blind spots, coverage-vs-correctness, markdown table parsing) — one situation, one page. Sources: ESLint RuleTester valid/invalid, Semgrep ruleid/ok, Google mutation testing, JSON Schema required, GFM tables spec.
+## [2026-07-30] ingest | backend +1 (NEW category common/storage): object-key-persistence — persist the upload response Key (+Bucket) and derive URLs on read; Location is produced by two different code paths split at the managed uploader part size (5 MiB default), so one column ends up holding percent-encoded and form-encoded keys and only multipart rows 404. Derived from an RTB presigned-URL regression; verified against S3 CompleteMultipartUpload API docs, ManagedUpload docs, aws-sdk-js managed_upload.js source, and issues #1158 (v2) / #5656 (v3).

wiki/backend/common/storage/object-key-persistence.md

@@ -0,0 +1,75 @@
+---
+id: backend-common-storage-object-key-persistence
+domain: backend
+category: storage
+applies_to: [aws-s3, aws-sdk-js-v2, aws-sdk-js-v3]
+confidence: verified
+sources:
+  - https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
+  - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html
+  - https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js
+  - https://github.com/aws/aws-sdk-js/issues/1158
+  - https://github.com/aws/aws-sdk-js-v3/issues/5656
+last_verified: 2026-07-30
+related: [backend-common-api-design-idempotency, backend-node-boundaries-runtime-validation]
+---
+
+# Persisting the Result of an Object-Storage Upload
+
+## When this applies
+
+You are writing the result of a managed/multipart object-storage upload
+(`s3.upload()`, `@aws-sdk/lib-storage` `Upload`, an equivalent transfer manager)
+into a database column, a message, or any store read back later — and you are
+choosing which response field to persist.
+
+## Do this
+
+1. **Persist the raw object key (`Key`) plus the bucket, and derive URLs at read
+   time.** `Key` is the field the service documents as "the object key of the
+   newly created object" and is what every later call (`GetObject`,
+   `getSignedUrl`, `HeadObject`, delete) takes as input.
+2. **Treat `Location` as display-only.** Its encoding and host are not a
+   documented contract, and the value comes from a different producer on each
+   upload path:
+
+| Upload path | Taken when | Who produces `Location` | Observed form |
+|-------------|------------|-------------------------|---------------|
+| Single part (`PutObject`) | Body ≤ the managed uploader's part size — 5 MiB (5,242,880 B) by default in aws-sdk-js | The SDK, joining `protocol + host + httpRequest.path` | Path-style percent-encoding: space → `%20` |
+| Multipart (`CompleteMultipartUpload`) | Body over that threshold | The S3 response, passed through after the SDK rewrites `%2F` back to `/` | Service encoding with only slashes repaired: space → `+` survives |
+
+3. **Build the read path from `Key`.** Sign or fetch with the stored key exactly
+   as stored; apply no decode step. A stored key needs no repair, so no decode
+   rule has to be guessed later.
+4. **When a column already holds URLs, migrate by extracting the key** — reverse
+   both encodings by upload size, verify each candidate with `HeadObject`, and
+   write `Key` back. Add the `Key` column before the code switch so old and new
+   rows are both readable ([backend-common-api-design-idempotency] for the
+   backfill's retry behavior).
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| A CDN or custom domain fronts the bucket | Store `Key` and compose `cdnBase + encodeURI(key)` at read time; in aws-sdk-js-v3 the multipart path returns the origin host in `Location` while the single-part path returns the custom domain (issue #5656) |
+| The key contains `+` as a literal character | Storing `Key` keeps it literal; a URL round-trip cannot distinguish it from an encoded space, which is why the encoded form is not a safe identifier |
+| A bug report says "only large files 404" | Read it as the single-part/multipart split, not as intermittency: the threshold is a deterministic byte boundary, so reproduce at part-size ± 1 byte |
+| Only `Location` was ever stored and the objects must be found now | Try both decodings per row (`+` → space and `+` literal), confirm with `HeadObject`, and record which rows stayed ambiguous |
+| `partSize` is configured above the default | The boundary moves to that value; read it from the uploader config rather than assuming 5 MiB |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| Save `uploadRes.Location` as the object's identifier | Save `uploadRes.Key` (with `Bucket`) and build URLs on read | `Location` is produced by two different code paths with two different encodings, so one column ends up holding both |
+| Add a `decodeURIComponent` on read to fix the broken rows | Store the raw key so the read path needs no decode | Standard URL decoding leaves `+` as a literal, so it repairs the single-part rows and leaves the multipart rows 404-ing |
+| Chase the mismatch as a nondeterministic uploader bug | Compare file sizes against the uploader's part size and reproduce at the boundary | The behaviour is decided by one byte-size comparison; naming it "intermittent" sends the fix to the upload layer instead of the persistence layer |
+| Normalize the encoding inside the upload wrapper | Change what the caller persists | The wrapper cannot fix rows already written, and the service keeps returning its own `Location` regardless |
+
+## Sources
+
+- https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html — response elements: `Key` is "the object key of the newly created object"; `Location` is documented only as "the URI that identifies the newly created object", with no encoding guarantee
+- https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html — callback data carries `Location`, `ETag`, `Bucket`, `Key`; `AWS.S3.ManagedUpload.minPartSize = 1024 * 1024 * 5` and the default `partSize` is 5 MB
+- https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js — `finishSinglePart` builds `data.Location` from `endpoint.protocol + '//' + endpoint.host + httpReq.path` and sets `data.Key` from `params.Key`; `finishMultiPart` takes the service's `Location` and applies only `.replace(/%2F/g, '/')`
+- https://github.com/aws/aws-sdk-js/issues/1158 — reported inconsistency: multipart returns `…/stream-uploads%2Fkokoko.gif` where single-part returns `…/stream-uploads/kokoko.gif`; `Key` is identical in both responses
+- https://github.com/aws/aws-sdk-js-v3/issues/5656 — same split in v3 `lib-storage`: `__uploadUsingPut` constructs `Location` client-side, `CompleteMultipartUploadCommand` does not

wiki/backend/index.md

@@ -5,7 +5,7 @@ three stack subtrees — route by concern first, stack second:
 
 | Subtree | Route there when |
 |---------|------------------|
-| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure |
+| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure, object-storage references |
 | [java](java/index.md) | You are writing/reviewing JVM backend code (Java/Kotlin, Spring, JPA/Hibernate) and the concern is stack-specific: entity mapping, persistence context, proxy pitfalls, JVM threads/memory |
 | [node](node/index.md) | You are writing/reviewing Node.js/TypeScript backend code: event-loop blocking, promise error handling, runtime validation at boundaries, graceful shutdown |
 | [python](python/index.md) | You are writing/reviewing Python backend code: GIL/concurrency model, pydantic validation, WSGI/ASGI workers, language traps |
@@ -26,6 +26,12 @@ Match your situation to a "load when" line; load only matching pages.
 | [idempotency](common/api-design/idempotency.md) | An endpoint with side effects (create, charge, send) can receive the same request twice — client retry after timeout, user double-submit, gateway retry; designing idempotency-key storage; deciding which operations are safe to retry |
 | [pagination-contract](common/api-design/pagination-contract.md) | Designing a list endpoint's request/response contract — cursor vs page-number, limit caps, total counts, expired-cursor behavior (the backing SQL/index → databases/query-optimization/keyset-pagination) |
 
+### storage
+
+| Page | Load when |
+|------|-----------|
+| [object-key-persistence](common/storage/object-key-persistence.md) | Persisting the result of an object-storage upload (`s3.upload()`, `lib-storage` `Upload`, a transfer manager) — choosing which response field goes in the DB column; building the read/signing path from a stored reference; migrating a column that holds URLs to keys; only large uploads 404 on read |
+
 ### reliability
 
 | Page | Load when |