fix(ci): run migrations in the API deploy, before the code that needs them (Sentry #7645807976) - #154
Merged
Conversation
… them deploy-api.yml had no migration step at all, so every schema change reached production only if someone remembered to run db:migrate by hand. batch_send .paused_at is what happened when nobody did: the column and the broadcast-reaper query that reads it shipped in the same commit (2b7a06c), the Lambda went live, and the reaper threw `column batch_send.paused_at does not exist` on every run against production -- 111 events before this was noticed. Migrating before deploying rather than after, for the reason 405063c already established for selfhost: `sst deploy` publishes each Lambda's new code the moment that function updates, well before the command returns, so migrating afterwards still leaves a window where new code runs against the old schema. Failing this way round is also the better failure -- a migration that dies leaves old code on the old schema, which works. Safe because migrations are expected to be backward compatible; a migration that removes or renames still has to be split expand/contract across two releases either way. Does not fix the currently-broken production database, which is still missing 0073_dazzling_puma and needs it applied out of band. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PAoM6kmDzrf5BxmbXn1Z51
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Auto-detected via Sentry issue #7645807976 —
column batch_send.paused_at does not exist, 111 events, ongoing.Root cause is not the reaper.
deploy-api.ymlhas never had a migration step, so every schema change reached production only if someone remembered to rundb:migrateby hand. Commit2b7a06c8addedbatch_send.paused_at, migration0073_dazzling_puma, and therunBroadcastReaperquery that reads the column — all in one commit. The Lambda deployed, the migration did not, and the reaper has thrown on every run since.Fix. Migrate before
sst deploy, for the reason405063c5already established for the selfhost path:sst deploypublishes each Lambda's new code the moment that function updates, well before the command returns, so migrating afterwards still leaves a window where new code runs against the old schema. That commit's own message namesbatch_send.paused_atas the motivating case — it landed for selfhost on Jul 31 and the same bug hit hosted production the next morning.Failing this way round is also the better failure: a migration that dies leaves old code on the old schema, which works.
The production database is still missing
0073_dazzling_pumaand the reaper is still throwing. Merging this only prevents recurrence.0073has to be applied out of band — this PR makes the next deploy carry its migrations, not this one.Sentry Context
error: column batch_send.paused_at does not exist(wrapped asError: Failed query: select …)runBroadcastReaperTest plan
baseline/deploy-api-workflow.test.ts(runs in CI viapnpm test:architecture)sst deployfails the ordering assertionsecrets.DATABASE_URLin theproductionenvironment permits DDLNote on scope
The test asserts workflow ordering rather than reproducing the SQL error, because the defect is a missing pipeline step — the repo's schema and migration were always correct. This mirrors the existing
selfhost-deploy-workflow.test.tsprecedent.Generated by sentry-autofix