Skip to content

fix(filler): defer the eager missing-mint reconciliation while the reader is catching up#69

Open
igorls wants to merge 1 commit into
atomicassets:mainfrom
igorls:fix/defer-init-mint-reconciliation-during-catchup
Open

fix(filler): defer the eager missing-mint reconciliation while the reader is catching up#69
igorls wants to merge 1 commit into
atomicassets:mainfrom
igorls:fix/defer-init-mint-reconciliation-during-catchup

Conversation

@igorls

@igorls igorls commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Problem

AtomicAssetsHandler.init() loops CALL update_atomicassets_mints (bounded 50k/call) + a full atomicassets_assets COUNT until the missing-mint backlog is under 50k. While the reader is catching up the update_atomicassets_mints job is deferred (shouldDeferDrain), so the backlog grows large — ~8M rows observed on WAX mainnet.

Running this reconciliation at boot then:

  • blocks the reader from ever starting (it loops ~160× on an 8M backlog), and
  • busts the connection statement_timeout on the 466M-row COUNT,

so any filler restart mid-catchup becomes an init crash-loop — the init promise rejects and the reader never starts (the process stays up but idle).

Fix

Gate the reconciliation on the reader's distance from head. shouldDeferDrain() can't be used here because reader.blocksUntilHead is still 0 before the reader starts, so compare the reader's persisted contract_readers.block_num to chain head directly and skip while far behind (>10k blocks). The deferred update_atomicassets_mints job fills the backlog once the reader reaches head. The reconciliation loop body is unchanged.

Validation

  • pnpm check-types clean.
  • On the WAX-mainnet cluster: a filler restart mid-catchup now logs Reader is ~N blocks behind head - skipping eager missing-mint reconciliation and the reader resumes immediately instead of crash-looping.

@igorls
igorls marked this pull request as ready for review June 9, 2026 01:52
Copilot AI review requested due to automatic review settings June 9, 2026 01:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Defers the expensive “missing mint number reconciliation” work during startup when the reader is far behind chain head, avoiding long startup stalls and potential crash-loops from DB statement_timeout.

Changes:

  • Adds a reader-lag gate by comparing the persisted reader position (contract_readers) to chainInfo.head_block_num.
  • Skips eager mint reconciliation when the reader is >10k blocks behind head, logging the deferral.
  • Avoids querying atomicassets_config entirely when reconciliation is skipped.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +237
const blocksBehindHead = chainInfo.head_block_num - (readerPositionQuery.rows[0] ? Number(readerPositionQuery.rows[0].block_num) : 0);
const skipMintReconciliation = blocksBehindHead > 10_000;
Comment on lines +236 to +237
const blocksBehindHead = chainInfo.head_block_num - (readerPositionQuery.rows[0] ? Number(readerPositionQuery.rows[0].block_num) : 0);
const skipMintReconciliation = blocksBehindHead > 10_000;
Comment on lines +243 to +245
const contractsQuery = skipMintReconciliation
? { rows: [] as any[] }
: await this.connection.database.query('SELECT * FROM atomicassets_config');
…ches up

AtomicAssetsHandler.init() loops `CALL update_atomicassets_mints` (bounded 50k/
call) + a full atomicassets_assets COUNT until the missing-mint backlog is under
50k. While the reader catches up the update_atomicassets_mints job is deferred
(shouldDeferDrain), so the backlog grows to millions of rows on large chains
(observed ~8M on WAX). Running the reconciliation at boot then blocks the reader
from ever starting AND busts the connection statement_timeout on the COUNT — so
any filler restart mid-catchup becomes an init crash-loop (the reader never
starts; the process sits idle).

Gate it on the reader's distance from head. shouldDeferDrain() is unusable here
because reader.blocksUntilHead is still 0 before the reader starts, so compare
the reader's persisted contract_readers.block_num to chain head directly and
skip the reconciliation while far behind. The deferred update_atomicassets_mints
job fills the backlog once the reader reaches head. Loop body unchanged.
@igorls
igorls force-pushed the fix/defer-init-mint-reconciliation-during-catchup branch from 0fc2d79 to 8c4ea75 Compare June 9, 2026 02:09

@robrigo robrigo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (FACINGS) — LGTM.

Verified chainInfo (get_info()), this.filler.reader.name, and positiveIntEnv all exist. The gate fails closed — unknown/non-numeric position → null → skip — which is the safe direction (never falls open into the expensive reconciliation during catch-up). Patch applies cleanly onto feat/atomicassets-v2.

Note: Copilot's three inline comments were on an earlier commit — all three are already addressed in the current head: the threshold is the env-tunable MINT_RECONCILIATION_MAX_LAG_BLOCKS constant, Number.isFinite + clamp handle the NaN case, and the skip branch uses a plain [] rows array (not {rows: [] as any[]}).

robrigo added a commit that referenced this pull request Jun 9, 2026
AtomicAssets v2 (dual ownership/renting, mutable templates, schema media types,
collection author swaps) was originally authored by Fabian Emilius in the WAX-OIG
eosio-contract-api fork; this repo carries it forward as a reconstruction onto 1.7.2.
Record the original authorship and co-attribute it. Also credits @igorls for the v2
catch-up/upgrade-safety fixes (#68/#69/#70).

Co-authored-by: Fabian Emilius <20770096+fabian-emilius@users.noreply.github.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
@igorls

igorls commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Rechecked against current main: the startup missing-mint reconciliation in src/filler/handlers/atomicassets/index.ts still runs ungated — no reader-lag check around the get_info() / atomicassets_config count-and-update_atomicassets_mints loop — so the catch-up startup stall this defers is still present. Branch is MERGEABLE and the gate fails closed (unknown/non-numeric reader position → skip), so it never falls open into the expensive reconciliation during catch-up. Ready to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants