GTID master failover recovery + integration test (GTID failover 3/3)

[driv3r]

What

Implements opt-in, GTID-only master failover recovery: when the source MySQL master fails over, Ghostferry reconnects the whole run to the promoted writer instead of dying. Includes a deterministic Go integration test that proves it end-to-end.

PR 3 of 3 of the GTID failover rework:

1. Introduce SourceRuntime (Introduce SourceRuntime to own the source connection (GTID failover 1/3) #451).
2. Migrate source consumers to SourceRuntime (Migrate source consumers to SourceRuntime (GTID failover 2/3) #452).
3. This PR — the failover feature + integration test.

Stacked on #452. Review/merge #451 and #452 first; this PR's diff is against #452.

How

Two commits.

Add GTID master failover recovery

Failover is an in-process resume onto a new source session, not a hand-repoint of each consumer:

• SourceReconnector is the seam between the streamer ("I lost the source; here's the GTID set I must not fall behind") and the Ferry, which owns the SourceRuntime. ferrySourceReconnector resolves the new writer, then SourceRuntime.ReplaceValidated opens one connection, validates it, and only publishes it if valid. Because every source consumer already reads from the SourceRuntime (Migrate source consumers to SourceRuntime (GTID failover 2/3) #452), that single swap repoints the entire run.
• Fail-closed validation (validateFailoverTarget): candidate must be a writer (not @@read_only), have @@GLOBAL.gtid_mode = ON, and its executed set must contain everything already applied downstream — the committed streamed set plus any in-flight transaction's GTID (tracked from GTIDEvent/XIDEvent). If a cutover stop target is recorded, the candidate must contain it too. An undecodable in-flight GTID aborts recovery. A candidate that fails validation is never published.
• On success the streamer rebuilds its syncer against the promoted writer and restarts StartSyncGTID from the last resumable set. DB/DBConfig swaps are under connMu; FlushAndStop reads currentDB() so cutover records the stop coordinate from the new writer. On recovery exhaustion the Run loop fatals and returns.
• Bounds go-mysql's same-host reconnect retries (MaxReconnectAttempts, default 3) when failover is enabled — otherwise the syncer retries the dead host forever and GetEvent only ever returns timeouts, so recovery would never trigger. Unchanged when failover is disabled; tunable via SyncerMaxReconnectAttempts.
• Config.MasterFailoverRecovery is validated (requires gtid mode + resolver) and wired onto the source streamer only.

Add Go integration test for master failover recovery

A deterministic Go test (test/go) that drives the ferry directly, avoiding the status-callback timing races of a Ruby harness test:

• sets mysql-3 (port 29293, already in the compose files) up as a GTID replica of the source (aligning gtid_purged so it holds the real data);
• runs the ferry, and only after row copy completes writes extra rows, waits for mysql-3 to replicate them, promotes mysql-3, and stops the source container for a genuine unrecoverable disconnect;
• synchronizes on SourceRuntime().Config().Port flipping to the promoted port before cutover;
• asserts the promoted master and target are identical and the ferry did not fatal.

GTID-only (skips otherwise); requires docker/podman to stop the source container (skips otherwise). start-mysql.sh now also starts/waits for mysql-3.

An earlier revision used a Ruby harness test; it was replaced because the coarse status hooks raced the cutover path and the 30s callback timeout, making recovery unreliable to exercise in CI. The Go test controls ordering directly.

Testing

• New Go unit tests: recovery gating, applied-set (committed + in-flight, fail-closed), GTID union helpers, ReplaceValidated publish/reject, reconnector resolver-error/nil-candidate paths, in-flight GTID tracking, config validation. -race clean.
• Go integration test passes deterministically (verified locally against MySQL 8.0 with podman, 3× in a row): source stopped → recovery onto mysql-3 → target matches promoted master. Skips correctly in file_position mode and without a container tool.
• Full test/go suite green in both file_position and gtid modes.
• go build + gofmt clean.