Fix livelock in WaitUntilCaughtUpWith#411
Open
sisuresh wants to merge 1 commit into
Open
Conversation
The gate from stellar#410 required the node to close to within 5 ledgers of the reference peer's live ledger, re-reading both every poll. A watcher that tracks the network at a constant offset (e.g. ~8 ledgers behind while also publishing checkpoints) advances at network rate and never closes the gap, so the wait spins until the CI stage times out. Snapshot the reference peer's ledger once at wait entry and wait for the node to reach that fixed target: a trailing-but-tracking node passes in seconds, while a genuinely stalled node (the original flake) still blocks until its next catchup round completes.
Contributor
There was a problem hiding this comment.
Pull request overview
Snapshots the reference peer’s ledger to prevent catch-up waiting from livelocking against a moving target.
Changes:
- Captures a fixed target ledger when waiting begins.
- Updates polling and logging to use that target.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The catchup gate from #410 compares against the reference peer's live ledger every poll. A watcher tracking the network at a constant offset (~8 ledgers behind while publishing checkpoints) never closes the gap, so
MissionDatabaseInplaceUpgradespins in the wait until the CI stage times out — observed in today's run:Fix: snapshot the reference ledger once at wait entry and wait for the node to reach that fixed target. A trailing-but-tracking node passes in seconds; a genuinely stalled node (the original flake) still blocks until its next catchup round completes.