Skip to content

fix(consul): one invalid node should not discard the remaining nodes of the service#13513

Merged
nic-6443 merged 2 commits into
apache:masterfrom
nic-6443:fix/consul-skip-invalid-node-12937
Jun 12, 2026
Merged

fix(consul): one invalid node should not discard the remaining nodes of the service#13513
nic-6443 merged 2 commits into
apache:masterfrom
nic-6443:fix/consul-skip-invalid-node-12937

Conversation

@nic-6443

Copy link
Copy Markdown
Member

Description

A single malformed entry in the consul health API response (an entry without the Service field, e.g. from an agent on a reclaimed cloud instance) currently wipes out the whole service. The per-node check in fetch_services_from_server() does goto CONTINUE, but the ::CONTINUE:: label sits outside the node loop, so one bad entry skips all the remaining nodes, the sort, and the up_services[key] = nodes assignment. If the malformed entry comes first, the service ends up with no nodes, update_all_services() deletes it from the shared dict, and requests start failing with "no valid upstream node".

The fix makes the skip local to the node loop (goto CONTINUE_NODE with the label at the end of the loop body) and logs a warning for the skipped entry. After the change, a malformed entry only drops that one node and the remaining healthy nodes keep serving traffic.

Added a regression test (t/discovery/consul-malformed-node.t) with a mock consul server returning one malformed entry followed by two valid ones; it fails without the fix and passes with it.

Which issue(s) this PR fixes:

Fixes #12937

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@nic-6443 nic-6443 marked this pull request as ready for review June 11, 2026 08:43
Copilot AI review requested due to automatic review settings June 11, 2026 08:43
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Jun 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a reliability bug in the Consul service discovery client where a single malformed entry (missing the Service field) could incorrectly skip the rest of the nodes for that service, potentially causing the service to disappear from discovery and break traffic routing.

Changes:

  • Make the “skip malformed node” logic local to the per-node loop (so one bad entry doesn’t discard the entire service’s node list) and emit a warning log for the skipped entry.
  • Add a regression test that mocks a Consul health response containing one malformed entry followed by valid nodes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
apisix/discovery/consul/client.lua Adjusts node-iteration control flow to skip only malformed nodes and logs a warning for the skipped entry.
t/discovery/consul-malformed-node.t Adds a regression test covering malformed health entries to ensure remaining valid nodes are still used.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apisix/discovery/consul/client.lua
Comment thread t/discovery/consul-malformed-node.t Outdated
@nic-6443 nic-6443 requested a review from membphis June 12, 2026 01:27

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. No blocking issues found.

@nic-6443 nic-6443 merged commit b1d91f1 into apache:master Jun 12, 2026
26 of 27 checks passed
@nic-6443 nic-6443 deleted the fix/consul-skip-invalid-node-12937 branch June 12, 2026 02:50
wistefan pushed a commit to wistefan/apisix that referenced this pull request Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: single node invalid will cause service unavailable

5 participants