Add broker drain API#18709
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #18709 +/- ##
============================================
- Coverage 65.45% 56.85% -8.61%
+ Complexity 1421 1 -1420
============================================
Files 3425 2628 -797
Lines 216303 156618 -59685
Branches 34266 25606 -8660
============================================
- Hits 141590 89040 -52550
+ Misses 63317 59961 -3356
+ Partials 11396 7617 -3779
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
left a comment
There was a problem hiding this comment.
Found one high-signal issue; see inline comment.
| } | ||
|
|
||
| private void startDrain() { | ||
| if (!_draining.compareAndSet(false, true)) { |
There was a problem hiding this comment.
startDrain() marks the broker draining before getConnectedHelixManager() and the Helix writes succeed. Because BaseBrokerStarter exposes the admin API before _participantHelixManager.connect(), a startup-window POST /drain (or any Helix write failure here) returns 500 but still leaves _draining set, so later queries are rejected until restart. Please make this transition rollback-safe or only flip the local drain state after the Helix preconditions/writes succeed.
d995f62 to
84eb1cd
Compare
dd0dbf9 to
d3fb3f4
Compare
7b4d3b7 to
3dd18d8
Compare
4c17828 to
14a496c
Compare
82927ea to
71baf18
Compare
71baf18 to
9dc7383
Compare
Summary
POST /drainandGET /drain/statusadmin APIs.BrokerShuttingDownwhile waiting for already accepted queries to finish.shutdownInProgress=true, remove it frombrokerResource, fail health checks while draining, and restore stale drain state on startup.User manual
To gracefully drain the broker and stop it after accepted queries finish:
curl -X POST "http://<broker-host>:<admin-port>/drain?timeoutMs=-1&shutdown=true"timeoutMs=-1usespinot.broker.delayShutdownTimeMs. Useshutdown=falsewhen an external orchestrator should terminate the process after drain completes:While draining, the broker health check returns 503 and new valid queries sent to this broker fail with
BrokerShuttingDown(212), so clients should retry another broker.Sample query
If this query is sent to a draining broker, the broker returns
BrokerShuttingDowninstead of accepting it for execution.Tests
./mvnw -pl pinot-broker -am -Dtest=BrokerDrainManagerTest,BrokerRequestHandlerDelegateTest -Dsurefire.failIfNoSpecifiedTests=false test./mvnw spotless:apply -pl pinot-broker,pinot-spi,pinot-controller./mvnw checkstyle:check -pl pinot-broker,pinot-spi,pinot-controller./mvnw license:format -pl pinot-broker,pinot-spi,pinot-controller./mvnw license:check -pl pinot-broker,pinot-spi,pinot-controllergit diff --check upstream/master...HEADReview notes
shutdown=truenow delays the asynchronous stop callback briefly so the drain API response is not raced by admin server shutdown.