Skip to content

fix(session-ingest): shrink cli_sessions_v2 status row lock#4322

Open
RSO wants to merge 3 commits into
mainfrom
fix/session-ingest-status-lock-duration
Open

fix(session-ingest): shrink cli_sessions_v2 status row lock#4322
RSO wants to merge 3 commits into
mainfrom
fix/session-ingest-status-lock-duration

Conversation

@RSO

@RSO RSO commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Reduces per-session row-lock hold time on cli_sessions_v2, the second major lock contributor identified in production Supabase lock-wait logs (the select "status" from "cli_sessions_v2" ... for update query). Over a representative 7-day window this exact query accounted for 1,440 acquired lock waits, dominated by ShareLock-on-transaction waits (max 110s) that the earlier ExclusiveLock-only investigation missed.

The previous applyMetadataChanges flow opened a JS transaction, took SELECT ... FOR UPDATE early, then held that row lock across a metadata UPDATE, a parent-session lookup + update, and a final full-row SELECT before commit. This change shrinks the locked critical section:

  • Status transitions now run as a single statement: a WITH locked AS (SELECT ... FOR UPDATE) CTE plus UPDATE ... RETURNING, returning both the locked previous status and the updated row. No multi-round-trip JS transaction holds the lock.
  • No-op status writes (same status re-sent) no longer stamp status_updated_at/updated_at, no longer hold the lock through a write, and emit no status notification.
  • Non-status metadata and parent-session updates moved out of the status lock path into their own idempotent UPDATE ... RETURNING statements.
  • The notification payload now comes from RETURNING instead of a final in-transaction SELECT.

Locked-status notification semantics are preserved: session.status.updated still reports the persisted previous status captured under the row lock, not the queued intake snapshot.

Verification

  • Reviewed production Supabase lock-wait logs (Axiom) to confirm the dominant wait mode and that long durations are lock waits on the PK index, not missing-index scans.
  • Manually traced the queue-consumer code path to confirm the lock is no longer held across parent lookup, parent update, or the final payload read.

Visual Changes

N/A

Reviewer Notes

  • The status transition is now raw SQL (db.execute(sql\...`)) because Drizzle's query builder can't express "return the locked previous value plus the updated row" in one statement. previous_statusis read via a CTE subquery inRETURNINGrather than relying onFROM`-alias visibility, for cross-version safety.
  • Behavior change worth confirming: a no-op status re-send no longer bumps updated_at/status_updated_at and emits no notification. This is intended (it was previously incidental lock-holding work), but flagging in case any consumer relied on the timestamp bump.
  • Non-status metadata, parent updates, and the status transition are now separate statements rather than one transaction. They remain individually idempotent and per-session; ordering across them is unchanged.

Reduce per-session row lock hold time on cli_sessions_v2 by removing the
multi-step JS transaction around status metadata. Status transitions now
run as a single SELECT ... FOR UPDATE + UPDATE ... RETURNING CTE, no-op
status writes no longer stamp timestamps or lock the row, and parent and
non-status metadata updates no longer execute while the status lock is
held.
Comment thread services/session-ingest/src/queue-consumer.ts Outdated
@kilo-code-bot

kilo-code-bot Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Executive Summary

The rewrite consolidates the metadata, parent, and status writes into a single atomic UPDATE ... RETURNING statement, resolving the prior cross-statement lock-holding WARNING with no new correctness issues found in the changed code.

Files Reviewed (2 files)
  • services/session-ingest/src/queue-consumer.ts
  • services/session-ingest/src/queue-consumer.test.ts
Previous Review Summaries (2 snapshots, latest commit 830a27d)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 830a27d)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0

Fix these issues in Kilo Cloud

Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/queue-consumer.ts 609 Wrapping updateSessionMetadata/updateSessionParent/updateSessionStatus in one db.transaction reintroduces cross-statement row-lock holding whenever metadata or parent changes are batched with a status change, since a plain UPDATE inside the transaction locks the row until commit — contradicting the accompanying comment's claim and undermining the PR's lock-shrinking goal for that case.
Files Reviewed (2 files)
  • services/session-ingest/src/queue-consumer.ts - 1 issue
  • services/session-ingest/src/queue-consumer.test.ts - 0 issues

Previous review (commit 2a81e84)

Status: 1 Issue Found | Recommendation: Address before merge

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 0

Fix these issues in Kilo Cloud

Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/queue-consumer.ts 598 Splitting the persistence path into separate committed statements means a later failure can leave Postgres permanently out of sync with the Durable Object replay path.
Files Reviewed (3 files)
  • services/session-ingest/src/queue-consumer.ts - 1 issue
  • services/session-ingest/src/queue-consumer.test.ts - 0 issues
  • services/session-ingest/src/types/user-connection-protocol.ts - 0 issues

Reviewed by claude-sonnet-5 · Input: 36 · Output: 15.9K · Cached: 948.7K

Review guidance: REVIEW.md from base branch main

…ction

applyMetadataChanges split persistence into three independently-committed
statements. If a later statement (e.g. the status CTE) failed after an
earlier one committed, Postgres was left permanently out of sync: the
queue consumer's DO only re-emits metadata changes that differ from its
last-sent value, so a retry after partial failure could arrive with an
empty/partial changes list.

Wrap the three writes in a single db.transaction so they commit or roll
back together, while keeping each write as one raw-SQL statement -
updateSessionStatus still uses a single WITH locked AS (SELECT ... FOR
UPDATE) ... UPDATE ... RETURNING statement, just run via tx.execute
instead of db.execute. This preserves the PR's goal of only holding the
cli_sessions_v2 row lock for that single statement, not across multiple
round trips.
Comment thread services/session-ingest/src/queue-consumer.ts Outdated
…tement

Replaces the db.transaction wrapping the metadata/parent/status writes with a
single UPDATE ... RETURNING guarded by a SELECT ... FOR UPDATE CTE. One
statement is inherently atomic (no torn subset can commit if a later part
fails) yet holds the row lock for exactly one round trip, never across a parent
lookup, a second write, or a final read -- resolving both review findings
without reintroducing cross-statement lock holding.

Parent FK validation is now a plain unlocked read before the write, so it never
extends the locked critical section. No-op status re-sends still take no lock
and bump no timestamps, and the notification semantics (previous status from the
locked row) are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant