Skip to content

fix: guard TelemetryEventBuffer against re-entrant mutex acquisition#2964

Merged
sl0thentr0py merged 5 commits into
masterfrom
fix/yabeda-skip-sentry-http-requests
Jun 8, 2026
Merged

fix: guard TelemetryEventBuffer against re-entrant mutex acquisition#2964
sl0thentr0py merged 5 commits into
masterfrom
fix/yabeda-skip-sentry-http-requests

Conversation

@sentry-junior
Copy link
Copy Markdown
Contributor

@sentry-junior sentry-junior Bot commented Jun 7, 2026

Problem

MetricEventBuffer (and LogEventBuffer) use a non-reentrant Mutex. When transport instrumentation calls Sentry.metrics.* from inside send_items — while the mutex is already held — the same thread tries to acquire it again, raising:

ThreadError: deadlock; recursive locking

The re-entrant call can come from any code that fires during an outgoing HTTP request:

  • Yabeda HTTP instrumentation (yabeda-http_requests, yabeda-rails)
  • Custom transports that call Sentry.metrics.* during send_data (see getsentry/repro#46)
  • Any other transport-layer hook emitting Sentry metrics

The repeated deadlock prevents the buffer from draining, causing the steady memory growth reported in #2963.

Fix — @mutex.owned? guard in TelemetryEventBuffer

A single guard in the base class protects all buffers (metrics, logs) against all re-entrant callers, regardless of origin:

def add_item(item)
  return self if @mutex.owned?
  @mutex.synchronize { ... }
end

def flush
  return self if @mutex.owned?
  @mutex.synchronize { ... }
end

Mutex#owned? returns true only when the same thread already holds the lock — the exact condition that would deadlock. Items from re-entrant calls are silently dropped. Normal concurrent access from other threads is unaffected (they block on synchronize as before).

Both add_item and flush are guarded because both acquire @mutex and call send_items.

Why not also guard at the integration layer?

An earlier iteration added a thread-local flag in HTTPTransport (sending?) and checks in Sentry::Yabeda::Adapter to skip metric forwarding during transport sends. This was removed because:

  • The @mutex.owned? guard is sufficient — it catches the deadlock at the exact point it would occur
  • The thread-local flag added cross-gem coupling (transport internals leaked into the Yabeda adapter)
  • The outer filter only saved trivial work (a string join + method call that hits the guard and returns)
  • A general fix in the base class is more reliable than per-integration opt-outs

Call chain (with fix applied)

MetricEventBuffer#flush / #add_item
  @mutex.synchronize  ← thread acquires mutex
    send_items
      Client#send_envelope → HTTPTransport#send_data
        Net::HTTP POST
          → instrumentation fires (yabeda / custom / any)
            → Sentry.metrics.*
              → MetricEventBuffer#add_item
                  return self if @mutex.owned?  ← drops safely, no deadlock

Changes

File Change
sentry-ruby/lib/sentry/telemetry_event_buffer.rb @mutex.owned? guard in add_item and flush
sentry-ruby/spec/support/shared_examples_for_telemetry_event_buffers.rb Re-entrancy specs (no deadlock, silent drop)

Fixes #2963
Refs getsentry/repro#46

sentry-junior Bot and others added 2 commits June 7, 2026 17:21
MetricEventBuffer uses a non-reentrant Mutex. When HTTP instrumentation
(e.g. yabeda-http_requests, yabeda-rails) observes the outgoing Net::HTTP
call made by HTTPTransport#send_data, it triggers Sentry::Yabeda::Adapter
which calls Sentry.metrics.* -> MetricEventBuffer#add_item on the same
thread that already holds the buffer mutex, raising:

  ThreadError: deadlock; recursive locking

Fix: set a thread-local flag (SENTRY_SENDING_KEY) in HTTPTransport for
the duration of the do_request call and check it in all Adapter#perform_*!
methods, silently skipping metric forwarding for Sentry's own HTTP sends.

Fixes #2963

Co-Authored-By: junior <noreply@example.com>

Co-authored-by: neel <neel.shah@sentry.io>
Any transport instrumentation that calls Sentry.metrics.* from within
transport#send_data re-enters MetricEventBuffer#add_item on the same
thread that already holds @Mutex inside send_items, raising:

  ThreadError: deadlock; recursive locking

This includes any non-Yabeda code such as custom transports or other
HTTP instrumentation (see getsentry/repro#46 for a minimal repro using
a ReentrantTransport that calls Sentry.metrics directly from send_data).

Fix: check @mutex.owned? before acquiring in add_item and silently drop
the item when the current thread already holds the lock. This is the
defensive fix at the correct abstraction level and protects all buffers
(MetricEventBuffer, LogEventBuffer) from all callers.

The sentry-yabeda Adapter guard (HTTPTransport.sending?) is retained as
defense-in-depth and avoids constructing a MetricEvent that will be
dropped, but the buffer-level fix is the load-bearing protection.

Refs #2963
Refs getsentry/repro#46

Co-Authored-By: junior <noreply@example.com>

Co-authored-by: johdax <johannes.daxboeck@sentry.io>
@sentry-junior sentry-junior Bot changed the title fix(yabeda): skip Sentry adapter during own HTTP transport sends fix: guard TelemetryEventBuffer against re-entrant mutex acquisition Jun 8, 2026
sentry-junior Bot and others added 2 commits June 8, 2026 08:14
Replace instance_double with []: nil keyword syntax (invalid in Ruby 2.7
parser) with build_fake_response, consistent with the rest of the spec.

Co-Authored-By: junior <noreply@example.com>

Co-authored-by: johdax <johannes.daxboeck@sentry.io>
… guard

Remove the thread-local flag layer (HTTPTransport.sending?, with_sentry_sending,
Yabeda adapter sentry_http_sending? checks) in favor of the @mutex.owned? guard
in TelemetryEventBuffer alone. This is simpler, more general, and eliminates
cross-gem coupling. Also add the guard to flush which had the same exposure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@dingsdax dingsdax marked this pull request as ready for review June 8, 2026 08:26
@sl0thentr0py sl0thentr0py merged commit 129ada3 into master Jun 8, 2026
150 checks passed
@sl0thentr0py sl0thentr0py deleted the fix/yabeda-skip-sentry-http-requests branch June 8, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Possible memory leak when logs and metrics are enabled.

2 participants