OTLP browser telemetry export: follow-up improvements (KERNEL-1596)

[archandatta]

tldr

Follow-up improvements to the in-VM OTLP browser-telemetry exporter (KERNEL-1596), post #308. Independent, reviewable commits. Off-by-default export is preserved; nothing changes for existing sessions unless the new toggle/knobs are set.

What changed

1. Externalize batch knobs — BTEL_OTLP_MAX_QUEUE_SIZE (2048), BTEL_OTLP_EXPORT_INTERVAL (1s), BTEL_OTLP_EXPORT_TIMEOUT (30s). Defaults match the OTel SDK's, so behavior is unchanged unless set; validated as > 0.
2. Per-session export toggle — new export.otlp.enabled on the telemetry API. Export is off by default and gated on the toggle, independent of whether an export destination is configured, so an always-injected relay does not generate work that gets dropped. The writer moves under a runtime-toggleable controller; toggling is best-effort (never fails a telemetry apply).
3. Drop / failure / exported metrics — queue-overflow drops (previously log-only) plus export failures and exported counts as Prometheus counters (kernel_otlp_records_dropped_total, kernel_otlp_export_failures_total, kernel_otlp_records_exported_total), owned above the writer so they stay monotonic across export toggles.
4. Crash/OOM severity — service_crashed and system_oom_kill now map to ERROR in the OTLP converter (were INFO), so consumers can alert on them.
5. Renderer-crash event — detect Inspector.targetCrashed (an "Aw, Snap!" renderer crash that leaves the browser process alive, so no service_crashed fires) and emit a page_crashed event, mapped to ERROR. Categorized as page (produced by the CDP monitor, captured only while the page category is enabled).

Live test results

Verified full-stack against the headless image + a real dockerized OpenTelemetry collector, driving the real telemetry API and a real Chromium over CDP:

• Batch knobs: custom BTEL_OTLP_* values surfaced in the boot config (otlp_max_queue_size=512 otlp_export_interval=500ms otlp_export_timeout=30s); export shows available, not started, at boot.
• Export toggle: GET /telemetry → 404 at boot; PUT with export.otlp.enabled:true → 201 echoing it. With export off but page/network capture on, exported_total froze (10 → 10) and the collector received nothing new; re-enabling resumed export (10 → 28), confirming the controller rebuilds the one-shot writer.
• Renderer crash: a real chrome://crash produced a page_crashed record at the collector with category=page, url=chrome://crash/, and SeverityText: ERROR.
• Converter fidelity: network_response carried promoted attributes (http.request.method, url.full, http.response.status_code=200); screenshot/monitor categories never reached the collector.
• Metrics: /metrics served kernel_otlp_records_dropped_total 0, kernel_otlp_export_failures_total 0, kernel_otlp_records_exported_total 28.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes the OTLP lifecycle (off-by-default, runtime toggle, tail-only replay) and telemetry API semantics, but export remains best-effort and existing VMs are unchanged unless the new toggle is enabled.

Overview
OTLP export no longer starts at boot when an endpoint is provisioned. Export is gated on export.otlp.enabled in the telemetry API (off by default). A runtime OTLPExportController starts/stops the sink; reconcileExport runs after PUT/PATCH under a separate exportMu so toggle-off drains do not block GET /telemetry. Re-enabled writers read from the stream tail so the ring is not replayed.

Config and observability: BTEL_OTLP_MAX_QUEUE_SIZE, BTEL_OTLP_EXPORT_INTERVAL, and BTEL_OTLP_EXPORT_TIMEOUT tune the OTel batch processor (validated at load). OTLPMetrics plus a drop-counting slog handler feed Prometheus counters for dropped, failed, and exported records.

Events: CDP Inspector.targetCrashed emits page_crashed (page category). OTLP severity maps service_crashed, system_oom_kill, and page_crashed to ERROR.

^{Reviewed by Cursor Bugbot for commit a31b83b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Drop counter wrapper loses attrs

Medium Severity

dropCountingHandler embeds slog.Handler and only overrides Handle, so promoted WithAttrs / WithGroup return the unwrapped base handler. Any logr/OTel path that names or attaches values before logging dropped log records bypasses the counter, leaving kernel_otlp_records_dropped_total stuck at 0 despite real queue overflow.

 

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

[cursor]

Stale URL on page crash

Medium Severity

page_crashed fills url from m.sessions attach-time target info, which is never updated on navigation. Other page events use the computed nav context’s current URL. After the first navigation, crash events report the wrong page URL even though the OpenAPI text describes the URL at crash time.

Additional Locations (1)

• server/lib/cdpmonitor/handlers.go#L730-L735

 

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

[Sayan-]

Approve — reviewed all 11 commits (correctness + adversarial), ran the affected Go test packages, and rebuilt the headless image to re-run the live test plan end-to-end against a real dockerized OTel collector. Everything checks out.

Commit-by-commit

• Externalize batch knobs — BTEL_OTLP_MAX_QUEUE_SIZE/EXPORT_INTERVAL/EXPORT_TIMEOUT, validated > 0, defaults match the OTel SDK so behavior is unchanged unless set. Applied only when > 0 in newOTLPStorage. Good.
• Per-session export toggle — export is off by default and gated on export.otlp.enabled, independent of whether an endpoint is provisioned; writer lifecycle moves under OTLPExportController (one-shot writer rebuilt per enable). Rollback-on-failure, PATCH-preserves / PUT-replaces semantics, and the no-op short-circuit all trace correctly.
• Drop/failure/exported metrics — verified the full chain against vendored SDK v0.20.0: global.Warn("dropped log records", "dropped", d uint64) → V(1).Info → logr→slog level Info-1 → the Info-1 handler threshold in main.go. attrUint64 handles all three type boxings. Counters owned above the writer so they stay monotonic across toggles.
• Crash/OOM → ERROR — service_crashed, system_oom_kill, page_crashed map to ERROR in the converter.
• Renderer-crash event — Inspector.enable added to enableDomains; Inspector.targetCrashed → page_crashed (category page). Untracked-session path falls back to target_type: other (valid enum) rather than "", with a regression test. Category gating confirmed: the monitor publishes via telemetrySession.Publish, which drops page events unless the page category is active.
• Shutdown ordering (Bugbot) — OTLP stop moved to main after the servers drain (mirrors s2Writer); Stop's ctx-exhausted branch closes the provider on a detached 2s budget so a hung flush can't orphan batch-export goroutines. Correct.

Tests

go test ./lib/events/ ./lib/cdpmonitor/ ./lib/metrics/ ./cmd/api/api/ ./cmd/config/ — all green. TestOTLPMetrics_CountsRealDrops forces a real batch-queue overflow through the SDK global logger, so a future SDK message rename fails CI instead of silently zeroing the metric — nice guard.

Live test plan (rebuilt headless image + real OTel collector)

Built the image and ran it against a dockerized otel-collector-contrib + an nginx target, driving the real telemetry API and real Chromium 148 over CDP (custom knobs 512 / 500ms / 30s). All PR claims reproduced:

• Boot config surfaced the knobs; OTLP export available (not started).
• GET /telemetry → 404 at boot; PUT with export.otlp.enabled:true → 201 echoing it.
• Real chrome://crash → page_crashed at collector with SeverityText: ERROR / SeverityNumber: Error(17), source.event=Inspector.targetCrashed, category=page.
• Converter fidelity: network_response carried http.request.method, url.full, http.response.status_code=Int(200); zero screenshot/monitor records reached the collector.
• Toggle: with export off, a full page navigation produced zero new exported records (exported_total frozen 22→22); re-enabling rebuilt the writer and resumed (22→54).
• /metrics: kernel_otlp_records_dropped_total 0, export_failures_total 0, records_exported_total climbing.

Non-blocking notes (no changes required)

1. reconcileExport holds monitorMu for up to otlpStopTimeout (5s) during a toggle-off drain; if the relay is unreachable this can briefly head-of-line-block concurrent GET/PUT/PATCH /telemetry. Latency only, not correctness — consider draining outside the lock if toggling gets frequent.
2. BTEL_OTLP_MAX_QUEUE_SIZE is only validated > 0 while the export batch size is fixed at 200; setting it below 200 won't crash but interacts oddly with the constant. Consider validating >= 200 or a doc note.
3. PUT with an omitted export block disables export (defaults false) — correct PUT-replace semantics and consistent with how omitted categories turn off, but a sharp edge for consumers; PATCH correctly preserves. Worth a sentence in the openapi description.

Nice, well-tested change. LGTM.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a31b83b. Configure here.}

[cursor]

Disable drain exports off-window

Medium Severity

On export disable, Stop cancels the read loop then Drains the ring with no seq cutoff. Events published after the toggle-off (capture still on) are forwarded, violating the “only while export is on” invariant and inflating kernel_otlp_records_exported_total.

Additional Locations (2)

• server/lib/events/otlpstorage.go#L290-L305
• server/lib/events/eventsstorage.go#L70-L87

 

^{Reviewed by Cursor Bugbot for commit a31b83b. Configure here.}