Cold start: single VssStore serves all Cashu + LDK reads synchronously — 19s vs 0.56s local (34x)

[hash-money]

Measurement (signet + Cashu, same config, only storage differs)

Storage	OrangeWallet::new
StorageConfig::Vss	19,050 ms
local SQLite	557 ms

Traced with RUST_LOG=info,cdk=debug,reqwest=debug,hyper=debug,rustls=debug: ~8s in the VSS store build (build_with_sigs_auth) + Cashu::init (the Cashu DB is VSS-backed), then ~10s in ldk_node build_with_store reading LDK state — the KV reads appear serial, one network round-trip each. Feerate cache (288ms) and chain sync are minor. fetch_mint_info is a background task and not implicated (earlier misattribution, corrected).

OrangeWallet builds one VssStore and passes it as the single store to both CashuKvDatabase and LdkNodeStore (src/lib.rs), so every cold start rebuilds all state over the network.

Fix directions (in rough order of ambition)

1. Batch/parallelize the VSS reads during node build — likely the low-risk win; today's serial per-key round-trips dominate.
2. Local-primary with async VSS replication (read local when the on-disk store is warm; write-through to VSS; reconcile in background). ⚠️ This one needs design care: VSS-at-boot is what catches newer remote state after a restore-on-another-device, so a naive local-first read risks resurrecting stale channel state. A safe shape might be: local-first only when a VSS version/etag check (one round-trip) confirms local is current.

Happy to test candidate branches against our staging VSS (Caddy-fronted, https://…/vss) — the 34× gap makes wins easy to verify. Repro recipe and full logs in our downstream issue: emergent-money/graduated-wallet#247.

[hash-money]

Additional evidence that this is per-operation, not just cold start: receive-URI generation shows the same signature. The trusted-tier quote flow (get_single_use_receive_uri → cashu mint quote) measured across three configurations, same mint:

Configuration	get-invoice wall clock
local-store wallet (wallet-cli)	103 ms
mint server alone (curl, HTTPS)	~700 ms
VSS-backed wallet (Android, mobile network, LSP connected)	16,650 ms

The ~16s gap is the store layer: the cashu wallet's writes during quote issuance each cost a synchronous VSS round-trip. So batching/caching in the KV path would pay off on the hot request path, not only at boot — and it strengthens the case for fix direction 1 (batch/parallelize) as the first step, since it helps both without touching the rollback-safety semantics that make direction 2 delicate.

[tnull]

Good callout!

We technically already parallelize initial reads in LDK Node for the most part, though we can surely further make some improvements there. I think some parallelization opportunities are also possible the side of Orange SDK.

However, the biggest bottleneck here seems to VSS's usage of bitreq which serializes all these reads again by making all requests through a single, non-pipelined connection, having us eat all these round-trips again. This was briefly mentioned here but I think we never followed up implementing an actual connection pool for bitreq. (cc @TheBlueMatt @tankyleo). Now tracking here: rust-bitcoin/corepc#658

[TheBlueMatt]

However, the biggest bottleneck here seems to VSS's usage of bitreq which serializes all these reads again by making all requests through a single, non-pipelined connection, having us eat all these round-trips again.

I thought we switched Vss-client to use pipelining? That should suffice to cut parallel requests latency down substantially.

[TheBlueMatt]

I'd really like to see trace-level logging here. 16 seconds is insanely high, even if requests are being sent serially that sounds like something else is going wrong.

[hash-money]

Trace-level logs captured (@TheBlueMatt) — Android device, mobile network, staging VSS behind Caddy/HTTPS. We enabled TRACE for vss_client_ng/bitreq + rustls DEBUG, millisecond timestamps. Full cold start + one receive-URI generation:

The numbers:

• 343 VSS requests in the window (277 GetObjectRequest, 24 ListKeyVersions, 42 PutObjectRequest)
• Perfectly serial: median inter-request gap 896 ms, a steady 9–10 requests per 10s for minutes
• ~Every request opens a fresh connection: 350 rustls client handshakes for 343 requests; 251 of 342 consecutive-request gaps contain a handshake. TLS 1.3 PSK resumption is working ("Resuming session") — but resumption still costs a new TCP connect + 1-RTT handshake per request.
• Cold start on this device: Initializing orange 17:17:12 → Creating LDK node 17:19:57 (164 s in the pre-LDK phase alone, ≈ 180 serialized VSS gets) → LDK start 17:20:15. Receive-URI generation on the same run: 14.2 s ≈ 16 serialized round-trips.

So the 16 s isn't "something else" — it's arithmetic: N serialized requests × (TCP connect + TLS resume + HTTP round-trip) on a mobile-RTT path (~0.9 s each). Our earlier 19 s measurement was from a VPS near the VSS host; the phone's RTT amplifies the same per-request cost ~10×. Re: pipelining having been switched on — at our pin (orange-sdk 4e9d230 → vss-client-ng 0.5.0), the handshake-per-request pattern says each request rides a new connection, so whatever pipelining exists isn't engaging on this path.

Sample window (repeating pattern, request → fresh connection → ~900 ms → next request):

2026-07-10 17:17:12.983 TRACE [vss_client_ng::client:88] Sending GetObjectRequest 3860541630712530198 for key cZ8kPdoVXlrvfzJQS9pZ
2026-07-10 17:17:13.542 DEBUG [rustls::client::hs:73] No cached session for DnsName("lsp.emergent.money")
2026-07-10 17:17:13.542 DEBUG [rustls::client::hs:132] Not resuming any session
2026-07-10 17:17:13.741 DEBUG [rustls::client::hs:615] Using ciphersuite TLS13_AES_128_GCM_SHA256
2026-07-10 17:17:13.742 DEBUG [rustls::client::hs:472] ALPN protocol is None
2026-07-10 17:17:14.344 TRACE [vss_client_ng::client:174] Sending ListKeyVersionsRequest 4624648138170831944 for key_prefix Some("
2026-07-10 17:17:14.560 DEBUG [rustls::client::hs:130] Resuming session
2026-07-10 17:17:14.990 DEBUG [rustls::client::hs:615] Using ciphersuite TLS13_AES_128_GCM_SHA256
2026-07-10 17:17:14.991 DEBUG [rustls::client::hs:472] ALPN protocol is None
2026-07-10 17:17:15.611 TRACE [vss_client_ng::client:88] Sending GetObjectRequest 15346573794384891948 for key vQ3MSyu3cTurwbOSE7L
2026-07-10 17:17:15.802 DEBUG [rustls::client::hs:130] Resuming session
2026-07-10 17:17:15.988 DEBUG [rustls::client::hs:615] Using ciphersuite TLS13_AES_128_GCM_SHA256
2026-07-10 17:17:15.989 DEBUG [rustls::client::hs:472] ALPN protocol is None
2026-07-10 17:17:16.403 TRACE [vss_client_ng::client:88] Sending GetObjectRequest 951519899227931550 for key vQ3MSyu3cTurwbOSE7LIK

Happy to re-run the same capture against any candidate branch — the setup is scripted now.

[hash-money]

New datapoint isolating the per-operation cost from mobile RTT: same-machine A/B (desktop, wired network, ~2 ms RTT to the Caddy-fronted VSS — not a phone), identical binary and config, only StorageConfig differs. Measured the receive/mint saga (mint-quote paid → proofs minted → PaymentReceived), 5,000-sat receive, 39 proofs:

Storage	Invoice creation	Quote paid → PaymentReceived
local SQLite	~1 s	~1 s (n=2)
VSS	10 s	60 s

Step breakdown for the VSS run: Paid notification → saga prepare 10 s → execute +6 s → post_mint +3 s → PaymentReceived +41 s. That 41 s tail is entirely after the mint's HTTP response is back — it's the proof/counter/saga-state persistence batch. So even with negligible network RTT, the hot path serializes enough store round-trips to take a minute; the earlier Android numbers (~896 ms/request) are the same arithmetic with mobile RTT multiplied in.

One more observation that might sharpen fix direction 2: our production VSS store holds 1,339 keys for a store whose LDK-only baseline (measured before the Cashu tier used VSS) was ~17 keys — i.e., the Cashu WalletDatabase dominates both key count and write churn. Since Wallet::restore() already re-derives Cashu proofs from the seed on a fresh install, it's worth asking whether CashuKvDatabase needs to live in VSS at all: making the Cashu store local-primary (VSS only for LDK state, which is what actually needs the rollback-safety semantics) would take the entire trusted-tier hot path off the network without touching the delicate channel-state design in direction 2. Happy to test that shape against our staging VSS too.

[tnull]

I thought we switched Vss-client to use pipelining? That should suffice to cut parallel requests latency down substantially.

We pipeline reads, but refrained from pipelining writes/removes due to idempotency concerns. While it now looks like the root cause might be the insane number of cashu-related keys, it's likely still very relevant as we require quite a few writes on each event handling and IIRC we also still have interleaving writes when we deserialize channelmanager etc.? So IMO using multiple connections would help a lot here to avoid head of line blocking.

[tnull]

One more observation that might sharpen fix direction 2: our production VSS store holds 1,339 keys for a store whose LDK-only baseline (measured before the Cashu tier used VSS) was ~17 keys — i.e., the Cashu WalletDatabase dominates both key count and write churn.

Okay, if these 1339 keys are indeed read on startup (sequentially or even in batches) the 15 seconds start making a lot more sense.

[tnull]

• ~Every request opens a fresh connection:

It seems bitreq currently marks the connection as dead if no keep-alive header is set, but HTTP 1.1 doesn't require keep-alive headers, and Caddy might just omit them: rust-bitcoin/corepc#659

[TheBlueMatt]

Can yall confirm if Caddy is returning Connection headers? Might have to just use curl to make an identical connection (or import the server private key into WireGuard, which can use that to decrypt SSL).

[hash-money]

Confirmed (@TheBlueMatt): Caddy omits the Connection header entirely on our VSS endpoint. Forced HTTP/1.1 against https://lsp.emergent.money/vss/getObject:

< HTTP/1.1 401 Unauthorized
< Alt-Svc: h3=":443"; ma=2592000
< Content-Length: 35
< Date: Sat, 11 Jul 2026 17:40:52 GMT
< Via: 1.1 Caddy
< Vss-Protocol-Version: 0
< Content-Type: application/octet-stream

No Connection: and no Keep-Alive: — which HTTP/1.1 §9.3 defines as persistent-by-default, but per rust-bitcoin/corepc#659 bitreq treats as connection-dead. That closes the loop on our capture: header omitted → bitreq drops the connection after every request → fresh TCP+TLS per VSS op → ~896 ms × N serialized.

(Caddy also negotiates HTTP/2 by default when offered — the bitreq path is presumably HTTP/1.1-only, which is fine once keep-alive semantics are honored.)

Standing offer stands: when the corepc fix (or a vss-client bump) is in a branch, we'll re-run the same A12 capture for before/after numbers.

[hash-money]

One more workload datapoint with a built-in control: a trusted-tier receive's CDK issue saga made 91 VSS requests in 46s (median 634ms apart, serial) — while in the same log window, cdk's mint HTTP calls reused pooled hyper connections and completed in ~200ms each. Same process, two HTTP stacks: hyper-with-pool is fine, bitreq-without-keep-alive crawls. Nicely isolates the fix surface to the vss-client/bitreq path.