rtksvr: stuck raw/rtcm decoder state for one input stream is never recovered by reconnecting the stream; only a full rtksvrstart() reset fixes it

[Loose-zhang]

Version

demo5, patch level b34L (VER_RTKLIB="demo5", PATCH_LEVEL="b34L")

Summary

When using rtkrcv with an ephemeris-only "corr" input stream (broadcast
ephemeris relayed from an NTRIP caster, e.g. BKG's BCEP00BKG0 mountpoint),
the per-stream RTCM3 decoder occasionally gets stuck after long continuous
operation (observed after ~3 days). Once stuck, satellite ephemeris (toe/toc)
stops updating and never recovers, even though:

• The underlying network stream is confirmed healthy (bytes are still
  flowing from the source).
• Restarting the process feeding that stream (an independent str2str
  relay in front of it) does NOT recover it.
• Only issuing a full restart inside rtkrcv (stop + start of the whole
  rtk server) recovers it.

This suggests the problem is not in the network/stream layer at all, but in
per-stream decoder state that lives inside the long-running rtkrcv process
and is never reset except on a full server restart.

Root cause (from code inspection)

init_raw() / init_rtcm(), which reset a stream's raw/rtcm decoder state
machine (nbyte, len, sync position, etc.), are only called in
rtksvrstart() in src/rtksvr.c:

for (i=0;i<3;i++) { /* input/log streams */
    ...
    /* initialize receiver raw and rtcm control */
    init_raw(svr->raw+i,formats[i]);
    init_rtcm(svr->rtcm+i);
    ...
}

There is no code path to reinitialize the decoder for a single input stream
(rover/base/corr) independently. Reconnecting the underlying stream_t
(automatically via the built-in timeout/reconnect logic, or manually by
restarting whatever process feeds that stream) only replaces the transport
socket — it never touches svr->raw[i] / svr->rtcm[i]. So if that
decoder's internal state gets desynchronized/wedged for any reason (e.g. a
malformed or truncated RTCM3 message straddling a read buffer boundary), it
appears to be unrecoverable short of a full rtksvrstart()/rtksvrstop()
cycle, which also resets the RTK filter state (rtkfree+rtkinit) and
therefore loses float/fixed ambiguity state — much more disruptive than
necessary for what should be an isolated, single-stream issue.

Setup used when this was observed

• inpstr1: rover raw observations (NTRIP)
• inpstr2: base raw observations (NTRIP)
• inpstr3: corr = broadcast ephemeris only, RTCM3, relayed via a local
  str2str -in ntrip://.../BCEP00BKG0 -out tcpsvr://:PORT process, consumed
  by rtkrcv as inpstr3-type=tcpcli, inpstr3-format=rtcm3
• misc-navmsgsel=all
• misc-timeout=10000, misc-reconnect=10000 (defaults)

Steps to reproduce

Not reliably reproducible on demand — it appears to be a rare edge case in
the byte stream that only shows up after multiple days of continuous
operation on a live multi-GNSS RTCM3 feed. Happy to capture and share trace
logs (misc-trace>=3) around the next occurrence if that's useful.

Suggested fix / feature request

Expose a way to reset a single input stream's decoder state (raw/rtcm init

• its read buffers) independently of the other streams and without
  resetting the RTK filter state — e.g. a rtksvrresetstream(rtksvr_t *svr, int index) that redoes what rtksvrstart() does for svr->raw[index] /
  svr->rtcm[index] / svr->buff[index] / svr->pbuf[index] only. This
  would let a user-level watchdog recover from a stuck stream decoder
  (detected e.g. via stale ephemeris timestamps) without forcing a full
  solution restart.

Alternatively, if there's a known failure mode in the RTCM3 byte-stream
parser that can leave it permanently desynchronized without a way to
self-resync, pointers to where to look would be appreciated.

[ourairquality]

Can you add some extra tracing to rtksrc.c update_eph(), check if it is still being called and also trace the time eph1->toe to see if there is an issue with that. Also the caller decoderaw() and the RTCM3 path, the result of input_rtcm3() and the error handling logic below. Also rtcm.c input_rtcm3(), trace the error paths there. Perhaps use trace level 1 for these to help avoid the trace file being too large. Shall try injecting errors into the stream to see if a lockup can be reproduced.

[ourairquality]

Injecting random errors and losses did not reproduce this issue. Did notice that the rtcm->time for the ephemeris corrections stream was not changing from its initial value and that is a problem as it needs to be maintained in the ballpark of the actual time for the RTCM time decoding to work. There is some code in decode_rtcm3() that adds a receiver option "-RT_INP" and adding that to this corrections stream did get the rtcm->time updating and that might be something to try. The real fix may be to keep rtcm->time updated in the ephemeris decoding paths too, at least in the ballpark. The MSM observation messages update the rtcm->time in mixed streams, see adjtime(). I've also been using an ephemeris only stream in some cases, for quick starts, and thank you for the report.

[Loose-zhang]

Thank you.@ourairquality

I will try setting:

inpstr3-rcvopt = -RT_INP

for the correction stream. As I understand it, this causes decode_rtcm3() to refresh rtcm->time from the current system time for each incoming message.

Our correction stream contains ephemeris messages only and no MSM observations, so without -RT_INP, rtcm->time may indeed remain close to its initial value. This provides a more specific and plausible explanation than my original assumption that the decoder state itself had become permanently stuck.

I will enable -RT_INP for the correction stream only and monitor it over several days. I will also add low-level tracing around update_eph(), the return value from input_rtcm3(), and the relevant error paths, including the decoded toe/toc values and the current rtcm->time.

Since the issue usually appears only after about three days, I cannot confirm the result immediately. I will report back after the long-running test, or provide traces from the next occurrence if it still happens.

Thank you also for testing random corruption and packet loss. If -RT_INP prevents the issue, that would strongly suggest that the root cause is a stale rtcm->time value in an ephemeris-only stream rather than an unrecoverable parser state.