feat: StreamCloud — Rust port of LingBot-Map (streaming 3D reconstruction)#612
Draft
ruvnet wants to merge 5 commits into
Draft
feat: StreamCloud — Rust port of LingBot-Map (streaming 3D reconstruction)#612ruvnet wants to merge 5 commits into
ruvnet wants to merge 5 commits into
Conversation
Add a self-contained Cargo workspace porting LingBot-Map (robbyant/lingbot-map, Apache-2.0) — a streaming feed-forward 3D reconstruction model — to Rust. Foundation (P0-P2, builds + tests green): - lingbot-memory: StreamingMemory over the real ruvector-core VectorDB API, replacing the FlashInfer paged KV cache with lock-free HNSW retrieval (O(log N) long-range drift correction, no VRAM ceiling). Long-range recall verified at 5000-frame range. - lingbot-tensor: ModelConfig (serde from config.json) + header-only safetensors WeightIndex (validate multi-GB checkpoints without loading them); candle weight materialization behind a feature flag. - ADR-0001..0005: workspace topology, ruvector memory layer, candle/safetensors backend, wgpu+WebGPU/WASM rendering & GitHub Pages deploy, MP4/PNG output. - PROGRESS.md: loop-state tracker driving the iterative build-out. - .cargo/config.toml: sparse crates.io protocol for sandboxed/offline builds. Excluded from the root workspace so it never perturbs the parent build and can be split into its own repo unchanged. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_016aHCtJGcejmWbzX2PXKtLH
…aming MP4 - lingbot-model: shared types (Frame/Point/PointCloud/KeyframeFeatures) and the Reconstructor trait. SyntheticReconstructor (pure-Rust, wasm-safe, default): monocular depth -> pinhole projection -> colored point cloud, with anchor context folded in as drift correction. candle GeometricContextTransformer (feature-gated) loads safetensors; both backends compile and are tested. - lingbot-io: FrameSink trait + PngSequenceSink + streaming Mp4Sink (openh264 H.264 encode -> mp4 mux). Output MP4 round-trips through the mp4 reader as a valid single AVC track; PNG export verified. - ADR-0006: synthetic-fallback strategy & checkpoint provenance. Tests green: lingbot-model (6) + lingbot-io (3), candle backend builds clean. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_016aHCtJGcejmWbzX2PXKtLH
- lingbot-pipeline: StreamingReconstructor wiring model + memory + io. Per frame: encode -> memory.retrieve_context(top_k) -> reconstruct(frame, anchors) -> insert_keyframe (the ADR-0002 KV-cache replacement). CPU orbit SoftwareRenderer (z-buffered point-cloud rasterizer showing real 3D parallax) + deterministic synthetic scene source. 5 tests pass. - lingbot-cli (bin `lingbot`): `render` runs the stream and exports a streaming MP4 + optional PNG sequence + final still; `inspect` reads a safetensors checkpoint header. Verified end-to-end: 480x360 MP4 + 48 PNG frames generated. - Reconstructor trait split into encode/reconstruct so anchors are retrieved between them; ModelConfig re-exported from lingbot-model. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_016aHCtJGcejmWbzX2PXKtLH
… port complete - lingbot-wasm: wasm-bindgen LingbotDemo exposing the streaming pipeline to JS; portable brute-force cosine anchor retrieval (native keeps ruvector HNSW). Cross-compiles to wasm32 (~500 KB). - demo/: WebGPU point-cloud renderer (2D canvas fallback) + static site loading the wasm-pack bundle; live stats (frame/points/anchors/keyframes). - .github/workflows/lingbot-pages.yml: build wasm-pack bundle + deploy demo/ to GitHub Pages (workflow_dispatch + path-filtered). - .github/workflows/build-lingbot.yml: workspace CI (fmt + clippy -D warnings + test + candle build + wasm build). - lingbot-tensor/load.rs: candle safetensors materialization helpers. - scene generator moved into lingbot-model (shared by native pipeline + wasm). - README/PROGRESS finalized; fmt + clippy clean. Verified: cargo fmt --check, clippy -D warnings, 25 native tests + 8 candle tests, wasm32 release build — all green. Native CLI produced a real MP4 + PNGs. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_016aHCtJGcejmWbzX2PXKtLH
Full rename of the standalone port:
- directory lingbot-map-rs/ → streamcloud-rs/
- crates lingbot-{memory,tensor,model,io,pipeline,cli,wasm} → streamcloud-*
- binary `lingbot` → `streamcloud`; wasm export LingbotDemo → StreamCloudDemo
- workflows → streamcloud-pages.yml / build-streamcloud.yml; root workspace
`exclude` updated to streamcloud-rs
- READMEs, ADRs, demo UI and all branding now read "StreamCloud"
Upstream attribution retained throughout: StreamCloud is described as a Rust
port of LingBot-Map (robbyant/lingbot-map, Apache-2.0), with HF/arXiv links and
the Apache-2.0 LICENSE kept for compliance.
Verified after rename: cargo fmt --check, clippy -D warnings, full test suite
(25 native + candle), wasm32 release build, and the renamed `streamcloud` binary
all green.
Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_016aHCtJGcejmWbzX2PXKtLH
ruvnet
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
StreamCloud is a standalone Rust port of LingBot-Map (
robbyant/lingbot-map, Apache-2.0, arXiv:2604.14141) — a feed-forward, streaming 3D reconstruction foundation model. Self-contained workspace atstreamcloud-rs/, excluded from the root build, splittable into its own repo.The headline architectural change: the FlashInfer paged KV cache (VRAM-bound, forgets past a sliding window) is replaced by a lock-free HNSW trajectory memory backed by
ruvector-core—O(log N)long-range drift correction with no VRAM ceiling.Status: complete (P0–P9, all ADRs)
streamcloud-memoryStreamingMemoryover the realruvector-coreVectorDBAPI (KV-cache → HNSW). Long-range recall verified at 5000 frames.streamcloud-tensorModelConfig+ header-only safetensorsWeightIndex+ candle weight loaders.streamcloud-modelSyntheticReconstructor(pure-Rust, wasm-safe, default) + candleGeometricContextTransformer(loads safetensors).streamcloud-ioFrameSink, PNG export, streaming H.264 MP4 (openh264→mp4; round-trips through the reader as a valid AVC track).streamcloud-pipelinestreamcloud-clistreamcloud render→ PNG + MP4,streamcloud inspect).streamcloud-wasm+demo/StreamCloudDemo+ WebGPU point-cloud renderer (2D fallback); static site.Demos
cargo run -p streamcloud-cli --release -- render --frames 60 --out out.mp4 --png-dir frames→ streaming MP4 + PNG sequence + final still (orbiting camera shows real 3D parallax).demo/, deployed by.github/workflows/streamcloud-pages.yml(path-filtered + manual dispatch).Verification (all green):
cargo fmt --check,cargo clippy --workspace -D warnings, 25 native tests + 8 candle tests,wasm32-unknown-unknownrelease build. CI:.github/workflows/build-streamcloud.yml.Honest scope notes
streamcloud-rs/directory (write access is scoped toruvnet/ruvector); structured to split out cleanly.Attribution
StreamCloud is an independent reimplementation crediting upstream LingBot-Map throughout (HF + arXiv links, Apache-2.0 LICENSE retained). Model weights remain the original authors' property under their license.
ADRs
0001 topology · 0002 ruvector memory · 0003 candle/safetensors · 0004 wgpu+WebGPU/Pages · 0005 MP4/PNG · 0006 synthetic fallback & provenance.
🤖 Generated with claude-flow