hostinet: support checkpoint/restore by shayonj · Pull Request #13404 · google/gvisor

shayonj · 2026-06-10T11:37:07Z

Host networking did not have checkpoint/restore support. A sandbox running with --network=host could not be checkpointed because hostinet.Stack was not savable, and restore did not define what should happen to open host sockets.

This change makes the hostinet stack savable and defines that restore contract. Host socket fds cannot cross a checkpoint boundary, so Socket.fd is tagged state:"nosave" and save does not touch host sockets. With checkpoint --leave-running, the original sandbox keeps serving traffic on its existing connections. If save fails, the sandbox is left unharmed.

Without --leave-running, the sandbox exits after save and the host kernel closes its sockets. That sends FIN or RST so remote peers do not hang on half-open connections.

In the restored sandbox, afterLoad sets each socket fd to -1. Host syscalls then fail with EBADF. Readiness reports hangup and error so epoll waiters wake immediately, and tasks that were blocked in I/O see EBADF when their syscall restarts. State() reports no TCP state for these sockets rather than logging a warning on every /proc/net/tcp read. Applications are expected to reconnect after restore.

The stack's host-derived state is not saved. This includes /proc/net/dev, /proc/net/snmp, TCP buffer sizes, and allowed socket types. During restore, runsc configures a fresh hostinet stack before seccomp filters are installed. ReplaceConfig then copies the destination host configuration into the deserialized stack and transfers ownership of the fresh proc net handles. The restored sandbox therefore uses the destination host configuration and limits.

A hostinet checkpoint contains no netstack state and a netstack checkpoint contains no hostinet state, so the checkpoint now records the network type in its metadata. Restore fails with a clear error when the network stack kind differs between checkpoint and restore, instead of panicking inside ReplaceConfig during kernel load. Sandbox and none networking both use netstack and remain interchangeable. Checkpoints taken before the network type was recorded skip this check.

The hostinet save_resume syscall tests are enabled because save does not touch host sockets. The save and restore syscall tests remain excluded because they expect sockets to keep working after restore.

Container tests cover restored socket behavior for TCP, UDP, epoll, and a blocked accept. They also cover remote peer close after checkpoint, continued service after checkpoint --leave-running, and the error when restoring a host networking checkpoint with sandbox networking. The focused hostinet unit tests and the hostinet save_resume syscall tests pass on x86_64. The checkpoint/restore user guide documents the restored socket behavior.

Fixes #6243

avagin · 2026-06-10T17:18:13Z

In the restored sandbox, afterLoad sets each socket fd to -1. Host syscalls then fail with EBADF. Readiness reports hangup and error so epoll waiters wake immediately, and tasks that were blocked in I/O see EBADF when their syscall restarts. State() reports no TCP state for these sockets rather than logging a warning on every /proc/net/tcp read. Applications are expected to reconnect after restore.

I think S/R should properly save and restore listening sockets. For connected sockets, it should be better to return ECONNABORTED rather than EBADF.

shayonj · 2026-06-10T17:56:18Z

Thanks for the quick feedback. Got it. I am def a bit new to this area and originally followed the unix socket close on save shape because i was thinking no host socket state crosses the checkpoint boundary and it being the simplest contract to reason about.

I did realize that the guest fd being still open in the sentry's fd table after restore means EBADF misdescribes what died I suppose, which is the connection is gone and not the descriptor. And listening sockets carry no connection state at all.

I'll update the PR. 2 quick questions:

If re-bind fails during restore, for example EADDRINUSE when restoring next to a still-running --leave-running source on the same host, I am thinking to fail the restore rather than degrade the socket. Let me know if you anticipate issues?
also should bound but unconnected UDP sockets be re-created too?
- established TCP connections need TCP_REPAIR-style surgery I reckon.

nybidari · 2026-06-25T16:34:13Z

+	restoredListeners.mu.Unlock()
+	for _, s := range sockets {
+		if err := s.restoreListener(); err != nil {
+			panic(fmt.Sprintf("failed to restore listening host socket (family=%d, addr=%x): %v", s.family, s.savedListener.addr, err))


This causes panic of the entire checkpoint/restore if restore failed for one connection.

We can instead log a warning that the restore was not successful for the socket and then continue to restore.

nybidari · 2026-06-25T16:34:16Z

+		if err := unix.SetsockoptString(fd, unix.SOL_SOCKET, unix.SO_BINDTODEVICE, l.bindToDevice); err != nil {
+			_ = unix.Close(fd)
+			return fmt.Errorf("setting SO_BINDTODEVICE to %q: %w", l.bindToDevice, err)
+		}


This can fail if the device/interface name changes during restore (if the restore happens on a different machine). Leave a comment on this method saying if anything fails, we will not restore this socket but the restore will continue for other sockets.

nybidari · 2026-06-25T16:34:24Z

 	}
+	if s.fd < 0 {
+		// Reading SO_ERROR after EPOLLERR must still succeed.
+		if level == linux.SOL_SOCKET && name == linux.SO_ERROR {


nybidari

Thanks for the PR! It largely looks good :) just a few comments.

nybidari · 2026-06-25T17:35:34Z

Regarding the question on bound but unconnected UDP sockets, it is not strictly required for now, but if we could support them that would be a great! We could also add support for it in the subsequent PRs.
I think one way to support would be to identify SOCK_DGRAM sockets in beforeSave that have a local address (via getsockname) and save their state, then during restoreListener, skip the unix.Listen call for those UDP sockets.

nybidari

LGTM

shayonj · 2026-06-25T18:35:41Z

ah! sorry forgot to hit submit 😅

Thanks for the review, it all makes sense. I will keep this PR focused on restoring TCP listening sockets and related changes. I can follow up with bound and unconnected udp socket restore separately. The shape you suggested sounds reasonable to me. I will do a bit more testing just to get my mental model updated and follow up

Host networking did not have checkpoint/restore support. A sandbox running with `--network=host` could not be checkpointed because `hostinet.Stack` was not savable, and restore did not define what should happen to open host sockets. This change makes the hostinet stack savable and defines that restore contract. Host socket fds cannot cross a checkpoint boundary, so `Socket.fd` is tagged `state:"nosave"` and save does not touch host sockets. With checkpoint `--leave-running`, the original sandbox keeps serving traffic on its existing connections. If save fails, the sandbox is left unharmed. Without `--leave-running`, the sandbox exits after save and the host kernel closes its sockets. That sends `FIN` or `RST` so remote peers do not hang on half-open connections. In the restored sandbox, `afterLoad` sets each socket fd to `-1`. Host syscalls then fail with `EBADF`. `Readiness` reports hangup and error so `epoll` waiters wake immediately, and tasks that were blocked in I/O see `EBADF` when their syscall restarts. `State()` reports no TCP state for these sockets rather than logging a warning on every `/proc/net/tcp` read. Applications are expected to reconnect after restore. The stack's host-derived state is not saved. This includes `/proc/net/dev`, `/proc/net/snmp`, TCP buffer sizes, and allowed socket types. During restore, runsc configures a fresh hostinet stack before seccomp filters are installed. `ReplaceConfig` then copies the destination host configuration into the deserialized stack and transfers ownership of the fresh proc net handles. The restored sandbox therefore uses the destination host configuration and limits. A hostinet checkpoint contains no netstack state and a netstack checkpoint contains no hostinet state, so the checkpoint now records the network type in its metadata. Restore fails with a clear error when the network stack kind differs between checkpoint and restore, instead of panicking inside `ReplaceConfig` during kernel load. Sandbox and none networking both use netstack and remain interchangeable. Checkpoints taken before the network type was recorded skip this check. The hostinet `save_resume` syscall tests are enabled because save does not touch host sockets. The save and restore syscall tests remain excluded because they expect sockets to keep working after restore. Container tests cover restored socket behavior for TCP, UDP, `epoll`, and a blocked accept. They also cover remote peer close after checkpoint, continued service after checkpoint `--leave-running`, and the error when restoring a host networking checkpoint with sandbox networking. The focused hostinet unit tests and the hostinet `save_resume` syscall tests pass on `x86_64`. The checkpoint/restore user guide documents the restored socket behavior. Fixes #6243 FUTURE_COPYBARA_INTEGRATE_REVIEW=#13404 from shayonj:hostinet-save-restore-v2 734480e PiperOrigin-RevId: 940513420

Host networking did not have checkpoint/restore support. A sandbox running with `--network=host` could not be checkpointed because `hostinet.Stack` was not savable, and restore did not define what should happen to open host sockets. This change makes the hostinet stack savable and defines that restore contract. Host socket fds cannot cross a checkpoint boundary, so `Socket.fd` is tagged `state:"nosave"` and save does not touch host sockets. With checkpoint `--leave-running`, the original sandbox keeps serving traffic on its existing connections. If save fails, the sandbox is left unharmed. Without `--leave-running`, the sandbox exits after save and the host kernel closes its sockets. That sends `FIN` or `RST` so remote peers do not hang on half-open connections. Listening sockets carry no connection state, so they are restored. Save records the bound address, the backlog, and the socket options needed to re-bind, and restore re-creates the host socket with `socket`, `bind`, and `listen` before tasks resume. Connections that were pending in the backlog at checkpoint time are lost. If the listen address cannot be bound on the restoring host, the restore fails. Sockets that were connected at checkpoint time cannot be restored. `afterLoad` sets their fd to `-1` and host socket operations return `ECONNABORTED`, which describes what was actually lost, since the guest file descriptor itself remains open in the sentry. `Readiness` reports hangup and error so `epoll` waiters wake immediately, reading `SO_ERROR` after `EPOLLERR` succeeds and returns `ECONNABORTED`, and such sockets appear as closed in `/proc/net/tcp`. Applications are expected to reconnect after restore. The stack's host-derived state is not saved. This includes `/proc/net/dev`, `/proc/net/snmp`, TCP buffer sizes, and allowed socket types. During restore, runsc configures a fresh hostinet stack before seccomp filters are installed. `ReplaceConfig` then copies the destination host configuration into the deserialized stack and transfers ownership of the fresh proc net handles. The restored sandbox therefore uses the destination host configuration and limits. A hostinet checkpoint contains no netstack state and a netstack checkpoint contains no hostinet state, so the checkpoint now records the network type in its metadata. Restore fails with a clear error when the network stack kind differs between checkpoint and restore, instead of panicking inside `ReplaceConfig` during kernel load. Sandbox and none networking both use netstack and remain interchangeable. Checkpoints that predate the metadata key are treated as netstack checkpoints. The hostinet `save_resume` syscall tests are enabled because save does not touch host sockets. The save and restore syscall tests remain excluded because they expect connected sockets to keep working after restore. Container tests cover `ECONNABORTED` on restored TCP and UDP sockets, `epoll` and `SO_ERROR` behavior, a blocked accept that completes against the re-created listener, remote peer close after checkpoint, continued service after checkpoint `--leave-running`, and the error when restoring a host networking checkpoint with sandbox networking. The focused hostinet unit tests and the hostinet `save_resume` syscall tests, including the listener-heavy `accept_bind` and `socket_inet_loopback` suites, pass on `x86_64`. The checkpoint/restore user guide documents the restored socket behavior.

shayonj · 2026-07-02T19:05:19Z

Looks like it needed a rebase, just did.

shayonj force-pushed the hostinet-save-restore-v2 branch from 8a82549 to 8076308 Compare June 10, 2026 11:48

shayonj force-pushed the hostinet-save-restore-v2 branch 2 times, most recently from c00626b to 3718b5c Compare June 10, 2026 23:32

shayonj mentioned this pull request Jun 17, 2026

checkpoint/restore not supported when using hostinet #6243

Open

nybidari self-requested a review June 22, 2026 19:35

nybidari reviewed Jun 25, 2026

View reviewed changes

nybidari requested changes Jun 25, 2026

View reviewed changes

shayonj force-pushed the hostinet-save-restore-v2 branch from 3718b5c to 734480e Compare June 25, 2026 16:58

nybidari approved these changes Jun 25, 2026

View reviewed changes

nybidari added ready to pull and removed ready to pull labels Jun 25, 2026

nybidari approved these changes Jun 26, 2026

View reviewed changes

nybidari added the ready to pull label Jun 26, 2026

copybara-service Bot mentioned this pull request Jun 30, 2026

hostinet: support checkpoint/restore #13599

Open

shayonj force-pushed the hostinet-save-restore-v2 branch from 734480e to 3695a73 Compare July 2, 2026 18:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

hostinet: support checkpoint/restore#13404

hostinet: support checkpoint/restore#13404
shayonj wants to merge 1 commit into
google:masterfrom
shayonj:hostinet-save-restore-v2

shayonj commented Jun 10, 2026

Uh oh!

avagin commented Jun 10, 2026

Uh oh!

shayonj commented Jun 10, 2026 •

edited

Loading

Uh oh!

nybidari Jun 25, 2026

Uh oh!

nybidari Jun 25, 2026

Uh oh!

nybidari Jun 25, 2026

Uh oh!

nybidari left a comment

Uh oh!

nybidari commented Jun 25, 2026

Uh oh!

nybidari left a comment

Uh oh!

shayonj commented Jun 25, 2026

Uh oh!

shayonj commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

shayonj commented Jun 10, 2026

Uh oh!

avagin commented Jun 10, 2026

Uh oh!

shayonj commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nybidari Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

nybidari Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

nybidari Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

nybidari left a comment

Choose a reason for hiding this comment

Uh oh!

nybidari commented Jun 25, 2026

Uh oh!

nybidari left a comment

Choose a reason for hiding this comment

Uh oh!

shayonj commented Jun 25, 2026

Uh oh!

shayonj commented Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shayonj commented Jun 10, 2026 •

edited

Loading