Every numaVM workspace is a Firecracker microVM. When a VM sits idle, we snapshot it — freeze the CPU, memory, and device state to disk — and reclaim the host resources. Restore takes 176ms. The VM resumes exactly where it left off.

The problem: if someone connects to a sleeping VM, the connection is refused. For HTTP, we had a workaround — Caddy catches the request, triggers a wake, shows a loading page. But for SSH, database ports, custom TCP services? Connection refused. No workaround, no explanation.

We fixed this. Every VM now gets a public IPv6 address, and any TCP connection to a sleeping VM wakes it up transparently. SSH, Postgres, Redis, a web server on port 3000 — doesn’t matter. The client sees a slow connect (1-2 seconds) instead of a refused connection.

This post explains how it works at the netfilter level and why most platforms can’t do this.

The landscape: how everyone else handles sleeping services

Scale-to-zero is not a new idea. But almost everyone implements it at Layer 7 (HTTP only), and almost nobody handles raw TCP wake.

Vercel avoids the problem entirely. Their Fluid Compute model keeps at least one function instance warm at all times (“scale to one, not scale to zero”). Predictive scaling pre-warms additional instances based on traffic patterns. They report zero cold starts for 99.37% of requests. But this is HTTP-only serverless functions on Lambda — there’s no VM, no SSH, no arbitrary TCP.

Fly.io runs Firecracker VMs and has a Fly Proxy that intercepts HTTP connections to stopped machines. When a request arrives, the proxy starts the machine and forwards the connection. They also support Firecracker snapshots for faster resume (~300ms). But SSH does not go through the Fly Proxy — if you flyctl ssh a stopped machine, it doesn’t wake. There’s an open community request for --auto-start on SSH. It’s unimplemented.

Railway built the most interesting approach outside of ours: Layer 4 socket activation using nftables. When a container goes idle, its port is added to an nftables redirect set. Incoming TCP is redirected to a socket activation daemon that starts the container. This works for any TCP protocol — Redis, Postgres, HTTP. Their engineering blog calls it “Do Apps Dream of Electric Sheep?” and it’s worth reading. The limitation: it’s container-based, and private network services can’t be woken because the Wireguard tunnel prevents traffic interception.

Render shuts down free-tier services after 15 minutes and cold-boots them on the next request. Wake time: 15-30 seconds. Only triggers on HTTP GET — POST requests to sleeping services don’t wake them. No TCP support.

Cloudflare Workers use V8 isolates that load in ~5ms, hidden behind the TLS handshake. Effectively zero cold start, but Workers are HTTP/WebSocket only. Raw TCP support (Socket Workers) is in development.

AWS Lambda uses Firecracker internally and added SnapStart (snapshot restore for Java, Python, .NET). But Lambda is event-driven — it responds to triggers from API Gateway, SQS, S3. There’s no concept of “a client TCP-connects to a Lambda and it wakes up.”

The pattern: platforms that support scale-to-zero mostly do it at HTTP. The few that handle TCP (Railway) are container-based. Nobody wakes a full VM on an arbitrary TCP connection.

That’s what we built.

Public IPv6 for every VM

Each VM gets a dedicated public IPv6 address from a pool configured on the host. Internally, VMs use ULA (Unique Local Address) on the bridge network. The host maps between them with ip6tables DNAT/SNAT:

Internet → [public IPv6] → PREROUTING DNAT → [ULA IPv6] → VM bridge → VM
VM → [ULA IPv6] → POSTROUTING SNAT → [public IPv6] → Internet

The IPv6 pool is configured via environment variable — either inline or from a file:

VM_IPV6_POOL=2001:db8::1,2001:db8::2,2001:db8::3
# or
VM_IPV6_POOL_FILE=/etc/numavm/ipv6-pool.txt

Each VM gets one address at creation time. It’s stored in the database and persists across snapshots and restores. The address is stable — DNS records and firewall rules elsewhere can point at it permanently.

Per-VM firewall rules

With a public IP comes the need for a firewall. Every VM gets a dedicated ip6tables chain (numavm-<vm-id>) with a default-deny policy:

ICMPv6 is always allowed (NDP, path MTU discovery — blocking ICMPv6 breaks IPv6 in subtle ways)
Established/related connections are always allowed
User-defined rules open specific ports from specific sources
Everything else is dropped

Rules are managed through the API and the dashboard:

POST /vms/:id/firewall
{
  "rules": [
    { "proto": "tcp", "port": 22, "source": "::/0", "description": "SSH" },
    { "proto": "tcp", "port": 5432, "source": "2001:db8::/32", "description": "Postgres from office" },
    { "proto": "tcp", "port": 3000, "source": "::/0", "description": "Web app" }
  ]
}

Rules take effect immediately on running VMs. For snapshotted VMs, they’re stored and applied on the next wake.

Wake-on-connect: the interesting part

Here’s the trick. When a VM is running, the packet path looks like this:

Client → [public IPv6]:22 → PREROUTING DNAT → [ULA]:22 → VM

The DNAT rule in PREROUTING rewrites the destination address before the kernel makes its routing decision. The kernel sees a packet destined for a ULA address on the bridge, forwards it to the VM, and the packet never reaches any local socket on the host.

When a VM is snapshotted, we tear down the DNAT rules. Now the packet’s destination remains the host’s own IPv6 address. The routing decision marks it for local delivery. If nobody is listening on that address+port, the connection is refused.

So we listen.

The control plane binds a TCP server on the VM’s public IPv6 for every port the VM exposes — all TCP firewall rules, plus port 22 for SSH. We call it the wake proxy. When a connection arrives:

Accept the connection with pauseOnConnect: true (no data is read from the socket)
Call ensureVMRunning(vmId) — restores from snapshot in ~176ms
Look up the VM’s internal IP from the database
Open a TCP connection to the VM’s internal IP on the same port
Resume the client socket and pipe the two together

Client → [public IPv6]:22 → wake-proxy → ensureVMRunning() → [VM IP]:22 → VM

The client sees a slow connect (1-2 seconds while the VM wakes and the proxy bridges the connection) instead of a refused connection. SSH handshakes complete normally. Database connections establish. HTTP requests get their response.

Why this works with netfilter

This relies on a fundamental property of the Linux networking stack, not a trick: PREROUTING happens before the routing decision.

When DNAT rules exist, the destination is rewritten, the routing decision sends the packet to FORWARD, and it reaches the VM. The wake proxy’s listening socket on INPUT is never consulted.

When DNAT rules are absent, the destination is the host’s own address, the routing decision sends the packet to INPUT, and the wake proxy accepts it.

The transition between these states is handled entirely by the presence or absence of DNAT rules. No reconfiguration of the proxy, no socket rebinding, no signaling between components.

There is one subtlety: conntrack. Netfilter’s NAT only evaluates rules on the first packet of a connection (the SYN). After that, the translation is recorded in conntrack and subsequent packets use the cached entry — even if the DNAT rule is removed. When we snapshot a VM and tear down DNAT rules, we also flush the relevant conntrack entries:

conntrack -D -d <public_ipv6>

Without this, established connections to a just-snapshotted VM would continue to be forwarded to the dead ULA address instead of reaching the wake proxy. This is the same class of bug that Kubernetes has hit repeatedly with stale conntrack entries during service endpoint changes.

Concurrent connections are safe

Multiple clients might connect simultaneously to a sleeping VM. ensureVMRunning() coalesces concurrent wake requests — the first caller triggers the restore, subsequent callers await the same promise. All connections are bridged once the VM is up.

Why SSH works well with a TCP proxy

SSH is a server-speaks-first protocol (RFC 4253). After the TCP handshake, the server sends its version string before the client sends anything. So during the 1-2 second wake delay, the SSH client is simply waiting for the server’s banner — normal behavior. Once the proxy pipes the sockets together, the VM’s sshd sends its version string through the pipe and the handshake proceeds as if the connection were direct.

The proxy completes the TCP handshake as itself, then opens a separate connection to the VM. The two connections have different sequence numbers and window sizes — standard TCP proxy behavior. The proxy remains in the data path for the lifetime of this connection. Once DNAT rules are re-applied, new connections bypass the proxy entirely and go directly to the VM.

Handling errors

If the wake fails (quota exceeded, VM in error state), the client socket is destroyed. The client sees a connection reset — not great, but honest. There’s no good way to send an SSH-level error message before the protocol handshake, so a clean close is the best we can do.

Bind errors (port conflicts, address unavailable) are logged and skipped. The proxy never crashes the control plane.

The order of operations matters

When snapshotting (VM going to sleep):

Remove DNAT rules — the wake proxy immediately starts catching new connections
Flush conntrack entries — stale entries stop forwarding to the about-to-die VM
Snapshot the VM

When restoring (VM waking up):

Restore the VM from snapshot
Add DNAT rules — new connections now go directly to the VM
Flush conntrack entries — any connections the wake proxy accepted during restore now reconnect directly

The wake proxy remains bound through the entire lifecycle. It’s either masked by DNAT (VM running) or active (VM sleeping). No socket churn, no rebinding races.

How it all fits together at startup

When the control plane starts:

reconcileRunningVMs() — reconcile in-memory state with surviving Firecracker processes
initWakeProxies() — bind wake-proxy listeners for every VM that has an IPv6 address
startSshProxy() — start the SSH proxy for subdomain-based access

For running VMs, the DNAT rules are re-applied during reconciliation, so the wake-proxy listeners are masked. For snapshotted VMs, there are no DNAT rules, and the proxy is active.

When firewall rules are updated, the proxy rebinds with the new set of ports. When a VM is deleted, the proxy is torn down.

What this enables

SSH to sleeping VMs: ssh dev@<ipv6> wakes the VM and completes the handshake
Database connections: psql -h <ipv6> wakes the VM and connects to Postgres
Any TCP service: Redis, custom daemons, web servers — if it has a firewall rule, it gets wake-on-connect
Direct HTTP access: curl http://[<ipv6>]:3000 wakes and proxies the connection
Stable addresses: The IPv6 persists across snapshots, so DNS records and remote firewall rules can point at it permanently
Zero config for users: If you have firewall rules set, wake-on-connect is automatic. No client-side changes needed.

Where we are vs. the field

Platform	Wake mechanism	Wake latency	TCP wake	SSH wake
numaVM	DNAT + wake proxy on Firecracker snapshots	~1-2s	Any TCP port	Yes
Fly.io	Fly Proxy + Firecracker suspend	~300ms (suspend) / ~2s (stop)	HTTP only	No
Railway	L4 nftables socket activation	<1s (container)	Any TCP port	No
Vercel	Scale to one (keep warm)	~0ms	N/A (HTTP functions)	No
Cloudflare	V8 isolate pre-warm during TLS	~0ms	No (HTTP/WS only)	No
Render	Cold boot on HTTP GET	15-30s	No	No

Railway’s L4 socket activation is the closest analog to our approach — they intercept at the TCP level too. The difference is we’re waking full Firecracker VMs with snapshot restore (176ms to a running Linux machine), and wake-on-connect works for every exposed port including SSH.

This is one of those features where the complexity is entirely in making it invisible. From the user’s perspective, their VM just has an IP address that always works.

numaVM gives every developer and AI agent their own Linux machine — a Firecracker microVM with persistent storage, SSH access, and sub-second snapshot restore. numavm.com