Build a mesh VPN · lesson 05

Encrypting the tunnel

ping works across the lab. tcpdump on the cloud bridge shows ICMP echo requests with your overlay addresses and payload in the clear. This lesson is session.c: the 70 lines that turn one tun0 packet into one DATA packet nobody on the path can read or forge.

in progress
20 min interactive

The cloud box can read your ping

Run the lesson 03 tunnel and capture on the bridge that stands in for the internet: tcpdump -i br0 -A. Each datagram from A to B carries a complete inner IPv4 packet. 100.64.0.1 > 100.64.0.2: ICMP echo request, then the payload bytes. Anyone on the path reads it. Worse, anyone on the path can write it: a forged datagram to B's UDP port lands in B's kernel as if it came from A. The threat-model row for the ISP in lesson 01 promised "outer IP:port pairs, packet sizes, timing; nothing inside". Right now it gets everything inside.

You saw the fix in the networking track's last lesson: authenticated encryption. Now you implement it. src/session.c is the whole data plane's crypto: one function to seal a packet, one to open it, and one to refuse packets you have already seen.

Where the keys come from

A struct session holds two 32-byte keys, tx_key for packets you send and rx_key for packets you receive, plus counters. Both sides derive them from the ephemeral Diffie-Hellman exchange in lesson 06, so that your tx_key is the peer's rx_key and vice versa; for this lesson, take them as two secrets both ends already agree on and look only at what the bytes on the wire become.

c
// src/session.h struct session { bool established; bool initiator; /* lesson 06 */ uint32_t local_index; /* peers put this in DATA they send to us */ uint32_t remote_index; /* we put this in DATA we send to them */ uint8_t tx_key[32], rx_key[32]; uint64_t tx_counter; /* nonce for the next packet we send */ uint64_t rx_highest; /* highest counter accepted so far */ uint64_t rx_bitmap; /* bit i set ⇔ counter (rx_highest − i) seen */ int64_t established_ms; int64_t retire_ms; /* lesson 06 */ }; // src/wire.h #define DATA_HDR_LEN 13 /* [type 1][receiver_index u32][counter u64] */ #define DATA_TAG_LEN 16 #define DATA_OVERHEAD (DATA_HDR_LEN + DATA_TAG_LEN) #define TUN_MTU 1400 /* 1400 + 29 + 28 outer IP/UDP fits a 1500 link */

The index fields are lesson 06's; for now, remote_index is a number the peer asked you to put at the front of every packet you send it. The MTU constant is why meshd up runs ip link set tun0 mtu 1400 up: an inner packet grows by 29 bytes here and 28 more in outer IP and UDP, and 1457 fits a 1500-byte link.

The DATA packet, byte by byte

Type byte 3. Then the receiver's index, 4 bytes little-endian. Then the counter, 8 bytes little-endian. Then the ciphertext, exactly as long as the plaintext, then 16 bytes of tag. wire.h chose little-endian for every multi-byte field, as WireGuard does, which is the opposite of the IP header you parsed in the networking track. That is a design choice, not a law; the put_u32le and put_u64le helpers exist so the two worlds never touch.

You saw this frame in the networking track; now the inner packet is the one you just read from tun0, and the header fields are the ones session_encrypt writes. Step through the visualization to see the same 56 bytes before and after, which fields stay readable, and what one flipped bit does to the tag.

DATA — one tun0 packet, sealed
Click next to start the walkthrough.

Sealing: session_encrypt

c
// src/session.c /* Nonce = four zero bytes ‖ counter (LE). The counter never repeats within * a session, and every session has fresh keys, so (key, nonce) is unique. */ static void make_nonce(uint64_t counter, uint8_t nonce[12]) { memset(nonce, 0, 4); put_u64le(nonce + 4, counter); } size_t session_encrypt(struct session *s, const uint8_t *plain, size_t len, uint8_t out[MAX_PKT], int64_t now) { if (!session_usable(s, now) || len + DATA_OVERHEAD > MAX_PKT) return 0; out[0] = PKT_DATA; put_u32le(out + 1, s->remote_index); put_u64le(out + 5, s->tx_counter); uint8_t nonce[12]; make_nonce(s->tx_counter, nonce); unsigned long long clen = 0; /* The 13-byte header is associated data: authenticated but not * encrypted, so a tampered index or counter fails verification. */ crypto_aead_chacha20poly1305_ietf_encrypt(out + DATA_HDR_LEN, &clen, plain, len, out, DATA_HDR_LEN, NULL, nonce, s->tx_key); s->tx_counter++; return DATA_HDR_LEN + (size_t)clen; }

The libsodium arguments, in order: output and its length (len + 16), plaintext and its length, then the associated data, which is out itself for DATA_HDR_LEN bytes, so the header you just wrote goes into the tag without being encrypted. The NULL is an unused parameter. Then the 12-byte nonce and the 32-byte key.

The nonce is the counter with four zero bytes in front, because the IETF variant wants 12 bytes and the counter is 8. Nothing random goes into it: a random 12-byte nonce would repeat after about 248 packets, and a counter repeats never, as long as it only goes up, which tx_counter++ guarantees. The receiver needs the nonce to decrypt and gets it for free from offset 5 of the header.

💡
Associated data is the part of the packet you want checked but not hidden. The receiver has to read the index to find the session and the counter to build the nonce before it can decrypt anything. So those 13 bytes stay in the clear. Feeding them in as associated data means the tag still covers them: change the counter in flight and the tag no longer matches, so the packet is dropped instead of decrypted under the wrong nonce.
🚫
Reuse a (key, nonce) pair once and ChaCha20 is broken for both packets. The cipher produces a keystream from the key and nonce and XORs it with the plaintext. Two packets under the same keystream XOR together to the XOR of their plaintexts, and an IP header is predictable enough to peel the rest off. Poly1305 fails in the same situation: two tags under one nonce leak the authentication key, and the attacker can forge. That is why the counter starts at 0 for every new session and never rewinds: a fresh key per session, a fresh counter per key.

The replay window

The tag stops forgery. It does not stop someone recording a valid packet and sending it again an hour later; the inner TCP would accept the replay, because its sequence numbers were replayed too. The counter solves this if the receiver tracks which counters it has seen. A set of every counter is unbounded, and accepting only counter > highest breaks on UDP, which reorders. The standard answer, from IPsec and WireGuard, is a window: the highest counter accepted plus one bit for each of the 64 just below it.

c
// src/session.c bool replay_check(struct session *s, uint64_t counter) { if (counter > s->rx_highest) { /* New high-water mark: slide the window forward. */ uint64_t shift = counter - s->rx_highest; s->rx_bitmap = shift >= 64 ? 0 : s->rx_bitmap << shift; s->rx_bitmap |= 1; s->rx_highest = counter; return true; } uint64_t behind = s->rx_highest - counter; if (behind >= 64) return false; /* too old to track → reject */ uint64_t bit = 1ull << behind; if (s->rx_bitmap & bit) return false; /* already seen → replay */ s->rx_bitmap |= bit; return true; }

Bit i of rx_bitmap means "counter rx_highest − i has been seen". Bit 0 is rx_highest itself. Walk three packets through it, starting from an empty session where rx_highest and rx_bitmap are both 0.

Counter 5 arrives. 5 > 0, so this is a new high-water mark. shift is 5. The bitmap 0 shifted left 5 is still 0; OR in bit 0 and it is 0b1. rx_highest becomes 5. The bitmap now says: 5 seen, 4 down to 0 not seen.

Counter 3 arrives late. 3 is not greater than 5, so behind is 2. Less than 64, so it is inside the window. bit is 1 << 2 = 0b100. The bitmap 0b1 AND 0b100 is 0: not seen. Set it; the bitmap is 0b101. Accepted. UDP reordered the packets and nothing was lost.

Counter 5 arrives again. behind is 0, bit is 0b1, and 0b101 & 0b1 is nonzero. Already seen. Rejected. A recorded packet, replayed, dies here.

Counter 80 arrives. New high-water mark, shift 75. Everything in the old window is now more than 64 behind, so the bitmap becomes 0b1, and counter 16 or below is "too old to track" whether or not it was seen. That is the trade: a packet delayed by more than 64 others is dropped, and the window costs 16 bytes of state.

⚠️
Shifting by the type's width is undefined, not zero. x << 75 on a uint64_t is UB; x86 masks the count to 6 bits, so you would get x << 11 and a window of stale bits that rejects fresh packets. That is what shift >= 64 ? 0 : … is for. Leave it out and the bug appears only after 64 packets are lost in a row, which is to say, only in production.

Opening: session_decrypt

c
// src/session.c int session_decrypt(struct session *s, const uint8_t *pkt, size_t len, uint8_t out[MAX_PKT], int64_t now) { if (!session_usable(s, now) || len < DATA_OVERHEAD || len > MAX_PKT) return -1; uint64_t counter = get_u64le(pkt + 5); uint8_t nonce[12]; make_nonce(counter, nonce); unsigned long long plen = 0; if (crypto_aead_chacha20poly1305_ietf_decrypt(out, &plen, NULL, pkt + DATA_HDR_LEN, len - DATA_HDR_LEN, pkt, DATA_HDR_LEN, nonce, s->rx_key) != 0) return -1; /* Replay check AFTER authentication: an attacker must not be able to * poison the window with forged counters. */ if (!replay_check(s, counter)) return -1; return (int)plen; }

The caller in meshd.c has already used the index at offset 1 to find the session; that lookup is lesson 06. Here, the counter is read from the header, the nonce rebuilt from it, and the ciphertext opened with the header as associated data. libsodium verifies the tag before it writes a single plaintext byte, and the comparison is constant-time inside the library; you never touch the tag yourself. A failed tag returns -1 and the packet is counted under bad_auth.

The order of the last two checks is deliberate. If the replay window ran first, an attacker could send a forged packet with counter 240, slide your window to it, and every genuine packet from the peer would be "too old" from then on. By authenticating first, only packets the peer really sent can move the window.

Keepalive is an empty DATA packet

session_encrypt(&p->cur, NULL, 0, out, now) is legal: zero bytes of plaintext, zero of ciphertext, a packet of exactly DATA_OVERHEAD = 29 bytes. path.c sends one when nothing has gone to a peer for 25 seconds, so the NAT mapping stays alive. The receiver's session_decrypt returns 0, meshd.c records that an authenticated packet arrived, and nothing is written to tun0. In a capture, a 29-byte UDP payload is a keepalive.

✅
Every packet from a peer is authenticated before anything else happens to it. The replay window, the roaming logic in lesson 09, and the source-address check from lesson 03 all run after session_decrypt returns. If it returns -1, the packet never influences any state. This is the one rule that makes the rest of the daemon safe to write.

Run it

The lab in test/run.sh captures on the cloud bridge and on A's tun0 while A pings B with the payload pattern MESHMESH. Run the whole thing (root on Linux, or the Docker lab from the README), then read the two captures yourself.

sh
make sudo ./test/run.sh # or: docker run --rm --privileged -v "$PWD":/work -w /work mesh-lab sh -c 'make && ./test/run.sh' # what the "internet" saw: only type-3 packets between the nodes tcpdump -nn -r /tmp/mesh-lab/cloud.pcap 'udp and not port 7000 and not port 7001 and not port 7002 and udp[8] = 3' | head -3 tcpdump -nn -r /tmp/mesh-lab/cloud.pcap icmp | wc -l tcpdump -r /tmp/mesh-lab/cloud.pcap -A | grep -c MESHMESH tcpdump -r /tmp/mesh-lab/tun.pcap -A | grep -c MESHMESH
sh
PASS: only type-3 DATA packets on the wire (10 seen, 0 other) PASS: no ICMP visible on the cloud side PASS: ping payload visible on tun0 (10) but not on the wire (0) ... 10:14:02.551 IP 10.0.0.2.51820 > 10.0.0.3.51820: UDP, length 113 10:14:02.552 IP 10.0.0.3.51820 > 10.0.0.2.51820: UDP, length 113 10:14:02.751 IP 10.0.0.2.51820 > 10.0.0.3.51820: UDP, length 113 0 0 10

Check the arithmetic: an 84-byte ICMP echo plus DATA_OVERHEAD 29 is 113, the UDP length tcpdump prints. udp[8] is the first payload byte, the type. Both addresses are the NATs' external sides, never 100.64.x.x. The pattern appears ten times on tun0, five requests and five replies, and zero times on the wire. While the nodes are still up, MESHD_SOCK=/tmp/mesh-lab/A.sock ./meshd status prints tx N rx M for each peer: those are tx_counter and rx_highest from this lesson, counting every DATA packet including keepalives.

one-line takeaway

A DATA packet is a 13-byte header the receiver must read, used as both nonce and associated data, followed by ciphertext and a tag; the counter in it never repeats going out and is checked against a 64-bit window coming in.

← previous Keys are identity next → The handshake