Cryptokey Routing — Build a mesh VPN

The destination is an IP. The recipient is a key.

Last lesson's naked.c had one peer, given on the command line. The real daemon has many, and when read(tun_fd) returns a packet for 100.64.0.7 it has to answer two questions before the packet can leave. Which peer is that? And where is that peer right now? The second question is lesson 09's; the address a peer sends from changes every time its laptop moves to a new Wi-Fi. The first question has a fixed answer, and it is not an address. It is a 32-byte public key.

That is the inversion to get used to. In a normal router the table maps a destination prefix to a next hop and an interface. Here it maps a destination prefix to an identity. The endpoint, the session keys, the relay flag all hang off that identity and can change under it. WireGuard calls the table cryptokey routing, and the mesh daemon's src/peers.c is a direct, deliberately plain implementation of it. Every lesson after this one adds fields to the row; this lesson is about the lookup.

struct peer

c

// src/peers.h
#define MAX_PEERS       64
#define MAX_ALLOWED      8
#define MAX_CANDIDATES   8

struct allowed_ip { uint32_t net; uint8_t prefix; };      /* host order net */

struct peer {
    uint8_t  pubkey[KEY_LEN];                          /* the row's identity           */
    uint32_t overlay_ip;                               /* 100.64.x.y                   */
    struct allowed_ip allowed[MAX_ALLOWED]; int n_allowed;   /* /32 first                    */
    struct endpoint candidates[MAX_CANDIDATES]; int n_candidates;   /* lesson 08 */
    struct endpoint endpoint; bool endpoint_ok;        /* lesson 09: the one that answered */
    bool via_relay;                                    /* lesson 10                    */
    struct session cur, prev;                          /* lessons 05–06                */
    uint64_t last_handshake_init_ts;                   /* lesson 06                    */
    int64_t  last_rx_ms, last_tx_ms, punching_since_ms;   /* lesson 09                 */

    /* working state: handshake in progress, one staged packet, PEERS_END diff flag */
    /* … see the repo; nothing below this line affects routing */
};

struct peer_table { struct peer peers[MAX_PEERS]; int n; };
          

Three fields matter today. pubkey is the identity; two rows never share one, and peer_add returns the existing row if you add a key twice. overlay_ip is the address the coordination server assigned to that key. allowed[] is every address range the key is allowed to use, as net/prefix pairs with the network in host byte order, like everything else the daemon compares. peer_set_overlay_ip always writes overlay_ip/32 into allowed[0]; anything a peer later advertises as a subnet route (lesson 11) is appended after it. A fresh peer therefore has exactly one allowed range, itself.

The table is a fixed array of 64, kept dense: peer_remove shifts the tail down with memmove, and no lookup depends on order. A struct peer is a few kilobytes because it carries a staged packet buffer, so the table lives in struct node, which cmd_up declares static rather than on the stack. Every lookup is a linear scan. With 64 rows that is faster than you can measure; WireGuard, which must handle thousands of peers in the kernel, uses a radix trie for allowed-IPs and a hash table for indices. Same semantics, different constant.

Outbound: longest prefix wins

The lookup is two functions. cidr_contains builds a mask from the prefix length and compares the masked bits. lookup_by_overlay_ip walks every allowed range of every peer and keeps the match with the longest prefix.

c

// src/peers.c
bool cidr_contains(uint32_t net, uint8_t prefix, uint32_t ip)
{
    if (prefix == 0)
        return true;
    uint32_t mask = prefix >= 32 ? 0xffffffffu : ~((1u << (32 - prefix)) - 1u);
    return (ip & mask) == (net & mask);
}

/* Longest-prefix match: a packet for 192.168.1.20 should go to the peer
 * advertising 192.168.1.0/24, not to one advertising 0.0.0.0/0. */
struct peer *lookup_by_overlay_ip(struct peer_table *t, uint32_t ip)
{
    struct peer *best = NULL;
    int best_prefix = -1;
    for (int i = 0; i < t->n; i++) {
        struct peer *p = &t->peers[i];
        for (int j = 0; j < p->n_allowed; j++) {
            const struct allowed_ip *a = &p->allowed[j];
            if (a->prefix > best_prefix && cidr_contains(a->net, a->prefix, ip)) {
                best = p;
                best_prefix = a->prefix;
            }
        }
    }
    return best;
}
          

Take 192.168.1.20 with two candidate rows: one peer advertised 192.168.1.0/24, another 192.168.0.0/16. Both masks match. The /24 wins because 24 > 16, and it should: the peer that claimed the smaller, more specific range is the one that actually has that LAN behind it. The /16 peer would still receive 192.168.7.7, which nobody else claimed. This is the same rule the kernel's own routing table applies, from the routing lesson, and for the same reason: a specific claim beats a general one.

Note what best_prefix = -1 buys. A row with prefix 0 (0.0.0.0/0, "send me everything", the exit-node case) matches every address but loses to any other match. And when nothing matches, best stays NULL, and the caller in meshd.c counts a no_peer drop. There is no default; a packet for an address no key owns goes nowhere. That one NULL is how the ACL in lesson 11 will work: the coordination server simply never tells you about a peer you may not reach, so its row does not exist, so the packet is dropped here.

⚠️

Why cidr_contains special-cases 0 and 32. The obvious mask, ~((1u << (32 - prefix)) - 1), shifts by 32 when the prefix is 0. Shifting a 32-bit value by 32 or more is undefined behavior in C; on x86 the hardware masks the count to 5 bits and you silently get a shift by 0, which makes the mask 0xffffffff, which makes 0.0.0.0/0 match nothing. The two early returns exist so that neither end of the range reaches the shift.

Step through the table below. Three peers, a packet from tun0, the prefix bits compared in purple, the winning row in green. Then the second half of the story.

cryptokey routing — which key owns which IP

Click next to start the walkthrough.

Inbound: a key may only speak for its own addresses

Routing outbound packets by key is half of the table's job. The other half runs on every packet that arrives. After lesson 05 the daemon will know, from the authentication tag, which key sent a packet. It then looks at the inner IP header, the one that will be written to tun0, and asks: is this source address one that key is allowed to use?

c

// src/peers.c
bool peer_allows_src(const struct peer *p, uint32_t ip)
{
    for (int i = 0; i < p->n_allowed; i++)
        if (cidr_contains(p->allowed[i].net, p->allowed[i].prefix, ip))
            return true;
    return false;
}

// src/meshd.c — the last three checks before a decrypted packet reaches the kernel
if (!ipv4_valid(plain, (size_t)plen) || !peer_allows_src(p, ipv4_src(plain))) {
    n->drops.bad_src++;
    return;
}
if (write(n->tun_fd, plain, (size_t)plen) < 0)
    log_warn("tun write: %m");
          

Without that check, consider peer 2, a legitimate member of your mesh at 100.64.0.12. It builds an IP packet with source 100.64.0.7, peer 1's address, and sends it to you through its perfectly valid encrypted session. The tag checks out: peer 2 really did send it. You write it to tun0. Your kernel now believes 100.64.0.7 said something, and every program that trusts that address (an sshd with an allow-list, a database bound to the overlay) is fooled. The encryption did its job. The authorization was missing. peer_allows_src is that authorization: the same allowed[] that decides where packets go decides where packets may come from.

🚫

Authenticated is not the same as authorized. A valid tag proves which key sent the datagram. It proves nothing about the addresses written inside the plaintext, because the sender wrote those. Skip the source check and any member of the mesh can spoof any other member to every program on your machine. The counter for this in meshd status is bad_src, and in a healthy mesh it reads 0.

Three more ways into the same row

The rest of peers.c is lookups by other keys, each used by one later lesson. lookup_by_pubkey is a memcmp over 32 bytes; the coordination client (lesson 07) uses it to apply a PEER line to the right row, and peer_add uses it to stay idempotent. lookup_by_rx_index takes the 32-bit index at bytes 1 to 4 of a DATA packet and finds the peer whose current or previous session owns it; lesson 06 explains why packets carry the receiver's index rather than the sender's key. lookup_by_endpoint finds a peer by the ip:port a datagram arrived from, and lesson 09 explains why that one is only a hint and never a proof. Each is a dozen lines and a linear scan. Read them now in src/peers.c; none has a surprise.

✅

The rule. A packet goes out to the key whose allowed-IPs best match its destination. A packet comes in only if its source is inside the sending key's allowed-IPs. Both halves use one table, and the coordination server is the only thing that writes to it. That pair of checks is the entire data-plane access-control story; Tailscale's ACLs, lesson 11, compile down to exactly which rows exist.

Run it

peers.c has no syscalls in it, so you can exercise it without root, a network, or the rest of the daemon. The harness below builds the three-peer table from the visualization, with fake 32-byte keys whose first byte is the peer number, and asks the two questions this lesson is about. Save it as lpm.c next to naked.c.

c

// lpm.c — this lesson only; not in the repo. Links against src/peers.o and src/util.o.
#include "peers.h"
#include <stdio.h>

static struct peer *add(struct peer_table *t, uint8_t id, const char *ip, const char *route)
{
    uint8_t key[KEY_LEN] = { id };                  /* a fake identity: first byte = id */
    struct peer *p = peer_add(t, key);
    uint32_t a, net; uint8_t prefix;
    parse_ip(ip, &a);
    peer_set_overlay_ip(p, a);                      /* allowed[0] = ip/32 */
    if (route && parse_cidr(route, &net, &prefix))
        peer_add_allowed(p, net, prefix);           /* allowed[1] = a subnet route */
    return p;
}

static void route(struct peer_table *t, const char *dst)
{
    uint32_t ip;
    parse_ip(dst, &ip);
    struct peer *p = lookup_by_overlay_ip(t, ip);
    if (p) printf("%-15s -> peer %u\n", dst, p->pubkey[0]);
    else   printf("%-15s -> no peer (drop)\n", dst);
}

int main(void)
{
    static struct peer_table t;                     /* 64 peers: too big for the stack */
    add(&t, 1, "100.64.0.7",  "192.168.1.0/24");
    add(&t, 2, "100.64.0.12", NULL);
    add(&t, 3, "100.64.0.3",  "192.168.0.0/16");

    route(&t, "100.64.0.7");
    route(&t, "192.168.1.20");                      /* /24 and /16 both match */
    route(&t, "192.168.7.7");                       /* only the /16 */
    route(&t, "10.9.9.9");

    uint32_t spoof;
    parse_ip("100.64.0.7", &spoof);                 /* peer 1's address… */
    printf("peer 2 may send from 100.64.0.7? %s\n",
           peer_allows_src(&t.peers[1], spoof) ? "yes" : "no (drop, bad_src)");   /* …claimed by peer 2 */
    return 0;
}
          

make                                 # for src/peers.o and src/util.o
gcc -std=c11 -O2 -Wall -Wextra -D_GNU_SOURCE -Isrc -o lpm lpm.c src/peers.o src/util.o -lsodium
./lpm
          

64.0.7      -> peer 1
168.1.20    -> peer 1
168.7.7     -> peer 3
9.9.9        -> no peer (drop)
peer 2 may send from 100.64.0.7? no (drop, bad_src)
          

Change peer 3's route to 192.168.1.0/25 and 192.168.1.20 moves to peer 3: a /25 is more specific than a /24. Then delete peer 1's route and watch 192.168.1.20 fall through to the /16. When the full system is running, the same table is visible in meshd status: each peer line ends with allowed 100.64.0.2/32, growing to allowed 100.64.0.2/32,192.168.1.0/24 after lesson 11, and the final line's bad_src and no_peer counters are these two checks keeping score.

one-line takeaway

The peer table maps address ranges to public keys: the longest matching prefix picks the key to encrypt for, and the same ranges decide which source addresses that key may ever claim.