TCP Is a Stream — Networking

Three sends, two recvs, zero messages

The sender calls send(fd, "HELLO", 5, 0), then "WORLD", then "!!". The receiver calls recv with a 7-byte buffer and gets "HELLOWO". It calls again and gets "RLD!!". Every byte arrived, in order. Not one recv matched a send.

This works on your laptop too, and it works worse in production. Over localhost the three sends often arrive as one 12-byte recv, so the program looks fine. Across a real network they arrive split wherever the kernel happened to cut them, and the program that assumed one recv is one message breaks in ways that depend on the weather.

Where the boundaries went

In the previous lesson you saw that send copies your bytes into the kernel's send buffer. That is the whole story. The buffer is an array of bytes. Your 5-byte send appends 5 bytes to it. The next send appends 5 more. There is no record of where one call ended and the next began, because TCP has no concept that would need it.

The kernel then drains that buffer into segments. The size of each segment depends on the maximum segment size negotiated during the handshake, on how many bytes happen to be sitting in the buffer, and on how much the receiver has said it can take. Twelve bytes might go as one segment or as two. Across the internet, a 100-kilobyte send becomes about 70 segments.

On the far side, segments are put back in sequence order into the receive buffer, another plain array of bytes. recv copies out whatever is there, up to the size you asked for. If 12 bytes have arrived and you ask for 7, you get 7. If 3 have arrived and you ask for 64, you get 3, right now, without waiting for more.

💡

TCP is a pipe, not a mailbox. It is the same model as the pipe from the systems track: write pushes bytes in one end, read pulls bytes out the other, and the pipe does not know what a "message" is. UDP keeps your boundaries. TCP erases them, on purpose, so it can repack bytes however the network needs.

tcp — a byte stream with no boundaries

Click next to start the walkthrough.

Partial writes

The same thing happens on the way in. send returns the number of bytes it copied into the send buffer, and that number can be less than you asked for. If the buffer has 2000 bytes of free space and you send 5000, you get 2000. The other 3000 were not sent. They are still in your buffer, and if you move on, they are gone.

With a blocking socket this is rare: send usually waits for room. It is not guaranteed to, and with non-blocking sockets it is routine. The only correct way to send N bytes is a loop that keeps going until N bytes have been accepted.

c

// send exactly len bytes, or fail. Returns 0 or -1.
int send_all(int fd, const void *buf, size_t len) {
    const char *p = buf;
    while (len > 0) {
        ssize_t n = send(fd, p, len, 0);
        if (n == -1) {
            if (errno == EINTR) continue;
            return -1;
        }
        p   += n;     // advance past what the kernel took
        len -= n;
    }
    return 0;
}

// receive exactly len bytes, or fail.
// Returns 0 on success, -1 on error or EOF.
int recv_exact(int fd, void *buf, size_t len) {
    char *p = buf;
    while (len > 0) {
        ssize_t n = recv(fd, p, len, 0);
        if (n == 0) return -1;  // peer closed too early
        if (n == -1) {
            if (errno == EINTR) continue;
            return -1;
        }
        p   += n;
        len -= n;
    }
    return 0;
}
          

Both loops have the same shape, and the shape is the lesson: the syscall tells you how many bytes it handled, you move your pointer by that much, and you call again. recv_exact treats a return of 0 as failure, because 0 means the peer sent FIN and no more bytes are ever coming. If you still needed some, the message was cut off.

🚫

Ignoring the return value of send loses data silently. send(fd, buf, 5000, 0); with no check can transmit 2000 bytes and report nothing wrong. The receiver gets a truncated message and your logs are clean. This is the single most common bug in hand-written network code.

Framing: putting the boundaries back

TCP gives you a stream. If your program thinks in messages, you have to draw the boundaries yourself, in the bytes. This is called framing, and there are two ways everyone does it.

Length prefix. Before each message, send its length as a fixed-size integer. The receiver reads exactly that many bytes for the length, decodes it, then reads exactly that many bytes for the body. Binary protocols do this. Four bytes, big-endian, is the convention, and it lets a message contain any bytes at all.

Delimiter. End each message with a byte that cannot appear inside it, usually \n. The receiver reads until it sees the delimiter. Text protocols do this. It is simpler to type at a terminal and harder to get right in code, because the delimiter can straddle two recv calls and you need to keep the leftover bytes for next time.

c · length-prefixed frames

#include <arpa/inet.h>     // htonl, ntohl
#include <stdint.h>
#include <stdlib.h>

// 1 MiB cap — never trust a length from the network
#define MAX_FRAME (1 << 20)

// wire format: [4-byte big-endian length][body]
int send_frame(int fd, const void *body, uint32_t len) {
    uint32_t be = htonl(len);  // to network order
    if (send_all(fd, &be, 4) == -1) return -1;
    return send_all(fd, body, len);
}

// returns a malloc'd body and sets *out_len;
// NULL on error, EOF, or oversize
char *recv_frame(int fd, uint32_t *out_len) {
    uint32_t be;
    if (recv_exact(fd, &be, 4) == -1) return NULL;

    uint32_t len = ntohl(be);
    // a hostile peer could send 0xFFFFFFFF
    if (len > MAX_FRAME) return NULL;

    char *body = malloc(len);
    if (!body) return NULL;
    if (recv_exact(fd, body, len) == -1) {
        free(body); return NULL;
    }

    *out_len = len;
    return body;
}
          

Now the sender's three messages come out as three messages, no matter how the kernel chopped the stream. The last steps of the visualization show the receive buffer holding 23 bytes with no boundaries, and recv_frame pulling exactly one message out of it.

✅

Decide on framing before you write the first send. Every protocol that runs on TCP has one. HTTP uses a delimiter for headers and a length (Content-Length) for the body. You will see exactly that in the next lesson. And always cap the length you accept: a 4-byte length field can ask you to malloc four gigabytes.

Why localhost lies

Small send calls in quick succession usually get merged. Partly this is the loopback interface being fast, and partly it is Nagle's algorithm: the kernel holds back small segments until the previous one is acknowledged, then sends everything it has collected in one go. Good for throughput, bad for a program that expects one segment per send, and the reason your "message per recv" code passes every local test and fails in the field.

⚠️

Do not fix framing bugs with TCP_NODELAY. Disabling Nagle makes your tests behave for a while and changes nothing about what TCP guarantees. Segments can still split and merge anywhere in the path. The fix is a length prefix or a delimiter. TCP_NODELAY is for latency, not correctness.

one-line takeaway

TCP moves bytes, not messages — loop until every byte is sent or received, and write the message boundaries into the bytes yourself.