Network Tutorial · 10

Every drop counted

A network stack that silently drops bad packets is correct and useless to debug. "It doesn't work" is not a bug report; it's a shrug. Alquist counts everything — every frame accepted, every packet refused, and why — so a shrug becomes a coordinate.

[o-o] |=| /_\

Silence is the right answer to a hostile packet on the wire. It is the wrong answer to the engineer holding the board. So we keep two faces: mute to the network, and a meticulous bookkeeper on the inside. Every rejection leaves a tally mark, filed by reason.

One counter per decision

The whole diagnostic surface is a single flat structure of unsigned counters (struct net_diag_stats). Crucially, it does not just count what worked — it counts each reason a packet was refused, because in networking the refusals are where the truth lives:

struct net_diag_stats {
    uint32_t frames_rx;            /* frames the pump handed to the brain     */
    uint32_t arp_rx;               /* ARP requests we answered                */
    uint32_t ipv4_rx;             /* IPv4 packets addressed to us            */
    uint32_t ipv4_drop_not_us;    /* wrong destination -- not our MAC/IP     */
    uint32_t ipv4_drop_hdr;       /* malformed/short header, or a fragment   */
    uint32_t ipv4_drop_csum;      /* header checksum failed                  */
    uint32_t icmp_rx, icmp_tx;    /* pings answered                          */
    uint32_t udp_rx, udp_tx;      /* datagrams in / echoed                   */
    uint32_t udp_drop_len;        /* length field dishonest                  */
    uint32_t udp_drop_csum;       /* checksum present and wrong              */
    uint32_t udp_drop_port;       /* nobody listening on that port           */
    uint32_t tcp_rx;              /* TCP segments parsed                     */
    uint32_t tcp_drop_parse_len;  /* segment too short to be TCP             */
    uint32_t tcp_drop_parse_csum; /* TCP checksum failed                     */
    uint32_t tcp_synack_tx;       /* handshakes we opened                    */
    uint32_t tcp_rst_tx;          /* segments we refused with RST            */
    uint32_t tcp_data_tx;         /* data/retransmit segments sent           */
    uint32_t tx_submit_ok;        /* frames the driver accepted             */
    uint32_t tx_submit_fail;      /* frames the driver refused (ring full)   */
};

Notice how the three IPv4 drop reasons are kept apart. "Nothing works" collapses instantly once you can see whether packets are arriving but addressed elsewhere (ipv4_drop_not_us climbing — a switching or ARP problem), arriving mangled (ipv4_drop_hdr — a framing or MTU problem), or arriving corrupt (ipv4_drop_csum — a wire or NIC problem). Same story for UDP: a datagram rejected for a bad length is a different diagnosis than one rejected because no port was open. The categories are the diagnosis.

Reading them: net stats

From the shell monitor, one command dumps the whole ledger, one counter per line, and a clear variant zeroes it so you can measure a single experiment cleanly:

alquist> net stats
frames_rx          1428
arp_rx                6
ipv4_rx            1402
ipv4_drop_not_us     18
ipv4_drop_hdr         0
ipv4_drop_csum        0
icmp_rx  / icmp_tx    4 / 4
udp_rx   / udp_tx    12 / 12
udp_drop_port         3
tcp_rx             1350
tcp_synack_tx         2
tcp_rst_tx            1
tcp_data_tx         690
...
alquist> net stats clear      # zero everything before the next test

This turns every experiment into a before/after measurement. Clear the counters, run one ping from the host, and icmp_rx and icmp_tx should each read exactly the number of echo requests you sent. If frames_rx climbed but icmp_rx did not, the packet arrived and was rejected earlier — and the drop counters tell you at which layer. Correctness becomes something you can count, not something you hope for.

Change-only delta logging

Polling net stats by hand is fine for a set-piece test; watching a live system is not. So there is a second mode: when logging is enabled, the stack samples the counters each pass and emits a compact one-line delta — but only when something changed:

static void net_diag_log_delta_if_changed(void)
{
    static struct net_diag_stats prev;      /* file-scope persistence, not a hidden global */
    struct net_diag_stats now;

    if (g_net_diag_log_enabled == 0u) return;   /* off by default -- opt in */

    net_diag_stats_read(&now);
    if (/* any field differs from prev */) {
        log_printf(LOG_INFO, "net",
                   "d fr+%u ar+%u ip+%u in+%u ih+%u ic+%u ir+%u it+%u ...",
                   now.frames_rx - prev.frames_rx, now.arp_rx - prev.arp_rx, ...);
    }
    prev = now;                              /* baseline for the next pass */
}

Two design choices earn their keep here. Delta, not absolute: the line shows what happened this interval (fr+3 ir+1 it+1 — three frames in, one ping answered), which is what you actually want when hunting a live event. Only-on-change: an idle box prints nothing, so the log stays silent until the network does something, and a burst of drops during an attack stands out instead of drowning in heartbeat noise. And it is off by default — observability you opt into, never overhead you pay for unasked.

Why counters, and not a packet capture

A big system reaches for tcpdump on the box itself. Alquist deliberately does not: an on-device capture buffer is unbounded memory, a parser for attacker-controlled bytes, and a privileged tap — three things this project spends its whole budget avoiding. Fixed counters cost twenty-odd words of RAM, can never overflow into a security problem, and answer the question you usually have (where are packets being lost?) directly. When you truly need the bytes, you capture them from the other end with tcpdump on the host — outside the box, where a capture buffer is someone else's memory. The board tells you the counts; the host tells you the contents.

Gotchas worth remembering

[^-^] |=| /_\

Now the stack can tell you not just that it dropped something, but why, and how often, and right now. Which means we can finally do the interesting thing: point hostile traffic at it, watch exactly which counter lights up, and see whether the wall we built actually holds.