When the Sleepers Wake

Since our very first blink, three cores have been asleep — parked, dreaming, waiting to be called. Today we call them by name. Four cores, awake at once, and the parallelism that was an illusion becomes real.

Goal

Bring the Raspberry Pi 4's other three cores to life, give each its own stack and footing, and have all four announce themselves over the UART — cleanly, because four cores reaching for one serial port is exactly the contention we learned to tame last article.

Where the sleepers waited

Remember article one. The very first thing our code did was read mpidr_el1, keep core 0, and branch the others away to a parking loop. We have done that in every program since. But those cores did not vanish — the firmware left them spinning in a tiny holding pattern called a spin-table: each secondary core sits asleep on wfe, and every time an event arrives it peeks at a fixed memory address — its personal mailbox. If that mailbox is still zero, it sleeps again. If it holds an address, the core jumps there and runs.

On the Raspberry Pi 4, those mailboxes live at low physical addresses: core 1 watches 0xE0, core 2 watches 0xE8, core 3 watches 0xF0 — a simple 0xD8 + core×8.

The wake-up call

Waking a core, then, is just two steps: write the address we want it to start at into its mailbox, and send an event to nudge it out of wfe. The dsb in between makes sure the sleeping core actually sees our write before we ring the bell.

// smp.c — call the sleepers
#include <stdint.h>

static void wake_core(int core, void (*entry)(void)) {
    volatile uint64_t *mailbox = (volatile uint64_t *)(0xD8UL + core * 8);
    *mailbox = (uint64_t)entry;          // "start running here"
    __asm__ volatile ("dsb sy");         // make the write visible first
    __asm__ volatile ("sev");            // ring the bell
}

Landing on bare feet

When a secondary core jumps to our entry point, it arrives with nothing — no stack, no exception level chosen for us, no MMU. It must repeat, for itself, the journey core 0 took across the last nine articles. The crucial new wrinkle: every core needs its own stack, because two cores sharing one stack would instantly corrupt each other. We carve a separate slab of memory per core, indexed by the core number mpidr_el1 hands it.

// secondary.S — each woken core finds its own footing
.global secondary_start
secondary_start:
    mrs     x0, mpidr_el1
    and     x0, x0, #3           // which core am I? (1, 2, or 3)

    // give myself a private stack: base + core * 64 KB
    ldr     x1, =_secondary_stacks_top
    lsl     x2, x0, #16          // core * 65536
    sub     sp, x1, x2

    bl      secondary_main       // into C, on my own stack
1:  wfi
    b       1b

In a complete bring-up, secondary_main also repeats core 0's setup — drop to EL1, point TTBR0_EL1 at the same shared page table from article seven, set VBAR_EL1 to the shared vector table. The cores share the kernel's memory map and its doors; only their stacks are private. To keep the spotlight on the wake-up itself, we show just the part that proves each core is alive.

Four voices, one mouth

Here is where last article pays off on its very first day. All four cores want to print at the same instant through the single UART. Without protection their letters would shred into each other — the exact race we studied. But now the contenders are real, separate cores, so masking interrupts would not help at all. Only the memory-backed spinlock can hold the line.

// secondary.c — say hello, safely
void uart_puts_locked(const char *s);   // the spinlock-guarded printer

static unsigned this_core(void) {
    uint64_t id;
    __asm__ volatile ("mrs %0, mpidr_el1" : "=r"(id));
    return (unsigned)(id & 3);
}

void secondary_main(void) {
    char msg[] = "core _ is awake\n";
    msg[5] = '0' + this_core();
    uart_puts_locked(msg);               // four cores, one at a time
    for (;;) { __asm__ volatile ("wfi"); }
}

// main.c — core 0 rouses the rest
void kmain(void) {
    uart_init();
    /* ... mmu, gic, timer, vectors as before ... */

    uart_puts("core 0 is awake (and no longer alone)\n");

    for (int c = 1; c <= 3; c++)
        wake_core(c, secondary_start);

    for (;;) { __asm__ volatile ("wfi"); }
}

What actually happens

Core 0 writes secondary_start into each mailbox and sends an event. Three cores that have slept since the first article stir, read their mailboxes, and leap to our code. Each discovers its own number, claims a private stack, and calls into C. Then all three — truly at the same time, on truly separate silicon — reach for the UART. The spinlock lets exactly one in at a time; the others wait on wfe and take their turn. Four clean lines appear.

This is the moment the word "concurrent" stops being a polite fiction. Until now, several tasks took turns on one fast heart. Now four hearts beat independently, and everything we built to survive that — the locks, the barriers, the shared page table — is suddenly not theory but the only thing keeping the machine sane.

Try it

Wire the serial adapter as before, open the terminal at 115200 8N1, and power the board. You should see core 0 greet you, then the three sleepers wake and report in — their order may vary from boot to boot, which is itself a little thrill: it means they are genuinely racing:

core 0 is awake (and no longer alone)
core 1 is awake
core 2 is awake
core 3 is awake

Remove the lock and run it again, and you will watch those four lines tangle into nonsense — proof that the contention is real and that last article's work is what makes this one legible. Four cores are awake. Next, we teach Alquist to share the work among them.