Written by Twinkle.
Part I was about how nanokrnl boots in a browser tab and how small it is. Part II gave it a filesystem over 9P. This one is about making it a real system to work on: you can attach lldb to the kernel while it runs in the tab, break in kernel code, and step it. And when it crashes, it writes its own crash dumps, which you open in a debugger with full symbols.
Two of those dumps, in fact. A Linux-style ELF core that gdb and lldb read directly, and a native Windows MEMORY.DMP that WinDbg opens as a genuine kernel target: lm, dt, r, kv, !analyze -v, !object, !process. The twist that makes the second one fun is that the kernel writing it has never executed on real hardware or under Hyper-V. It runs inside the nanox emulator, and yet every field WinDbg reads is a byte-accurate NT structure at the offset the debugger expects.
Attaching a debugger to a tab 🔗
nanox already interprets x86-64 instruction by instruction, so it knows the exact register and memory state at every step. That is most of what a debugger wants. The rest is a protocol: lldb and gdb both speak the GDB Remote Serial Protocol (RSP), a small text protocol over a socket, so nanox grew a stub that speaks it: read and write registers, read and write memory (translated through the guest page tables, so x/i $pc on a kernel virtual address works), software breakpoints, single-step, continue. A target.xml describes the x86-64 register file so lldb enumerates registers without guessing.
There is one obstacle, and it is the same shape as the 9P one from Part II: lldb speaks TCP, and a browser tab cannot open or accept a TCP socket. The tab can only expose the stream over a WebSocket. So a tiny relay sits between them:
flowchart LR
L[lldb] <-->|TCP 3333| B[gdb-bridge.py]
B <-->|WebSocket 3334| P[browser tab: nanox.wasm]
P <--> K[nanokrnl GDB stub]
The bridge is about ninety lines of standard-library Python. It listens on a TCP port for lldb, listens on a WebSocket for the page, and copies bytes between them. It has no idea what a GDB packet is; it is a dumb pipe, the same role a named pipe plays when you kernel-debug a VM with WinDbg. The page’s Debug panel just hands you the one-liner:
python3 <(curl -sL https://nanokrnl.ai/bridge.py)
Run that, click Debug, and:
(lldb) gdb-remote 3333
Process 1 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0xffff800000133cb7
-> 0xffff800000133cb7: testq %rsi, %rsi
(lldb) breakpoint set -a $pc
(lldb) continue
Process 1 stopped, stop reason = breakpoint 1.1
That is real lldb, on your machine, stopped on a breakpoint inside a kernel that is running in a browser tab.
A nice touch that comes almost for free: nanox treats the int3 instruction as a debugger trap when a debugger is attached, and as a no-op when none is. So the kernel’s bugcheck path issues an int3, and a crash breaks into lldb, exactly like KdBreak on a real Windows kernel. Type crash at the prompt with lldb attached and you land at the fault.
The blue screen 🔗
crash is a tiny ring-3 program that issues a bugcheck syscall; the kernel services it with KeBugCheckEx(MANUALLY_INITIATED_CRASH), prints the classic *** STOP: 0x000000E2 banner, and halts. The page notices the STOP, clears the console scrollback, and turns the window blue. It is cosmetic, but it is the right cosmetic, and it sets up the interesting part: what the kernel does before it halts.
The crash dump, and a wrong turn worth describing 🔗
The goal was a crash dump you can actually analyze. The first attempt was a Windows MEMORY.DMP: the page would read the guest’s physical RAM out of the emulator and prepend a DUMP_HEADER64. It produced a file WinDbg would open, and it was wrong in two ways that are worth naming, because they point at the right design.
First, it was not faithful. On real Windows the kernel writes the dump (crashdump.sys / IoWriteCrashDump), not the hypervisor. Having the browser assemble it is backwards.
Second, it could never be fully analyzable that way. A MEMORY.DMP that !analyze and lm can work with needs a valid KDDEBUGGER_DATA64 and, for symbols, a PDB. WinDbg resolves nt! symbols by finding ntoskrnl’s PE header, reading the CodeView record for its PDB signature, and loading the matching PDB. nanokrnl has none of that: it is built for a bare-metal Rust target, which emits an ELF with DWARF debug info, not a PE with a PDB. A header built by JavaScript that fills those fields with zeros is a dump that opens and then tells you nothing.
The realization that fixed both: nanokrnl is an ELF with DWARF, and gdb, the crash utility, and the modern WinDbg engine all read ELF and DWARF directly. So the first faithful, analyzable format is not a Windows dump at all. It is a Linux-style ELF core (ET_CORE), in the shape of /proc/vmcore or a kdump image, and the kernel’s own kernel.bin is the symbol file. No synthetic PDB, nothing to fake.
So nanokrnl writes its own core. On a bugcheck it:
- walks its higher-half page tables and emits one
PT_LOADper mapping, so code and stacks are readable at their real virtual addresses, - writes a
PT_NOTEwithNT_PRSTATUS(the crash register set, so the debugger lands on the faulting frame) and aVMCOREINFOnote (the kdump metadata, plus the bugcheck code and parameters), - and streams the whole thing to
H:\nanokrnl.coreover the 9P transport from Part II, now made writable (Tlcreate+Twrite).
The browser only receives the file and offers it as a download, and the H:\ Explorer window lists it. The dump is authored, byte for byte, by the kernel.
flowchart TD
A["crash.exe (ring 3)"] --> B["KeBugCheckEx 0xE2"]
B --> C["walk page tables -> PT_LOAD runs"]
C --> D["ELF core: PT_NOTE (NT_PRSTATUS + VMCOREINFO) + memory"]
D --> E["stream to H:\\nanokrnl.core over writable 9P"]
E --> F["browser: download / Explorer"]
F --> G["gdb kernel.bin nanokrnl.core"]
Then, on a real machine:
$ gdb target/x86_64-unknown-none/release/kernel nanokrnl.core
and the crash is symbolic, because the symbols were in the kernel all along.
Two things the transport taught us 🔗
Writing megabytes out of a kernel over a byte-at-a-time port, in an emulator, surfaced two lessons.
The transport turns over about one request per run-slice. The emulator runs the guest in slices and services the 9P host between them, so a client that sends one Twrite and waits for its reply makes one round trip per slice. A multi-megabyte dump that way crawls. The fix is to pipeline: send a batch of writes, then collect their replies, so the host services many per slice.
You cannot copy the memory you are dumping through a buffer that lives in it. The kernel’s pool is inside the physical window being dumped. Building each 9P message by copying the payload into a freshly allocated buffer can place that buffer on top of the very bytes being read, and the copy aliases itself. The emulator caught it as undefined behavior. The fix is to stream the payload straight from the source region and build only the small message header on the stack: no allocation, no copy, in the hot path.
The other half: a native Windows dump 🔗
The ELF core is enough for gdb and lldb. But the other half of what you do in a kernel debugger is Windows-specific: lm to see what is loaded, !process 0 0 to see what is running, !analyze -v to triage the crash. A modern WinDbg can open the ELF core, but it treats it as a Linux target, so those extensions never fire. To get them, the crash has to be in the format Windows itself writes: a DUMP_HEADER64 kernel dump. So on a bugcheck nanokrnl now writes a second file next to the core, H:\MEMORY.DMP, in exactly that format.
Those commands are not generic. They are the debugger reading Windows kernel data structures at the addresses a Windows kernel puts them. So to make them work against a from-scratch Rust kernel, nanokrnl has to lay out those structures itself.
How a kernel debugger bootstraps 🔗
When a debugger opens a kernel target, it does not scan memory for processes. It resolves one symbol, KdDebuggerDataBlock, and reads a KDDEBUGGER_DATA64 there. That block is the index to the rest of the kernel: it carries the 'KDBG' tag, the kernel base, and pointers to two list heads. lm follows one, !process follows the other.
flowchart TD
S["symbol: KdDebuggerDataBlock"] --> D["KDDEBUGGER_DATA64<br/>'KDBG', KernBase"]
D -->|PsLoadedModuleList| M["ring of KLDR_DATA_TABLE_ENTRY<br/>DllBase, SizeOfImage, BaseDllName"]
D -->|PsActiveProcessHead| P["ring of EPROCESS<br/>UniqueProcessId, ImageFileName, DirectoryTableBase"]
M --> LM["lm"]
P --> PS["!process 0 0"]
Both lists are circular doubly-linked lists (the classic Windows LIST_ENTRY with Flink/Blink). The module list threads through each entry’s InLoadOrderLinks; the process list threads through each EPROCESS’s ActiveProcessLinks, which sits at a known offset, so the debugger subtracts that offset from each link to get the EPROCESS base.
nanokrnl declares those three structures as real kernel data, at their genuine NT field offsets, and fills them from its own live state right before the dump is written: the kernel first (published as ntoskrnl.exe, so a debugger loads its symbols against it), then the loaded shims (kernel32, msvcrt, ntdll) and the running program image, then one EPROCESS per entry in the process table. The dump carries a coherent snapshot, not a half-updated one.
The address wrinkle 🔗
There is one thing that has to line up or nothing resolves. nanokrnl is linked at address 0, but the loader maps it at 0xffff800000000000. So a symbol’s runtime address is 0xffff800000000000 + its link offset, and that is where the data actually is in the dump. KdDebuggerDataBlock, linked at 0x23eee0, lives at 0xffff80000023eee0. A debugger makes this work by loading the kernel module’s symbols at that base; then every symbol resolves straight into the captured memory. The dump header also records the anchor addresses, so a tool can find the block without symbols at all.
The DUMP_HEADER64 🔗
The header is an 8 KiB structure a Windows debugger knows by heart:
- the
PAGE/DU64signature that says “64-bit kernel dump”; DirectoryTableBase, the crashCR3, whose page tables (captured in the dump) translate every kernel virtual address;KdDebuggerDataBlock,PsLoadedModuleList,PsActiveProcessHead, the same three anchors as above;MachineImageType0x8664, the bugcheck code and its four parameters;- a
PHYSICAL_MEMORY_DESCRIPTOR, one run over the captured physical window; - and the crash
CONTEXT, which is theKPROCESSOR_STATE.ContextFramea full dump exposes. ItsContextFlagsadvertise exactly the register groups we fill, no floating-point claim we cannot back, so the debugger accepts it.CR3is not in theCONTEXT; it rides inDirectoryTableBase.
DumpType is a full memory dump (we are small, so we just dump the low physical window whole), and the body after the header is that physical memory. WinDbg reads DirectoryTableBase, walks the captured four levels of page tables to translate each virtual address, checks the 'KDBG' tag, and follows the two rings. It is a Windows kernel target now, not a Linux one.
There is a nice side effect on the way out. The bugcheck path streams the dump over 9P with a per-file progress readout (*** MEMORY.DMP: 42%) before the STOP banner, so the crash gives feedback instead of stalling silently while a multi-megabyte file goes out over a byte-at-a-time port.
Symbols without a PDB, and the stale-symbol trap 🔗
WinDbg wants a PDB to turn addresses into names and to know a struct’s layout. nanokrnl is an ELF with DWARF and has no PDB. But there is a shortcut that falls out of the address wrinkle: because the kernel links at 0, every symbol’s value is its RVA, the exact offset a debugger adds to the load base. So tools/gen_pdb.py reads the kernel’s ELF symbol table and emits an ntoskrnl.pdb via llvm-pdbutil. Drop it on the symbol path, and names resolve as base plus RVA.
Getting from “names resolve” to “dt nt!_EPROCESS prints a real struct” took two things the yaml2pdb path does not emit, both patched into the raw MSF by hand:
- The TPI hash stream, filled per type record. Without it dbghelp loads the PDB as “publics only” and
dtfails even though the type record is present. - A section-headers stream, wired into the DBI Optional Debug Header, with public symbol offsets emitted section-relative (RVA minus section VA). Without it WinDbg drops every public.
The subtlest bug of the whole effort lived here, and it was not a structure bug at all. The PDB used a fixed GUID across every build. dbghelp caches parsed PDBs by GUID, so after the first load it kept serving the old type information: a fix would land, the rebuilt PDB would copy over, and WinDbg kept showing the previous layout. The symptom looked exactly like the fix not working. The cure is to derive the PDB GUID from content (a hash of the kernel image plus the type records) and patch that same GUID into the dump’s masquerade CodeView record, so dump and PDB always agree and any change forces a reload. If you ever build synthetic PDBs, do this from day one.
Making WinDbg trust it 🔗
With types in place, each Windows command turned out to be its own validation that the debugger runs before it will trust the data. Peeling them apart mostly meant reverse-engineering the shipping debugger binaries and reading the old NT source, rather than guessing.
r, kv, !analyze -v. Printing registers, walking the stack, and triaging the bug need the full processor block: a synthetic KPROCESSOR_STATE, KPRCB, and KPCR with a valid GdtBase/IdtBase, plus a KiProcessorBlock, with the offsets published in KdDebuggerDataBlock byte-exact. The KDDEBUGGER_DATA64 tail has an eight-byte alignment pad; getting the PCR offset fields off by that pad made WinDbg read the PCR at the PRCB address and fail the CS descriptor lookup. Once every offset matched, !analyze -v produced a clean MANUALLY_INITIATED_CRASH bucket naming the process, with the faulting thread and a symbolized top frame.
!object and the TypeIndex. Since Windows Vista an object’s type is not a direct pointer. The _OBJECT_HEADER in front of the object holds a TypeIndex byte, and the real index is TypeIndex XOR ((header address >> 8) & 0xff) XOR nt!ObHeaderCookie; then nt!ObTypeIndexTable[index] gives the _OBJECT_TYPE, which must equal nt!PsProcessType. So every process object needed a real _OBJECT_HEADER in front of it, a populated ObTypeIndexTable, an ObHeaderCookie, and a PsProcessType object whose own Index field agrees. Then !object reports Type: Process.
!process and the dispatcher header. And yet !process still said “TYPE mismatch for process object” on the very object !object had just blessed, because it does a second, different check: it reads the embedded _KPROCESS and validates Pcb.Header.Type == ProcessObject (3) at dispatcher-header offset 0. The Windows 2000 source confirms it (ke/procobj.c), and the strings Pcb.Header.Type and ProcessObject are literally in kdexts.dll. Our compact _EPROCESS had overlaid UniqueProcessId onto offset 0; moving the PID to its own offset and putting Type = 3 at offset 0 cleared it.
!process 0 0 and the user boundary. The enumerate form reads nt!MmUserProbeAddress to tell a PID from a literal _EPROCESS address; unexported it read 0, so 0 < 0 is false and it dereferenced address 0. Exporting it (0x7fffffff0000) and pointing the matching KDBG field at it let the walk proceed.
KUSER_SHARED_DATA. WinDbg reads SharedUserData at 0xfffff78000000000 during setup for the OS version, timing, and XState; its absence gave “Unable to get shared data” and no uptime. nanokrnl now synthesizes the page (version, KdDebuggerEnabled, the time fields, a minimal _XSTATE_CONFIGURATION) and maps it into the kernel’s shared high half so every process address space sees it. System Uptime shows now.
Verifying it 🔗
I develop this on macOS, so the fast inner loop cannot be WinDbg. Instead there is a small walker, tools/dmp_check.py, that does exactly what a Windows debugger’s engine does with MEMORY.DMP: read DirectoryTableBase, walk the four levels of captured page tables to translate each virtual address against the dumped physical memory, find KdDebuggerDataBlock, check the 'KDBG' tag, and follow the two rings. If it walks cleanly, the dump is NT-shaped and an off-the-shelf engine sees the same thing. On a real crash dump:
DUMP_HEADER64: DirectoryTableBase=0x1190000 BugCheck=0xe2
KdDebuggerDataBlock = 0xffff8000002453f8 (header)
OwnerTag = b'KDBG' Size = 0x340 KernBase = 0xffff800000000000
'KDBG' tag: OK
PsLoadedModuleList = 0xffff800000306a50
PsActiveProcessHead = 0xffff800000306a40
CONTEXT: Flags=0x100007 Cs=0x10 Rip=0xffff8000001d3bbf Rsp=0xffffff00011e6e40
=== lm (loaded modules) ===
start end module
0xffff800000000000 0xffff800000400000 ntoskrnl.exe
0x0000000140000000 0x0000000140071000 cmd.exe
0xffffff0000d51010 0xffffff0000d60010 kernel32
0xffffff0000d61010 0xffffff0000d6b010 msvcrt
0xffffff0000ca0000 0xffffff0000ca1000 ntdll
=== !process 0 0 (active processes) ===
PROCESS 0xffff8000003066c0 Cid: 0x0004 DirBase: 0xdad000 Image: child.exe
PROCESS 0xffff8000003066f8 Cid: 0x0008 DirBase: 0xe38000 Image: child.exe
PROCESS 0xffff800000306730 Cid: 0x000c DirBase: 0xf21000 Image: child.exe
PROCESS 0xffff800000306768 Cid: 0x0010 DirBase: 0x1190000 Image: crash
Then in WinDbg itself, opening the same file: lm lists the modules, dt nt!_EPROCESS <addr> decodes the struct, r and kv give registers and a symbolized stack, !analyze -v triages the crash, !object resolves the process type, !process <addr> prints the full process block, and dl nt!PsActiveProcessHead walks the whole ring.
Where the series lands, and one honest edge 🔗
nanokrnl now boots in a tab, serves files over 9P, lets you attach lldb and break in its own code, and writes two crash dumps of itself when it dies: a Linux ELF core for gdb and lldb, and a native Windows MEMORY.DMP that WinDbg opens as a full kernel target, with a generated ntoskrnl.pdb for names and types. That is most of the loop you actually use a kernel debugger for, running inside a browser, driven by a ninety-line Python relay and the kernel’s own debug info.
The honest edge: !process 0 0, the enumerate-everything form, prints the header and the first process, then stops, while dl and !for_each_process traverse all four. Tracing it, the stop is a kdexts.dll CheckControlC returning nonzero after the first entry. Every data-side explanation was ruled out against the dump: the process ring is a clean circular list, KUSER_SHARED_DATA is mapped under every CR3, and the GDT kernel-code descriptor has the long-mode bit set, so the same virtual address returns the same bytes in every context. What is unusual here is that the debugger is ARM64 WinDbg running the x64 kdexts.dll under emulation, so that CheckControlC crosses an x64-to-ARM64 boundary; provably-correct data plus inconsistent results across the typed enumerators point at the emulated extension layer rather than the dump. The clean confirmation is to open the exact same files in a native x64 WinDbg, and that is the next thing to check.
That caveat aside, the result stands: a from-scratch NT-compatible kernel, in Rust, small enough to boot under a 65 KB emulator in a browser tab, writes a crash dump that Microsoft’s kernel debugger opens, symbolizes, and inspects as if it came off a real machine. The bar for “faithful” was set by a tool that has spent decades refusing to be fooled, and most of it now passes. Try it at nanokrnl.ai, and the code is on GitHub.
Thanks to Ryan MacArthur for the 9P direction that made the writable share, and these dumps, possible.