# A Windows Kernel in a Browser Tab, Part III: Debugging It, and the Crash Dumps It Writes Itself URL: https://www.msuiche.com/posts/nanokrnl-debugging-crash-dump/ Date: 2026-07-05 Author: Matt Suiche Tags: nanokrnl, nanox, Rust, Windows Kernel, ntoskrnl, lldb, gdb, WinDbg, Debugging, Crash Dump, ELF, DWARF, KDBG, KDDEBUGGER_DATA64, MEMORY.DMP, DUMP_HEADER64, PDB, EPROCESS, 9P, WebAssembly > You can attach lldb to nanokrnl while it runs in a browser tab, break in kernel code, and hit it. When it bugchecks it writes two crash dumps of itself: a Linux ELF core symbolized by its own DWARF, and a native Windows MEMORY.DMP that WinDbg opens as a full kernel target, with lm, dt, r, kv, !analyze -v, !object and !process all working. From a kernel that has never run on real hardware. --- *Written by Twinkle.* [Part I](/posts/nanokrnl-cold-boot-fast-boot/) was about how nanokrnl boots in a browser tab and how small it is. [Part II](/posts/nanokrnl-9p-host-filesystem/) gave it a filesystem over 9P. This one is about making it a real system to work on: you can attach **lldb** to the kernel while it runs in the tab, break in kernel code, and step it. And when it crashes, it writes **its own crash dumps**, which you open in a debugger with full symbols. Two of those dumps, in fact. A Linux-style ELF core that gdb and lldb read directly, and a native Windows `MEMORY.DMP` that WinDbg opens as a genuine kernel target: `lm`, `dt`, `r`, `kv`, `!analyze -v`, `!object`, `!process`. The twist that makes the second one fun is that the kernel writing it has never executed on real hardware or under Hyper-V. It runs inside the nanox emulator, and yet every field WinDbg reads is a byte-accurate NT structure at the offset the debugger expects. ## Attaching a debugger to a tab nanox already interprets x86-64 instruction by instruction, so it knows the exact register and memory state at every step. That is most of what a debugger wants. The rest is a protocol: lldb and gdb both speak the **GDB Remote Serial Protocol** (RSP), a small text protocol over a socket, so nanox grew a stub that speaks it: read and write registers, read and write memory (translated through the guest page tables, so `x/i $pc` on a kernel virtual address works), software breakpoints, single-step, continue. A `target.xml` describes the x86-64 register file so lldb enumerates registers without guessing. There is one obstacle, and it is the same shape as the 9P one from Part II: **lldb speaks TCP, and a browser tab cannot open or accept a TCP socket.** The tab can only expose the stream over a WebSocket. So a tiny relay sits between them: ```mermaid flowchart LR L[lldb] <-->|TCP 3333| B[gdb-bridge.py] B <-->|WebSocket 3334| P[browser tab: nanox.wasm] P <--> K[nanokrnl GDB stub] ``` The bridge is about ninety lines of standard-library Python. It listens on a TCP port for lldb, listens on a WebSocket for the page, and copies bytes between them. It has no idea what a GDB packet is; it is a dumb pipe, the same role a named pipe plays when you kernel-debug a VM with WinDbg. The page's Debug panel just hands you the one-liner: ```text python3 <(curl -sL https://nanokrnl.ai/bridge.py) ``` Run that, click Debug, and: ```text (lldb) gdb-remote 3333 Process 1 stopped * thread #1, stop reason = signal SIGTRAP frame #0: 0xffff800000133cb7 -> 0xffff800000133cb7: testq %rsi, %rsi (lldb) breakpoint set -a $pc (lldb) continue Process 1 stopped, stop reason = breakpoint 1.1 ``` That is real lldb, on your machine, stopped on a breakpoint inside a kernel that is running in a browser tab. A nice touch that comes almost for free: nanox treats the `int3` instruction as a debugger trap when a debugger is attached, and as a no-op when none is. So the kernel's bugcheck path issues an `int3`, and a crash **breaks into lldb**, exactly like `KdBreak` on a real Windows kernel. Type `crash` at the prompt with lldb attached and you land at the fault. ## The blue screen `crash` is a tiny ring-3 program that issues a bugcheck syscall; the kernel services it with `KeBugCheckEx(MANUALLY_INITIATED_CRASH)`, prints the classic `*** STOP: 0x000000E2` banner, and halts. The page notices the STOP, clears the console scrollback, and turns the window blue. It is cosmetic, but it is the right cosmetic, and it sets up the interesting part: what the kernel does *before* it halts. ## The crash dump, and a wrong turn worth describing The goal was a crash dump you can actually analyze. The first attempt was a Windows `MEMORY.DMP`: the page would read the guest's physical RAM out of the emulator and prepend a `DUMP_HEADER64`. It produced a file WinDbg would open, and it was wrong in two ways that are worth naming, because they point at the right design. First, it was **not faithful**. On real Windows the *kernel* writes the dump (crashdump.sys / `IoWriteCrashDump`), not the hypervisor. Having the browser assemble it is backwards. Second, it could **never be fully analyzable that way**. A `MEMORY.DMP` that `!analyze` and `lm` can work with needs a valid `KDDEBUGGER_DATA64` and, for symbols, a **PDB**. WinDbg resolves `nt!` symbols by finding ntoskrnl's PE header, reading the CodeView record for its PDB signature, and loading the matching PDB. nanokrnl has none of that: it is built for a bare-metal Rust target, which emits an **ELF** with **DWARF** debug info, not a PE with a PDB. A header built by JavaScript that fills those fields with zeros is a dump that opens and then tells you nothing. The realization that fixed both: **nanokrnl is an ELF with DWARF, and gdb, the `crash` utility, and the modern WinDbg engine all read ELF and DWARF directly.** So the first faithful, analyzable format is not a Windows dump at all. It is a **Linux-style ELF core** (`ET_CORE`), in the shape of `/proc/vmcore` or a kdump image, and the kernel's own `kernel.bin` is the symbol file. No synthetic PDB, nothing to fake. So nanokrnl writes its own core. On a bugcheck it: - walks its higher-half page tables and emits one `PT_LOAD` per mapping, so code and stacks are readable at their real virtual addresses, - writes a `PT_NOTE` with `NT_PRSTATUS` (the crash register set, so the debugger lands on the faulting frame) and a `VMCOREINFO` note (the kdump metadata, plus the bugcheck code and parameters), - and streams the whole thing to `H:\nanokrnl.core` over the 9P transport from Part II, now made writable (`Tlcreate` + `Twrite`). The browser only *receives* the file and offers it as a download, and the H:\ Explorer window lists it. The dump is authored, byte for byte, by the kernel. ```mermaid flowchart TD A["crash.exe (ring 3)"] --> B["KeBugCheckEx 0xE2"] B --> C["walk page tables -> PT_LOAD runs"] C --> D["ELF core: PT_NOTE (NT_PRSTATUS + VMCOREINFO) + memory"] D --> E["stream to H:\\nanokrnl.core over writable 9P"] E --> F["browser: download / Explorer"] F --> G["gdb kernel.bin nanokrnl.core"] ``` Then, on a real machine: ```text $ gdb target/x86_64-unknown-none/release/kernel nanokrnl.core ``` and the crash is symbolic, because the symbols were in the kernel all along. ## Two things the transport taught us Writing megabytes out of a kernel over a byte-at-a-time port, in an emulator, surfaced two lessons. **The transport turns over about one request per run-slice.** The emulator runs the guest in slices and services the 9P host between them, so a client that sends one `Twrite` and waits for its reply makes one round trip per slice. A multi-megabyte dump that way crawls. The fix is to **pipeline**: send a batch of writes, then collect their replies, so the host services many per slice. **You cannot copy the memory you are dumping through a buffer that lives in it.** The kernel's pool is inside the physical window being dumped. Building each 9P message by copying the payload into a freshly allocated buffer can place that buffer on top of the very bytes being read, and the copy aliases itself. The emulator caught it as undefined behavior. The fix is to stream the payload straight from the source region and build only the small message header on the stack: no allocation, no copy, in the hot path. ## The other half: a native Windows dump The ELF core is enough for gdb and lldb. But the other half of what you do in a kernel debugger is Windows-specific: `lm` to see what is loaded, `!process 0 0` to see what is running, `!analyze -v` to triage the crash. A modern WinDbg can open the ELF core, but it treats it as a *Linux* target, so those extensions never fire. To get them, the crash has to be in the format Windows itself writes: a `DUMP_HEADER64` kernel dump. So on a bugcheck nanokrnl now writes a second file next to the core, `H:\MEMORY.DMP`, in exactly that format. Those commands are not generic. They are the debugger reading Windows kernel data structures at the addresses a Windows kernel puts them. So to make them work against a from-scratch Rust kernel, nanokrnl has to lay out those structures itself. ### How a kernel debugger bootstraps When a debugger opens a kernel target, it does not scan memory for processes. It resolves one symbol, `KdDebuggerDataBlock`, and reads a `KDDEBUGGER_DATA64` there. That block is the index to the rest of the kernel: it carries the `'KDBG'` tag, the kernel base, and pointers to two list heads. `lm` follows one, `!process` follows the other. ```mermaid flowchart TD S["symbol: KdDebuggerDataBlock"] --> D["KDDEBUGGER_DATA64
'KDBG', KernBase"] D -->|PsLoadedModuleList| M["ring of KLDR_DATA_TABLE_ENTRY
DllBase, SizeOfImage, BaseDllName"] D -->|PsActiveProcessHead| P["ring of EPROCESS
UniqueProcessId, ImageFileName, DirectoryTableBase"] M --> LM["lm"] P --> PS["!process 0 0"] ``` Both lists are circular doubly-linked lists (the classic Windows `LIST_ENTRY` with `Flink`/`Blink`). The module list threads through each entry's `InLoadOrderLinks`; the process list threads through each `EPROCESS`'s `ActiveProcessLinks`, which sits at a known offset, so the debugger subtracts that offset from each link to get the `EPROCESS` base. nanokrnl declares those three structures as real kernel data, at their genuine NT field offsets, and fills them from its own live state right before the dump is written: the kernel first (published as `ntoskrnl.exe`, so a debugger loads its symbols against it), then the loaded shims (`kernel32`, `msvcrt`, `ntdll`) and the running program image, then one `EPROCESS` per entry in the process table. The dump carries a coherent snapshot, not a half-updated one. ### The address wrinkle There is one thing that has to line up or nothing resolves. nanokrnl is linked at address 0, but the loader maps it at `0xffff800000000000`. So a symbol's runtime address is `0xffff800000000000 + its link offset`, and that is where the data actually is in the dump. `KdDebuggerDataBlock`, linked at `0x23eee0`, lives at `0xffff80000023eee0`. A debugger makes this work by loading the kernel module's symbols at that base; then every symbol resolves straight into the captured memory. The dump header also records the anchor addresses, so a tool can find the block without symbols at all. ### The DUMP_HEADER64 The header is an 8 KiB structure a Windows debugger knows by heart: - the `PAGE` / `DU64` signature that says "64-bit kernel dump"; - `DirectoryTableBase`, the crash `CR3`, whose page tables (captured in the dump) translate every kernel virtual address; - `KdDebuggerDataBlock`, `PsLoadedModuleList`, `PsActiveProcessHead`, the same three anchors as above; - `MachineImageType` `0x8664`, the bugcheck code and its four parameters; - a `PHYSICAL_MEMORY_DESCRIPTOR`, one run over the captured physical window; - and the crash `CONTEXT`, which is the `KPROCESSOR_STATE.ContextFrame` a full dump exposes. Its `ContextFlags` advertise exactly the register groups we fill, no floating-point claim we cannot back, so the debugger accepts it. `CR3` is not in the `CONTEXT`; it rides in `DirectoryTableBase`. `DumpType` is a full memory dump (we are small, so we just dump the low physical window whole), and the body after the header is that physical memory. WinDbg reads `DirectoryTableBase`, walks the captured four levels of page tables to translate each virtual address, checks the `'KDBG'` tag, and follows the two rings. It is a Windows kernel target now, not a Linux one. There is a nice side effect on the way out. The bugcheck path streams the dump over 9P with a per-file progress readout (`*** MEMORY.DMP: 42%`) before the STOP banner, so the crash gives feedback instead of stalling silently while a multi-megabyte file goes out over a byte-at-a-time port. ## Symbols without a PDB, and the stale-symbol trap WinDbg wants a PDB to turn addresses into names and to know a struct's layout. nanokrnl is an ELF with DWARF and has no PDB. But there is a shortcut that falls out of the address wrinkle: because the kernel links at 0, every symbol's value *is* its RVA, the exact offset a debugger adds to the load base. So `tools/gen_pdb.py` reads the kernel's ELF symbol table and emits an `ntoskrnl.pdb` via `llvm-pdbutil`. Drop it on the symbol path, and names resolve as base plus RVA. Getting from "names resolve" to "`dt nt!_EPROCESS` prints a real struct" took two things the `yaml2pdb` path does not emit, both patched into the raw MSF by hand: - The **TPI hash stream**, filled per type record. Without it dbghelp loads the PDB as "publics only" and `dt` fails even though the type record is present. - A **section-headers stream**, wired into the DBI Optional Debug Header, with public symbol offsets emitted section-relative (RVA minus section VA). Without it WinDbg drops every public. The subtlest bug of the whole effort lived here, and it was not a structure bug at all. The PDB used a fixed GUID across every build. **dbghelp caches parsed PDBs by GUID**, so after the first load it kept serving the old type information: a fix would land, the rebuilt PDB would copy over, and WinDbg kept showing the previous layout. The symptom looked exactly like the fix not working. The cure is to derive the PDB GUID from content (a hash of the kernel image plus the type records) and patch that same GUID into the dump's masquerade CodeView record, so dump and PDB always agree and any change forces a reload. If you ever build synthetic PDBs, do this from day one. ## Making WinDbg trust it With types in place, each Windows command turned out to be its own validation that the debugger runs before it will trust the data. Peeling them apart mostly meant reverse-engineering the shipping debugger binaries and reading the old NT source, rather than guessing. **`r`, `kv`, `!analyze -v`.** Printing registers, walking the stack, and triaging the bug need the full processor block: a synthetic `KPROCESSOR_STATE`, `KPRCB`, and `KPCR` with a valid `GdtBase`/`IdtBase`, plus a `KiProcessorBlock`, with the offsets published in `KdDebuggerDataBlock` byte-exact. The `KDDEBUGGER_DATA64` tail has an eight-byte alignment pad; getting the PCR offset fields off by that pad made WinDbg read the PCR at the PRCB address and fail the CS descriptor lookup. Once every offset matched, `!analyze -v` produced a clean `MANUALLY_INITIATED_CRASH` bucket naming the process, with the faulting thread and a symbolized top frame. **`!object` and the TypeIndex.** Since Windows Vista an object's type is not a direct pointer. The `_OBJECT_HEADER` in front of the object holds a `TypeIndex` byte, and the real index is `TypeIndex XOR ((header address >> 8) & 0xff) XOR nt!ObHeaderCookie`; then `nt!ObTypeIndexTable[index]` gives the `_OBJECT_TYPE`, which must equal `nt!PsProcessType`. So every process object needed a real `_OBJECT_HEADER` in front of it, a populated `ObTypeIndexTable`, an `ObHeaderCookie`, and a `PsProcessType` object whose own `Index` field agrees. Then `!object` reports `Type: Process`. **`!process` and the dispatcher header.** And yet `!process` still said "TYPE mismatch for process object" on the very object `!object` had just blessed, because it does a *second, different* check: it reads the embedded `_KPROCESS` and validates `Pcb.Header.Type == ProcessObject` (3) at dispatcher-header offset 0. The Windows 2000 source confirms it (`ke/procobj.c`), and the strings `Pcb.Header.Type` and `ProcessObject` are literally in `kdexts.dll`. Our compact `_EPROCESS` had overlaid `UniqueProcessId` onto offset 0; moving the PID to its own offset and putting `Type = 3` at offset 0 cleared it. **`!process 0 0` and the user boundary.** The enumerate form reads `nt!MmUserProbeAddress` to tell a PID from a literal `_EPROCESS` address; unexported it read 0, so `0 < 0` is false and it dereferenced address 0. Exporting it (`0x7fffffff0000`) and pointing the matching KDBG field at it let the walk proceed. **`KUSER_SHARED_DATA`.** WinDbg reads `SharedUserData` at `0xfffff78000000000` during setup for the OS version, timing, and XState; its absence gave "Unable to get shared data" and no uptime. nanokrnl now synthesizes the page (version, `KdDebuggerEnabled`, the time fields, a minimal `_XSTATE_CONFIGURATION`) and maps it into the kernel's shared high half so every process address space sees it. System Uptime shows now. ## Verifying it I develop this on macOS, so the fast inner loop cannot be WinDbg. Instead there is a small walker, `tools/dmp_check.py`, that does exactly what a Windows debugger's engine does with `MEMORY.DMP`: read `DirectoryTableBase`, walk the four levels of captured page tables to translate each virtual address against the dumped physical memory, find `KdDebuggerDataBlock`, check the `'KDBG'` tag, and follow the two rings. If it walks cleanly, the dump is NT-shaped and an off-the-shelf engine sees the same thing. On a real crash dump: ```text DUMP_HEADER64: DirectoryTableBase=0x1190000 BugCheck=0xe2 KdDebuggerDataBlock = 0xffff8000002453f8 (header) OwnerTag = b'KDBG' Size = 0x340 KernBase = 0xffff800000000000 'KDBG' tag: OK PsLoadedModuleList = 0xffff800000306a50 PsActiveProcessHead = 0xffff800000306a40 CONTEXT: Flags=0x100007 Cs=0x10 Rip=0xffff8000001d3bbf Rsp=0xffffff00011e6e40 === lm (loaded modules) === start end module 0xffff800000000000 0xffff800000400000 ntoskrnl.exe 0x0000000140000000 0x0000000140071000 cmd.exe 0xffffff0000d51010 0xffffff0000d60010 kernel32 0xffffff0000d61010 0xffffff0000d6b010 msvcrt 0xffffff0000ca0000 0xffffff0000ca1000 ntdll === !process 0 0 (active processes) === PROCESS 0xffff8000003066c0 Cid: 0x0004 DirBase: 0xdad000 Image: child.exe PROCESS 0xffff8000003066f8 Cid: 0x0008 DirBase: 0xe38000 Image: child.exe PROCESS 0xffff800000306730 Cid: 0x000c DirBase: 0xf21000 Image: child.exe PROCESS 0xffff800000306768 Cid: 0x0010 DirBase: 0x1190000 Image: crash ``` Then in WinDbg itself, opening the same file: `lm` lists the modules, `dt nt!_EPROCESS ` decodes the struct, `r` and `kv` give registers and a symbolized stack, `!analyze -v` triages the crash, `!object` resolves the process type, `!process ` prints the full process block, and `dl nt!PsActiveProcessHead` walks the whole ring. ## Where the series lands, and one honest edge nanokrnl now boots in a tab, serves files over 9P, lets you attach lldb and break in its own code, and writes two crash dumps of itself when it dies: a Linux ELF core for gdb and lldb, and a native Windows `MEMORY.DMP` that WinDbg opens as a full kernel target, with a generated `ntoskrnl.pdb` for names and types. That is most of the loop you actually use a kernel debugger for, running inside a browser, driven by a ninety-line Python relay and the kernel's own debug info. The honest edge: `!process 0 0`, the enumerate-everything form, prints the header and the first process, then stops, while `dl` and `!for_each_process` traverse all four. Tracing it, the stop is a `kdexts.dll` `CheckControlC` returning nonzero after the first entry. Every data-side explanation was ruled out against the dump: the process ring is a clean circular list, `KUSER_SHARED_DATA` is mapped under every CR3, and the GDT kernel-code descriptor has the long-mode bit set, so the same virtual address returns the same bytes in every context. What is unusual here is that the debugger is ARM64 WinDbg running the x64 `kdexts.dll` under emulation, so that `CheckControlC` crosses an x64-to-ARM64 boundary; provably-correct data plus inconsistent results across the typed enumerators point at the emulated extension layer rather than the dump. The clean confirmation is to open the exact same files in a native x64 WinDbg, and that is the next thing to check. That caveat aside, the result stands: a from-scratch NT-compatible kernel, in Rust, small enough to boot under a 65 KB emulator in a browser tab, writes a crash dump that Microsoft's kernel debugger opens, symbolizes, and inspects as if it came off a real machine. The bar for "faithful" was set by a tool that has spent decades refusing to be fooled, and most of it now passes. Try it at [nanokrnl.ai](https://nanokrnl.ai), and the code is on [GitHub](https://github.com/msuiche/nanokrnl). *Thanks to [Ryan MacArthur](https://x.com/maceip) for the 9P direction that made the writable share, and these dumps, possible.*