Steering a Loop: Control Vectors Meet the Looped Transformer

Sun, 06 Sep 2026 00:00:00 +0200

Nanbeige4.2-3B is the first open-weight Looped Transformer anyone can download: a 22-layer stack executed twice, 44 effective layer-passes, 3B non-embedding parameters, and benchmark numbers that embarrass dense models four times its size. It buys that capacity with repetition rather than parameters.

The lineage runs from the Universal Transformer through Huginn (a recurrent block looped a variable number of times at inference) and ByteDance Seed’s Ouro (looped computation trained into pretraining itself, with an entropy-regularized objective that learns how many iterations an input deserves), to Google’s Mixture-of-Recursions, which routes each token to its own recursion depth. On September 1st, The Information reported that OpenAI’s Astra is recurrent-depth: latent iteration in place of visible chain-of-thought, which is why the safety community spent the first week of September arguing about monitorability. OpenAI has not published architecture details, so treat the report as a strong rumor. Either way, depth has become a runtime quantity; Nanbeige4.5 is already training with the token-routed machinery in its config.

Inkling on Two DGX Sparks: The Only vLLM Lane, and the Two Walls Behind It

Fri, 04 Sep 2026 23:00:00 +0200

Thinking Machines shipped Inkling-Small at the end of July: a ~300B-total, ~10B-active hybrid MoE (short-conv plus relative-bias attention), natively multimodal, 1M context, and - the part that matters for homelab hardware - released as NVFP4 from day one. 170.7 GB of weights. Two DGX Sparks hold 243 GB of unified memory. You can see where this is going.

For five weeks, though, every Spark recipe for it - drowzeys’ champion image, MiaAI’s wrapper, and the half-dozen forks - ran on SGLang with a custom-baked image, for a simple reason: SGLang had Inkling support in early August, and vLLM only gained it this week, in v0.28.0. The engine we build everything on could not even load the model before Monday.

Weightless on Matt Suiche

Steering a Loop: Control Vectors Meet the Looped Transformer

Inkling on Two DGX Sparks: The Only vLLM Lane, and the Two Walls Behind It