Cohere details a megakernel serving engine for North Mini Code

Cohere released an early research serving engine for North Mini Code that runs the model’s BF16 decode path as a single persistent megakernel on one NVIDIA H100. The technical deep dive and public repository describe a working server that combines the kernel with continuous batching, paged KV-cache management, ragged-sequence attention and OpenAI-style API routes.

The release turns a low-level kernel design into a serving stack, but within narrow boundaries: it is specialized for North Mini Code, targets H100 SM90a hardware, supports batch sizes from 1 through 8 and uses the megakernel only for decode. Prefill still runs through ordinary PyTorch kernels and pauses decoding.

For each forward pass, the engine launches one persistent CUDA thread block on each of the H100’s 132 streaming multiprocessors. It divides model operations into tiled tasks and tracks dependencies with counters in global memory. That arrangement is designed to avoid the repeated kernel launches, synchronization points and intermediate memory traffic of a conventional decode path. Most work follows task lists assembled by the host and distributed round-robin. Variable-sized attention and mixture-of-experts work instead enters shared queues, where workers can claim tiles through local atomic work stealing.

Continuous batching is split across two layers. A Python control plane accepts requests, performs prefill and decides when to admit, evict or reshape requests. A native C++ loop owns decode. Before Python changes the live batch, it parks that loop, updates the state and resumes it.

The attention path keeps per-request KV data in paged cache tables. Because active requests can have different sequence lengths, the engine calculates an attention split count for each request from its current length, places the resulting tiles in a shared queue and lets GPU workers drain the work dynamically. Paged storage avoids requiring every request’s KV cache to occupy one contiguous allocation, while the queue lets the assigned attention work track each request’s live sequence length.

The server exposes OpenAI-style chat-completions and completions routes, along with a models route. Cohere also lists streaming and tool calling as supported features. Those interfaces and scheduling features are not unique to this implementation: vLLM’s documentation also lists continuous batching, PagedAttention-based KV-memory management and an OpenAI-compatible API server.

Cohere says its decode-only test reached 292 tokens per second at batch size 1, equal to 62% of a theoretical limit it calculated from the model’s active weights and H100 bandwidth, and 1.58 times its vLLM result. The comparison used vLLM v0.24 with FlashAttention-3 and Triton mixture-of-experts backends, disabled prefill, a synthetic KV cache and 1,000 generated tokens, according to Cohere. NVIDIA lists 3.35 TB/s of memory bandwidth for the H100 SXM, the figure Cohere used in that calculation. None of the opened sources independently reproduced the result.

In five separate end-to-end runs with real prompts, prefill and continuous batching at batch size 8, Cohere reports 1.25 to 1.41 times higher average decode throughput than vLLM. The two engines generated different total token counts, and the comparison has not been independently replicated. Cohere also reports mean scores of 38.9% plus or minus 1.6% on SciCode and 70.3% plus or minus 1.1% on LiveCodeBench v6 across seven megakernel runs, compared with 38.2% and 70.3% for vLLM. Those accuracy results likewise remain unverified outside Cohere’s tests.

North Mini Code is a mixture-of-experts coding model with 30 billion total parameters. Cohere’s model announcement rounded the active count to 3 billion, while the serving post gives 3.3 billion active parameters per token. DataPhoenix separately covered Cohere’s Agentic Task Ecosystem study of public MCP tools.

The engine follows earlier Hazy Research work that combined a Llama-1B forward pass into one megakernel using per-SM instruction sequences and global-memory counters. Cohere’s repository labels the new implementation an early, model-specific research release rather than a general-purpose inference engine.