AMD details automatic FlashInfer kernel routing on ROCm 10.1
AMD lists four Instinct GPUs for the source build, which adds FP8 prefill and a separate kernel that overlaps prompt processing with token generation.
AMD on November 20 detailed automatic kernel routing in FlashInfer on ROCm 10.1. In the FlashInfer 0.6.18+amd.2 source build for four Instinct GPUs, the language-model inference library chooses between AMD’s AI Tensor Engine for ROCm (AITER) and FlashInfer’s built-in HIP kernels for supported attention operations.
AMD lists the MI300X and MI325X on CDNA3 (gfx942), and the MI350X and MI355X on CDNA4 (gfx950).
FlashInfer selects a backend automatically by default for single-request prefill, batch prefill with paged or ragged key-value caches, and batch decode. In backend="auto" mode, FlashInfer checks whether AITER supports the call’s data types, cache layout, attention mask and head dimensions, then applies rules for the GPU architecture. It falls back to HIP where supported if AITER cannot handle the call. The rules can also keep some compatible calls on HIP, such as certain short-query prefill operations.
After plan(), the batch prefill and decode wrappers report the choice in wrapper.backend as aiter or fa2, the name for built-in HIP attention.
AMD lists 8-bit floating-point (FP8) prefill for prompt processing and POD-Attention, a separate HIP kernel that overlaps prompt processing with token generation. POD-Attention rejects FP8 key-value caches.
Fused mixture-of-experts operations use AITER only. For batch decode under CUDA graph capture, automatic routing can select AITER when the caller sets max_seq_len and the call passes the other compatibility checks. Without that setting, automatic batch decode stays on HIP during graph capture.
AMD says its port retains the decode, prefill and multi-head latent attention wrapper signatures from FlashInfer v0.6.18, released upstream on August 29. Some CUDA-only options, including graph capture for multi-head latent attention, remain unsupported on ROCm. The AMD source release followed on October 22.
AMD specifies ROCm 10.1.0, PyTorch 2.12.0, Python 3.12 and Ubuntu 24.04 for building FlashInfer from source. The setup uses amd-aiter 0.1.21.post2 built from source. AMD lists no prebuilt FlashInfer wheel or container image for this release. Its instructions specify building the development image as a regular user from the release/0.6.18+amd.2 branch, then installing FlashInfer from that checkout inside the container.
More news

Kioxia and Dell detail a single-rack object store with 157 PB of raw capacity

Together AI launches Storage Accelerator for GPU clusters

AWS adds distributed KV caching to ElastiCache for self-hosted LLMs
