News

Exa reports 47% embedding efficiency gain on AWS AI chips

Exa says layout fixes, lower-precision matrix math and custom AWS Neuron kernels improved performance and cost per million embedded documents by 47% from its Inferentia2 baseline. The result has not been independently reproduced.

D
Nov 18, 2026 · 3 min read

Exa says it improved performance and cost per 1 million embedded documents by 47% after changing tensor layouts, numerical precision and kernels in its AWS Neuron pipeline. The figure comes from a jointly authored AWS and Exa case study, not an independent benchmark, and the post does not disclose raw throughput or cost figures.

The work supported Exa’s asynchronous pipeline for embedding structured web documents into a vector database. Exa said a model migration required it to re-embed 30 billion high-priority documents while the live pipeline kept running, temporarily doubling the embedding load.

Exa’s initial baseline was a data-parallel FP16 model on an inf2.xlarge instance, with each workload assigned to one NeuronCore. The team first aligned weight layout with the model’s execution schedule to eliminate unnecessary transposes. It then enabled compiler autocasting, allowing matrix multiplications to use lower precision while preserving a wider dynamic range. AWS lists inf2.xlarge with one Inferentia2 chip and 32 GB of accelerator memory, but those specifications do not validate Exa’s performance claim.

Specialized kernels came next. Exa said a prebuilt attention kernel delivered an additional gain. Profiling then showed the model’s SwiGLU feed-forward block spilling intermediate data to high-bandwidth memory and loading it back. Exa wrote a weight-stationary fused SwiGLU kernel that kept weights in place and combined operations to remove those stalls. The company used the high-level Neuron Kernel Interface for layouts, loops and memory operations, dropping to lower-level instructions for selected tensor operations. AWS’s NKI documentation confirms that the interface gives developers control over tiling, scheduling, memory pressure and instruction-level parallelism.

Exa also rewrote attention around the model’s actual head shapes to avoid computation on padding. It folded attention operations into single matrix multiplications where a generic kernel had required several steps, reducing inner-loop work and trips to accelerator memory.

For each optimization, Exa said it compiled a Neuron executable, checked a forward pass against a baseline dataset, profiled another pass and ran a 30-minute saturation benchmark on hardware. Changes had to produce embeddings with at least 0.999 cosine similarity to the canonical model. That issuer-defined threshold checks numerical closeness; it is not an independent accuracy test and does not establish downstream retrieval quality.

The published result leaves several performance questions open. AWS and Exa did not provide documents-per-second measurements, cost inputs, per-change gains, batch size, sequence-length distribution, the autocast format or the Neuron SDK and compiler version. The headline rounds the result to a 50% throughput gain, while the body reports a 47% improvement in performance and cost per million documents without separately quantifying throughput and cost. The supplied research identified no raw logs, executable artifact or third-party reproduction.

Exa said the same process runs on a Trainium2 NeuronCore and scales through data parallelism, but the published measurements are for Inferentia2 and include no separate Trainium2 configuration, price basis or benchmark. The workload also differs from prior Trainium serving releases: Amazon’s multi-node Rufus deployment used model sharding, hybrid parallelism and vLLM for interactive large-language-model serving, while AWS’s speculative-decoding work measured latency and cost per generated token for decode-heavy LLM inference.

More news