AssemblyAI runs speech inference on EC2 Spot GPUs with queued retries

AssemblyAI runs its speech-to-text inference platform on a mix of Amazon EC2 Spot and On-Demand GPU capacity, according to an AWS case study. Amazon SQS holds transcription jobs for workers, allowing interrupted work to return to the queue for another healthy worker instead of tying a request to one interruptible instance.

When EC2 prepares to reclaim a Spot instance, an EventBridge event invokes a Lambda interruption handler. Jobs that need more time than the warning window return to the main SQS queue, where another healthy GPU worker can pick them up on a different instance type or in another Availability Zone. EC2 normally issues the warning two minutes before it stops or terminates an instance, although AWS says notices are best-effort and hibernation does not receive the two-minute lead time.

AssemblyAI uses On-Demand instances as an always-on baseline and Spot capacity for demand spikes. Its Spot pool spans multiple Availability Zones and the g4dn, g5 and g6 GPU families, including g4dn.xlarge, g4dn.2xlarge, g5.xlarge, g5.4xlarge and g6.2xlarge instances. Amazon ECS managed scaling adds GPU instances as pending inference work grows. The deployment assigns one inference model to each GPU across what AWS described as thousands of instances.

Each request enters an SQS queue before a GPU-backed ECS task retrieves the audio from Amazon S3, runs the inference pipeline and writes the completed transcription to a results store. AWS said individual jobs usually last from 15 seconds to several minutes. Work that can finish within the interruption window completes normally, while longer work returns to the queue. SQS makes an unfinished message available for another attempt after its visibility timeout expires; setting the timeout to zero makes it immediately available to other consumers. Jobs that exhaust their retries go to a dead-letter queue for investigation.

AWS reported that the Spot design cut AssemblyAI’s inference-compute cost by 70% compared with On-Demand pricing while compute demand grew nearly fourfold year over year. It also said customers see a small increase in processing time after an interruption while the transcription still completes, and that the design maintains AssemblyAI’s service-level commitments. The case study did not provide the methodology or raw measurements behind the cost, growth, latency or availability claims.

AssemblyAI operates independent deployments in AWS’s US West (Oregon) and Europe (London) regions. Each region has separate SQS queues, ECS clusters, capacity providers and GPU fleets, which AWS said supports customers’ data-residency requirements.