AWS documents prefix-aware routing for SageMaker real-time endpoints

AWS documentation describes prefix-aware routing for Amazon SageMaker real-time inference endpoints. Teams can keep requests with matching prompt beginnings on the same instance until a configured concurrency limit is reached, giving shared-prefix workloads request affinity without preventing SageMaker from redirecting traffic when that instance is too busy.

Teams enable PREFIX_AWARE in the production variant routing configuration. SageMaker then sends requests that share a prompt prefix to the same instance. When that instance reaches its configured maximum number of in-flight requests, SageMaker falls back to an instance with more available capacity.

The prefix-aware configuration exposes two workload-specific controls. PrefixLength can be set from 1,024 to 65,536. For InvokeEndpoint and InvokeEndpointWithResponseStream, SageMaker measures the prefix in bytes from the start of the request body. For the OpenAI-compatible API, it counts characters from message text.

AWS advises setting the prefix length to cover shared content, such as a system prompt, plus enough unique content to distribute traffic across instances. The guidance makes the setting a workload-specific routing tradeoff rather than a universal optimum.

The other control, ConcurrencyThreshold, can be set from 1 to 1,024. It defines the maximum number of in-flight requests allowed on the prefix-selected instance before capacity-based fallback applies, so prefix affinity does not override endpoint capacity indefinitely.

The standard invocation API and streaming invocation API also accept an optional stable PrefixAwareId of one to 64 characters. Requests carrying the same prefix and identifier route to the same instance. AWS recommends distinct identifiers when separate applications may send the same prefix.

For context on the company’s other SageMaker infrastructure work, DataPhoenix previously covered AWS’s HyperPod InstantStart control plane.