AWS adds usage metering to SageMaker bidirectional streams

AWS has introduced usage-based AWS Marketplace metering for Amazon SageMaker AI bidirectional streaming inference. The option lets model providers report software use as a live session runs. AWS named Deepgram as the launch partner.

The metering applies to the provider’s AWS Marketplace software charge. Customers still pay SageMaker AI infrastructure costs for the instances running the endpoint, including while the endpoint remains provisioned. AWS’s Marketplace pricing documentation describes inference pricing as an additional software charge, not a replacement for those infrastructure costs.

The mechanism adds a second WebSocket connection alongside the session’s existing data connection. A model server opts in by returning X-Amzn-SageMaker-Metadata-Stream-Supported: true in the data WebSocket upgrade response. SageMaker AI then opens a companion metadata WebSocket, using /invocations-bidirectional-stream-metadata by default. Both connections carry the same request ID, allowing the server to associate metering records with the correct session. The underlying SageMaker container contract uses /invocations-bidirectional-stream as the default data path.

Providers report consumption during the session by sending one JSON record in each metadata WebSocket text frame. Every record uses the fixed inference.count dimension and a nonnegative ConsumedUnits value for usage since the previous record, rather than a running total. The provider defines one unit; AWS’s examples include a minute of speech-to-text audio or a block of text-to-speech characters. An optional client token makes duplicate records idempotent on the same connection.

AWS recommends reporting units incrementally, followed by a final record for any unreported usage when the data session ends. AWS limits each metadata frame to one record and no more than one record per second per connection. Bidirectional sessions can last up to 30 minutes. The companion path addresses a response-header constraint: once a streaming response begins, the count in an HTTP response header cannot be updated as more usage accrues.

If the metadata connection cannot open or drops, inference continues by default, but subsequent usage is not reported. Providers can instead set the metadata-stream Required header to true, making a failure stop the request or session. AWS named Deepgram as the launch partner and said its model server calculates metering automatically for usage-based speech-to-text and text-to-speech billing through AWS Marketplace.