Amazon SageMaker AI adds usage-based metering for bidirectional streaming

Until now, AWS Marketplace software metering for SageMaker AI real-time inference relied on a unit count sent in an HTTP response header, which does not work for streams where the total is known only when the session ends. The new approach adds a second WebSocket, a metadata connection, alongside the inference stream. The model server opts in with the X-Amzn-SageMaker-Metadata-Stream-Supported header, and SageMaker AI opens the companion connection at /invocations-bidirectional-stream-metadata, with both connections sharing the X-Amzn-SageMaker-Request-Id header.

The model server sends one JSON metering record per text frame, with a ConsumedUnits value and an optional ClientToken for idempotency. Records should be sent incrementally as usage accrues, and the metadata stream is limited to one message per second per connection and 512 bytes per message. A setting called X-Amzn-SageMaker-Metadata-Stream-Required controls whether inference stops when the metadata connection fails; it defaults to false.

SageMaker AI endpoint infrastructure charges still apply. Model providers set a per-unit price for real-time inference on their AWS Marketplace listing, and the unit definition is up to the provider.

Model providers building streaming products can now bill per unit consumed during a session rather than through hourly pricing, but they must define the unit and validate reported records against AWS Marketplace reports.

Agent developers deploying real-time speech models will see usage flow into their AWS bill, while endpoint infrastructure charges continue separately. Teams running their own model servers should plan for the metadata connection and decide whether inference should stop if it fails.