The latest Data Phoenix news tagged inference-infrastructure.
AWS has added NVIDIA Multi-Instance GPU partitioning to SageMaker real-time inference components, giving separately deployed models fixed slices of a…
IBM Research and Red Hat say llm-d combined cache-aware routing, distributed KV caching and split inference stages to serve GLM-5.2 on an existing…
AWS documentation describes prefix-aware routing for SageMaker real-time endpoints, including controls for prefix length, concurrency and…