Together AI launches Storage Accelerator for GPU clusters
The distributed read cache uses storage on GPU nodes to cut repeat downloads of model weights, checkpoints and training data. Together AI says it comes at no additional cost.
Together AI launched Storage Accelerator on November 19. The distributed read cache is available at no additional cost with Together GPU Clusters. Together’s documentation says the feature needs to be enabled for the customer’s organization. The cache targets a bottleneck that can leave GPUs waiting for data when workers repeatedly download model weights, checkpoints or training datasets from S3-compatible object storage or Hugging Face Hub.
The cache runs on each GPU node and splits files into blocks stored on NVMe drives across the cluster. The first request for uncached data fetches it from the source. Later requests can read cached blocks from the same node or from peers. When several workers request the same uncached file at once, the cache is designed to combine their requests into one fetch.
Customers configure existing S3-compatible or Hugging Face clients to use an endpoint inside the cluster. The service supports up to three S3-compatible backends and one Hugging Face Hub token per cluster. Ordinary S3 writes pass through to the source. The endpoint does not support multipart uploads.
In a company-run preview test using single-stream transfers on one H100 node, Together AI reported reading a 2 GiB object at 3.0–3.4 GB/s from a warm cache, about 20 times the 143–169 MB/s it measured directly from Cloudflare R2. Together reported warm-cache read times of 5 ms for a 5 KB object and 8 ms for a 2.4 MB object. The preview did not measure reads from peer nodes.
Together illustrated how the cache could cut egress. In the company’s example, a 200-node cluster fetching the same checkpoint would make one download from remote storage with the cache, compared with 200 without it. The launch post and documentation give no measured reduction in customer egress bills.
The cache is designed for files that stay the same. Together’s documentation says that if a file is overwritten under the same key, cached blocks can persist until eviction. A read that spans the change can return a mix of old and new data.
More news

AWS adds distributed KV caching to ElastiCache for self-hosted LLMs

NVIDIA updates DAQIRI for real-time scientific instruments

AWS turns on continuous SageMaker inference monitoring by default
