OpenAI returns Habitat storage to shared Cosmos DB containers

OpenAI has returned its Habitat storage platform to shared Azure Cosmos DB containers, making multi-tenancy the default again after dedicated containers became difficult to manage at scale. The company said selective replication, in-region caching and a Rust rewrite cut global median read latency from about 120 milliseconds to about 10 milliseconds.

Habitat supports ChatGPT, the API, Codex and internal services. OpenAI said it responds in milliseconds for more than 1 billion users across 40 regions while managing more than 500 petabytes of data. The scale and performance figures come from OpenAI.

OpenAI had moved from one shared container to dedicated, single-tenant containers to isolate workloads. It said the dedicated model made capacity planning unmanageable as the number of internal tenants grew. OpenAI reported no provisioning or noisy-neighbor incidents in the months after reverting to the shared design, though that operating claim has not been independently audited.

To limit interference within a shared container, Habitat throttles the hottest keys first and uses priority tiers so offline work is restricted before interactive traffic. OpenAI also described a co-engineered serverless mode that provides 10,000 request units per second for each physical partition and splits partitions according to use. Microsoft’s partitioning documentation says provisioned throughput is divided across physical partitions and a hot logical partition can trigger rate limits. Its priority-execution documentation describes the feature as best effort and says the public version is not supported on serverless accounts, making OpenAI’s combined implementation distinct from the standard public configuration.

OpenAI said its largest Cosmos DB account stores tens of petabytes and handles more than 20 million queries per second. Microsoft has separately said OpenAI operates Cosmos DB across petabytes of data and millions of queries per second, but it did not confirm the exact account-level figures.

Rather than replicate every large dataset worldwide, Habitat reserves broad replication for small, latency-sensitive records such as authentication metadata. OpenAI said some accounts replicate to all 40 Azure regions it serves, while others keep at least one copy on each continent. The company said storage, power, cooling and regional capacity limit replication of larger datasets.

For data that is not replicated globally, Habitat places a distributed Valkey lookaside cache near request-serving systems. A lookaside cache checks a nearby copy first, then retrieves the record from the database after a miss. Habitat consumes the Azure Cosmos DB change feed to invalidate cached copies worldwide within seconds after underlying data changes, OpenAI said. Microsoft documents the change feed as an asynchronous, incremental record that applications can process in parallel across regions, but does not verify OpenAI’s seconds-level result.

OpenAI attributed the reported 12-fold reduction in median read latency to selective replication, in-region caching and the Rust service described in Part I. The post does not disclose its benchmark method, time series or product-level distribution.

OpenAI’s retrospective links the architecture to two failover problems. The company said a late-2024 power outage exposed multi-hour failover times for a large Cosmos DB account. It said a failback about a month later caused a connection stampede that overwhelmed the original region’s gateway. OpenAI’s contemporaneous incident report confirmed a data-center power failure, database impact and error rates above 90% for several products, but did not identify Cosmos DB or document the later failback.

OpenAI said it now co-operates Cosmos DB with Azure through shared tests and dashboards, direct engineering channels, coordinated monitoring, SDK work and defined escalation paths. According to the company, Habitat’s change-data-capture system processes millions of changes per second across thousands of containers and millions of partitions, feeding more than 100 Kafka topics used for cache invalidation, secondary indexes, data movement, analytics and other real-time functions.