Together AI makes automatic GPU-cluster repair controls generally available

Together AI has made configurable automatic repair controls generally available for Together GPU Clusters, letting operators decide which GPU, host and fabric faults the platform can remediate without human approval. The controls add per-fault and per-node consent, wait-for-idle settings and safety limits for actions that can interrupt workloads or erase local data.

The update expands the human-approved node-repair workflow Together AI described in July. At that time, the system detected a node-level problem and recommended an action, but an operator had to approve the repair. Together AI said it was still developing fully automatic options for specific failure modes.

In its October 27 general-availability announcement, Together AI said operators can put a cluster in fully automatic mode while selecting the fault classes that may run unattended. Faults without consent continue to wait for approval, and operators can exclude an individual node from automatic repair. Consent can also be set for individual GPU Xid codes. NVIDIA defines an Xid message as a driver error report written to an operating-system kernel or event log and cautions that the code is a debugging guide, not proof of a single root cause.

The platform combines GPU and operating-system signals collected inside a guest virtual machine with host-side checks for fabric and reachability faults. Together AI says every repair recommendation requires a failed health check and includes supporting evidence. Signals without a mapped action may generate an alert but do not trigger a repair. The company’s health-check documentation says continuous passive checks can generate node-repair recommendations automatically.

Operators can hold an approved automatic repair until a Kubernetes node has no GPU-requesting pod or a Slurm node has no allocation. The announcement describes a two-hour default maximum wait, a 24-hour cap and an option to wait indefinitely. Together AI’s current node-repair documentation adds that a new cluster’s starting policy depends on account settings. It also warns that a cluster starting in fully automatic mode without saved wait settings may interrupt work as soon as the node is cordoned.

Automatic repairs can progress from reboot to quick reprovision and migration to a new host. Reprovisioning and migration destroy data stored locally in the virtual machine, while removing a node for return merchandise authorization remains a human-initiated action. For a recurring fault, the system can move to a heavier action after the same repair has been used twice on the same host. The documentation warns that an automatically handled reboot fault can therefore escalate to a destructive reprovision or migration without another review.

Together AI says its safeguards include signal-persistence thresholds, an approximately 30-minute post-repair cooldown, cluster-sized concurrency limits, one active repair per node pool and a circuit breaker for bursts of correlated failures. Migration also requires current or recent evidence of a critical alert. The company says customers and its internal inference platform have used the capability to improve resiliency and reduce downtime, but it published no customer count, recovery-time data or measured downtime reduction. Node-label remediation contracts for customer-supplied detectors remain in preview.