Polars brings distributed queries to existing Ray clusters
Polars’ Ray integration runs the distributed engine behind Nanook across existing clusters, handling query planning, partitioning and shuffles.
Polars has introduced polars-nanook-ray, an integration that runs distributed Polars queries on Ray clusters that organizations already operate. It hosts the scheduler and workers for the distributed engine behind Polars Nanook as Ray actors.
The integration takes on orchestration work that would otherwise fall to developers wrapping Polars code in Ray tasks. Polars says a manual approach can be enough for a simple per-file map: split the input, mark the processing function with @ray.remote, and fan out tasks. But joins and group-bys that cross data partitions require developers to choose a partitioning scheme, repartition between stages, implement a shuffle and connect the tasks so records with matching keys reach the same partition.
With the Nanook engine running on Ray, Polars says it plans the query as a whole, partitions the data and performs the shuffles between stages. Ray is responsible for placing the scheduler and worker actors and keeping them alive. The published example retains a lazy Polars pipeline containing a join, filter, group-by, aggregation and sort, then sends it to a Ray-backed context with .remote(ctx) for execution rather than collecting it on one machine.
That division extends to scaling. Polars says its engine decides how many workers a query requests, while Ray decides where those actors run. Each worker requests configured CPU and memory resources, and the deployment uses Ray’s SPREAD strategy to place workers across nodes with available capacity. If the underlying Ray cluster has autoscaling enabled, Ray can add nodes when requested workers do not fit. Ray’s actor documentation confirms that actor placement considers declared resources and scheduling strategy, although it does not independently validate the integration’s end-to-end behavior.
The package requires Python 3.11 or later. Polars says it must be installed in the environment of every Ray node, using the same Ray version as the cluster; the integration repository says nodes running the scheduler and workers must be Linux hosts. The deployment requires a Nanook license file or a valid service account. Polars also says deployed clusters include a dashboard showing query operations, row flow and data shuffled between workers. The announcement does not include benchmark or cost-comparison results for the integration.
More news

Apache Airflow 3.4.0 adds bounded task loops and AI agent workflows

MotherDuck says its Azure service is live in two regions

ClickHouse launches managed observability service in public beta
