I turned on the NVIDIA RAPIDS Accelerator for a set of Spark SQL and PySpark jobs on Google Kubernetes Engine and expected those jobs to finish several times faster. The Qualification Tool agreed. It placed several workloads in the Large speedup category and estimated a 4–5× GPU speedup.

I then compared end-to-end wall-clock time with the same jobs on CPU. Some of the RAPIDS runs finished slower.

This is the gap I care about: what that tool is scoring, where the extra minutes went on GKE, and the three rules I kept after the benchmarks. The manifests, the qualification wrapper, and the full configuration notes live in here. Copy the YAML from there. The rest of this piece is why I stopped treating GPU Spark as a cluster-wide default.

The stack I actually ran

The deployment was Apache Spark 3.5.5 with RAPIDS Accelerator 25.12.0 on CUDA 12. Executors landed on g2-standard-32 nodes, each with one NVIDIA L4. The Spark Operator submitted SparkApplicationmanifests. The driver stayed on CPU nodes. Each executor requested one GPU.

That is an ordinary production shape for this plugin. Spark RAPIDS is a Spark SQL plugin. It loads through spark.plugins=com.nvidia.spark.SQLPluginand moves supported operators — joins, aggregations, sorts, Parquet reads — onto the GPU through cuDF. Application code stays as it is. Unsupported operators, many UDFs, and some file formats remain on the CPU.

I built the executor image in three layers: a CPU Spark base, a CUDA layer, then a workload image that copies the RAPIDS JAR and NVIDIA’s GPU discovery script to/opt/sparkRapidsPlugin/. Spark refuses to schedule a GPU resource when that script is missing. A second copy of the JAR on the classpath causes its own failures, so the image keeps a single copy.

The image build, the node labels, and the minimum Spark properties are written up in here ,including a minimal manifest and a full one with GPU affinity and tolerations. The plugin did load.

Driver logs showed:

INFO RapidsPluginUtils: RAPIDS Accelerator 25.12.0 using cudf 25.12.0INFO DriverPluginContainer: Initialized driver component for plugin com.nvidia.spark.SQLPlugin

The surprise was the clock.

NVIDIA’s Qualification Tool reads Spark event logs and estimates how much of a job can move to the GPU. I enabled event logging to a GCS prefix and ran the wrapper from the repo: bash ./scripts/qualify.sh gs://<bucket>/spark-events/spark-<app-id>

The fields I read were the estimated GPU speedup, the speedup category, and the share of stage time spent in unsupported operators. A Large category with a small unsupported share looks like a green light. The tool also writes combined.conf, a Spark config aimed at that application. I used that file as the starting point for tuning.

On several production jobs the estimate landed around 4–5×, and unsupported operators accounted for a modest slice of stage duration. Stop at that CSV and the GPU looks like an obvious win.

The tool is scoring SQL stage work it believes the GPU can run. It is a planning aid. It does not include the time GKE spends placing an L4, and it does not include the minutes an executor spends Pending. My production number was submit-to-done for the same workload, next to a CPU baseline. Those two clocks measure different intervals. Both were honest. They did not match.

Where the time went

Four costs showed up once I recorded more than stage duration.

GPU nodes take time to appear. g2-standard-32 capacity is regional, and a cold node is part of the job. On-demand pools were steadier than spot for work that cared about latency. Dynamic allocation still had to stay inside the number of GPUs the pool could give me. When spark.dynamicAllocation.maxExecutorsasked for more GPUs than the region had, executors sat Pending and the wall clock grew while the SQL plan looked fine.

Scheduling swallowed the speedup. A stage the tool expects to shrink by several times can still lose if the cluster spends minutes placing executor pods. I started writing down Pending time beside SQL time. On the runs that lost to CPU, the SQL portion often looked better. The placement portion did not.

Unsupported work pulls the job back to the CPU. Text output, some Iceberg versions, Cartesian-style joins, and PySpark UDFs stay on the CPU. With spark.rapids.sql.explain=ALL I could see which operators fell back. I set that property back to NONEafter I understood a job, because the explain output is for diagnosis. A Large estimate can still sit next to enough CPU fallback, and a skewed shuffle, that the GPU run loses on the clock.

**One GPU per executor leaves cores idle until you share it.**On a 16-core executor, Spark’s default is one runnable task per GPU. The log warns that the remaining cores will sit unused. The change that moved utilization was small:

spark.task.resource.gpu.amount=0.001,spark.rapids.sql.concurrentGpuTasks=3

Several tasks then share the L4. I would set those two properties on every GPU executor, next to a batch size the job can hold in memory. They do nothing for a job whose time is dominated by node startup.

GPU out-of-memory during joins was the other failure I hit in practice. Lowering spark.rapids.sql.batchSizeBytes, easing concurrentGpuTasks, or repartitioning a skewed key cleared it. Those fixes apply after you have already decided the job belongs on a GPU.

The three rules I kept

I stopped asking whether the cluster should run RAPIDS. I started asking whether the workload should.

Qualify before the plugin is enabled on a job. Capture event logs, then run scripts/qualify.sh. A Medium or Not Recommended category means I leave the job on CPU. A Large estimate is permission to run a wall-clock trial.

**Measure submit-to-done.**Include node provisioning and Pending executors, and compare that time with a CPU run of the same job. Stage duration inside the GPU plan is a lab figure. The figure that decides the rollout is whether the job returned sooner.

Leave RAPIDS opt-in. I did not enable the plugin on every SparkApplication. Shared settings — executor size, batch size, concurrent GPU tasks — traveled better across similar jobs than a hand-tuned file per application id. Qualification output from one representative job in a class was enough to seed that shared config. Heavy supported SQL, large scans, and shuffle-heavy joins stayed candidates. Jobs built around unsupported operators stayed on CPU.
The manifests that encode this split are in the repo: a minimal SparkApplication, a production-style manifest, and a full example with GPU affinity and tolerations, all underexamples/. The qualification helper and the log-fetch scripts are under scripts/. Copy those files. They already set the classpath, the discovery-script path, and the Kryo registrator.

What I would repeat on a new cluster

I would build the CUDA image again, pin executors to L4 nodes, and leave the driver on CPU. I would want this line in the driver log before I trusted a run:

INFO RapidsPluginUtils: RAPIDS Accelerator 25.12.0 using cudf 25.12.0

I would qualify a CPU event log before spending GPU quota. I would compare wall-clock time on two or three representative jobs before I touched the rest of the pipeline. I would read a 4–5× estimate as a hypothesis that still has to survive the scheduler.

Spark RAPIDS accelerates the SQL it supports. On GKE that accelerator sits behind a node pool, a device plugin, and a regional GPU quota. Leave those out of the measurement and the qualification CSV will keep promising a speedup the job does not deliver.

The configuration, the example manifests, and the longer enablement guide are in here That is the repo I wish I had opened on day one.