spark.shuffle.manager
文章介绍了Spark中的三种ShuffleManager:SortShuffleManager、Celeborn的SparkShuffleManager和RssShuffleManager,并提供了相应的配置参数和文档链接。 2025-7-21 00:42:37 Author: dyrnq.com(查看原文) 阅读量:17 收藏

org.apache.spark.shuffle.sort.SortShuffleManager

(org.apache.spark.shuffle.sort.SortShuffleManager)[https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala]

org.apache.spark.shuffle.celeborn.SparkShuffleManager

--conf spark.shuffle.manager=org.apache.spark.shuffle.celeborn.SparkShuffleManager \
--conf spark.shuffle.service.enabled=false

org.apache.spark.shuffle.celeborn.SparkShuffleManager

org.apache.spark.shuffle.RssShuffleManager

spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
spark.rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999

https://uniffle.apache.org/docs/intro


文章来源: https://dyrnq.com/spark-shuffle-manager/
如有侵权请联系:admin#unsafe.sh