Package org.apache.sysds.runtime.controlprogram.context

Class SparkExecutionContext.SparkClusterConfig

java.lang.Object
org.apache.sysds.runtime.controlprogram.context.SparkExecutionContext.SparkClusterConfig
Enclosing class:
SparkExecutionContext
public static class SparkExecutionContext.SparkClusterConfig extends Object
Captures relevant spark cluster configuration properties, e.g., memory budgets and degree of parallelism. This configuration abstracts legacy (< Spark 1.6) and current configurations and provides a unified view.

  • Field Details

    • RESERVED_SYSTEM_MEMORY_BYTES

      public static final long RESERVED_SYSTEM_MEMORY_BYTES
      See Also:

  • Constructor Details

    • SparkClusterConfig

      public SparkClusterConfig()

    • SparkClusterConfig

      public SparkClusterConfig(org.apache.spark.SparkConf sconf)

  • Method Details

    • getBroadcastMemoryBudget

      public long getBroadcastMemoryBudget()

    • getDataMemoryBudget

      public long getDataMemoryBudget(boolean min, boolean refresh)

    • getNumExecutors

      public int getNumExecutors()

    • getDefaultParallelism

      public int getDefaultParallelism(boolean refresh)

    • analyzeSparkConfiguationLegacy

      public void analyzeSparkConfiguationLegacy(org.apache.spark.SparkConf conf)

    • analyzeSparkConfiguation

      public void analyzeSparkConfiguation(org.apache.spark.SparkConf conf)

    • toString

      public String toString()
      Overrides:
      toString in class Object