Package org.apache.sysds.runtime.matrix.data

Class LibMatrixAgg

java.lang.Object

org.apache.sysds.runtime.matrix.data.LibMatrixAgg

public class LibMatrixAgg
extends Object

MB: Library for matrix aggregations including ak+, uak+ for all combinations of dense and sparse representations, and corrections. Those are performance-critical operations because they are used on combiners/reducers of important operations like tsmm, mvmult, indexing, but also basic sum/min/max/mean, row*, col*, etc. Specific handling is especially required for all non sparse-safe operations in order to prevent unnecessary worse asymptotic behavior. This library currently covers the following opcodes: ak+, uak+, uark+, uack+, uasqk+, uarsqk+, uacsqk+, uamin, uarmin, uacmin, uamax, uarmax, uacmax, ua*, uamean, uarmean, uacmean, uavar, uarvar, uacvar, uarimax, uaktrace, cumk+, cummin, cummax, cum*, tak+, cm, cov TODO next opcode extensions: a+, colindexmax

Field Summary

Fields

Modifier and Type	Field	Description
static final long	PAR_NUMCELL_THRESHOLD1

Method Summary

Modifier and Type	Method	Description
static void	aggregateBinaryMatrix(MatrixBlock in, MatrixBlock aggVal, MatrixBlock aggCorr, boolean deep)	Core incremental matrix aggregate (ak+) as used in mapmult, tsmm, cpmm, etc.
static void	aggregateBinaryMatrix(MatrixBlock in, MatrixBlock aggVal, AggregateOperator aop)	Core incremental matrix aggregate (ak+) as used for uack+ and acrk+.
static CmCovObject	aggregateCmCov(MatrixBlock in1, MatrixBlock in2, MatrixBlock in3, ValueFunction fn)	Single threaded Covariance and Central Moment operations CM = Central Moment COV = Covariance
static CmCovObject	aggregateCmCov(MatrixBlock in1, MatrixBlock in2, MatrixBlock in3, ValueFunction fn, int k)	Multi threaded Covariance and Central Moment operations CM = Central Moment COV = Covariance
static MatrixBlock	aggregateTernary(MatrixBlock in1, MatrixBlock in2, MatrixBlock in3, MatrixBlock ret, AggregateTernaryOperator op)
static MatrixBlock	aggregateTernary(MatrixBlock in1, MatrixBlock in2, MatrixBlock in3, MatrixBlock ret, AggregateTernaryOperator op, int k)
static void	aggregateUnaryMatrix(MatrixBlock in, MatrixBlock out, AggregateUnaryOperator uaop)
static void	aggregateUnaryMatrix(MatrixBlock in, MatrixBlock out, AggregateUnaryOperator uaop, boolean allowReformatToSparse)
static void	aggregateUnaryMatrix(MatrixBlock in, MatrixBlock out, AggregateUnaryOperator uaop, int k)
static MatrixBlock	aggregateUnaryMatrix(AggregateUnaryOperator op, MatrixBlock in, MatrixValue result, int blen, MatrixIndexes indexesIn, boolean inCP)
static void	countAgg(double[] a, int[] c, int[] aix, int ai, int len)
static void	countAgg(double[] a, int[] c, int ai, int len)
static MatrixBlock	cumaggregateUnaryMatrix(MatrixBlock in, MatrixBlock out, UnaryOperator uop)
static MatrixBlock	cumaggregateUnaryMatrix(MatrixBlock in, MatrixBlock out, UnaryOperator uop, double[] agg)
static MatrixBlock	cumaggregateUnaryMatrix(MatrixBlock in, MatrixBlock out, UnaryOperator uop, int k)
static void	groupedAggregate(MatrixBlock groups, MatrixBlock target, MatrixBlock weights, MatrixBlock result, int numGroups, Operator op)
static void	groupedAggregate(MatrixBlock groups, MatrixBlock target, MatrixBlock weights, MatrixBlock result, int numGroups, Operator op, int k)
static boolean	isSupportedUnaryAggregateOperator(AggregateUnaryOperator op)
static boolean	isSupportedUnaryOperator(UnaryOperator op)
static MatrixBlock	prepareAggregateUnaryOutput(MatrixBlock in, AggregateUnaryOperator op, MatrixValue result, int blen)
static void	recomputeIndexes(MatrixBlock out, AggregateUnaryOperator op, int blen, MatrixIndexes ix)	Recompute outputs (e.g., maxindex or minindex) according to block indexes from MR.
static boolean	satisfiesMultiThreadingConstraints(MatrixBlock in, int k)
static boolean	satisfiesMultiThreadingConstraints(MatrixBlock in, MatrixBlock out, AggregateUnaryOperator uaop, int k)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

PAR_NUMCELL_THRESHOLD1

public static final long PAR_NUMCELL_THRESHOLD1

See Also:

• Constant Field Values

Method Details

aggregateBinaryMatrix

public static void aggregateBinaryMatrix(MatrixBlock in,
 MatrixBlock aggVal,
 MatrixBlock aggCorr,
 boolean deep)

Core incremental matrix aggregate (ak+) as used in mapmult, tsmm, cpmm, etc. Note that we try to keep the current aggVal and aggCorr in dense format in order to allow efficient access according to the dense/sparse input.

Parameters:

in - input matrix

aggVal - current aggregate values (in/out)

aggCorr - current aggregate correction (in/out)

deep - deep copy flag

aggregateBinaryMatrix

public static void aggregateBinaryMatrix(MatrixBlock in,
 MatrixBlock aggVal,
 AggregateOperator aop)

Core incremental matrix aggregate (ak+) as used for uack+ and acrk+. Embedded correction values. DOES NOT EVALUATE SPARSITY SINCE IT IS USED IN INCREMENTAL AGGREGATION

Parameters:

in - matrix block

aggVal - aggregate operator

aop - aggregate operator

aggregateUnaryMatrix

public static MatrixBlock aggregateUnaryMatrix(AggregateUnaryOperator op,
 MatrixBlock in,
 MatrixValue result,
 int blen,
 MatrixIndexes indexesIn,
 boolean inCP)

aggregateUnaryMatrix

public static void aggregateUnaryMatrix(MatrixBlock in,
 MatrixBlock out,
 AggregateUnaryOperator uaop)

aggregateUnaryMatrix

public static void aggregateUnaryMatrix(MatrixBlock in,
 MatrixBlock out,
 AggregateUnaryOperator uaop,
 boolean allowReformatToSparse)

aggregateUnaryMatrix

public static void aggregateUnaryMatrix(MatrixBlock in,
 MatrixBlock out,
 AggregateUnaryOperator uaop,
 int k)

cumaggregateUnaryMatrix

public static MatrixBlock cumaggregateUnaryMatrix(MatrixBlock in,
 MatrixBlock out,
 UnaryOperator uop)

cumaggregateUnaryMatrix

public static MatrixBlock cumaggregateUnaryMatrix(MatrixBlock in,
 MatrixBlock out,
 UnaryOperator uop,
 double[] agg)

cumaggregateUnaryMatrix

public static MatrixBlock cumaggregateUnaryMatrix(MatrixBlock in,
 MatrixBlock out,
 UnaryOperator uop,
 int k)

aggregateCmCov

public static CmCovObject aggregateCmCov(MatrixBlock in1,
 MatrixBlock in2,
 MatrixBlock in3,
 ValueFunction fn)

Single threaded Covariance and Central Moment operations CM = Central Moment COV = Covariance

Parameters:

in1 - Main input matrix

in2 - Second input matrix

in3 - Third input matrix (not output since output is returned)

fn - Value function to apply

Returns:

Central Moment or Covariance object

aggregateCmCov

public static CmCovObject aggregateCmCov(MatrixBlock in1,
 MatrixBlock in2,
 MatrixBlock in3,
 ValueFunction fn,
 int k)

Multi threaded Covariance and Central Moment operations CM = Central Moment COV = Covariance

Parameters:

in1 - Main input matrix

in2 - Second input matrix

in3 - Third input matrix (not output since output is returned)

fn - Value function to apply

k - Parallelization degree

Returns:

Central Moment or Covariance object

aggregateTernary

public static MatrixBlock aggregateTernary(MatrixBlock in1,
 MatrixBlock in2,
 MatrixBlock in3,
 MatrixBlock ret,
 AggregateTernaryOperator op)

aggregateTernary

public static MatrixBlock aggregateTernary(MatrixBlock in1,
 MatrixBlock in2,
 MatrixBlock in3,
 MatrixBlock ret,
 AggregateTernaryOperator op,
 int k)

groupedAggregate

public static void groupedAggregate(MatrixBlock groups,
 MatrixBlock target,
 MatrixBlock weights,
 MatrixBlock result,
 int numGroups,
 Operator op)

groupedAggregate

public static void groupedAggregate(MatrixBlock groups,
 MatrixBlock target,
 MatrixBlock weights,
 MatrixBlock result,
 int numGroups,
 Operator op,
 int k)

isSupportedUnaryAggregateOperator

public static boolean isSupportedUnaryAggregateOperator(AggregateUnaryOperator op)

isSupportedUnaryOperator

public static boolean isSupportedUnaryOperator(UnaryOperator op)

satisfiesMultiThreadingConstraints

public static boolean satisfiesMultiThreadingConstraints(MatrixBlock in,
 MatrixBlock out,
 AggregateUnaryOperator uaop,
 int k)

satisfiesMultiThreadingConstraints

public static boolean satisfiesMultiThreadingConstraints(MatrixBlock in,
 int k)

recomputeIndexes

public static void recomputeIndexes(MatrixBlock out,
 AggregateUnaryOperator op,
 int blen,
 MatrixIndexes ix)

Recompute outputs (e.g., maxindex or minindex) according to block indexes from MR. TODO: this should not be part of block operations but of the MR instruction.

Parameters:

out - matrix block

op - aggregate unary operator

blen - number of rows/cols in a block

ix - matrix indexes

countAgg

public static void countAgg(double[] a,
 int[] c,
 int[] aix,
 int ai,
 int len)

countAgg

public static void countAgg(double[] a,
 int[] c,
 int ai,
 int len)

prepareAggregateUnaryOutput

public static MatrixBlock prepareAggregateUnaryOutput(MatrixBlock in,
 AggregateUnaryOperator op,
 MatrixValue result,
 int blen)