Package org.apache.sysds.api.jmlc

Class Connection

java.lang.Object
org.apache.sysds.api.jmlc.Connection
All Implemented Interfaces:
Closeable, AutoCloseable
public class Connection extends Object implements Closeable
Interaction with SystemDS using the JMLC (Java Machine Learning Connector) API is initiated with a Connection object. The JMLC API is designed after JDBC. A DML script is precompiled by calling the prepareScript(String, String[], String[]) method or the prepareScript(String, Map, String[], String[]) method on the Connection object, which returns a PreparedScript object. Note that this is similar to calling a prepareStatement method on a JDBC Connection object.

Following this, input variable data is passed to the script by calling the setFrame, setMatrix, and setScalar methods of the PreparedScript object. The script is executed via PreparedScript's executeScript method, which returns a ResultVariables object, which is similar to a JDBC ResultSet. Data can be read from a ResultVariables object by calling its getFrame and getMatrix methods.

  • Constructor Details

    • Connection

      public Connection()
      Connection constructor, the starting point for any other JMLC API calls.

    • Connection

      public Connection(CompilerConfig.ConfigType... cconfigs)
      Connection constructor, the starting point for any other JMLC API calls. This variant allows to enable a set of boolean compiler configurations.
      Parameters:
      cconfigs - one or many boolean compiler configurations to enable.

    • Connection

      public Connection(DMLConfig dmlconfig, CompilerConfig.ConfigType... cconfigs)
      Connection constructor, the starting point for any other JMLC API calls. This variant allows to pass a global dml configuration and enable a set of boolean compiler configurations.
      Parameters:
      dmlconfig - a dml configuration.
      cconfigs - one or many boolean compiler configurations to enable.

    • Connection

      public Connection(DMLConfig dmlconfig)
      Connection constructor, the starting point for any other JMLC API calls. This variant allows to pass a global dml configuration.
      Parameters:
      dmlconfig - a dml configuration.

  • Method Details

    • setConfigTypes

      public void setConfigTypes(boolean activate, CompilerConfig.ConfigType... cconfigs)
      Sets compiler configs.
      Parameters:
      activate - activate or disable
      cconfigs - the configs to set

    • setStatistics

      public void setStatistics(boolean stats)
      Sets a boolean flag indicating if runtime statistics should be gathered Same behavior as in "MLContext.setStatistics()"
      Parameters:
      stats - boolean value with true indicating statistics should be gathered

    • setLineage

      public void setLineage(boolean lt)
      Sets a boolean flag indicating if lineage trace should be captured
      Parameters:
      lt - boolean value with true indicating lineage should be captured

    • gatherMemStats

      public void gatherMemStats(boolean stats)
      Sets a boolean flag indicating if memory profiling statistics should be gathered. The option is false by default.
      Parameters:
      stats - boolean value with true indicating memory statistics should be gathered

    • prepareScript

      public PreparedScript prepareScript(String script, String[] inputs, String[] outputs)
      Prepares (precompiles) a script and registers input and output variables.
      Parameters:
      script - string representing the DML or PyDML script
      inputs - string array of input variables to register
      outputs - string array of output variables to register
      Returns:
      PreparedScript object representing the precompiled script

    • prepareScript

      public PreparedScript prepareScript(String script, Map<String,String> args, String[] inputs, String[] outputs)
      Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.
      Parameters:
      script - string representing the DML or PyDML script
      args - map of input parameters ($) and their values
      inputs - string array of input variables to register
      outputs - string array of output variables to register
      Returns:
      PreparedScript object representing the precompiled script

    • prepareScript

      public PreparedScript prepareScript(String script, Map<String,String> nsscripts, Map<String,String> args, String[] inputs, String[] outputs)
      Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.
      Parameters:
      script - string representing of the DML or PyDML script
      nsscripts - map (name, script) of the DML or PyDML namespace scripts
      args - map of input parameters ($) and their values
      inputs - string array of input variables to register
      outputs - string array of output variables to register
      Returns:
      PreparedScript object representing the precompiled script

    • close

      public void close()
      Close connection to SystemDS, which clears the thread-local DML and compiler configurations.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • readScript

      public String readScript(String fname) throws IOException
      Read a DML or PyDML file as a string.
      Parameters:
      fname - the filename of the script
      Returns:
      string content of the script file
      Throws:
      IOException - if IOException occurs

    • readDoubleMatrix

      public double[][] readDoubleMatrix(String fname) throws IOException
      Reads an input matrix in arbitrary format from HDFS into a dense double array. NOTE: this call currently only supports default configurations for CSV.
      Parameters:
      fname - the filename of the input matrix
      Returns:
      matrix as a two-dimensional double array
      Throws:
      IOException - if IOException occurs

    • readDoubleMatrix

      public double[][] readDoubleMatrix(String fname, Types.FileFormat fmt, long rows, long cols, int blen, long nnz) throws IOException
      Reads an input matrix in arbitrary format from HDFS into a dense double array. NOTE: this call currently only supports default configurations for CSV.
      Parameters:
      fname - the filename of the input matrix
      fmt - file format
      rows - number of rows in the matrix
      cols - number of columns in the matrix
      blen - block length
      nnz - number of non-zero values, -1 indicates unknown
      Returns:
      matrix as a two-dimensional double array
      Throws:
      IOException - if IOException occurs

    • convertToDoubleMatrix

      public double[][] convertToDoubleMatrix(String input, String meta) throws IOException
      Converts an input string representation of a matrix in csv or textcell format into a dense double array. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
      Parameters:
      input - string matrix in csv or textcell format
      meta - string representing SystemDS matrix metadata in JSON format
      Returns:
      matrix as a two-dimensional double array
      Throws:
      IOException - if IOException occurs

    • convertToDoubleMatrix

      public double[][] convertToDoubleMatrix(String input, int rows, int cols) throws IOException
      Converts an input string representation of a matrix in textcell format into a dense double array.
      Parameters:
      input - string matrix in textcell format
      rows - number of rows in the matrix
      cols - number of columns in the matrix
      Returns:
      matrix as a two-dimensional double array
      Throws:
      IOException - if IOException occurs

    • convertToDoubleMatrix

      public double[][] convertToDoubleMatrix(InputStream input, int rows, int cols) throws IOException
      Converts an input stream of a string matrix in textcell format into a dense double array.
      Parameters:
      input - InputStream to a string matrix in textcell format
      rows - number of rows in the matrix
      cols - number of columns in the matrix
      Returns:
      matrix as a two-dimensional double array
      Throws:
      IOException - if IOException occurs

    • convertToDoubleMatrix

      public double[][] convertToDoubleMatrix(InputStream input, int rows, int cols, String format) throws IOException
      Converts an input stream of a string matrix in csv or textcell format into a dense double array.
      Parameters:
      input - InputStream to a string matrix in csv or textcell format
      rows - number of rows in the matrix
      cols - number of columns in the matrix
      format - input format of the given stream
      Returns:
      matrix as a two-dimensional double array
      Throws:
      IOException - if IOException occurs

    • convertToMatrix

      public MatrixBlock convertToMatrix(String input, String meta) throws IOException
      Converts an input string representation of a matrix in csv or textcell format into a matrix block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
      Parameters:
      input - string matrix in csv or textcell format
      meta - string representing SystemDS matrix metadata in JSON format
      Returns:
      matrix as a matrix block
      Throws:
      IOException - if IOException occurs

    • convertToMatrix

      public MatrixBlock convertToMatrix(InputStream input, String meta) throws IOException
      Converts an input stream of a string matrix in csv or textcell format into a matrix block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
      Parameters:
      input - InputStream to a string matrix in csv or textcell format
      meta - string representing SystemDS matrix metadata in JSON format
      Returns:
      matrix as a matrix block
      Throws:
      IOException - if IOException occurs

    • convertToMatrix

      public MatrixBlock convertToMatrix(String input, int rows, int cols) throws IOException
      Converts an input string representation of a matrix in textcell format into a matrix block.
      Parameters:
      input - string matrix in textcell format
      rows - number of rows in the matrix
      cols - number of columns in the matrix
      Returns:
      matrix as a matrix block
      Throws:
      IOException - if IOException occurs

    • convertToMatrix

      public MatrixBlock convertToMatrix(InputStream input, int rows, int cols) throws IOException
      Converts an input stream of a string matrix in text format into a matrix block.
      Parameters:
      input - InputStream to a string matrix in text format
      rows - number of rows in the matrix
      cols - number of columns in the matrix
      Returns:
      matrix as a matrix block
      Throws:
      IOException - if IOException occurs

    • convertToMatrix

      public MatrixBlock convertToMatrix(InputStream input, int rows, int cols, String format) throws IOException
      Converts an input stream of a string matrix in csv or text format into a matrix block.
      Parameters:
      input - InputStream to a string matrix in csv or text format
      rows - number of rows in the matrix
      cols - number of columns in the matrix
      format - input format of the given stream
      Returns:
      matrix as a matrix block
      Throws:
      IOException - if IOException occurs

    • readStringFrame

      public String[][] readStringFrame(String fname) throws IOException
      Reads an input frame in arbitrary format from HDFS into a dense string array. NOTE: this call currently only supports default configurations for CSV.
      Parameters:
      fname - the filename of the input frame
      Returns:
      frame as a two-dimensional string array
      Throws:
      IOException - if IOException occurs

    • readStringFrame

      public String[][] readStringFrame(String fname, Types.FileFormat fmt, long rows, long cols) throws IOException
      Reads an input frame in arbitrary format from HDFS into a dense string array. NOTE: this call currently only supports default configurations for CSV.
      Parameters:
      fname - the filename of the input frame
      fmt - file format type
      rows - number of rows in the frame
      cols - number of columns in the frame
      Returns:
      frame as a two-dimensional string array
      Throws:
      IOException - if IOException occurs

    • convertToStringFrame

      public String[][] convertToStringFrame(String input, String meta) throws IOException
      Converts an input string representation of a frame in csv or textcell format into a dense string array. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
      Parameters:
      input - string frame in csv or textcell format
      meta - string representing SystemDS frame metadata in JSON format
      Returns:
      frame as a two-dimensional string array
      Throws:
      IOException - if IOException occurs

    • convertToStringFrame

      public String[][] convertToStringFrame(String input, int rows, int cols) throws IOException
      Converts an input stream of a string frame in textcell format into a dense string array.
      Parameters:
      input - string frame in textcell format
      rows - number of rows in the frame
      cols - number of columns in the frame
      Returns:
      frame as a two-dimensional string array
      Throws:
      IOException - if IOException occurs

    • convertToStringFrame

      public String[][] convertToStringFrame(InputStream input, int rows, int cols) throws IOException
      Converts an input stream of a string frame in textcell format into a dense string array.
      Parameters:
      input - InputStream to a string frame in text format
      rows - number of rows in the frame
      cols - number of columns in the frame
      Returns:
      frame as a two-dimensional string array
      Throws:
      IOException - if IOException occurs

    • convertToStringFrame

      public String[][] convertToStringFrame(InputStream input, int rows, int cols, String format) throws IOException
      Converts an input stream of a string frame in csv or text format into a dense string array.
      Parameters:
      input - InputStream to a string frame in csv or text format
      rows - number of rows in the frame
      cols - number of columns in the frame
      format - input format of the given stream
      Returns:
      frame as a two-dimensional string array
      Throws:
      IOException - if IOException occurs

    • convertToFrame

      public FrameBlock convertToFrame(String input, String meta) throws IOException
      Converts an input string representation of a frame in csv or textcell format into a frame block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
      Parameters:
      input - string frame in csv or textcell format
      meta - string representing SystemDS frame metadata in JSON format
      Returns:
      frame as a frame block
      Throws:
      IOException - if IOException occurs

    • convertToFrame

      public FrameBlock convertToFrame(InputStream input, String meta) throws IOException
      Converts an input stream of a string frame in csv or textcell format into a frame block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.
      Parameters:
      input - InputStream to a string frame in csv or textcell format
      meta - string representing SystemDS frame metadata in JSON format
      Returns:
      frame as a frame block
      Throws:
      IOException - if IOException occurs

    • convertToFrame

      public FrameBlock convertToFrame(String input, int rows, int cols) throws IOException
      Converts an input string representation of a frame in textcell format into a frame block.
      Parameters:
      input - string frame in textcell format
      rows - number of rows in the frame
      cols - number of columns in the frame
      Returns:
      frame as a frame block
      Throws:
      IOException - if IOException occurs

    • convertToFrame

      public FrameBlock convertToFrame(InputStream input, int rows, int cols) throws IOException
      Converts an input stream of a string frame in textcell format into a frame block.
      Parameters:
      input - InputStream to a string frame in textcell format
      rows - number of rows in the frame
      cols - number of columns in the frame
      Returns:
      frame as a frame block
      Throws:
      IOException - if IOException occurs

    • convertToFrame

      public FrameBlock convertToFrame(InputStream input, int rows, int cols, String format) throws IOException
      Converts an input stream of a frame in csv or textcell format into a frame block.
      Parameters:
      input - InputStream to a string frame in csv or textcell format
      rows - number of rows in the frame
      cols - number of columns in the frame
      format - input format of the given stream
      Returns:
      frame as a frame block
      Throws:
      IOException - if IOException occurs

    • readTransformMetaDataFromFile

      public FrameBlock readTransformMetaDataFromFile(String spec, String metapath) throws IOException
      Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.
      Parameters:
      spec - transform specification as json string
      metapath - hdfs file path to meta data directory
      Returns:
      FrameBlock object representing transform metadata
      Throws:
      IOException - if IOException occurs

    • readTransformMetaDataFromFile

      public FrameBlock readTransformMetaDataFromFile(String spec, String metapath, String colDelim) throws IOException
      Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.
      Parameters:
      spec - transform specification as json string
      metapath - hdfs file path to meta data directory
      colDelim - separator for processing column names in the meta data file 'column.names'
      Returns:
      FrameBlock object representing transform metadata
      Throws:
      IOException - if IOException occurs

    • readTransformMetaDataFromPath

      public FrameBlock readTransformMetaDataFromPath(String metapath) throws IOException
      Reads transform meta data from the class path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.
      Parameters:
      metapath - resource path to meta data directory
      Returns:
      FrameBlock object representing transform metadata
      Throws:
      IOException - if IOException occurs

    • readTransformMetaDataFromPath

      public FrameBlock readTransformMetaDataFromPath(String spec, String metapath) throws IOException
      Reads transform meta data from the class path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.
      Parameters:
      spec - transform specification as json string
      metapath - resource path to meta data directory
      Returns:
      FrameBlock object representing transform metadata
      Throws:
      IOException - if IOException occurs

    • readTransformMetaDataFromPath

      public FrameBlock readTransformMetaDataFromPath(String spec, String metapath, String colDelim) throws IOException
      Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.
      Parameters:
      spec - transform specification as json string
      metapath - resource path to meta data directory
      colDelim - separator for processing column names in the meta data file 'column.names'
      Returns:
      FrameBlock object representing transform metadata
      Throws:
      IOException - if IOException occurs