Package org.apache.sysds.runtime.transform.tokenize.applier

Class TokenizerApplier

java.lang.Object
org.apache.sysds.runtime.transform.tokenize.applier.TokenizerApplier
All Implemented Interfaces:
Serializable
Direct Known Subclasses:
TokenizerApplierCount, TokenizerApplierHash, TokenizerApplierPosition
public abstract class TokenizerApplier extends Object implements Serializable
See Also:

  • Field Details

  • Constructor Details

    • TokenizerApplier

      public TokenizerApplier(int numIdCols, int maxTokens, boolean wideFormat, boolean applyPadding)

  • Method Details

    • applyInternalRepresentation

      public int applyInternalRepresentation(DocumentRepresentation[] internalRepresentation, FrameBlock out)

    • build

      public void build(DocumentRepresentation[] internalRepresentation, int inputRowStart, int blk)

    • getBuildTasks

      public List<DependencyTask<?>> getBuildTasks(DocumentRepresentation[] internalRepresentation)

    • getApplyTasks

      public List<DependencyTask<?>> getApplyTasks(DocumentRepresentation[] internalRepresentation, FrameBlock out)

    • getOutSchema

      public abstract Types.ValueType[] getOutSchema()

    • hasPadding

      public boolean hasPadding()

    • getMaxTokens

      public int getMaxTokens()

    • getMaxNumRows

      public int getMaxNumRows(int inRows)

    • getNumRows

      public abstract int getNumRows(DocumentRepresentation[] internalRepresentation)

    • getOutputRow

      public <T, E> int getOutputRow(int inputRowStart, List<Map<T,E>> internalData)

    • getOutputRow

      public int getOutputRow(int inputRowStart, DocumentRepresentation[] internalData)

    • getNumCols

      public long getNumCols()

    • isWideFormat

      public boolean isWideFormat()

    • allocateInternalMeta

      public void allocateInternalMeta(int numDocuments)