Package org.apache.sysds.runtime.transform.tokenize.builder

Class TokenizerBuilderNgram

java.lang.Object

org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilder

org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit

org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram

All Implemented Interfaces:

Serializable

public class TokenizerBuilderNgram
extends TokenizerBuilderWhitespaceSplit

See Also:

• Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
int	maxGram
int	minGram
org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram.NgramType	ngramType

Fields inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit

regex

Constructor Summary

Constructors

Constructor	Description
TokenizerBuilderNgram(int[] idCols, int tokenizeCol, org.apache.wink.json4j.JSONObject params)

Method Summary

Modifier and Type	Method	Description
void	createInternalRepresentation(FrameBlock in, DocumentRepresentation[] internalRepresentation, int rowStart, int blk)
List<Token>	splitIntoNgrams(Token token, int minGram, int maxGram)

Methods inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderWhitespaceSplit

splitToTokens

Methods inherited from class org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilder

createInternalRepresentation, getTasks

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

minGram

public int minGram

maxGram

public int maxGram

ngramType

public org.apache.sysds.runtime.transform.tokenize.builder.TokenizerBuilderNgram.NgramType ngramType

Constructor Details

TokenizerBuilderNgram

public TokenizerBuilderNgram(int[] idCols,
 int tokenizeCol,
 org.apache.wink.json4j.JSONObject params)
                      throws org.apache.wink.json4j.JSONException

Throws:

org.apache.wink.json4j.JSONException

Method Details

splitIntoNgrams

public List<Token> splitIntoNgrams(Token token,
 int minGram,
 int maxGram)

createInternalRepresentation

public void createInternalRepresentation(FrameBlock in,
 DocumentRepresentation[] internalRepresentation,
 int rowStart,
 int blk)

Overrides:

createInternalRepresentation in class TokenizerBuilderWhitespaceSplit