org.apache.spark.sql.sources

Class OutputWriter

java.lang.Object

org.apache.spark.sql.sources.OutputWriter

public abstract class OutputWriter
extends java.lang.Object

::Experimental:: OutputWriter is used together with HadoopFsRelation for persisting rows to the underlying file system. Subclasses of OutputWriter must provide a zero-argument constructor. An OutputWriter instance is created and initialized when a new output file is opened on executor side. This instance is used to persist rows to this single output file.

Since:

1.4.0

Constructor Summary

Constructors

Constructor and Description
OutputWriter()

Method Summary

Methods

Modifier and Type	Method and Description
abstract void	close() Closes the OutputWriter.
protected void	initConverter(StructType dataSchema)
abstract void	write(Row row) Persists a single row.
protected void	writeInternal(org.apache.spark.sql.catalyst.InternalRow row)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OutputWriter

public OutputWriter()

Method Detail

write

public abstract void write(Row row)

Persists a single row. Invoked on the executor side. When writing to dynamically partitioned tables, dynamic partition columns are not included in rows to be written.

Parameters:

row - (undocumented)

Since:

1.4.0

close

public abstract void close()

Closes the OutputWriter. Invoked on the executor side after all rows are persisted, before the task output is committed.

Since:

1.4.0

initConverter

protected void initConverter(StructType dataSchema)

writeInternal

protected void writeInternal(org.apache.spark.sql.catalyst.InternalRow row)