Append to file in HDFS (CDH 5.4.5)

Question

Brand new to HDFS here.

I've got this small section of code to test out appending to a file:

val path: Path = new Path("/tmp", "myFile")
val config = new Configuration()
val fileSystem: FileSystem = FileSystem.get(config)
val outputStream = fileSystem.append(path)
outputStream.writeChars("what's up")
outputStream.close()

It is failing with this message:

Not supported
java.io.IOException: Not supported
    at org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java:352)
    at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1163)

I looked at the source for ChecksumFileSystem.java, and it seems to be hardcoded to not support appending:

@Override
public FSDataOutputStream append(Path f, int bufferSize,
    Progressable progress) throws IOException {
  throw new IOException("Not supported");
}

How to make this work? Is there some way to change the default file system to some other implementation that does support append?

Answer 1

It turned out that I needed to actually run a real hadoop namenode and datanode. I am new to hadoop and did not realize this. Without this, it will use your local filesystem which is a ChecksumFileSystem, which does not support append. So I followed the blog post here to get it up and running on my system, and now I am able to append.

Answer 2

The append method has to be called on outputstream not on filesystem. filesystem.get() is just used to connect to your HDFS. First set dfs.support.append as true in hdfs-site.xml

<property>
       <name>dfs.support.append</name>
       <value>true</value>
</property> 

stop all your demon services using stop-all.sh and restart it again using start-all.sh. Put this in your main method.

String fileuri = "hdfs/file/path"
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(fileuri),conf);
FSDataOutputStream out = fs.append(new Path(fileuri));
PrintWriter writer = new PrintWriter(out);
writer.append("I am appending this to my file");
writer.close();
fs.close();