How to avoid creation of .crc files when parquet files are created

Question

I am using parquet framework to write parquet files. I create the parquet writer with this constructor--

public class ParquetBaseWriter<T extends HashMap> extends ParquetWriter<T> {
    public ParquetBaseWriter(Path file, HashMap<String, SchemaField> mySchema,
                             CompressionCodecName compressionCodecName, int blockSize,
                             int pageSize) throws IOException {
        super(file, ParquetBaseWriter.<T>writeSupport(mySchema),
                compressionCodecName, blockSize, pageSize, DEFAULT_IS_DICTIONARY_ENABLED, false);
    }

Eachtime a parquet file is created, A .crc file corresponding to it also gets created on the disk. How can I avoid the creation of that .crc file? Is there a flag or something which I have to set?

Thanks

Answer 1

You could see this google groups discussion about the crc files: https://groups.google.com/a/cloudera.org/forum/#!topic/cdk-dev/JR45MsLeyTE

TL;DR - crc files don't take up any overhead in the NN namespace. They're not HDFS data files, they are meta files in the data directories. You will see them in your local filesystem if you use the "file:///" URI.