Multiples Hadoop FileSystem instances

Question

I have a class (I removes try/catch for readability):

public class HadoopFileSystem {

    private FileSystem m_fileSystem = null;

    public HadoopFileSystem() {
        Configuration l_configuration = new Configuration();
        l_configuration .set("fs.default.name", "hdfs://localhost:9100");
        l_configuration .set("mapred.job.tracker", "localhost:9101");

        m_fileSystem = FileSystem.get(l_configuration );

    }

    public void close() {
        m_fileSystem.close();
    }

    public void createFile(String a_pathDFS) {
        m_fileSystem.create(new Path(a_pathDFS));
    }

}

In my program, I a first HadoopFileSysem object, I don't close it.

Then I create a second HadoopFileSysem object, and I close it.

Finally, when I want to use a function on m_fileSystem in my first object, I have the error : java.io.IOException: Filesystem closed

But I did'nt close it !

Here's a little code to illustrate my problem :

HadoopFileSystem h1 = new HadoopFileSystem();
HadoopFileSystem h2 = new HadoopFileSystem();

if(h1 == h2)
    System.out.println("=="); // No print
if(h1.equals(h2))
    System.out.println("equals"); // No print

h2.close();
h1.createFile("test.test"); // ERROR : java.io.IOException: Filesystem closed
h1.close();

Why ?

Answer 1

m_fileSystem = FileSystem.get(l_configuration ); is a static call even if you have two different objects created. You need to find a way to not make this call static for two different objects.

Try this out to do away with the problem,

conf.setBoolean("fs.hdfs.impl.disable.cache", true);