Error Hadoop on Windows via Cygwin: Could Not Locate null\bin\winutils.exe

Question

When I set HADOOP_HOME=/cygdrive/c/ecosystem/hadoop-2.5.1 and then attempt to run bin/hadoop fs or bin/hadoop hadoop-streaming.jar from Cygwin, I receive the following error:

    ERROR [main] util.Shell (Shell.java:getWinUtilsPath(373)) - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:363)
        at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:432)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:478)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
Exception in thread "main" java.lang.RuntimeException: core-site.xml not found
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2269)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2195)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2112)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:989)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:961)
        at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1299)
        at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:319)
        at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:479)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
        at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

I can access bin/hadoop version along with classpath and most other commands. I thought that maybe I needed a Windows style path so that it could conform to the \bin\winutils style. So, I set HADOOP_HOME=c:\ecosystem\hadoop-2.5.1 and this error went away. However, now I receive a link error when running in Cygwin. I tested hadoop with HADOOP_HOME=c:\ecosystem\hadoop-2.5.1 via Windows Command, and I can run a MapReduce job without issue.

I want to successfully run hadoop in Cygwin, and I don't want to deal with link errors. If I keep HADOOP_HOME=/cygdrive/c/ecosystem/hadoop-2.5.1, is there any way that I can set hadoop-config or some other file to look for the Windows style path if the Cygwin path fails or is null? Maybe something like this in hadoop-config:

if ["$HADOOP_HOME" = ""]; then
  HADOOP_HOME=$(cygpath -w "$HADOOP_HOME")
fi

Obviously this particular code didn't work in hadoop-config, but I suppose you'll get the general idea. Any help would be appreciated.

Answer 1

I solved a similar error with Eclipse when I was trying to execute Hadoop MR program locally in Windows by setting the environment variable HADOOP_HOME through Eclipse: Run->Run Configurations->Environment.

Answer 2

Since I was able to successfully operate with Hadoop via the Windows command, it didn't make much sense to waste time trying to figure out how to make it work with cygwin. Instead, for my greater Hadoop ecosystem, I simply built it on linux. It's super easy to use, and is by far the best decision I've made.