Another MapReduce issue with MacOs

I faced another macos issue w/ MapReduce on MacOs:

 

$HADOOP_HOME/bin/hadoop jar target/hadoop-tutorial-1.0-SNAPSHOT-jar-with-dependencies.jar fr.finaxys.tutorials.hadoop.TraceCount -archives /user/pierre/AtomLogs.zip tracecount

Exception in thread « main » java.io.IOException: Mkdirs failed to create /var/folders/5w/rckc21hd0kz2qg0_3r7g60b00000gn/T/hadoop-unjar3988465705021626998/META-INF/license

at org.apache.hadoop.util.RunJar.ensureDirectory(RunJar.java:128)

at org.apache.hadoop.util.RunJar.unJar(RunJar.java:104)

at org.apache.hadoop.util.RunJar.unJar(RunJar.java:81)

at org.apache.hadoop.util.RunJar.run(RunJar.java:209)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Solutions can be found here:

http://stackoverflow.com/questions/10522835/hadoop-java-io-ioexception-mkdirs-failed-to-create-some-path

It is a case sensitive issue with MacOs.

For now, I wasn’t able to make the trick with maven working so I’m removing the LICENCE file manually from the archive:

zip -d target/hadoop-tutorial-1.0-SNAPSHOT-jar-with-dependencies.jar META-INF/LICENSE

 

YARN Test on MacosX

Hi,

I’m currently playing a little with hdfs and yarn (Hadoop 2.7.1) on my laptop (Darwin Kernel Version 14.5.0 / OS X Yosemite Version 10.10.5) . Until then, I was mainly doing HBase development and I thought it could be nice if I learn some basic stuff such as Map Reduce programming before starting with Spark.

While I was trying the Map Reduce tutorial, I fall on this error:

15/08/17 22:59:29 INFO mapreduce.Job: Job job_1439843536878_0001 failed with state FAILED due to: Application application_1439843536878_0001 failed 2 times due to AM Container for appattempt_1439843536878_0001_000002 exited with  exitCode: 127

For more detailed output, check application tracking page:http://MacBook-Pro-de-Pierre.local:8088/cluster/app/application_1439843536878_0001Then, click on links to logs of each attempt.

Diagnostics: Exception from container-launch.

Container id: container_1439843536878_0001_02_000001

Exit code: 127

Stack trace: ExitCodeException exitCode=127:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)

at org.apache.hadoop.util.Shell.run(Shell.java:456)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 127

Failing this attempt. Failing the application.

15/08/17 22:59:29 INFO mapreduce.Job: Counters: 0

This simple command fix the issue:

sudo ln -s /usr/bin/java /bin/java

Hope this helps.

Pierre