2.1 Hadoop Eclipse Plugin 配置及安装

时间：2014-05-23 02:44:30 收藏：0 阅读：354

Hadoop Eclipse 开发工具主要分为

1、根据Hadoop版本生成插件

2、安装Hadoop Eclipse插件

3、配置Hadoop目录

4、配置Hadoop连接

5、新一个MapReduce工程

WordCount.java

MapReduce——WordCount问题总结

参考：http://blog.sina.com.cn/s/blog_7fcb1aef0100zpux.html

正成功输入出后信息:

14/05/21 23:06:47 INFO input.FileInputFormat: Total input paths to process : 2
14/05/21 23:06:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/05/21 23:06:47 WARN snappy.LoadSnappy: Snappy native library not loaded
14/05/21 23:06:48 INFO mapred.JobClient: Running job: job_201405220635_0009
14/05/21 23:06:49 INFO mapred.JobClient: map 0% reduce 0%
14/05/21 23:06:59 INFO mapred.JobClient: map 50% reduce 0%
14/05/21 23:07:00 INFO mapred.JobClient: map 100% reduce 0%
14/05/21 23:07:09 INFO mapred.JobClient: map 100% reduce 33%
14/05/21 23:07:11 INFO mapred.JobClient: map 100% reduce 100%
14/05/21 23:07:13 INFO mapred.JobClient: Job complete: job_201405220635_0009
14/05/21 23:07:13 INFO mapred.JobClient: Counters: 29
14/05/21 23:07:13 INFO mapred.JobClient: Job Counters
14/05/21 23:07:13 INFO mapred.JobClient: Launched reduce tasks=1
14/05/21 23:07:13 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=17386
14/05/21 23:07:13 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/05/21 23:07:13 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/05/21 23:07:13 INFO mapred.JobClient: Launched map tasks=2
14/05/21 23:07:13 INFO mapred.JobClient: Data-local map tasks=2
14/05/21 23:07:13 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=12160
14/05/21 23:07:13 INFO mapred.JobClient: File Output Format Counters
14/05/21 23:07:13 INFO mapred.JobClient: Bytes Written=15
14/05/21 23:07:13 INFO mapred.JobClient: FileSystemCounters
14/05/21 23:07:13 INFO mapred.JobClient: FILE_BYTES_READ=52
14/05/21 23:07:13 INFO mapred.JobClient: HDFS_BYTES_READ=252
14/05/21 23:07:13 INFO mapred.JobClient: FILE_BYTES_WRITTEN=177419
14/05/21 23:07:13 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=15
14/05/21 23:07:13 INFO mapred.JobClient: File Input Format Counters
14/05/21 23:07:13 INFO mapred.JobClient: Bytes Read=22
14/05/21 23:07:13 INFO mapred.JobClient: Map-Reduce Framework
14/05/21 23:07:13 INFO mapred.JobClient: Map output materialized bytes=58
14/05/21 23:07:13 INFO mapred.JobClient: Map input records=2
14/05/21 23:07:13 INFO mapred.JobClient: Reduce shuffle bytes=58
14/05/21 23:07:13 INFO mapred.JobClient: Spilled Records=8
14/05/21 23:07:13 INFO mapred.JobClient: Map output bytes=38
14/05/21 23:07:13 INFO mapred.JobClient: CPU time spent (ms)=5610
14/05/21 23:07:13 INFO mapred.JobClient: Total committed heap usage (bytes)=336338944
14/05/21 23:07:13 INFO mapred.JobClient: Combine input records=4
14/05/21 23:07:13 INFO mapred.JobClient: SPLIT_RAW_BYTES=230
14/05/21 23:07:13 INFO mapred.JobClient: Reduce input records=4
14/05/21 23:07:13 INFO mapred.JobClient: Reduce input groups=2
14/05/21 23:07:13 INFO mapred.JobClient: Combine output records=4
14/05/21 23:07:13 INFO mapred.JobClient: Physical memory (bytes) snapshot=428146688
14/05/21 23:07:13 INFO mapred.JobClient: Reduce output records=2
14/05/21 23:07:13 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2140233728
14/05/21 23:07:13 INFO mapred.JobClient: Map output records=4

错误解决方法：

14/05/21 23:14:11 INFO mapred.JobClient: Cleaning up the staging area hdfs://192.168.1.53:9000/app/hadoop/hadoop/tmp/mapred/staging/hadoop/.staging/job_201405220635_0010
14/05/21 23:14:11 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:973)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:93)

cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory newout already exists

说明目录已经存在了，需要删除后再执行，命令如下：

hadoop fs -rmr newout

2.1 Hadoop Eclipse Plugin 配置及安装,布布扣,bubuko.com