转自：http://my.oschina.net/leejun2005/blog/276891?utm_source=tuicool&utm_medium=referral

1、计数器 简介

在许多情况下，一个用户需要了解待分析的数据，尽管这并非所要执行的分析任务 的核心内容。以统计数据集中无效记录数目的任务为例，如果发现无效记录的比例 相当高，那么就需要认真思考为何存在如此多无效记录。是所采用的检测程序存在 缺陷，还是数据集质量确实很低，包含大量无效记录？如果确定是数据集的质量问 题，则可能需要扩大数据集的规模，以增大有效记录的比例，从而进行有意义的 分析。 
计数器是一种收集作业统计信息的有效手段，用于质量控制或应用级统计。计数器 还可辅助诊断系统故障。如果需要将日志信息传输到map或reduce任务，更好的 方法通常是尝试传输计数器值以监测某一特定事件是否发生。对于大型分布式作业 而言，使用计数器更为方便。首先，获取计数器值比输出日志更方便，其次，根据 计数器值统计特定事件的发生次数要比分析一堆日志文件容易得多。 

2 、内置计数器

Hadoop为每个作业维护若干内置计数器, 以描述该作业的各项指标。例如，某些计数器记录已处理的字节数和记录数，使用户可监控已处理的输入数据量和已产生的输出数据量，并以此对 job 做适当的优化。

14/06/08 15:13:35 INFO mapreduce.Job: Counters: 46
  File System Counters
  FILE: Number of bytes read=159
  FILE: Number of bytes written=159447
  FILE: Number of read operations=0
  FILE: Number of large read operations=0
  FILE: Number of write operations=0
  HDFS: Number of bytes read=198
  HDFS: Number of bytes written=35
  HDFS: Number of read operations=6
  HDFS: Number of large read operations=0
  HDFS: Number of write operations=2
  Job Counters 
  Launched map tasks=1
  Launched reduce tasks=1
  Rack-local map tasks=1
  Total time spent by all maps in occupied slots (ms)=3896
  Total time spent by all reduces in occupied slots (ms)=9006
  Map-Reduce Framework
  Map input records=3
  Map output records=12
  Map output bytes=129
  Map output materialized bytes=159
  Input split bytes=117
  Combine input records=0
  Combine output records=0
  Reduce input groups=4
  Reduce shuffle bytes=159
  Reduce input records=12
  Reduce output records=4
  Spilled Records=24
  Shuffled Maps =1
  Failed Shuffles=0
  Merged Map outputs=1
  GC time elapsed (ms)=13
  CPU time spent (ms)=3830
  Physical memory (bytes) snapshot=537718784
  Virtual memory (bytes) snapshot=7365263360
  Total committed heap usage (bytes)=2022309888
  Shuffle Errors
  BAD_ID=0
  CONNECTION=0
  IO_ERROR=0
  WRONG_LENGTH=0
  WRONG_MAP=0
  WRONG_REDUCE=0
  File Input Format Counters 
  Bytes Read=81
  File Output Format Counters 
  Bytes Written=35

计数器由其关联任务维护，并定期传到tasktracker，再由tasktracker传给 jobtracker.因此，计数器能够被全局地聚集。详见第 hadoop 权威指南第170页的“进度和状态的更新”小节。与其他计数器（包括用户定义的计数器）不同，内置的作业计数器实际上 由jobtracker维护，不必在整个网络中发送。 
一个任务的计数器值每次都是完整传输的，而非自上次传输之后再继续数未完成的传输，以避免由于消息丢失而引发的错误。另外，如果一个任务在作业执行期间失 败，则相关计数器值会减小。仅当一个作业执行成功之后，计数器的值才是完整可 靠的。 

3、 用户定义的Java计数器

MapReduce允许用户编写程序来定义计数器，计数器的值可在mapper或reducer 中增加。多个计数器由一个Java枚举(enum)类型来定义，以便对计数器分组。一 个作业可以定义的枚举类型数量不限，各个枚举类型所包含的字段数量也不限。枚 举类型的名称即为组的名称，枚举类型的字段就是计数器名称。计数器是全局的。 换言之，MapReduce框架将跨所有map和reduce聚集这些计数器，并在作业结束 时产生一个最终结果。

Note1： 需要说明的是，不同的 hadoop 版本定义的方式会有些许差异。

（1）在0.20.x版本中使用counter很简单,直接定义即可，如无此counter，hadoop会自动添加此counter. 
Counter ct = context.getCounter("INPUT_WORDS", "count"); 
ct.increment(1);   
（2）在0.19.x版本中,需要定义enum 
enum MyCounter {INPUT_WORDS }; 
reporter.incrCounter(MyCounter.INPUT_WORDS, 1); 
RunningJob job = JobClient.runJob(conf); 
Counters c = job.getCounters(); 
long cnt = c.getCounter(MyCounter.INPUT_WORDS);

Notice2： 使用计数器需要清楚的是它们都存储在jobTracker的内存里。Mapper/Reducer 任务序列化它们，连同更新状态被发送。为了运行正常且jobTracker不会出问题，计数器的数量应该在10-100个，计数器不仅仅只用来聚合MapReduce job的统计值。新版本的hadoop限制了计数器的数量，以防给jobTracker带来损害。你最不想看到的事情就是由于定义上百个计数器而使jobTracker宕机。 

下面咱们来看一个计数器的实例（以下代码请运行在 0.20.1 版本以上）： 

3.1 测试数据：

hello world 2013 mapreduce
hello world 2013 mapreduce
hello world 2013 mapreduce

3.2 代码：

/**
 * Project Name:CDHJobs
 * File Name:MapredCounter.java
 * Package Name:tmp
 * Date:2014-6-8下午2:12:48
 * Copyright (c) 2014, decli#qq.com All Rights Reserved.
 *
 */

package tmp;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Counter;
import org.apache.hadoop.mapreduce.CounterGroup;
import org.apache.hadoop.mapreduce.Counters;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCountWithCounter {

  static enum WordsNature {
    STARTS_WITH_DIGIT, STARTS_WITH_LETTER, ALL
  }

  /**
   * The map class of WordCount.
   */
  public static class TokenCounterMapper extends Mapper<Object, Text, Text, IntWritable> {

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
      StringTokenizer itr = new StringTokenizer(value.toString());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
      }
    }
  }

  /**
   * The reducer class of WordCount
   */
  public static class TokenCounterReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException,
        InterruptedException {
      int sum = 0;

      String token = key.toString();
      if (StringUtils.isNumeric(token)) {
        context.getCounter(WordsNature.STARTS_WITH_DIGIT).increment(1);
      } else if (StringUtils.isAlpha(token)) {
        context.getCounter(WordsNature.STARTS_WITH_LETTER).increment(1);
      }
      context.getCounter(WordsNature.ALL).increment(1);

      for (IntWritable value : values) {
        sum += value.get();
      }
      context.write(key, new IntWritable(sum));
    }
  }

  /**
   * The main entry point.
   */
  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = new Job(conf, "WordCountWithCounter");
    job.setJarByClass(WordCountWithCounter.class);
    job.setMapperClass(TokenCounterMapper.class);
    job.setReducerClass(TokenCounterReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path("/tmp/dsap/rawdata/june/a.txt"));
    FileOutputFormat.setOutputPath(job, new Path("/tmp/dsap/rawdata/june/a_result"));
    int exitCode = job.waitForCompletion(true) ? 0 : 1;

    Counters counters = job.getCounters();
    Counter c1 = counters.findCounter(WordsNature.STARTS_WITH_DIGIT);
    System.out.println("-------------->>>>: " + c1.getDisplayName() + ": " + c1.getValue());

    // The below example shows how to get built-in counter groups that Hadoop provides basically.
    for (CounterGroup group : counters) {
      System.out.println("==========================================================");
      System.out.println("* Counter Group: " + group.getDisplayName() + " (" + group.getName() + ")");
      System.out.println("  number of counters in this group: " + group.size());
      for (Counter counter : group) {
        System.out.println("  ++++ " + counter.getDisplayName() + ": " + counter.getName() + ": "
            + counter.getValue());