如何设置Samza的metrics

时间:2014-04-29 23:50:43   收藏:0   阅读:843

参考这个里边对API的调用

http://samza.incubator.apache.org/learn/documentation/0.7.0/container/metrics.html
参考这里列出的可用配置
http://samza.incubator.apache.org/learn/documentation/0.7.0/jobs/configuration-table.html
在正常的配置之外,需要额外添加的配置。添加这些配置以后,就可以在StreamTask里使用metrics的API来提供metrics信息
 
  1. 需要哪些配置?
    1. 设置用哪种reporter factory,Samza自带有Kafka和JMX两种
    2. 设置reporter使用的stream,包括设置这个stream的serde
    3. 注册这个reporter
#设置用来做为输出流的Stream
#设置samza-metrics这个stream的system
streams.samza-metrics.system= kafka 
#设置samza-metrics这个stream的名字,即在kafka里对应的topic的名字
streams.samza-metrics.stream= samza-metrics

#配置reporter
#设置reporter factory这里使用的MetricsSnapshotReporterFactory,把JSON格式的metrics发给Kafka。这个class要是Factory的名字,samza当前的document里的配置是错的
metrics.reporter.samza-metrics.class= org.apache.samza.metrics.reporter.MetricsSnapshotReporterFactory
 
#samza-metrics这个reporter10s发送一次metrics
metrics.reporter.samza-metrics.window.ms= 10000
 
#定义samza-metrics这个reporter对应的stream。此reporter使用kafka这个system下的samza-metrics这个stream。注意一定要使用
#system.stream这种写法,即写成kafka.samza-metrics
#这个配置是必须的,但是Samza的config文档里没有列出
metrics.reporter.samza-metrics.stream= kafka.samza-metrics
 
#注册samza-metrics这个reporter
metrics.reporters= samza-metrics

#配置serde
Serializers 定义可以使用的Serde factory的名字
serializers.registry.json.class= org.apache.samza.serializers.JsonSerdeFactory
serializers.registry.metrics.class= org.apache.samza.serializers.MetricsSnapshotSerdeFactory
 
#定义samza-metrics这个stream,用来做消息serde的Serde factory为metrics。由于这个stream输出的是metrics的信息,因此必须使用这个serde factory
systems.kafka.streams.samza-metrics.samza.msg.serde= metrics
 
 
最后会有三种metrics。StramTask的、SystemProducer的、SystemConsumer的

{"metrics":{"org.apache.samza.container.TaskInstanceMetrics":{"process-calls":28575767,"messages-sent":0,"commit-calls":12,"window-skipped":30204500,"kafka-pizza-offset":"36950462","commit-skipped":30204488,"send-skipped":30204500,"window-calls":0,"send-calls":0},"class hs.samza.simple.SimpleKafkaTask":{"messageCount":28500000}},"header":{"reset-time":1398672217312,"job-id":"1","time":1398672937854,"host":"hadoop-node-1","container-name":"samza-container-2","source":"Partition-2","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}


{"metrics":{"org.apache.samza.system.kafka.KafkaSystemProducerMetrics":{"kafka-producer-sends":38,"kafka-partition-2-producer-buffer-size":0,"kafka-flushes":38,"kafka-metricssnapshotreporterfactory-producer-buffer-size":0,"kafka-samza-container-2-producer-buffer-size":0,"kafka-producer-reconnects":0,"kafka-flush-sizes":38}},"header":{"reset-time":1398672217312,"job-id":"1","time":1398672937855,"host":"hadoop-node-1","container-name":"samza-container-2","source":"MetricsSnapshotReporterFactory","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}


{"metrics":{"org.apache.samza.system.SystemConsumersMetrics":{"blocking-poll-timeout":10,"kafka-messages-per-poll":15667419,"chose-object":29071417,"kafka-ssp-fetches-per-poll":1501156,"max-buffered-messages-per-stream-partition":1000,"ssps-needed-by-chooser":1,"kafka-pizza-messages-chosen":29071417,"unprocessed-messages":0,"chose-null":1483650,"kafka-polls":15667420,"poll-timeout":10},"org.apache.samza.metrics.JvmMetrics":{"threads-runnable":5,"mem-heap-committed-mb":313.5625,"threads-new":0,"mem-non-heap-committed-mb":24.75,"mem-heap-used-mb":93.080475,"mem-non-heap-used-mb":24.63375,"threads-terminated":0,"ps marksweep-gc-time-millis":41,"ps scavenge-gc-count":380,"ps scavenge-gc-time-millis":16289,"gc-time-millis":16330,"threads-blocked":0,"threads-timed-waiting":6,"ps marksweep-gc-count":1,"threads-waiting":4,"gc-count":381},"org.apache.samza.container.SamzaContainerMetrics":{"process-null-envelopes":1483649,"process-envelopes":29071417,"process-calls":30555067,"commit-calls":30555066,"window-calls":30555066,"send-calls":30555066},"org.apache.samza.system.chooser.RoundRobinChooserMetrics":{"buffered-messages":0},"org.apache.samza.system.kafka.KafkaSystemConsumerMetrics":{"kafka-pizza-3-offset-change":37437828,"poll-count":15667420,"kafka-10.5.132.122-9092-topic-partitions":1,"no-more-messages-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":false,"kafka-10.5.132.122-9092-messages-read":36263,"blocking-poll-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":0,"kafka-pizza-3-bytes-read":1512471142,"kafka-pizza-3-messages-read":29071417,"kafka-10.5.132.122-9092-skipped-fetch-requests":8,"blocking-poll-timeout-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":728147,"kafka-pizza-3-messages-behind-high-watermark":0,"buffered-message-count-SystemStreamPartition [partition=Partition [partition=3], system=kafka, stream=pizza]":0,"kafka-10.5.132.122-9092-bytes-read":1512471142,"kafka-10.5.132.122-9092-reconnects":0},"org.apache.samza.system.kafka.KafkaSystemProducerMetrics":{"kafka-producer-sends":0,"kafka-flushes":14,"kafka-partition-3-producer-buffer-size":0,"kafka-producer-reconnects":0,"kafka-flush-sizes":0},"org.apache.samza.system.SystemProducersMetrics":{"partition-3-sends":0,"partition-3-flushes":14,"flushes":14,"sends":0}},"header":{"reset-time":1398672219649,"job-id":"1","time":1398673001854,"host":"hd-e.cdh","container-name":"samza-container-3","source":"samza-container-3","job-name":"my-samza-test","samza-version":"0.0.1","version":"0.0.1"}}

 

看着头都大了。应该还是JMX那种reporter看着舒服点。以后再试吧……

如何设置Samza的metrics,码迷,mamicode.com

评论(0
© 2014 mamicode.com 版权所有 京ICP备13008772号-2  联系我们:gaon5@hotmail.com
迷上了代码!