[Programmatic cluster monitoring] How to use the HBase monitoring APIs

[techbuddy]

Hi all Hbase veterans, gurus, newbies,

I'm trying to figure out how to go about programmatically monitoring an Hbase cluster using the APIs listed @
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/monitoring/package-summary.html. I couldn't land any webdoc explaining how to use these APIs.. I know about the monitoring alternatives of using Ganglia ,JMX etc. But surprisingly, I couldn't retrieve any documentation/tutorial on using the above APIs..
Any pointers would be highly appreciated.

-thanks
Santosh

[stack-3]

On Mon, Oct 1, 2012 at 3:17 PM, techbuddy <[hidden email]> wrote:

> Hi all Hbase veterans, gurus, newbies,
>
> I'm trying to figure out how to go about programmatically monitoring an
> Hbase cluster using the APIs listed @
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/monitoring/package-summary.html.
> I couldn't land any webdoc explaining how to use these APIs.. I know about
> the monitoring alternatives of using Ganglia ,JMX etc. But surprisingly, I
> couldn't retrieve any documentation/tutorial on using the above APIs..
> Any pointers would be highly appreciated.
>

If you look at the javadoc you'll see they all have the annotation:

@InterfaceAudience.Private

This is how we flag stuff that is for server-side only, not for public
consumption.

These classes are used by the server variously adding stuff to UIs, etc.

Regards monitoring stuff programmatically, what do you mean?  Why
would you want to write a monitoring tool when there are already
myriad?  If you insist, you could do something like grab the json
content of the /jmx servlet run on each server or you could enable
metrics and catch the firehose by parsing ganglia formatter messages
or the text form you can have dumped to a file, etc.

Let us know more what you need so we can help you out.

St.Ack

[techbuddy]

Thanks for clarifying the usage part out Stack!

As for programmatic monitoring, I was trying to figure out how to extend the already available metrics capture mechanism (available for Region server and Master server processes) to dump some custom metrics into a file (using the TimestampedFileContext) for downstream consumption by our graphing solution. These custom metrics could be like # of minor compactions that happened since the last poll time , or the rate of minor compactions over a time window etc.

In other words,I was wondering if HBase monitoring framework supports some kind of plugin development.
So, I'm not really looking at building a brand new monitoring solution, but rather a way to extend the available set of metrics.

Does that make sense or do you think the available set of metrics (for RegionServer,Master Server) are good enough.

-thanks
tb
PS : The available set of metrics, for e.g. the RegionServer are like Block cache metrics, compaction metrics,IO, store metrics etc

[Otis Gospodnetic]

Hi,

Have a look at
https://github.com/sematext/HBaseMetricsContext  +
http://blog.sematext.com/2011/07/31/extending-hadoop-metrics/ -- this
may lead you in the right direction if you really really need to do
this.... although I'm not sure if that stuff is outdated now.

If you are just trying to get existing HBase metrics there is SPM
(http://sematext.com/spm/index.html), Ganglia, etc.

Otis


On Tue, Oct 2, 2012 at 1:52 AM, techbuddy <[hidden email]> wrote:

> Thanks for clarifying the usage part out Stack!
>
> As for programmatic monitoring, I was trying to figure out how to extend the
> already available metrics capture mechanism (available for Region server and
> Master server processes) to dump some *custom *metrics into a file (using
> the TimestampedFileContext) for downstream consumption by our graphing
> solution. These custom metrics could be like # of minor compactions that
> happened since the last poll time , or the rate of minor compactions over a
> time window etc.
>
> In other words,I was wondering if HBase monitoring framework supports some
> kind of plugin development.
> So, I'm not really looking at building a brand new monitoring solution, but
> rather a way to extend the available set of metrics.
>
> Does that make sense or do you think the available set of metrics (for
> RegionServer,Master Server) are good enough.
>
> -thanks
> tb
> PS : The available set of metrics, for e.g. the RegionServer are like Block
> cache metrics, compaction metrics,IO, store metrics etc
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/Programmatic-cluster-monitoring-How-to-use-the-HBase-monitoring-APIs-tp4032453p4032495.html
> Sent from the HBase User mailing list archive at Nabble.com.

[techbuddy]

Thanks for that link Otis!  This indeed allows completely overriding the default monitoring by Hbase, however what we are looking at really is capturing some additional metrics over and above what the monitoring is already generating.

So, we figured a way to achieve that through co-processors as they seem to nicely fit into our requirements, which is like capturing metrics for example # of compactions that got triggered in a time window. The interface http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html exposes a bunch of callbacks on important internal events as well as client API calls (get's, put's, delete's etc). We are planning to log entries (into the same metrics file where hbase monitoring framework writes the stats to through the FileContext/TimestampedFileContext class) on some of these internal events, and use our homegrown graphing solution to publish the reports.

[stack-3]

On Mon, Oct 1, 2012 at 10:52 PM, techbuddy <[hidden email]> wrote:
> As for programmatic monitoring, I was trying to figure out how to extend the
> already available metrics capture mechanism (available for Region server and
> Master server processes) to dump some *custom *metrics into a file (using
> the TimestampedFileContext) for downstream consumption by our graphing
> solution. These custom metrics could be like # of minor compactions that
> happened since the last poll time , or the rate of minor compactions over a
> time window etc.
>

These sound like metrics that are of general interest.  Consider
making a patch and we'll integrate it (be careful -- metrics is partly
revamped moved to hadoop metrics2... would be great if you could add
metrics2 format rather than the old stuff).

> In other words,I was wondering if HBase monitoring framework supports some
> kind of plugin development.

No.

There has been a paucity up to this but it is improving w/ 0.94 and beyond.

> Does that make sense or do you think the available set of metrics (for
> RegionServer,Master Server) are good enough.
>

No.


St.Ack