Quantcast

[Programmatic cluster monitoring] How to use the HBase monitoring APIs

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Programmatic cluster monitoring] How to use the HBase monitoring APIs

techbuddy
Hi all Hbase veterans, gurus, newbies,

I'm trying to figure out how to go about programmatically monitoring an Hbase cluster using the APIs listed @
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/monitoring/package-summary.html. I couldn't land any webdoc explaining how to use these APIs.. I know about the monitoring alternatives of using Ganglia ,JMX etc. But surprisingly, I couldn't retrieve any documentation/tutorial on using the above APIs..
Any pointers would be highly appreciated.

-thanks
Santosh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Programmatic cluster monitoring] How to use the HBase monitoring APIs

stack-3
On Mon, Oct 1, 2012 at 3:17 PM, techbuddy <[hidden email]> wrote:

> Hi all Hbase veterans, gurus, newbies,
>
> I'm trying to figure out how to go about programmatically monitoring an
> Hbase cluster using the APIs listed @
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/monitoring/package-summary.html.
> I couldn't land any webdoc explaining how to use these APIs.. I know about
> the monitoring alternatives of using Ganglia ,JMX etc. But surprisingly, I
> couldn't retrieve any documentation/tutorial on using the above APIs..
> Any pointers would be highly appreciated.
>

If you look at the javadoc you'll see they all have the annotation:

@InterfaceAudience.Private

This is how we flag stuff that is for server-side only, not for public
consumption.

These classes are used by the server variously adding stuff to UIs, etc.

Regards monitoring stuff programmatically, what do you mean?  Why
would you want to write a monitoring tool when there are already
myriad?  If you insist, you could do something like grab the json
content of the /jmx servlet run on each server or you could enable
metrics and catch the firehose by parsing ganglia formatter messages
or the text form you can have dumped to a file, etc.

Let us know more what you need so we can help you out.

St.Ack
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Programmatic cluster monitoring] How to use the HBase monitoring APIs

techbuddy
Thanks for clarifying the usage part out Stack!

As for programmatic monitoring, I was trying to figure out how to extend the already available metrics capture mechanism (available for Region server and Master server processes) to dump some custom metrics into a file (using the TimestampedFileContext) for downstream consumption by our graphing solution. These custom metrics could be like # of minor compactions that happened since the last poll time , or the rate of minor compactions over a time window etc.

In other words,I was wondering if HBase monitoring framework supports some kind of plugin development.
So, I'm not really looking at building a brand new monitoring solution, but rather a way to extend the available set of metrics.

Does that make sense or do you think the available set of metrics (for RegionServer,Master Server) are good enough.

-thanks
tb
PS : The available set of metrics, for e.g. the RegionServer are like Block cache metrics, compaction metrics,IO, store metrics etc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Programmatic cluster monitoring] How to use the HBase monitoring APIs

Otis Gospodnetic
Hi,

Have a look at
https://github.com/sematext/HBaseMetricsContext  +
http://blog.sematext.com/2011/07/31/extending-hadoop-metrics/ -- this
may lead you in the right direction if you really really need to do
this.... although I'm not sure if that stuff is outdated now.

If you are just trying to get existing HBase metrics there is SPM
(http://sematext.com/spm/index.html), Ganglia, etc.

Otis


On Tue, Oct 2, 2012 at 1:52 AM, techbuddy <[hidden email]> wrote:

> Thanks for clarifying the usage part out Stack!
>
> As for programmatic monitoring, I was trying to figure out how to extend the
> already available metrics capture mechanism (available for Region server and
> Master server processes) to dump some *custom *metrics into a file (using
> the TimestampedFileContext) for downstream consumption by our graphing
> solution. These custom metrics could be like # of minor compactions that
> happened since the last poll time , or the rate of minor compactions over a
> time window etc.
>
> In other words,I was wondering if HBase monitoring framework supports some
> kind of plugin development.
> So, I'm not really looking at building a brand new monitoring solution, but
> rather a way to extend the available set of metrics.
>
> Does that make sense or do you think the available set of metrics (for
> RegionServer,Master Server) are good enough.
>
> -thanks
> tb
> PS : The available set of metrics, for e.g. the RegionServer are like Block
> cache metrics, compaction metrics,IO, store metrics etc
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/Programmatic-cluster-monitoring-How-to-use-the-HBase-monitoring-APIs-tp4032453p4032495.html
> Sent from the HBase User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Programmatic cluster monitoring] How to use the HBase monitoring APIs

techbuddy
Thanks for that link Otis!  This indeed allows completely overriding the default monitoring by Hbase, however what we are looking at really is capturing some additional metrics over and above what the monitoring is already generating.

So, we figured a way to achieve that through co-processors as they seem to nicely fit into our requirements, which is like capturing metrics for example # of compactions that got triggered in a time window. The interface http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html exposes a bunch of callbacks on important internal events as well as client API calls (get's, put's, delete's etc). We are planning to log entries (into the same metrics file where hbase monitoring framework writes the stats to through the FileContext/TimestampedFileContext class) on some of these internal events, and use our homegrown graphing solution to publish the reports.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Programmatic cluster monitoring] How to use the HBase monitoring APIs

stack-3
In reply to this post by techbuddy
On Mon, Oct 1, 2012 at 10:52 PM, techbuddy <[hidden email]> wrote:
> As for programmatic monitoring, I was trying to figure out how to extend the
> already available metrics capture mechanism (available for Region server and
> Master server processes) to dump some *custom *metrics into a file (using
> the TimestampedFileContext) for downstream consumption by our graphing
> solution. These custom metrics could be like # of minor compactions that
> happened since the last poll time , or the rate of minor compactions over a
> time window etc.
>

These sound like metrics that are of general interest.  Consider
making a patch and we'll integrate it (be careful -- metrics is partly
revamped moved to hadoop metrics2... would be great if you could add
metrics2 format rather than the old stuff).

> In other words,I was wondering if HBase monitoring framework supports some
> kind of plugin development.

No.

There has been a paucity up to this but it is improving w/ 0.94 and beyond.

> Does that make sense or do you think the available set of metrics (for
> RegionServer,Master Server) are good enough.
>

No.


St.Ack
Loading...