Quantcast

Major compaction issues in hbase

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Major compaction issues in hbase

yanivG
Hi,
I am using hbase 0.94.6 version. We are developing a java application and using it as our storage for a high load web application.
The average ration is about 500 requests per second.
From time to time I see high latency (more then 300 ms!!!) when trying to store data object within hbase (using Put).

We disabled major compaction and set it to run manually once a week.
By examining the logs during those high latency , I see that major compaction was executed.
I saw a post (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM) which explains it.
The question is how can I avoid this? Should I ran the major compaction once a day in low load hours? will it eliminate the sporadic major compaction?
Thanks,
Y
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Major compaction issues in hbase

Ted Yu-3
You can use off peak compaction: https://issues.apache.org/jira/browse/HBASE-4463

See also http://www.slideshare.net/cloudera/hbasecon-2013-compaction-improvements-in-apache-hbase

On Feb 2, 2014, at 1:14 PM, yanivG <[hidden email]> wrote:

> Hi,
> I am using hbase 0.94.6 version. We are developing a java application and
> using it as our storage for a high load web application.
> The average ration is about 500 requests per second.
> From time to time I see high latency (more then 300 ms!!!) when trying to
> store data object within hbase (using Put).
>
> We disabled major compaction and set it to run manually once a week.
> By examining the logs during those high latency , I see that major
> compaction was executed.
> I saw a post
> (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM) which
> explains it.
> The question is how can I avoid this? Should I ran the major compaction once
> a day in low load hours? will it eliminate the sporadic major compaction?
> Thanks,
> Y
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Major compaction issues in hbase

Jean-Marc Spaggiari
In reply to this post by yanivG
Hi Yaniv,

When there is more than 3 files for a region (default setting), HBase will
trigger a minor compaction for this region. However, if all the regions
need to be compacted, then it will promote it as a major compaction. So
even by disabling major compaction so can still see some of it in the logs.

If you can run major compaction one a day it's better than once a week.
Also, this might be related to your design (too many column families?),
your usage and your configuration (compaction triggers, memory
configuration, etc.).

JM


2014-02-02 yanivG <[hidden email]>:

> Hi,
> I am using hbase 0.94.6 version. We are developing a java application and
> using it as our storage for a high load web application.
> The average ration is about 500 requests per second.
> From time to time I see high latency (more then 300 ms!!!) when trying to
> store data object within hbase (using Put).
>
> We disabled major compaction and set it to run manually once a week.
> By examining the logs during those high latency , I see that major
> compaction was executed.
> I saw a post
> (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM)
> which
> explains it.
> The question is how can I avoid this? Should I ran the major compaction
> once
> a day in low load hours? will it eliminate the sporadic major compaction?
> Thanks,
> Y
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Major compaction issues in hbase

Kevin O'dell
In reply to this post by yanivG
Hi Y,

  That may help, but a coulple quick questions:

Are you doing any bulk loads?

What is your avg flush size through out the day?

After a day before how many storefiles have you built up?

I would look at your memory sizing along with scheduling daily compactions.


On Sun, Feb 2, 2014 at 4:14 PM, yanivG <[hidden email]> wrote:

> Hi,
> I am using hbase 0.94.6 version. We are developing a java application and
> using it as our storage for a high load web application.
> The average ration is about 500 requests per second.
> From time to time I see high latency (more then 300 ms!!!) when trying to
> store data object within hbase (using Put).
>
> We disabled major compaction and set it to run manually once a week.
> By examining the logs during those high latency , I see that major
> compaction was executed.
> I saw a post
> (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM)
> which
> explains it.
> The question is how can I avoid this? Should I ran the major compaction
> once
> a day in low load hours? will it eliminate the sporadic major compaction?
> Thanks,
> Y
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>



--
Kevin O'Dell
Systems Engineer, Cloudera
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Major compaction issues in hbase

Vladimir Rodionov-2
I think, the major issue here - not a sudden major compaction, but why
everything, literally, stalls when major compaction kicks off? Is there any
clear explanation, somewhere, how does major compaction can affect tput,
latency, availability of HBase? 300ms latency is possible, of course , as
an rare outlier, but if its 90% percentile - its totally different story.

-Vladimir


On Sun, Feb 2, 2014 at 1:49 PM, Kevin O'dell <[hidden email]>wrote:

> Hi Y,
>
>   That may help, but a coulple quick questions:
>
> Are you doing any bulk loads?
>
> What is your avg flush size through out the day?
>
> After a day before how many storefiles have you built up?
>
> I would look at your memory sizing along with scheduling daily compactions.
>
>
> On Sun, Feb 2, 2014 at 4:14 PM, yanivG <[hidden email]> wrote:
>
> > Hi,
> > I am using hbase 0.94.6 version. We are developing a java application and
> > using it as our storage for a high load web application.
> > The average ration is about 500 requests per second.
> > From time to time I see high latency (more then 300 ms!!!) when trying to
> > store data object within hbase (using Put).
> >
> > We disabled major compaction and set it to run manually once a week.
> > By examining the logs during those high latency , I see that major
> > compaction was executed.
> > I saw a post
> > (https://groups.google.com/forum/#!topic/nosql-databases/YwE894gq1qM)
> > which
> > explains it.
> > The question is how can I avoid this? Should I ran the major compaction
> > once
> > a day in low load hours? will it eliminate the sporadic major compaction?
> > Thanks,
> > Y
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592.html
> > Sent from the HBase Developer mailing list archive at Nabble.com.
> >
>
>
>
> --
> Kevin O'Dell
> Systems Engineer, Cloudera
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Major compaction issues in hbase

yanivG
In reply to this post by Kevin O'dell
Hi Kevin,
1. No I am not doing bulk loads.
2. I don't know. Where can I see this metric? the load is about 200 requests per second * 5K of data.
3. I don't know.

Memory sizing? You mean heap size, etc?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Major compaction issues in hbase

yanivG
In reply to this post by Jean-Marc Spaggiari
JM,
Can you elaborate on: "Also, this might be related to your design (too many column families?),
your usage and your configuration (compaction triggers, memory
configuration, etc.). "?

We have 3 CF.
Compaction triggers?
memory configuration - do you mean GC settings?

Y
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Major compaction issues in hbase

Nick Dimiduk
In reply to this post by yanivG
"Memory sizing" generally means how much RAM you have allocated to HBase.
The amount of heap, the total size of your memstores and the number of
them. This will impact the size of an individual memstore and thus the size
of the new HFiles flushed to disk. This impacts the frequency and character
of compactions. More column families => more memstores => (usually) smaller
flush sizes => more compactions.


On Mon, Feb 3, 2014 at 4:34 AM, yanivG <[hidden email]> wrote:

> Hi Kevin,
> 1. No I am not doing bulk loads.
> 2. I don't know. Where can I see this metric? the load is about 200
> requests
> per second * 5K of data.
> 3. I don't know.
>
> Memory sizing? You mean heap size, etc?
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Major-compaction-issues-in-hbase-tp4055592p4055604.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>
Loading...