What is the cause for RegionTooBusyException?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

What is the cause for RegionTooBusyException?

jeff saremi
We're getting errors like this. Where should we be looking into to solve this?


Failed 69261 actions: RegionTooBusyException: 12695 times, RemoteWithExtrasException: 56566 times

thanks

Jeff

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

Ted Yu-3
Have you checked region server log ?
Please take a look at the following method in HRegion:

  private void checkResources() throws RegionTooBusyException {

...

    if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {

      blockedRequestsCount.increment();

      requestFlush();

      throw new RegionTooBusyException("Above memstore limit, " +

Which hbase release are you using ?

Cheers

On Fri, May 19, 2017 at 3:59 PM, jeff saremi <[hidden email]> wrote:

> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

James Moore
In reply to this post by jeff saremi
That error appears to be coming from a batch call, 12695 out of 69261
operations failed with a RegionTooBusyException Some of the causes can be.

1. A full MemStore such as if you write to the MemStore faster than it can
flush or if it's too small to fit incoming writes
2. Too many storefiles for the region without compaction.  The default
number of store files which will start blocking inbound writes is 15
3. If the region is in the process of being closed but has long running
operations such as a scanner fetch active.

You should be able to see some additional info for the exact cause of the
exception in your region servers logs or in the message of a specific
RegionTooBusy exception.

cheers,

--James

On Fri, May 19, 2017 at 6:59 PM, jeff saremi <[hidden email]> wrote:

> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

jeff saremi
thanks James for the hints

________________________________
From: James Moore <[hidden email]>
Sent: Friday, May 19, 2017 7:42:02 PM
To: [hidden email]
Subject: Re: What is the cause for RegionTooBusyException?

That error appears to be coming from a batch call, 12695 out of 69261
operations failed with a RegionTooBusyException Some of the causes can be.

1. A full MemStore such as if you write to the MemStore faster than it can
flush or if it's too small to fit incoming writes
2. Too many storefiles for the region without compaction.  The default
number of store files which will start blocking inbound writes is 15
3. If the region is in the process of being closed but has long running
operations such as a scanner fetch active.

You should be able to see some additional info for the exact cause of the
exception in your region servers logs or in the message of a specific
RegionTooBusy exception.

cheers,

--James

On Fri, May 19, 2017 at 6:59 PM, jeff saremi <[hidden email]> wrote:

> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

jeff saremi
In reply to this post by Ted Yu-3
Thanks Ted. I will look deeper as you suggested

________________________________
From: Ted Yu <[hidden email]>
Sent: Friday, May 19, 2017 4:18:12 PM
To: [hidden email]
Subject: Re: What is the cause for RegionTooBusyException?

Have you checked region server log ?
Please take a look at the following method in HRegion:

  private void checkResources() throws RegionTooBusyException {

...

    if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {

      blockedRequestsCount.increment();

      requestFlush();

      throw new RegionTooBusyException("Above memstore limit, " +

Which hbase release are you using ?

Cheers

On Fri, May 19, 2017 at 3:59 PM, jeff saremi <[hidden email]> wrote:

> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

jeff saremi
while I'm still trying to find anything useful in the logs, my question is why isn't HBase self managing this?
In my 2-3 decades of using databases i had never had to stop a write operation to do anything such as compactions or whatever. The only time a write would fail would be if the db runs out of physical storage. The client could slowness due to many writes but throwing exceptions was unheard of.


________________________________
From: jeff saremi <[hidden email]>
Sent: Friday, May 19, 2017 8:18:59 PM
To: [hidden email]
Subject: Re: What is the cause for RegionTooBusyException?

Thanks Ted. I will look deeper as you suggested

________________________________
From: Ted Yu <[hidden email]>
Sent: Friday, May 19, 2017 4:18:12 PM
To: [hidden email]
Subject: Re: What is the cause for RegionTooBusyException?

Have you checked region server log ?
Please take a look at the following method in HRegion:

  private void checkResources() throws RegionTooBusyException {

...

    if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {

      blockedRequestsCount.increment();

      requestFlush();

      throw new RegionTooBusyException("Above memstore limit, " +

Which hbase release are you using ?

Cheers

On Fri, May 19, 2017 at 3:59 PM, jeff saremi <[hidden email]> wrote:

> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

stack-3
On Mon, May 22, 2017 at 9:31 AM, jeff saremi <[hidden email]> wrote:

> while I'm still trying to find anything useful in the logs, my question is
> why isn't HBase self managing this?
>

It should do better here, yes (I thought TooBusy retried but I am not
finding it at the mo.). Exception is thrown for such as the reasons James
lists -- in essence out of resources --  including the case where we fail
to obtain lock inside the configured timeouts (row lock on write or region
lock doing bulk load). As James notes, you should see the too busy dumped
into the regionserver log at time of issue. Having this, you can figure
what resource is crimped. Is there no more detail on client side on the
root of the TooBusy exceptions?


Thanks,
S



>
> ________________________________
> From: jeff saremi <[hidden email]>
> Sent: Friday, May 19, 2017 8:18:59 PM
> To: [hidden email]
> Subject: Re: What is the cause for RegionTooBusyException?
>
> Thanks Ted. I will look deeper as you suggested
>
> ________________________________
> From: Ted Yu <[hidden email]>
> Sent: Friday, May 19, 2017 4:18:12 PM
> To: [hidden email]
> Subject: Re: What is the cause for RegionTooBusyException?
>
> Have you checked region server log ?
> Please take a look at the following method in HRegion:
>
>   private void checkResources() throws RegionTooBusyException {
>
> ...
>
>     if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
>
>       blockedRequestsCount.increment();
>
>       requestFlush();
>
>       throw new RegionTooBusyException("Above memstore limit, " +
>
> Which hbase release are you using ?
>
> Cheers
>
> On Fri, May 19, 2017 at 3:59 PM, jeff saremi <[hidden email]>
> wrote:
>
> > We're getting errors like this. Where should we be looking into to solve
> > this?
> >
> >
> > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > RemoteWithExtrasException: 56566 times
> >
> > thanks
> >
> > Jeff
> >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

jeff saremi
Hi Stack

no there are no details in the exception. I mentioned that in another thread. When you perform a Batch operation, I believe no details will be communicated. I am not sure about individual Put's though. That makes it hard to go through logs cause we don't know out of hundreds of RS's which logs to look at

I have an issue with this exception being thrown period. I think the resource management needs a lot of work. I will soon post another note about my impression of this whole thing.

Jeff

________________________________
From: [hidden email] <[hidden email]> on behalf of Stack <[hidden email]>
Sent: Friday, May 26, 2017 12:05:36 AM
To: Hbase-User
Subject: Re: What is the cause for RegionTooBusyException?

On Mon, May 22, 2017 at 9:31 AM, jeff saremi <[hidden email]> wrote:

> while I'm still trying to find anything useful in the logs, my question is
> why isn't HBase self managing this?
>

It should do better here, yes (I thought TooBusy retried but I am not
finding it at the mo.). Exception is thrown for such as the reasons James
lists -- in essence out of resources --  including the case where we fail
to obtain lock inside the configured timeouts (row lock on write or region
lock doing bulk load). As James notes, you should see the too busy dumped
into the regionserver log at time of issue. Having this, you can figure
what resource is crimped. Is there no more detail on client side on the
root of the TooBusy exceptions?


Thanks,
S



>
> ________________________________
> From: jeff saremi <[hidden email]>
> Sent: Friday, May 19, 2017 8:18:59 PM
> To: [hidden email]
> Subject: Re: What is the cause for RegionTooBusyException?
>
> Thanks Ted. I will look deeper as you suggested
>
> ________________________________
> From: Ted Yu <[hidden email]>
> Sent: Friday, May 19, 2017 4:18:12 PM
> To: [hidden email]
> Subject: Re: What is the cause for RegionTooBusyException?
>
> Have you checked region server log ?
> Please take a look at the following method in HRegion:
>
>   private void checkResources() throws RegionTooBusyException {
>
> ...
>
>     if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
>
>       blockedRequestsCount.increment();
>
>       requestFlush();
>
>       throw new RegionTooBusyException("Above memstore limit, " +
>
> Which hbase release are you using ?
>
> Cheers
>
> On Fri, May 19, 2017 at 3:59 PM, jeff saremi <[hidden email]>
> wrote:
>
> > We're getting errors like this. Where should we be looking into to solve
> > this?
> >
> >
> > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > RemoteWithExtrasException: 56566 times
> >
> > thanks
> >
> > Jeff
> >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

James Moore
One mechanism for revealing the error in question is to print one of the
individual exceptions which are included in the batch calls response.  We
use this in a few places to allow inspection of individual Exceptions you
can see an example of how to do this over here
https://github.com/apache/hbase/blob/master/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java#L1222


On Fri, May 26, 2017 at 12:47 PM, jeff saremi <[hidden email]>
wrote:

> Hi Stack
>
> no there are no details in the exception. I mentioned that in another
> thread. When you perform a Batch operation, I believe no details will be
> communicated. I am not sure about individual Put's though. That makes it
> hard to go through logs cause we don't know out of hundreds of RS's which
> logs to look at
>
> I have an issue with this exception being thrown period. I think the
> resource management needs a lot of work. I will soon post another note
> about my impression of this whole thing.
>
> Jeff
>
> ________________________________
> From: [hidden email] <[hidden email]> on behalf of Stack <
> [hidden email]>
> Sent: Friday, May 26, 2017 12:05:36 AM
> To: Hbase-User
> Subject: Re: What is the cause for RegionTooBusyException?
>
> On Mon, May 22, 2017 at 9:31 AM, jeff saremi <[hidden email]>
> wrote:
>
> > while I'm still trying to find anything useful in the logs, my question
> is
> > why isn't HBase self managing this?
> >
>
> It should do better here, yes (I thought TooBusy retried but I am not
> finding it at the mo.). Exception is thrown for such as the reasons James
> lists -- in essence out of resources --  including the case where we fail
> to obtain lock inside the configured timeouts (row lock on write or region
> lock doing bulk load). As James notes, you should see the too busy dumped
> into the regionserver log at time of issue. Having this, you can figure
> what resource is crimped. Is there no more detail on client side on the
> root of the TooBusy exceptions?
>
>
> Thanks,
> S
>
>
>
> >
> > ________________________________
> > From: jeff saremi <[hidden email]>
> > Sent: Friday, May 19, 2017 8:18:59 PM
> > To: [hidden email]
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Thanks Ted. I will look deeper as you suggested
> >
> > ________________________________
> > From: Ted Yu <[hidden email]>
> > Sent: Friday, May 19, 2017 4:18:12 PM
> > To: [hidden email]
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Have you checked region server log ?
> > Please take a look at the following method in HRegion:
> >
> >   private void checkResources() throws RegionTooBusyException {
> >
> > ...
> >
> >     if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
> >
> >       blockedRequestsCount.increment();
> >
> >       requestFlush();
> >
> >       throw new RegionTooBusyException("Above memstore limit, " +
> >
> > Which hbase release are you using ?
> >
> > Cheers
> >
> > On Fri, May 19, 2017 at 3:59 PM, jeff saremi <[hidden email]>
> > wrote:
> >
> > > We're getting errors like this. Where should we be looking into to
> solve
> > > this?
> > >
> > >
> > > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > > RemoteWithExtrasException: 56566 times
> > >
> > > thanks
> > >
> > > Jeff
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

jeff saremi
@James, thank you very much. That was extremely helpful

________________________________
From: James Moore <[hidden email]>
Sent: Friday, May 26, 2017 10:24:42 AM
To: [hidden email]
Subject: Re: What is the cause for RegionTooBusyException?

One mechanism for revealing the error in question is to print one of the
individual exceptions which are included in the batch calls response.  We
use this in a few places to allow inspection of individual Exceptions you
can see an example of how to do this over here
https://github.com/apache/hbase/blob/master/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java#L1222


On Fri, May 26, 2017 at 12:47 PM, jeff saremi <[hidden email]>
wrote:

> Hi Stack
>
> no there are no details in the exception. I mentioned that in another
> thread. When you perform a Batch operation, I believe no details will be
> communicated. I am not sure about individual Put's though. That makes it
> hard to go through logs cause we don't know out of hundreds of RS's which
> logs to look at
>
> I have an issue with this exception being thrown period. I think the
> resource management needs a lot of work. I will soon post another note
> about my impression of this whole thing.
>
> Jeff
>
> ________________________________
> From: [hidden email] <[hidden email]> on behalf of Stack <
> [hidden email]>
> Sent: Friday, May 26, 2017 12:05:36 AM
> To: Hbase-User
> Subject: Re: What is the cause for RegionTooBusyException?
>
> On Mon, May 22, 2017 at 9:31 AM, jeff saremi <[hidden email]>
> wrote:
>
> > while I'm still trying to find anything useful in the logs, my question
> is
> > why isn't HBase self managing this?
> >
>
> It should do better here, yes (I thought TooBusy retried but I am not
> finding it at the mo.). Exception is thrown for such as the reasons James
> lists -- in essence out of resources --  including the case where we fail
> to obtain lock inside the configured timeouts (row lock on write or region
> lock doing bulk load). As James notes, you should see the too busy dumped
> into the regionserver log at time of issue. Having this, you can figure
> what resource is crimped. Is there no more detail on client side on the
> root of the TooBusy exceptions?
>
>
> Thanks,
> S
>
>
>
> >
> > ________________________________
> > From: jeff saremi <[hidden email]>
> > Sent: Friday, May 19, 2017 8:18:59 PM
> > To: [hidden email]
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Thanks Ted. I will look deeper as you suggested
> >
> > ________________________________
> > From: Ted Yu <[hidden email]>
> > Sent: Friday, May 19, 2017 4:18:12 PM
> > To: [hidden email]
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Have you checked region server log ?
> > Please take a look at the following method in HRegion:
> >
> >   private void checkResources() throws RegionTooBusyException {
> >
> > ...
> >
> >     if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
> >
> >       blockedRequestsCount.increment();
> >
> >       requestFlush();
> >
> >       throw new RegionTooBusyException("Above memstore limit, " +
> >
> > Which hbase release are you using ?
> >
> > Cheers
> >
> > On Fri, May 19, 2017 at 3:59 PM, jeff saremi <[hidden email]>
> > wrote:
> >
> > > We're getting errors like this. Where should we be looking into to
> solve
> > > this?
> > >
> > >
> > > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > > RemoteWithExtrasException: 56566 times
> > >
> > > thanks
> > >
> > > Jeff
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: What is the cause for RegionTooBusyException?

stack-3
In reply to this post by jeff saremi
On Fri, May 26, 2017 at 9:47 AM, jeff saremi <[hidden email]> wrote:

>
> I have an issue with this exception being thrown period. I think the
> resource management needs a lot of work. I will soon post another note
> about my impression of this whole thing.
>
>
Please. Would appreciate your input Jeff.
Thanks,
St.Ack



> Jeff
>
> ________________________________
> From: [hidden email] <[hidden email]> on behalf of Stack <
> [hidden email]>
> Sent: Friday, May 26, 2017 12:05:36 AM
> To: Hbase-User
> Subject: Re: What is the cause for RegionTooBusyException?
>
> On Mon, May 22, 2017 at 9:31 AM, jeff saremi <[hidden email]>
> wrote:
>
> > while I'm still trying to find anything useful in the logs, my question
> is
> > why isn't HBase self managing this?
> >
>
> It should do better here, yes (I thought TooBusy retried but I am not
> finding it at the mo.). Exception is thrown for such as the reasons James
> lists -- in essence out of resources --  including the case where we fail
> to obtain lock inside the configured timeouts (row lock on write or region
> lock doing bulk load). As James notes, you should see the too busy dumped
> into the regionserver log at time of issue. Having this, you can figure
> what resource is crimped. Is there no more detail on client side on the
> root of the TooBusy exceptions?
>
>
> Thanks,
> S
>
>
>
> >
> > ________________________________
> > From: jeff saremi <[hidden email]>
> > Sent: Friday, May 19, 2017 8:18:59 PM
> > To: [hidden email]
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Thanks Ted. I will look deeper as you suggested
> >
> > ________________________________
> > From: Ted Yu <[hidden email]>
> > Sent: Friday, May 19, 2017 4:18:12 PM
> > To: [hidden email]
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Have you checked region server log ?
> > Please take a look at the following method in HRegion:
> >
> >   private void checkResources() throws RegionTooBusyException {
> >
> > ...
> >
> >     if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
> >
> >       blockedRequestsCount.increment();
> >
> >       requestFlush();
> >
> >       throw new RegionTooBusyException("Above memstore limit, " +
> >
> > Which hbase release are you using ?
> >
> > Cheers
> >
> > On Fri, May 19, 2017 at 3:59 PM, jeff saremi <[hidden email]>
> > wrote:
> >
> > > We're getting errors like this. Where should we be looking into to
> solve
> > > this?
> > >
> > >
> > > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > > RemoteWithExtrasException: 56566 times
> > >
> > > thanks
> > >
> > > Jeff
> > >
> > >
> >
>
Loading...