How to get useful info from Client exceptions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to get useful info from Client exceptions

jeff saremi
We get errors like below which are not helping at all.

For instance in this case we have no clue what server/region it is talking about

Is there a setting we missed or somehow we could augment this information to help us? We're using hbase 1.2.5


Failed 10456 actions: UnknownHostException: 10456 times,
Stack Trace: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 10456 actions: UnknownHostException: 10456 times,
        at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:258)
        at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$2000(AsyncProcess.java:238)
        at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1817)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240)
        at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:146)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1028)

thanks



Reply | Threaded
Open this post in threaded view
|

Re: How to get useful info from Client exceptions

Ted Yu-3
The exception is composed this way (see AsyncProcess):

      return new RetriesExhaustedWithDetailsException(new
ArrayList<Throwable>(throwables),
          new ArrayList<Row>(actions), new ArrayList<String>(addresses));

In your case there were 10456 UnknownHostException's.

Was there any other clue in client side log ?
I think one improvement we can do is selectively logging which host(s) was
involved in the UnknownHostException's.

BTW was hbase-site.xml on the classpath of your client ?

On Tue, May 23, 2017 at 3:28 PM, jeff saremi <[hidden email]> wrote:

> We get errors like below which are not helping at all.
>
> For instance in this case we have no clue what server/region it is talking
> about
>
> Is there a setting we missed or somehow we could augment this information
> to help us? We're using hbase 1.2.5
>
>
> Failed 10456 actions: UnknownHostException: 10456 times,
> Stack Trace: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 10456 actions: UnknownHostException: 10456 times,
>         at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.makeException(AsyncProcess.java:258)
>         at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.access$2000(AsyncProcess.java:238)
>         at org.apache.hadoop.hbase.client.AsyncProcess.
> waitForAllPreviousOpsAndReset(AsyncProcess.java:1817)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.
> backgroundFlushCommits(BufferedMutatorImpl.java:240)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.
> mutate(BufferedMutatorImpl.java:146)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1028)
>
> thanks
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: How to get useful info from Client exceptions

jeff saremi
Thansk Ted
We might have a stale hbase-site.xml deployed along the code which runs on Spark.

however what we set is the Zookeeper quorum for the right cluster.

I was told that the third Batch starts to throw errors like these and the first two are fine. But then again I'm not running this myself

We're going to increase the log level to DEBUG to see if we see anything.

________________________________
From: Ted Yu <[hidden email]>
Sent: Tuesday, May 23, 2017 3:49:27 PM
To: [hidden email]
Subject: Re: How to get useful info from Client exceptions

The exception is composed this way (see AsyncProcess):

      return new RetriesExhaustedWithDetailsException(new
ArrayList<Throwable>(throwables),
          new ArrayList<Row>(actions), new ArrayList<String>(addresses));

In your case there were 10456 UnknownHostException's.

Was there any other clue in client side log ?
I think one improvement we can do is selectively logging which host(s) was
involved in the UnknownHostException's.

BTW was hbase-site.xml on the classpath of your client ?

On Tue, May 23, 2017 at 3:28 PM, jeff saremi <[hidden email]> wrote:

> We get errors like below which are not helping at all.
>
> For instance in this case we have no clue what server/region it is talking
> about
>
> Is there a setting we missed or somehow we could augment this information
> to help us? We're using hbase 1.2.5
>
>
> Failed 10456 actions: UnknownHostException: 10456 times,
> Stack Trace: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 10456 actions: UnknownHostException: 10456 times,
>         at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.makeException(AsyncProcess.java:258)
>         at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.access$2000(AsyncProcess.java:238)
>         at org.apache.hadoop.hbase.client.AsyncProcess.
> waitForAllPreviousOpsAndReset(AsyncProcess.java:1817)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.
> backgroundFlushCommits(BufferedMutatorImpl.java:240)
>         at org.apache.hadoop.hbase.client.BufferedMutatorImpl.
> mutate(BufferedMutatorImpl.java:146)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1028)
>
> thanks
>
>
>
>