Quantcast

[Stand alone - distributed mode] HBase master isn't initializing completely

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Stand alone - distributed mode] HBase master isn't initializing completely

techbuddy
Hi,

I've a standalone Hbase cluster configured in a distributed mode (i.e. the ZK, Master and the RegServer all running in separate JVMs on the same host).

The HBase master doesn't seem to be intializing successfully. This has started happening since I encountered the stop-hbase script going into an indefinite loop followed by a forced kill of the three Hbase processes (Zk,Master and Region server). I did manually kill these processes couple of times earlier too, but the Hbase master never stalled partially initialized. I have also removed the zookeeper and the hbase/test/hbase folders in an effort to clean things up before triggering the start-hbase script.

Symptoms

1. The create table command on the Hbase shell errors out with a PleaseHoldException (Master in initializing)

2. On the Admin UI, the regon server lists only the ROOT and the single META regions. It doesn't list one single region of the user tables. However ,a list command on the shell indeed lists all the user tables (that were created in the past)

3. On the Master logs, I see the following locate region failures and retries on the META table

2012-10-11 14:09:03,970 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=materialstranger.corp.XXX.com, port=60020}, attempt=77 of 100 failed; retrying after sleep of 32172 because: Connection refused
2012-10-11 14:09:03,971 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc; serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc; serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=materialstranger.corp.XXX.com, port=60020}, attempt=78 of 100 failed; retrying after sleep of 32275 because: Connection refused
2012-10-11 14:09:36,198 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc; serverName=materialstranger.corp.XXX.com,60020,1349987514876


Any pointers on how to restore the cluster to a clean ,working state would be highly appreciated.

-SB


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

答复: [Stand alone - distributed mode] HBase master isn't initializing completely

谢良
Is there any WARNING/ERROR in HDFS logfile ?
Please ensure zk&hdfs are in healthy status firstly
Could you provide your hbse version, it'll be great:)
________________________________________
发件人: techbuddy [[hidden email]]
发送时间: 2012年10月12日 5:11
收件人: [hidden email]
主题: [Stand alone - distributed mode] HBase master isn't initializing completely

Hi,

I've a standalone Hbase cluster configured in a distributed mode (i.e. the
ZK, Master and the RegServer all running in separate JVMs on the same host).

The HBase master doesn't seem to be intializing successfully. This has
started happening since I encountered the stop-hbase script going into an
indefinite loop followed by a forced kill of the three Hbase processes
(Zk,Master and Region server). I did manually kill these processes couple of
times earlier too, but the Hbase master never stalled partially initialized.
I have also removed the zookeeper and the hbase/test/hbase folders in an
effort to clean things up before triggering the start-hbase script.

Symptoms

1. The create table command on the Hbase shell errors out with a
PleaseHoldException (Master in initializing)

2. On the Admin UI, the regon server lists only the ROOT and the single META
regions. It doesn't list one single region of the user tables. However ,a
list command on the shell indeed lists all the user tables (that were
created in the past)

3. On the Master logs, I see the following locate region failures and
retries on the META table

2012-10-11 14:09:03,970 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=materialstranger.corp.XXX.com, port=60020}, attempt=77 of 100
failed; retrying after sleep of 32172 because: Connection refused
2012-10-11 14:09:03,971 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876
2012-10-11 14:09:36,197 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=materialstranger.corp.XXX.com, port=60020}, attempt=78 of 100
failed; retrying after sleep of 32275 because: Connection refused
2012-10-11 14:09:36,198 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location,
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@1bbbafc;
serverName=materialstranger.corp.XXX.com,60020,1349987514876


Any pointers on how to restore the cluster to a clean ,working state would
be highly appreciated.

-SB






--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869.html
Sent from the HBase User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 答复: [Stand alone - distributed mode] HBase master isn't initializing completely

techbuddy
I'm using the local filesystem for the underlying filesystem of hbase, and the Hbase version is 0.94.0
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 答复: [Stand alone - distributed mode] HBase master isn't initializing completely

Suraj Varma
Shutdown the cluster and remove all the *.pid files from the
configured tmp.dir - especially of the master. Then bring up the
cluster again. See if this resolves it.

Also - did you check your hosts file and refer to the host
configuration specified in the hbase online guide?
--S

On Fri, Oct 12, 2012 at 12:27 AM, techbuddy <[hidden email]> wrote:
> I'm using the local filesystem for the underlying filesystem of hbase, and
> the Hbase version is 0.94.0
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/Stand-alone-distributed-mode-HBase-master-isn-t-initializing-completely-tp4032869p4032891.html
> Sent from the HBase User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Stand alone - distributed mode] HBase master isn't initializing completely

abhishek1015
In reply to this post by techbuddy
I am facing same problem.

@techbuddy: Is this problem resolved?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Stand alone - distributed mode] HBase master isn't initializing completely

techbuddy
We root caused the problem to corruption of the .META. table data,that was being caused by a bug in one of our co-processors.
The co-processor was returning a null internalScanner in one of the RegionObserver callbacks.

I'd suggest you run the hbck tool, that does an hbase file system check, if your cluster is starting up. If not, try running the OfflineMetaRepair tool. Also, visit the master and region server logs for more information.


On Sat, Jun 29, 2013 at 9:59 PM, abhishek1015 [via Apache HBase] <[hidden email]> wrote:
I am facing same problem.

@techbuddy: Is this problem resolved?


To unsubscribe from [Stand alone - distributed mode] HBase master isn't initializing completely, click here.
NAML

Loading...