hbase-master-server slept
Locked
 
|
Our hbase-master-server was shutdown with following message.
Hbase is runnig in Distributed mode in a single node. I checked that GC completed in a very short time at the time of output the WARN. In addition the other system that is running in the same architecture doesn't output the following WARN messsage and works well. So I think that this is not due to a long GC pause. Do you have any idea about the problem? 2013-01-30 03:07:48,582 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 28970ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-01-30 03:07:48,583 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 36902ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-01-30 03:07:48,585 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 39989ms for sessionid 0x13c84cebfce0000, closing socket connection and attempting reconnect 2013-01-30 03:07:48,586 INFO org.apache.zookeeper.ClientCnxn: Client session timed out, have not heard from server in 39987ms for sessionid 0x13c84cebfce0001, closing socket connection and attempting reconnect 2013-01-30 03:07:52,779 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server VM_11/192.168.152.1:2181 2013-01-30 03:07:52,789 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to VM_11/192.168.152.1:2181, initiating session 2013-01-30 03:07:52,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server VM_11/192.168.152.1:2181 2013-01-30 03:07:52,793 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to VM_11/192.168.152.1:2181, initiating session 2013-01-30 03:07:52,794 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13c84cebfce0001 has expired, closing socket connection 2013-01-30 03:07:52,794 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, trying to reconnect. 2013-01-30 03:07:52,794 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Trying to reconnect to zookeeper. 2013-01-30 03:07:52,795 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=VM_11:2181 sessionTimeout=180000 watcher=hconnection 2013-01-30 03:07:52,812 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x13c84cebfce0000 has expired, closing socket connection 2013-01-30 03:07:52,813 FATAL org.apache.hadoop.hbase.master.HMaster: master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 2013-01-30 03:07:52,813 INFO org.apache.hadoop.hbase.master.HMaster: Aborting 2013-01-30 03:07:52,813 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2013-01-30 03:07:52,813 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server VM_11/192.168.152.1:2181 2013-01-30 03:07:52,814 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to VM_11/192.168.152.1:2181, initiating session 2013-01-30 03:07:52,815 ERROR org.apache.hadoop.hbase.master.HMaster: Region server serverName=VM_11,60020,1359437833300, load=(requests=0, regions=3, usedHeap=45, maxHeap=997) reported a fatal error: ABORTING region server serverName=VM_11,60020,1359437833300, load=(requests=0, regions=3, usedHeap=45, maxHeap=997): regionserver:60020-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002 regionserver:60020-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002 received expired from ZooKeeper, aborting Cause: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 2013-01-30 03:07:52,820 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server VM_11/192.168.152.1:2181, sessionid = 0x13c84cebfce0005, negotiated timeout = 40000 2013-01-30 03:07:52,841 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Reconnected successfully. This disconnect could have been caused by a network partition or a long-running GC pause, either way it's recommended that you verify your environment. 2013-01-30 03:07:52,841 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2013-01-30 03:07:53,614 INFO org.apache.hadoop.hbase.master.LogCleaner: master-VM_11:60000.oldLogCleaner exiting 2013-01-30 03:07:54,251 INFO org.apache.hadoop.hbase.master.HMaster$2: VM_11:60000-BalancerChore exiting 2013-01-30 03:07:54,251 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads 2013-01-30 03:07:54,251 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000 2013-01-30 03:07:54,252 INFO org.apache.hadoop.hbase.master.HMaster: Stopping infoServer 2013-01-30 03:07:54,325 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60010 2013-01-30 03:07:54,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting 2013-01-30 03:07:54,326 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: exiting 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: exiting 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: exiting 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: exiting 2013-01-30 03:07:54,327 INFO org.apache.hadoop.hbase.master.CatalogJanitor: VM_11:60000-CatalogJanitor exiting 2013-01-30 03:07:54,328 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting 2013-01-30 03:07:54,328 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder 2013-01-30 03:07:54,337 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) 2013-01-30 03:07:54,337 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) 2013-01-30 03:07:54,337 ERROR org.apache.hadoop.hbase.master.ActiveMasterManager: master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) 2013-01-30 03:07:54,337 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4743bf3d 2013-01-30 03:07:54,337 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x13c84cebfce0005 has been closed. 2013-01-30 03:07:54,338 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x13c84cebfce0005 2013-01-30 03:07:54,339 INFO org.apache.zookeeper.ZooKeeper: Session: 0x13c84cebfce0005 closed 2013-01-30 03:07:54,339 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to null has been closed. 2013-01-30 03:07:54,339 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting 2013-01-30 03:07:54,339 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2013-01-30 03:07:54,339 INFO org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor: VM_11:60000.timeoutMonitor exiting |
Re: hbase-master-server slept
|
Regards, So,
Can you provide more information about your setup? - HBase version - Hadoop version - Operating System - Java version On 02/08/2013 03:55 AM, So Hibino wrote: > Our hbase-master-server was shutdown with following message. > Hbase is runnig in Distributed mode in a single node. Can you share your .conf files? > I checked that GC completed in a very short time at the time of output the > WARN. > In addition the other system that is running in the same architecture > doesn't output the following WARN messsage and works well. > So I think that this is not due to a long GC pause. > > Do you have any idea about the problem? > > 2013-01-30 03:07:48,582 WARN org.apache.hadoop.hbase.util.Sleeper: We slept > 28970ms instead of 1000ms, this is likely due to a long garbage collecting > pause and it's usually bad, see > http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired Todd wrote a series of posts in Cloudera´s blog about Java Long GC pauses, HBase and Zookeeper. It´s a great read: http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/ http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-2/ > 2013-01-30 03:07:48,583 WARN org.apache.hadoop.hbase.util.Sleeper: We slept > 36902ms instead of 10000ms, this is likely due to a long garbage collecting > pause and it's usually bad, see > http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired > 2013-01-30 03:07:48,585 INFO org.apache.zookeeper.ClientCnxn: Client session > timed out, have not heard from server in 39989ms for sessionid > 0x13c84cebfce0000, closing socket connection and attempting reconnect > 2013-01-30 03:07:48,586 INFO org.apache.zookeeper.ClientCnxn: Client session > timed out, have not heard from server in 39987ms for sessionid > 0x13c84cebfce0001, closing socket connection and attempting reconnect > 2013-01-30 03:07:52,779 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server VM_11/192.168.152.1:2181 > 2013-01-30 03:07:52,789 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to VM_11/192.168.152.1:2181, initiating session > 2013-01-30 03:07:52,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server VM_11/192.168.152.1:2181 > 2013-01-30 03:07:52,793 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to VM_11/192.168.152.1:2181, initiating session > 2013-01-30 03:07:52,794 INFO org.apache.zookeeper.ClientCnxn: Unable to > reconnect to ZooKeeper service, session 0x13c84cebfce0001 has expired, > closing socket connection > 2013-01-30 03:07:52,794 INFO > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > This client just lost it's session with ZooKeeper, trying to reconnect. > 2013-01-30 03:07:52,794 INFO > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Trying to reconnect to zookeeper. > 2013-01-30 03:07:52,795 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, connectString=VM_11:2181 sessionTimeout=180000 > watcher=hconnection > 2013-01-30 03:07:52,812 INFO org.apache.zookeeper.ClientCnxn: Unable to > reconnect to ZooKeeper service, session 0x13c84cebfce0000 has expired, > closing socket connection > 2013-01-30 03:07:52,813 FATAL org.apache.hadoop.hbase.master.HMaster: > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > received expired from ZooKeeper, aborting > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > 2013-01-30 03:07:52,813 INFO org.apache.hadoop.hbase.master.HMaster: > Aborting > 2013-01-30 03:07:52,813 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2013-01-30 03:07:52,813 INFO org.apache.zookeeper.ClientCnxn: Opening socket > connection to server VM_11/192.168.152.1:2181 > 2013-01-30 03:07:52,814 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to VM_11/192.168.152.1:2181, initiating session > 2013-01-30 03:07:52,815 ERROR org.apache.hadoop.hbase.master.HMaster: Region > server serverName=VM_11,60020,1359437833300, load=(requests=0, regions=3, > usedHeap=45, maxHeap=997) reported a fatal error: > ABORTING region server serverName=VM_11,60020,1359437833300, > load=(requests=0, regions=3, usedHeap=45, maxHeap=997): > regionserver:60020-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002 > regionserver:60020-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002 > received expired from ZooKeeper, aborting > Cause: > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > > 2013-01-30 03:07:52,820 INFO org.apache.zookeeper.ClientCnxn: Session > establishment complete on server VM_11/192.168.152.1:2181, sessionid = > 0x13c84cebfce0005, negotiated timeout = 40000 > 2013-01-30 03:07:52,841 INFO > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Reconnected successfully. This disconnect could have been caused by a > network partition or a long-running GC pause, either way it's recommended > that you verify your environment. > 2013-01-30 03:07:52,841 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2013-01-30 03:07:53,614 INFO org.apache.hadoop.hbase.master.LogCleaner: > master-VM_11:60000.oldLogCleaner exiting > 2013-01-30 03:07:54,251 INFO org.apache.hadoop.hbase.master.HMaster$2: > VM_11:60000-BalancerChore exiting > 2013-01-30 03:07:54,251 DEBUG org.apache.hadoop.hbase.master.HMaster: > Stopping service threads > 2013-01-30 03:07:54,251 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60000 > 2013-01-30 03:07:54,252 INFO org.apache.hadoop.hbase.master.HMaster: > Stopping infoServer > 2013-01-30 03:07:54,325 INFO org.mortbay.log: Stopped > SelectChannelConnector@0.0.0.0:60010 > 2013-01-30 03:07:54,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 5 on 60000: exiting > 2013-01-30 03:07:54,326 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC > Server listener on 60000 > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 9 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 8 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 7 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 6 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 4 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 3 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 1 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.hbase.master.CatalogJanitor: > VM_11:60000-CatalogJanitor exiting > 2013-01-30 03:07:54,328 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 0 on 60000: exiting > 2013-01-30 03:07:54,328 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC > Server Responder > 2013-01-30 03:07:54,337 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > Unable to get data of znode /hbase/master > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /hbase/master > at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) > at > org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) > 2013-01-30 03:07:54,337 ERROR > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > Received unexpected KeeperException, re-throwing exception > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /hbase/master > at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) > at > org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) > 2013-01-30 03:07:54,337 ERROR > org.apache.hadoop.hbase.master.ActiveMasterManager: > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > Error deleting our own master address node > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /hbase/master > at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) > at > org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) > 2013-01-30 03:07:54,337 DEBUG > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker > org.apache.hadoop.hbase.catalog.CatalogTracker@4743bf3d > 2013-01-30 03:07:54,337 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > The connection to hconnection-0x13c84cebfce0005 has been closed. > 2013-01-30 03:07:54,338 INFO > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Closed zookeeper sessionid=0x13c84cebfce0005 > 2013-01-30 03:07:54,339 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x13c84cebfce0005 closed > 2013-01-30 03:07:54,339 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > The connection to null has been closed. > 2013-01-30 03:07:54,339 INFO org.apache.hadoop.hbase.master.HMaster: HMaster > main thread exiting > 2013-01-30 03:07:54,339 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2013-01-30 03:07:54,339 INFO > org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor: > VM_11:60000.timeoutMonitor exiting > > > > -- > View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192.html > Sent from the HBase User mailing list archive at Nabble.com. -- Marcos Ortiz Valmaseda, Product Manager && Data Scientist at UCI Blog: http://marcosluis2186.posterous.com Twitter: @marcosluis2186 <http://twitter.com/marcosluis2186> |
Re: hbase-master-server slept
|
On Fri, Feb 8, 2013 at 1:26 AM, Marcos Ortiz <[hidden email]> wrote:
> Regards, So, > Can you provide more information about your setup? > - HBase version > - Hadoop version > - Operating System > - Java version This would be helpful. I would also like to see that GC log please. > Did you check the link? > Todd wrote a series of posts in Cloudera´s blog about Java Long GC pauses, > HBase and Zookeeper. > It´s a great read: > http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/ > http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-2/ The master doesn't have memstores so this wouldn't help. In fact it's pretty rare that we see the master with GC issues. I recall seing issues with time travelling (machine clock's too slow and ntpd resets it) or on EC2 where sometimes you'd see random machine pauses out of nowhere (although that was a long time ago and haven't used EC2 since). J-D |
|
In reply to this post by So Hibino
What zookeeper version are you using ?
Is the ensemble managed by HBase ? Can you check the zookeeper log on 192.168.152.1<http://192.168.152.1:2181/> ? Use pastebin to show us the log if necessary. Thanks On Fri, Feb 8, 2013 at 12:55 AM, So Hibino <[hidden email]> wrote: > Our hbase-master-server was shutdown with following message. > Hbase is runnig in Distributed mode in a single node. > I checked that GC completed in a very short time at the time of output the > WARN. > In addition the other system that is running in the same architecture > doesn't output the following WARN messsage and works well. > So I think that this is not due to a long GC pause. > > Do you have any idea about the problem? > > 2013-01-30 03:07:48,582 WARN org.apache.hadoop.hbase.util.Sleeper: We slept > 28970ms instead of 1000ms, this is likely due to a long garbage collecting > pause and it's usually bad, see > http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired > 2013-01-30 03:07:48,583 WARN org.apache.hadoop.hbase.util.Sleeper: We slept > 36902ms instead of 10000ms, this is likely due to a long garbage collecting > pause and it's usually bad, see > http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired > 2013-01-30 03:07:48,585 INFO org.apache.zookeeper.ClientCnxn: Client > session > timed out, have not heard from server in 39989ms for sessionid > 0x13c84cebfce0000, closing socket connection and attempting reconnect > 2013-01-30 03:07:48,586 INFO org.apache.zookeeper.ClientCnxn: Client > session > timed out, have not heard from server in 39987ms for sessionid > 0x13c84cebfce0001, closing socket connection and attempting reconnect > 2013-01-30 03:07:52,779 INFO org.apache.zookeeper.ClientCnxn: Opening > socket > connection to server VM_11/192.168.152.1:2181 > 2013-01-30 03:07:52,789 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to VM_11/192.168.152.1:2181, initiating session > 2013-01-30 03:07:52,777 INFO org.apache.zookeeper.ClientCnxn: Opening > socket > connection to server VM_11/192.168.152.1:2181 > 2013-01-30 03:07:52,793 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to VM_11/192.168.152.1:2181, initiating session > 2013-01-30 03:07:52,794 INFO org.apache.zookeeper.ClientCnxn: Unable to > reconnect to ZooKeeper service, session 0x13c84cebfce0001 has expired, > closing socket connection > 2013-01-30 03:07:52,794 INFO > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > This client just lost it's session with ZooKeeper, trying to reconnect. > 2013-01-30 03:07:52,794 INFO > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Trying to reconnect to zookeeper. > 2013-01-30 03:07:52,795 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, connectString=VM_11:2181 sessionTimeout=180000 > watcher=hconnection > 2013-01-30 03:07:52,812 INFO org.apache.zookeeper.ClientCnxn: Unable to > reconnect to ZooKeeper service, session 0x13c84cebfce0000 has expired, > closing socket connection > 2013-01-30 03:07:52,813 FATAL org.apache.hadoop.hbase.master.HMaster: > > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > received expired from ZooKeeper, aborting > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired > at > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361) > at > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279) > at > > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > 2013-01-30 03:07:52,813 INFO org.apache.hadoop.hbase.master.HMaster: > Aborting > 2013-01-30 03:07:52,813 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2013-01-30 03:07:52,813 INFO org.apache.zookeeper.ClientCnxn: Opening > socket > connection to server VM_11/192.168.152.1:2181 > 2013-01-30 03:07:52,814 INFO org.apache.zookeeper.ClientCnxn: Socket > connection established to VM_11/192.168.152.1:2181, initiating session > 2013-01-30 03:07:52,815 ERROR org.apache.hadoop.hbase.master.HMaster: > Region > server serverName=VM_11,60020,1359437833300, load=(requests=0, regions=3, > usedHeap=45, maxHeap=997) reported a fatal error: > ABORTING region server serverName=VM_11,60020,1359437833300, > load=(requests=0, regions=3, usedHeap=45, maxHeap=997): > > regionserver:60020-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002 > > regionserver:60020-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002 > received expired from ZooKeeper, aborting > Cause: > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired > at > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361) > at > > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279) > at > > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) > > 2013-01-30 03:07:52,820 INFO org.apache.zookeeper.ClientCnxn: Session > establishment complete on server VM_11/192.168.152.1:2181, sessionid = > 0x13c84cebfce0005, negotiated timeout = 40000 > 2013-01-30 03:07:52,841 INFO > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Reconnected successfully. This disconnect could have been caused by a > network partition or a long-running GC pause, either way it's recommended > that you verify your environment. > 2013-01-30 03:07:52,841 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2013-01-30 03:07:53,614 INFO org.apache.hadoop.hbase.master.LogCleaner: > master-VM_11:60000.oldLogCleaner exiting > 2013-01-30 03:07:54,251 INFO org.apache.hadoop.hbase.master.HMaster$2: > VM_11:60000-BalancerChore exiting > 2013-01-30 03:07:54,251 DEBUG org.apache.hadoop.hbase.master.HMaster: > Stopping service threads > 2013-01-30 03:07:54,251 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60000 > 2013-01-30 03:07:54,252 INFO org.apache.hadoop.hbase.master.HMaster: > Stopping infoServer > 2013-01-30 03:07:54,325 INFO org.mortbay.log: Stopped > SelectChannelConnector@0.0.0.0:60010 > 2013-01-30 03:07:54,326 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 5 on 60000: exiting > 2013-01-30 03:07:54,326 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC > Server listener on 60000 > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 9 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 8 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 7 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 6 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 4 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 3 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 1 on 60000: exiting > 2013-01-30 03:07:54,327 INFO org.apache.hadoop.hbase.master.CatalogJanitor: > VM_11:60000-CatalogJanitor exiting > 2013-01-30 03:07:54,328 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 0 on 60000: exiting > 2013-01-30 03:07:54,328 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC > Server Responder > 2013-01-30 03:07:54,337 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: > > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > Unable to get data of znode /hbase/master > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:118) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) > at > > org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) > 2013-01-30 03:07:54,337 ERROR > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > Received unexpected KeeperException, re-throwing exception > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:118) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) > at > > org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) > 2013-01-30 03:07:54,337 ERROR > org.apache.hadoop.hbase.master.ActiveMasterManager: > > master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000 > Error deleting our own master address node > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /hbase/master > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:118) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:577) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:554) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAsAddress(ZKUtil.java:648) > at > > org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:202) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:318) > 2013-01-30 03:07:54,337 DEBUG > org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker > org.apache.hadoop.hbase.catalog.CatalogTracker@4743bf3d > 2013-01-30 03:07:54,337 DEBUG > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > The connection to hconnection-0x13c84cebfce0005 has been closed. > 2013-01-30 03:07:54,338 INFO > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Closed zookeeper sessionid=0x13c84cebfce0005 > 2013-01-30 03:07:54,339 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x13c84cebfce0005 closed > 2013-01-30 03:07:54,339 DEBUG > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > The connection to null has been closed. > 2013-01-30 03:07:54,339 INFO org.apache.hadoop.hbase.master.HMaster: > HMaster > main thread exiting > 2013-01-30 03:07:54,339 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2013-01-30 03:07:54,339 INFO > org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor: > VM_11:60000.timeoutMonitor exiting > > > > -- > View this message in context: > http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192.html > Sent from the HBase User mailing list archive at Nabble.com. > |
|
|
In reply to this post by So Hibino
Hi,
>The master doesn't have memstores so this wouldn't help. In fact it's >pretty rare that we see the master with GC issues. I recall seing >issues with time travelling (machine clock's too slow and ntpd resets >it) or on EC2 where sometimes you'd see random machine pauses out of >nowhere (although that was a long time ago and haven't used EC2 >since). We doesn't use EC2,but this server works with KVM. The software version, the logs, the conf files are shown below. software version ---------------------------------------- HBase version: 0.90.6-cdh3u4 Hadoop version: 0.20.2+923.256-1 Zookeeper version: 3.3.5+19.1-1 Operating System: CentOS release 5.8 Linux kernel version: 2.6.18-308.el5 Java version: 1.6.0_31 ---------------------------------------- master log ------------------ 2013-02-12 00:10:24,309 DEBUG org.apache.hadoop.hbase.master.LoadBalancer: Server information: VM_11,60020,1359691508001=3 2013-02-12 00:10:24,310 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=1 regions=3 average=3.0 mostloaded=3 leastloaded=3 2013-02-12 00:10:24,318 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s) 2013-02-12 00:13:21,105 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 13417ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:13:55,239 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 34132ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:13:55,242 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 24949ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:14:18,441 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 73255ms instead of 60000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:14:18,442 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 23203ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:14:18,444 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 14017ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:15:24,358 DEBUG org.apache.hadoop.hbase.master.LoadBalancer: Server information: VM_11,60020,1359691508001=3 2013-02-12 00:15:24,358 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=1 regions=3 average=3.0 mostloaded=3 leastloaded=3 2013-02-12 00:15:24,361 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s) ------------------ master GC log ------------------ 2013-02-11T23:46:37.285+0900: 902498.189: [GC 902498.189: [DefNew: 17041K->16K(19136K), 0.0017450 secs] 20049K->3025K(83008K) icms_dc=0 , 0.0018270 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2013-02-12T00:35:25.628+0900: 905426.532: [GC 905426.532: [DefNew: 17040K->18K(19136K), 0.0017430 secs] 20049K->3026K(83008K) icms_dc=0 , 0.0018370 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2013-02-12T01:20:26.110+0900: 908127.014: [GC 908127.014: [DefNew: 17034K->27K(19136K), 0.0023420 secs] 20043K->3036K(83008K) icms_dc=0 , 0.0025090 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] ------------------- region log ------------------- 2013-02-12 00:00:09,968 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3022, hits=3015, hitRatio=99.76%%, cachingAccesses=3015, cachingHits=3012, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN 2013-02-12 00:05:09,971 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3023, hits=3016, hitRatio=99.76%%, cachingAccesses=3016, cachingHits=3013, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN 2013-02-12 00:10:12,109 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3024, hits=3017, hitRatio=99.76%%, cachingAccesses=3017, cachingHits=3014, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN 2013-02-12 00:15:09,969 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3025, hits=3018, hitRatio=99.76%%, cachingAccesses=3018, cachingHits=3015, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN 2013-02-12 00:20:09,970 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3026, hits=3019, hitRatio=99.76%%, cachingAccesses=3019, cachingHits=3016, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN ------------------- region GC log ------------------ 2013-02-11T22:31:11.315+0900: 897964.350: [GC 897964.350: [DefNew: 17062K->35K(19136K), 0.0036000 secs] 40262K->23234K(83008K) icms_dc=0 , 0.0037710 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2013-02-12T00:27:13.313+0900: 904926.348: [GC 904926.348: [DefNew: 17059K->43K(19136K), 0.0020250 secs] 40258K->23243K(83008K) icms_dc=0 , 0.0021130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2013-02-12T02:23:52.114+0900: 911925.149: [GC 911925.149: [DefNew: 17067K->43K(19136K), 0.0018170 secs] 40267K->23243K(83008K) icms_dc=0 , 0.0019330 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] --------------------- zookeeper log ------------------- no logs at that time ------------------- hbase-site.xml ---------------------- <configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.rootdir</name> <value>/var/lib/hbase/cache/${user.name}/root</value> </property> <property> <name>hbase.tmp.dir</name> <value>/var/lib/hbase/cache/${user.name}/tmp</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>VM_11</value> </property> </configuration> ------------------------ zoo.cfg ---------------------- tickTime=2000 initLimit=10 syncLimit=5 dataDir=/var/zookeeper clientPort=2181 server.0=VM_11:2888:3888 ---------------------- |
Re: hbase-master-server slept
|
|
Well my friend, my first advice is to update your completed infrastructure:
- Update your Hadoop to 1.x branch - Update HBase to 0.94.4 - Update Zookeeper to 3.4.5 Or simply update your CDH version to 4.1 or 4.2 ----- Mensaje original ----- De: "So Hibino" <[hidden email]> Para: [hidden email] Enviados: Lunes, 11 de Febrero 2013 23:06:25 Asunto: Re: hbase-master-server slept Hi, >The master doesn't have memstores so this wouldn't help. In fact it's >pretty rare that we see the master with GC issues. I recall seing >issues with time travelling (machine clock's too slow and ntpd resets >it) or on EC2 where sometimes you'd see random machine pauses out of >nowhere (although that was a long time ago and haven't used EC2 >since). We doesn't use EC2,but this server works with KVM. The software version, the logs, the conf files are shown below. software version ---------------------------------------- HBase version: 0.90.6-cdh3u4 Hadoop version: 0.20.2+923.256-1 Zookeeper version: 3.3.5+19.1-1 Operating System: CentOS release 5.8 Linux kernel version: 2.6.18-308.el5 Java version: 1.6.0_31 ---------------------------------------- master log ------------------ 2013-02-12 00:10:24,309 DEBUG org.apache.hadoop.hbase.master.LoadBalancer: Server information: VM_11,60020,1359691508001=3 2013-02-12 00:10:24,310 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=1 regions=3 average=3.0 mostloaded=3 leastloaded=3 2013-02-12 00:10:24,318 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s) 2013-02-12 00:13:21,105 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 13417ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:13:55,239 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 34132ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:13:55,242 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 24949ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:14:18,441 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 73255ms instead of 60000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:14:18,442 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 23203ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:14:18,444 WARN org.apache.hadoop.hbase.util.Sleeper: We slept 14017ms instead of 1000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2013-02-12 00:15:24,358 DEBUG org.apache.hadoop.hbase.master.LoadBalancer: Server information: VM_11,60020,1359691508001=3 2013-02-12 00:15:24,358 INFO org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing. servers=1 regions=3 average=3.0 mostloaded=3 leastloaded=3 2013-02-12 00:15:24,361 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s) ------------------ master GC log ------------------ 2013-02-11T23:46:37.285+0900: 902498.189: [GC 902498.189: [DefNew: 17041K->16K(19136K), 0.0017450 secs] 20049K->3025K(83008K) icms_dc=0 , 0.0018270 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 2013-02-12T00:35:25.628+0900: 905426.532: [GC 905426.532: [DefNew: 17040K->18K(19136K), 0.0017430 secs] 20049K->3026K(83008K) icms_dc=0 , 0.0018370 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 2013-02-12T01:20:26.110+0900: 908127.014: [GC 908127.014: [DefNew: 17034K->27K(19136K), 0.0023420 secs] 20043K->3036K(83008K) icms_dc=0 , 0.0025090 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] ------------------- region log ------------------- 2013-02-12 00:00:09,968 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3022, hits=3015, hitRatio=99.76%%, cachingAccesses=3015, cachingHits=3012, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN 2013-02-12 00:05:09,971 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3023, hits=3016, hitRatio=99.76%%, cachingAccesses=3016, cachingHits=3013, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN 2013-02-12 00:10:12,109 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3024, hits=3017, hitRatio=99.76%%, cachingAccesses=3017, cachingHits=3014, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN 2013-02-12 00:15:09,969 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3025, hits=3018, hitRatio=99.76%%, cachingAccesses=3018, cachingHits=3015, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN 2013-02-12 00:20:09,970 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.94 MB, max=199.59 MB, blocks=3, accesses=3026, hits=3019, hitRatio=99.76%%, cachingAccesses=3019, cachingHits=3016, cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN ------------------- region GC log ------------------ 2013-02-11T22:31:11.315+0900: 897964.350: [GC 897964.350: [DefNew: 17062K->35K(19136K), 0.0036000 secs] 40262K->23234K(83008K) icms_dc=0 , 0.0037710 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2013-02-12T00:27:13.313+0900: 904926.348: [GC 904926.348: [DefNew: 17059K->43K(19136K), 0.0020250 secs] 40258K->23243K(83008K) icms_dc=0 , 0.0021130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 2013-02-12T02:23:52.114+0900: 911925.149: [GC 911925.149: [DefNew: 17067K->43K(19136K), 0.0018170 secs] 40267K->23243K(83008K) icms_dc=0 , 0.0019330 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] --------------------- zookeeper log ------------------- no logs at that time ------------------- hbase-site.xml ---------------------- <configuration> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.rootdir</name> <value>/var/lib/hbase/cache/${user.name}/root</value> </property> <property> <name>hbase.tmp.dir</name> <value>/var/lib/hbase/cache/${user.name}/tmp</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>VM_11</value> </property> </configuration> ------------------------ zoo.cfg ---------------------- tickTime=2000 initLimit=10 syncLimit=5 dataDir=/var/zookeeper clientPort=2181 server.0=VM_11:2888:3888 ---------------------- -- View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038406.html Sent from the HBase User mailing list archive at Nabble.com. -- Marcos Ortiz Valmaseda, Product Manager && Data Scientist at UCI Blog : http://marcosluis2186.posterous.com LinkedIn: http://www.linkedin.com/in/marcosluis2186 Twitter : @marcosluis2186 |
|
|
Hi,
The other system with the same version works well and don't output any WARN message. So I think this isn't problem of CDH version. Thank you, So |
Re: hbase-master-server slept
|
|
In reply to this post by So Hibino
On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email]> wrote:
> master log Unfortunately there's not much in there apart from the weird sleep times. At this point I'd consider OS-level issues. > We doesn't use EC2,but this server works with KVM. Can you post the dmesg and syslog spanning the time of the issue? Please use pastebin.com as it can be really hard to read the logs in emails. Thx, J-D |
|
|
Hi,
> Can you post the dmesg and syslog spanning the time of the issue? Is the syslog you mean /var/log/message ? On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote: > On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote: > > > master log > > Unfortunately there's not much in there apart from the weird sleep > times. At this point I'd consider OS-level issues. > > > We doesn't use EC2,but this server works with KVM. > > Can you post the dmesg and syslog spanning the time of the issue? > Please use pastebin.com as it can be really hard to read the logs in > emails. > > Thx, > > J-D > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > If you reply to this email, your message will be added to the discussion below: > > NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> -- H24.07より組織名が変わりました。 NTTサービスエボリューション研究所 日比野 壮 （So Hibino） 〒239-0847 横須賀市光の丘1-1 (320C) TEL : 046-859-4582 FAX : 046-855-1282 e-mail : [hidden email] |
Re: hbase-master-server slept
|
|
messages and syslog please, just for the time spanning the issue.
J-D On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[hidden email]> wrote: > Hi, > >> Can you post the dmesg and syslog spanning the time of the issue? > Is the syslog you mean /var/log/message ? > > > On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote: >> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote: >> >> > master log >> >> Unfortunately there's not much in there apart from the weird sleep >> times. At this point I'd consider OS-level issues. >> >> > We doesn't use EC2,but this server works with KVM. >> >> Can you post the dmesg and syslog spanning the time of the issue? >> Please use pastebin.com as it can be really hard to read the logs in >> emails. >> >> Thx, >> >> J-D >> >> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --- >> If you reply to this email, your message will be added to the discussion below: >> >> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > -- > H24.07より組織名が変わりました。 > > NTTサービスエボリューション研究所 > 日比野 壮 （So Hibino） > > 〒239-0847 横須賀市光の丘1-1 (320C) > TEL : 046-859-4582 FAX : 046-855-1282 > e-mail : [hidden email] > > > > > > -- > View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html > Sent from the HBase User mailing list archive at Nabble.com. |
|
|
Hi,
I pasted dmseg and syslog on pastebin.com. URL is below. dmseg http://pastebin.com/t3i3KRJ8 syslog http://pastebin.com/LcBRYMyX Any messages wasn't output for the time spanning the isuue. thank you, So On 2013/02/13 9:31, Jean-Daniel Cryans [via Apache HBase] wrote: > > > messages and syslog please, just for the time spanning the issue. > > J-D > > On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[hidden email]> wrote: >> Hi, >> >>> Can you post the dmesg and syslog spanning the time of the issue? >> Is the syslog you mean /var/log/message ? >> >> >> On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote: >>> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote: >>> >>> > master log >>> >>> Unfortunately there's not much in there apart from the weird sleep >>> times. At this point I'd consider OS-level issues. >>> >>> > We doesn't use EC2,but this server works with KVM. >>> >>> Can you post the dmesg and syslog spanning the time of the issue? >>> Please use pastebin.com as it can be really hard to read the logs in >>> emails. >>> >>> Thx, >>> >>> J-D >>> >>> >>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> --- >>> If you reply to this email, your message will be added to the discussion below: >>> >>> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> >> >> -- >> H24.07より組織名が変わりました。 >> >> NTTサービスエボリューション研究所 >> 日比野 壮 （So Hibino） >> >> 〒239-0847 横須賀市光の丘1-1 (320C) >> TEL : 046-859-4582 FAX : 046-855-1282 >> e-mail : [hidden email] >> >> >> >> >> >> -- >> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html >> Sent from the HBase User mailing list archive at Nabble.com. > > > > > _______________________________________________ > If you reply to this email, your message will be added to the discussion below: > http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038461.html > > To unsubscribe from hbase-master-server slept, visit -- H24.07より組織名が変わりました。 NTTサービスエボリューション研究所 日比野 壮 （So Hibino） 〒239-0847 横須賀市光の丘1-1 (320C) TEL : 046-859-4582 FAX : 046-855-1282 e-mail : [hidden email] |
|
|
In reply to this post by Jean-Daniel Cryans
Hi,
The VM was rebooted last night because of the host server memory error. So the dmesg was cleared at the time of the isuue. The below dmesg is new and the issue has not been happened yet. Thank you, So On 2013/02/14 15:10, So Hibino wrote: > Hi, > > I pasted dmseg and syslog on pastebin.com. > > URL is below. > > dmseg > http://pastebin.com/t3i3KRJ8 > > syslog > http://pastebin.com/LcBRYMyX > > Any messages wasn't output for the time spanning the isuue. > > thank you, > So > > > On 2013/02/13 9:31, Jean-Daniel Cryans [via Apache HBase] wrote: >> >> >> messages and syslog please, just for the time spanning the issue. >> >> J-D >> >> On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[hidden email]> wrote: >>> Hi, >>> >>>> Can you post the dmesg and syslog spanning the time of the issue? >>> Is the syslog you mean /var/log/message ? >>> >>> >>> On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote: >>>> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote: >>>> >>>> > master log >>>> >>>> Unfortunately there's not much in there apart from the weird sleep >>>> times. At this point I'd consider OS-level issues. >>>> >>>> > We doesn't use EC2,but this server works with KVM. >>>> >>>> Can you post the dmesg and syslog spanning the time of the issue? >>>> Please use pastebin.com as it can be really hard to read the logs in >>>> emails. >>>> >>>> Thx, >>>> >>>> J-D >>>> >>>> >>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > --! >>> --- >>>> If you reply to this email, your message will be added to the discussion below: >>>> >>>> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >>> >>> -- >>> H24.07より組織名が変わりました。 >>> >>> NTTサービスエボリューション研究所 >>> 日比野 壮 （So Hibino） >>> >>> 〒239-0847 横須賀市光の丘1-1 (320C) >>> TEL : 046-859-4582 FAX : 046-855-1282 >>> e-mail : [hidden email] >>> >>> >>> >>> >>> >>> -- >>> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html >>> Sent from the HBase User mailing list archive at Nabble.com. >> >> >> >> >> _______________________________________________ >> If you reply to this email, your message will be added to the discussion below: >> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038461.html >> >> To unsubscribe from hbase-master-server slept, visit > > -- H24.07より組織名が変わりました。 NTTサービスエボリューション研究所 日比野 壮 （So Hibino） 〒239-0847 横須賀市光の丘1-1 (320C) TEL : 046-859-4582 FAX : 046-855-1282 e-mail : [hidden email] |
Re: hbase-master-server slept
|
First,
In the world of Hadoop, if it ain't broke don't fix it, may not be the best advice. HBase is still evolving at a good pace and you want to be closer to the latest releases. CDH4 is stable so that I would agree that going to CDH4 would be best. Second. You are running this as a single machine within the VM. What does the hardware look like? Number of cores, physical or virtual ? How much memory? What type of disks and where are they? ( attached or SAN?) Sent from a remote device. Please excuse any typos... Mike Segel On Feb 14, 2013, at 1:16 AM, So Hibino <[hidden email]> wrote: > Hi, > > The VM was rebooted last night because of the host server memory error. > So the dmesg was cleared at the time of the isuue. > > The below dmesg is new and the issue has not been happened yet. > > Thank you, > So > > On 2013/02/14 15:10, So Hibino wrote: >> Hi, >> >> I pasted dmseg and syslog on pastebin.com. >> >> URL is below. >> >> dmseg >> http://pastebin.com/t3i3KRJ8 >> >> syslog >> http://pastebin.com/LcBRYMyX >> >> Any messages wasn't output for the time spanning the isuue. >> >> thank you, >> So >> >> >> On 2013/02/13 9:31, Jean-Daniel Cryans [via Apache HBase] wrote: >>> >>> >>> messages and syslog please, just for the time spanning the issue. >>> >>> J-D >>> >>> On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[hidden email]> wrote: >>>> Hi, >>>> >>>>> Can you post the dmesg and syslog spanning the time of the issue? >>>> Is the syslog you mean /var/log/message ? >>>> >>>> >>>> On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote: >>>>> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote: >>>>> >>>>>> master log >>>>> >>>>> Unfortunately there's not much in there apart from the weird sleep >>>>> times. At this point I'd consider OS-level issues. >>>>> >>>>>> We doesn't use EC2,but this server works with KVM. >>>>> >>>>> Can you post the dmesg and syslog spanning the time of the issue? >>>>> Please use pastebin.com as it can be really hard to read the logs in >>>>> emails. >>>>> >>>>> Thx, >>>>> >>>>> J-D >>>>> >>>>> >>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! > -! >> --! >>>> --- >>>>> If you reply to this email, your message will be added to the discussion below: >>>>> >>>>> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>> >>>> >>>> -- >>>> H24.07より組織名が変わりました。 >>>> >>>> NTTサービスエボリューション研究所 >>>> 日比野 壮 （So Hibino） >>>> >>>> 〒239-0847 横須賀市光の丘1-1 (320C) >>>> TEL : 046-859-4582 FAX : 046-855-1282 >>>> e-mail : [hidden email] >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html >>>> Sent from the HBase User mailing list archive at Nabble.com. >>> >>> >>> >>> >>> _______________________________________________ >>> If you reply to this email, your message will be added to the discussion below: >>> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038461.html >>> >>> To unsubscribe from hbase-master-server slept, visit > > -- > H24.07より組織名が変わりました。 > > NTTサービスエボリューション研究所 > 日比野 壮 （So Hibino） > > 〒239-0847 横須賀市光の丘1-1 (320C) > TEL : 046-859-4582 FAX : 046-855-1282 > e-mail : [hidden email] > > > > > > -- > View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038516.html > Sent from the HBase User mailing list archive at Nabble.com. |
|
|
Hi,
We consider to update the version of Hbase. The VM spec is below. CPU:2 Core MEMORY:4GB We don't know the hardware spec of host server, because we owe the VM from a VPS provider. Additionally, I check vmstat for the time spanning the isuue. No batch and online job worked on this server for this time. But the cpu usage was very high suddenly. Is it a clue? 2013/01/17 16:05:51 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ 2013/01/17 16:05:51 r b swpd free buff cache si so bi bo in cs us sy id wa st 2013/01/18 09:15:45 0 1 0 3113832 126920 573332 0 0 0 3 1001 121 0 0 100 0 0 2013/01/18 09:15:50 0 1 0 3113832 126924 573332 0 0 0 18 1002 125 0 0 100 0 0 2013/01/18 09:15:55 0 1 0 3113832 126924 573332 0 0 0 2 1001 123 0 0 100 0 0 2013/01/18 09:16:00 0 1 0 3113832 126924 573332 0 0 0 15 1001 122 0 0 100 0 0 2013/01/18 09:16:05 0 0 0 3113708 126940 573352 0 0 0 35 1002 140 0 0 100 0 0 2013/01/18 09:16:10 0 0 0 3113708 126944 573352 0 0 0 7 1001 122 0 0 100 0 0 2013/01/18 09:16:15 0 1 0 3113708 126944 573352 0 0 0 6 1001 122 0 0 100 0 0 2013/01/18 09:16:20 0 1 0 3113708 126944 573352 0 0 0 1 1001 122 0 0 100 0 0 2013/01/18 09:16:25 0 1 0 3113708 126948 573348 0 0 0 7 1001 123 0 0 100 0 0 2013/01/18 09:16:30 0 1 0 3113708 126948 573352 0 0 0 28 1001 130 0 0 100 0 0 2013/01/18 09:16:35 0 1 0 3113708 126948 573352 0 0 0 12 1001 128 0 0 100 0 0 2013/01/18 09:16:40 0 1 0 3113708 126956 573352 0 0 0 9 1001 127 0 0 100 0 0 2013/01/18 09:16:45 0 1 0 3113708 126956 573352 0 0 0 5 1001 127 0 0 100 0 0 2013/01/18 09:16:50 0 1 0 3113708 126956 573352 0 0 0 3 1001 128 0 0 100 0 0 2013/01/18 09:16:55 0 1 0 3113708 126964 573348 0 0 0 6 1001 128 0 0 99 0 0 2013/01/18 09:17:00 0 1 0 3113708 126972 573348 0 0 0 3 1001 127 0 0 100 0 0 2013/01/18 09:17:05 0 0 0 3113708 126976 573372 0 0 0 25 1002 142 0 0 100 0 0 2013/01/18 09:17:10 0 0 0 3113708 126976 573372 0 0 0 53 1002 128 0 0 100 0 0 2013/01/18 09:17:15 0 0 0 3113708 126980 573372 0 0 0 11 1001 127 0 0 100 0 0 2013/01/18 09:17:20 0 0 0 3113708 126984 573368 0 0 0 3 1001 126 0 0 100 0 0 2013/01/18 09:17:25 0 1 0 3113708 126992 573364 0 0 0 6 1001 127 0 0 100 0 0 2013/01/18 09:17:30 0 1 0 3113708 126992 573376 0 0 0 2 1001 125 0 0 100 0 0 2013/01/18 09:17:35 0 1 0 3113708 127000 573372 0 0 0 10 1001 128 0 0 100 0 0 2013/01/18 09:17:40 0 1 0 3113708 127000 573376 0 0 0 2 1001 126 0 0 100 0 0 2013/01/18 09:17:46 2 1 0 3113692 127004 573376 0 0 0 11 806 29 2 2 95 2 0 2013/01/18 09:17:53 1 1 0 3113708 127008 573376 0 0 0 10 1043 15 0 10 40 50 0 2013/01/18 09:18:00 1 1 0 3113708 127008 573376 0 0 0 4 1364 15 8 0 67 25 0 2013/01/18 09:18:06 3 0 0 3113584 127008 573396 0 0 0 26 1591 32 8 17 67 8 0 2013/01/18 09:18:12 1 0 0 3113584 127012 573396 0 0 0 21 1004 10 0 0 89 11 0 2013/01/18 09:18:17 1 1 0 3113584 127020 573388 0 0 0 34 1392 12 10 0 90 0 0 2013/01/18 09:18:53 2 2 0 3113584 127020 573396 0 0 0 6 1075 15 0 25 75 0 0 2013/01/18 09:18:58 1 1 0 3113584 127020 573396 0 0 0 14 6534 18 2 0 98 0 0 2013/01/18 09:19:04 2 0 0 3113576 127032 573392 0 0 0 9 1604 20 0 9 27 64 0 2013/01/18 09:19:11 3 1 0 3113584 127044 573416 0 0 0 41 1579 29 17 17 67 0 0 2013/01/18 09:19:17 1 1 0 3113584 127048 573416 0 0 0 3 705 10 0 0 50 50 0 2013/01/18 09:19:23 1 1 0 3113584 127060 573412 0 0 0 7 1684 15 0 6 88 6 0 2013/01/18 09:19:28 1 2 0 3113584 127060 573416 0 0 0 33 1153 21 10 20 60 10 0 2013/01/18 09:19:33 1 1 0 3113584 127060 573416 0 0 0 6 861 16 0 0 44 56 0 2013/01/18 09:19:39 1 1 0 3113584 127068 573416 0 0 0 6 1036 16 0 0 78 22 0 2013/01/18 09:19:44 0 1 0 3113584 127072 573416 0 0 0 3 1007 10 14 14 57 14 0 2013/01/18 09:19:49 0 1 0 3113584 127076 573416 0 0 0 6 1369 18 0 0 91 9 0 2013/01/18 09:19:56 2 0 0 3113584 127084 573408 0 0 0 8 596 11 20 20 60 0 0 2013/01/18 09:20:01 1 1 0 3113584 127084 573416 0 0 0 6 1421 10 0 8 92 0 0 2013/01/18 09:20:16 2 2 0 3113444 127096 573432 0 0 0 60 804 26 20 0 20 60 0 2013/01/18 09:20:21 0 1 0 3113460 127100 573432 0 0 0 10 3578 20 4 4 32 61 0 2013/01/18 09:20:26 1 1 0 3113460 127100 573432 0 0 0 13 833 16 0 14 71 14 0 2013/01/18 09:20:32 1 1 0 3113460 127100 573432 0 0 0 0 802 8 0 0 0 100 0 2013/01/18 09:20:40 1 1 0 3113460 127104 573432 0 0 0 8 1175 9 10 10 0 80 0 2013/01/18 09:20:46 1 1 0 3113460 127108 573432 0 0 0 7 1850 14 0 0 87 13 0 2013/01/18 09:20:51 0 1 0 3113460 127112 573432 0 0 0 14 817 7 14 0 57 29 0 2013/01/18 09:20:56 2 1 0 3113460 127120 573432 0 0 0 6 1250 16 0 0 56 44 0 2013/01/18 09:21:05 5 1 0 3113452 127128 573432 0 0 0 3 1302 16 0 0 100 0 0 2013/01/18 09:21:14 1 1 0 3113336 127136 573460 0 0 0 48 1400 27 20 20 60 0 0 2013/01/18 09:21:23 1 1 0 3113336 127136 573460 0 0 0 35 1899 11 0 0 100 0 0 2013/01/18 09:21:28 0 1 0 3113336 127144 573456 0 0 0 10 1963 17 5 5 21 68 0 2013/01/18 09:21:33 1 1 0 3113336 127148 573460 0 0 0 2 761 13 0 20 40 40 0 2013/01/18 09:21:38 0 1 0 3113336 127148 573460 0 0 0 0 1346 17 10 10 0 80 0 2013/01/18 09:21:44 1 1 0 3113336 127148 573460 0 0 0 2 622 12 0 0 0 100 0 2013/01/18 09:21:49 1 0 0 3113336 127156 573460 0 0 0 11 1433 14 10 0 70 20 0 2013/01/18 09:21:54 0 0 0 3113336 127160 573456 0 0 0 11 1190 13 0 0 80 20 0 2013/01/18 09:22:00 3 1 0 3113336 127160 573460 0 0 0 2 981 14 0 9 91 0 0 2013/01/18 09:22:09 1 0 0 3113336 127168 573476 0 0 0 45 1125 26 25 38 25 13 0 2013/01/18 09:22:15 1 1 0 3113336 127168 573480 0 0 0 1 1294 9 0 9 91 0 0 2013/01/18 09:22:21 1 1 0 3113336 127172 573480 0 0 0 11 1696 14 8 8 85 0 0 2013/01/18 09:22:26 0 1 0 3113336 127176 573480 0 0 0 8 1123 12 0 0 57 43 0 2013/01/18 09:22:31 1 2 0 3113336 127184 573480 0 0 0 33 1000 15 10 0 70 20 0 2013/01/18 09:22:36 1 1 0 3113336 127188 573480 0 0 0 3 880 12 0 14 43 43 0 2013/01/18 09:22:41 0 1 0 3113336 127188 573480 0 0 0 7 1399 32 0 1 96 3 0 2013/01/18 09:22:46 0 1 0 3113336 127188 573480 0 0 0 6 1001 121 0 0 100 0 0 2013/01/18 09:22:51 0 1 0 3113336 127196 573476 0 0 0 3 1001 121 0 0 100 0 0 2013/01/18 09:22:56 0 1 0 3113336 127196 573480 0 0 0 10 1001 127 0 0 100 0 0 2013/01/18 09:23:01 0 0 0 3113336 127204 573476 0 0 0 33 1002 141 0 0 100 0 0 2013/01/18 09:23:06 0 0 0 3113336 127204 573500 0 0 0 7 1001 127 0 0 100 0 0 2013/01/18 09:23:11 0 1 0 3113336 127204 573500 0 0 0 26 1000 124 0 0 100 0 0 2013/01/18 09:23:16 0 0 0 3113336 127212 573496 0 0 0 8 1002 128 0 0 100 0 0 2013/01/18 09:23:21 0 1 0 3113336 127216 573496 0 0 0 3 1001 124 0 0 100 0 0 2013/01/18 09:23:26 0 1 0 3113336 127216 573500 0 0 0 6 1001 127 0 0 100 0 0 2013/01/18 09:23:31 0 0 0 3113336 127220 573496 0 0 0 9 1002 128 0 0 100 0 0 2013/01/18 09:23:36 0 1 0 3113336 127224 573496 0 0 0 6 1001 126 0 0 100 0 0 2013/01/18 09:23:41 1 1 0 3113336 127232 573496 0 0 0 4 1001 115 0 0 100 0 0 -------------------------------------------- On 2013/02/14 22:10, Michel Segel [via Apache HBase] wrote: > > > First, > In the world of Hadoop, if it ain't broke don't fix it, may not be the best advice. > HBase is still evolving at a good pace and you want to be closer to the latest releases. > CDH4 is stable so that I would agree that going to CDH4 would be best. > > Second. > You are running this as a single machine within the VM. > > What does the hardware look like? > Number of cores, physical or virtual ? > How much memory? > > What type of disks and where are they? > ( attached or SAN?) > > > > > Sent from a remote device. Please excuse any typos... > > Mike Segel > > On Feb 14, 2013, at 1:16 AM, So Hibino <[hidden email]> wrote: > >> Hi, >> >> The VM was rebooted last night because of the host server memory error. >> So the dmesg was cleared at the time of the isuue. >> >> The below dmesg is new and the issue has not been happened yet. >> >> Thank you, >> So >> >> On 2013/02/14 15:10, So Hibino wrote: >>> Hi, >>> >>> I pasted dmseg and syslog on pastebin.com. >>> >>> URL is below. >>> >>> dmseg >>> http://pastebin.com/t3i3KRJ8 >>> >>> syslog >>> http://pastebin.com/LcBRYMyX >>> >>> Any messages wasn't output for the time spanning the isuue. >>> >>> thank you, >>> So >>> >>> >>> On 2013/02/13 9:31, Jean-Daniel Cryans [via Apache HBase] wrote: >>>> >>>> >>>> messages and syslog please, just for the time spanning the issue. >>>> >>>> J-D >>>> >>>> On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[hidden email]> wrote: >>>>> Hi, >>>>> >>>>>> Can you post the dmesg and syslog spanning the time of the issue? >>>>> Is the syslog you mean /var/log/message ? >>>>> >>>>> >>>>> On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote: >>>>>> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote: >>>>>> >>>>>>> master log >>>>>> >>>>>> Unfortunately there's not much in there apart from the weird sleep >>>>>> times. At this point I'd consider OS-level issues. >>>>>> >>>>>>> We doesn't use EC2,but this server works with KVM. >>>>>> >>>>>> Can you post the dmesg and syslog spanning the time of the issue? >>>>>> Please use pastebin.com as it can be really hard to read the logs in >>>>>> emails. >>>>>> >>>>>> Thx, >>>>>> >>>>>> J-D >>>>>> >>>>>> >>>>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------! >> -! >>> --! >>>>> --- >>>>>> If you reply to this email, your message will be added to the discussion below: >>>>>> >>>>>> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>> >>>>> >>>>> -- >>>>> H24.07より組織名が変わりました。 >>>>> >>>>> NTTサービスエボリューション研究所 >>>>> 日比野 壮 （So Hibino） >>>>> >>>>> 〒239-0847 横須賀市光の丘1-1 (320C) >>>>> TEL : 046-859-4582 FAX : 046-855-1282 >>>>> e-mail : [hidden email] >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html >>>>> Sent from the HBase User mailing list archive at Nabble.com. >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> If you reply to this email, your message will be added to the discussion below: >>>> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038461.html >>>> >>>> To unsubscribe from hbase-master-server slept, visit >> >> -- >> H24.07より組織名が変わりました。 >> >> NTTサービスエボリューション研究所 >> 日比野 壮 （So Hibino） >> >> 〒239-0847 横須賀市光の丘1-1 (320C) >> TEL : 046-859-4582 FAX : 046-855-1282 >> e-mail : [hidden email] >> >> >> >> >> >> -- >> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038516.html >> Sent from the HBase User mailing list archive at Nabble.com. > > > > > _______________________________________________ > If you reply to this email, your message will be added to the discussion below: > http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038520.html > > To unsubscribe from hbase-master-server slept, visit -- H24.07より組織名が変わりました。 NTTサービスエボリューション研究所 日比野 壮 （So Hibino） 〒239-0847 横須賀市光の丘1-1 (320C) TEL : 046-859-4582 FAX : 046-855-1282 e-mail : [hidden email] |
«
Return to HBase User
|
1 view|%1 views
Loading...
| Powered by Nabble | Edit this page |