Quantcast

hBase - the server has too many connections (maxClientConn property set to 0 does not help)

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

hBase - the server has too many connections (maxClientConn property set to 0 does not help)

glumet
Hello,
I use hbase to store my data which comes from a crawler. These data are indexed from hbase to Solr. But I have a problem with hBase. As I have larger DB now, everytime I try to index data from hbase the error shown below occures. This is the part of log file hadoop.log

There is something like Caused by: This could be a sign that the server has too many connections (30 is the default). and I found here on SO that this could be beacues of maxClientConnections property so I set up this value to 0 but this solution did not help.

2013-11-11 15:30:22,638 ERROR zookeeper.ClientCnxn - Error while calling watcher
java.lang.NullPointerException
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:315)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1340)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:343)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:488)
2013-11-11 15:30:27,925 WARN  mapred.FileOutputCommitter - Output path is null in cleanup
2013-11-11 15:30:27,925 WARN  mapred.LocalJobRunner - job_local1497313370_0001
org.apache.gora.util.GoraException: java.lang.RuntimeException: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
at org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:568)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:127)
at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
... 6 more
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:109)
... 8 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 14 more


Do you know where is the problem please?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

Jean-Marc Spaggiari
So this occurs when you run you MR job to index HBase data from Solr? How
many maps  do you have at the same time? Is that more than 30? If so, can
you try to get than lower to see if  that helps? Else you can see if
maxClientCnxns in ZK side can be increased?

JM


2013/11/12 glumet <[hidden email]>

> Hello,
> I use hbase to store my data which comes from a crawler. These data are
> indexed from hbase to Solr. But I have a problem with hBase. As I have
> larger DB now, everytime I try to index data from hbase the error shown
> below occures. This is the part of log file hadoop.log
>
> There is something like Caused by: This could be a sign that the server has
> too many connections (30 is the default). and I found here on SO that this
> could be beacues of maxClientConnections property so I set up this value to
> 0 but this solution did not help.
>
> /2013-11-11 15:30:22,638 ERROR zookeeper.ClientCnxn - Error while calling
> watcher
> java.lang.NullPointerException
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:315)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1340)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:343)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:488)
> 2013-11-11 15:30:27,925 WARN  mapred.FileOutputCommitter - Output path is
> null in cleanup
> 2013-11-11 15:30:27,925 WARN  mapred.LocalJobRunner -
> job_local1497313370_0001
> org.apache.gora.util.GoraException: java.lang.RuntimeException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> at
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
> at
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
> at
>
> org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
> at
>
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:568)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to
> connect to ZooKeeper but the connection closes immediately. This could be a
> sign that the server has too many connections (30 is the default). Consider
> inspecting your ZK server logs for that error and then make sure you are
> reusing HBaseConfiguration as often as you can. See HTable's javadoc for
> more information.
> at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:127)
> at
>
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
> at
>
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
> ... 6 more
> Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is
> able to connect to ZooKeeper but the connection closes immediately. This
> could be a sign that the server has too many connections (30 is the
> default). Consider inspecting your ZK server logs for that error and then
> make sure you are reusing HBaseConfiguration as often as you can. See
> HTable's javadoc for more information.
> at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)
> at
>
> org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
> at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:109)
> ... 8 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
> at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
> ... 14 more/
>
> Do you know where is the problem please?
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-tp4052728.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

glumet
My data are stored in hBase. And I am trying to put them from hBase to Solr. I have written my own mapper and reducer (I put it here if you are interested http://pastebin.com/jsKqiZTw). I have to admit that I do not know where I can check how
many maps do I have at the same time and where I can change a value... is it possible that problem is in bad re-using of org.apache.hadoop.conf.Configuration in my code?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

Jean-Marc Spaggiari
I look very quickly at your code and I have not seend anything wrong so
far. I will take a deeper look.

For the number of maps/slots available, you can take a look at the
jobtracker interface http://youserver:50030/jobtracker.jsp

I will take another look at your code now and keep you posted.


2013/11/12 glumet <[hidden email]>

> My data are stored in hBase. And I am trying to put them from hBase to
> Solr.
> I have written my own mapper and reducer (I put it here if you are
> interested http://pastebin.com/jsKqiZTw). I have to admit that I do not
> know
> where I can check how
> many maps do I have at the same time and where I can change a value... is
> it
> possible that problem is in bad re-using of
> org.apache.hadoop.conf.Configuration in my code?
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-tp4052728p4052730.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

Jean-Marc Spaggiari
So really, I don't think it's from you code.

Looking at the stacktrace, seems that Gora is creating one connection per
task, and your number of concurrent tasks might be bigger than 30.

Since tasks might run in different nodes, it's a big difficult to share the
connection betweek the tasks.

so I guess you 2 best options are to reduce the tasks to make sure you have
less than 30 at a time, or to increase the ZK limit...

JM


2013/11/12 Jean-Marc Spaggiari <[hidden email]>

> I look very quickly at your code and I have not seend anything wrong so
> far. I will take a deeper look.
>
> For the number of maps/slots available, you can take a look at the
> jobtracker interface http://youserver:50030/jobtracker.jsp
>
> I will take another look at your code now and keep you posted.
>
>
> 2013/11/12 glumet <[hidden email]>
>
>> My data are stored in hBase. And I am trying to put them from hBase to
>> Solr.
>> I have written my own mapper and reducer (I put it here if you are
>> interested http://pastebin.com/jsKqiZTw). I have to admit that I do not
>> know
>> where I can check how
>> many maps do I have at the same time and where I can change a value... is
>> it
>> possible that problem is in bad re-using of
>> org.apache.hadoop.conf.Configuration in my code?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-tp4052728p4052730.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

glumet
Thank you! I really appreciate you answers because I learned many new things about hbase during last two days. It looks that my jobtracker is not listening at port 50030, but I noticed there are other ports 60010 and 60030

http://ir.lmcloud.vse.cz:60010/master.jsp
http://ir.lmcloud.vse.cz:60030/regionserver.jsp

I looked into the hbase-webapps folder and there are only two subfolders (master and regionserver) and under subfolders WEB-INF in web.xml file there are noticed only /master.jsp, /table.jsp, /zk.jsp (master's WEB-INF) and /regionserver.jsp (regionserver's WEB-INF) so maybe I have no jobtracker.jsp file or anything...

As you have written "reduce the task to have less than 30" so is this configured somewhere in hbase-site.xml? The second thing "increase the ZK limit" I tried to increase hbase.zookeeper.property.maxClientCnxns to 5000 and then to 0 (unlimited) but it failed in both cases...
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

glumet
I have found that somebody had the same exception here and he solved it with editing a /etc/hosts file. So I changed my lines to (previously the second line ip was 127.0.1.1) :

127.0.0.1       localhost
127.0.0.1       ir.lmcloud.vse.cz       ir

I tried it now and nothing has changed - it is running for a while (I mean for one hour and a half) and then Exception showed abowe in the first post appears. I really do not know where is the problem. I run hBase in standalone mode... tried to set the property for maxClientCnxns to 0, 5000, 1000000 but I still get the exception after some time.



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

Jean-Marc Spaggiari
Hi,

JobTracker is MapReduce...

So you should look on the MapReduce side.

Few links which might be interesting for you.

First, HBase ports =>
http://blog.cloudera.com/blog/2013/07/guide-to-using-apache-hbase-ports/
Hadoop =>
http://blog.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/

Take a look at all those UIs and ports.

To limit the number of tasks you might be able to do that on your code side
or on the MapReduce config file.

JM


2013/11/14 glumet <[hidden email]>

> I have found that somebody had the same exception  here
> <
> http://stackoverflow.com/questions/4942378/zookeeperconnectionexception-in-hbase-standalone-mode
> >
> and he solved it with editing a /etc/hosts file. So I changed my lines to
> (previously the second line ip was 127.0.1.1) :
>
> 127.0.0.1       localhost
> 127.0.0.1       ir.lmcloud.vse.cz       ir
>
> I tried it now and nothing has changed - it is running for a while (I mean
> for one hour and a half) and then Exception showed abowe in the first post
> appears. I really do not know where is the problem. I run hBase in
> standalone mode... tried to set the property for maxClientCnxns to 0, 5000,
> 1000000 but I still get the exception after some time.
>
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-tp4052728p4052804.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

Renato Marroquín Mogrovejo
Hi,

Did you give this [1] a look? Looks like the error you are getting.


Renato M.

[1] https://wiki.apache.org/nutch/ErrorMessagesInNutch2


2013/11/14 Jean-Marc Spaggiari <[hidden email]>

> Hi,
>
> JobTracker is MapReduce...
>
> So you should look on the MapReduce side.
>
> Few links which might be interesting for you.
>
> First, HBase ports =>
> http://blog.cloudera.com/blog/2013/07/guide-to-using-apache-hbase-ports/
> Hadoop =>
> http://blog.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/
>
> Take a look at all those UIs and ports.
>
> To limit the number of tasks you might be able to do that on your code side
> or on the MapReduce config file.
>
> JM
>
>
> 2013/11/14 glumet <[hidden email]>
>
> > I have found that somebody had the same exception  here
> > <
> >
> http://stackoverflow.com/questions/4942378/zookeeperconnectionexception-in-hbase-standalone-mode
> > >
> > and he solved it with editing a /etc/hosts file. So I changed my lines to
> > (previously the second line ip was 127.0.1.1) :
> >
> > 127.0.0.1       localhost
> > 127.0.0.1       ir.lmcloud.vse.cz       ir
> >
> > I tried it now and nothing has changed - it is running for a while (I
> mean
> > for one hour and a half) and then Exception showed abowe in the first
> post
> > appears. I really do not know where is the problem. I run hBase in
> > standalone mode... tried to set the property for maxClientCnxns to 0,
> 5000,
> > 1000000 but I still get the exception after some time.
> >
> >
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-tp4052728p4052804.html
> > Sent from the HBase User mailing list archive at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

glumet
Hi, I don't think that this is the same error as I have. My error and error from [1] have in common this warning: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default) but nothing else. But... I will try to replace hbase .jar with the newest because I tried to include mapred properties into my hbase-site.xml but no progress again.

<property>
<name>mapred.map.tasks</name>
<value>2</value>
</property>

<property>
<name>mapred.reduce.tasks</name>
<value>1</value>
</property>

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>2</value>
</property>

<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>2</value>
</property>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

Ted Yu-3
You should modify mapred properties in mapred-site.xml

Cheers


On Thu, Nov 14, 2013 at 9:44 AM, glumet <[hidden email]> wrote:

> Hi, I don't think that this is the same error as I have. My error and error
> from [1] have in common this warning: /HBase is able to connect to
> ZooKeeper
> but the connection closes immediately. This could be a sign that the server
> has too many connections (30 is the default)/ but nothing else. But... I
> will try to replace hbase .jar with the newest because I tried to include
> mapred properties into my hbase-site.xml but no progress again.
>
> <property>
> <name>mapred.map.tasks</name>
> <value>2</value>
> </property>
>
> <property>
> <name>mapred.reduce.tasks</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <value>2</value>
> </property>
>
> <property>
> <name>mapred.tasktracker.reduce.tasks.maximum</name>
> <value>2</value>
> </property>
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-tp4052728p4052816.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

glumet
I see. But I don't have such a file in my file system because I use only hbase, not hadoop.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: hBase - the server has too many connections (maxClientConn property set to 0 does not help)

glumet
Ok, I finally figured it out. I don't know what exactly solved the problem but I tried to combine these things:

1) property maxClientConnections set up to:

                        <property>
                                <name>hbase.zookeeper.property.maxClientCnxns</name>
                                <value>0</value>
                        </property>

2) increased time out

<property>
<name>zookeeper.session.timeout</name>
<value>1200000</value>
</property>

  <property>
    <name>hbase.zookeeper.property.tickTime</name>
    <value>6000</value>
  </property>

3) Nutch 2.1 + HBase 0.90.4 cluster settings

Then I killed all java processes, start and stop hBase and then run rolling-restart.sh and everything works fine now.
Loading...