[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[xmlblaster] server hangs when changing the size maximum of the DispathcWorkerPool

To: <xmlblaster at server.xmlBlaster.org>
Subject: [xmlblaster] server hangs when changing the size maximum of the DispathcWorkerPool
From: "Xavier Roques" <xRoques at infovista.com>
Date: Thu, 27 Sep 2007 18:04:11 +0200
In-reply-to: <46FA906C.9050601 at marcelruff.info>
References: 46FA3C73.3040605 at marcelruff.info <20070926124615.af31f4b0 at mail.openroadsconsulting.com> <0329C0D74DD1D64F98E6C6B43B5C7810031F9EDF at frlulms01.iv.local> <46FA906C.9050601 at marcelruff.info>
Reply-to: xmlblaster at server.xmlBlaster.org
Sender: owner-xmlblaster at server.xmlBlaster.org
Thread-index: AcgAX7NWqorcCbl0Q4yOZdcfECq3BgAvSj1A
Thread-topic: [xmlblaster] server hangs when changing the size maximum of the DispathcWorkerPool
Hi Marcel,

I'm using your last fix in order to avoid dead lock.
But I encountered a new problem.

I changed my server configuration in order to have less threads.

So I added the following lines: 
dispatch/callback/maximumPoolSize=10
dispatch/callback/minimumPoolSize=5

But therefore I was adding some clients to my server, suddenly, I was
not more able to connect to my server.
It seems to be hung.

Below it the dump of the threads (Unfortunately the Dispathworker 8 and
9 are missing )

Thanks in advance for your answer,

Xavier.
 
 "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-7" daemon
prio=10 tid=0x010fb400 nid=0xbb3 waiting for monitor entry
[0xd308f000..0xd308fb70]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:103)
 	- waiting to lock <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xdf5776b0> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp
atchManager.java:674)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D
ispatchManager.java:888)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
 	at java.lang.Thread.run(Thread.java:619)
 
 
 "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-6" daemon
prio=10 tid=0x0015a000 nid=0x199 waiting for monitor entry
[0xd49ff000..0xd49ff870]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:103)
 	- waiting to lock <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xdff773b8> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp
atchManager.java:674)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D
ispatchManager.java:888)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
 	at java.lang.Thread.run(Thread.java:619)
 
 
 "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-5" daemon
prio=10 tid=0x007a7000 nid=0x171 waiting for monitor entry
[0xd3c5f000..0xd3c5f870]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:103)
 	- waiting to lock <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xdd854e80> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp
atchManager.java:674)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D
ispatchManager.java:888)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
 	at java.lang.Thread.run(Thread.java:619)
 
 "XmlBlaster.SubjectEntryShuffler" daemon prio=10 tid=0x007a9000
nid=0x170 in Object.wait() [0xd323f000..0xd323f9f0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at java.lang.Object.wait(Object.java:485)
 	at
EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.put(Unknown Source)
 	- locked <0xe1a81d48> (a
EDU.oswego.cs.dl.util.concurrent.LinkedNode)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$WaitWhenBlocked.blockedA
ction(Unknown Source)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(Unknown Source)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:107)
 	- locked <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xe16a3f48> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.notifyAboutNewEntry(Dispatc
hManager.java:652)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.putPost(DispatchManager.jav
a:570)
 	at
org.xmlBlaster.util.queue.ram.RamQueuePlugin.put(RamQueuePlugin.java:764
)
 	at
org.xmlBlaster.authentication.SessionInfo.queueMessage(SessionInfo.java:
497)
 	at
org.xmlBlaster.authentication.SubjectInfo.forwardToSessionQueue(SubjectI
nfo.java:659)
 	at
org.xmlBlaster.authentication.SubjectInfo.forwardToSessionQueue(SubjectI
nfo.java:588)
 	at
org.xmlBlaster.engine.SubjectEntryShuffler.run(SubjectEntryShuffler.java
:88)
 	at java.lang.Thread.run(Thread.java:619)
 
 
 "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-4" daemon
prio=10 tid=0x00c02c00 nid=0xcd waiting for monitor entry
[0xd3b6f000..0xd3b6fbf0]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:103)
 	- waiting to lock <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xdad7fe90> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp
atchManager.java:674)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D
ispatchManager.java:888)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
 	at java.lang.Thread.run(Thread.java:619)
 
 "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-3" daemon
prio=10 tid=0x00b76400 nid=0xcc waiting for monitor entry
[0xd3b9e000..0xd3b9fa70]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:103)
 	- waiting to lock <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xe21c2d50> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.notifyAboutNewEntry(Dispatc
hManager.java:652)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.putPost(DispatchManager.jav
a:570)
 	at
org.xmlBlaster.util.queue.ram.RamQueuePlugin.put(RamQueuePlugin.java:764
)
 	at
org.xmlBlaster.engine.TopicHandler.invokeCallback(TopicHandler.java:1390
)
 	at
org.xmlBlaster.engine.TopicHandler.invokeCallbackAndHandleFailure(TopicH
andler.java:1222)
 	at
org.xmlBlaster.engine.TopicHandler.publish(TopicHandler.java:741)
 	at
org.xmlBlaster.engine.RequestBroker.publish(RequestBroker.java:1591)
 	at
org.xmlBlaster.engine.RequestBroker.publish(RequestBroker.java:1368)
 	at
org.xmlBlaster.engine.RequestBroker.publish(RequestBroker.java:1356)
 	at
org.xmlBlaster.engine.XmlBlasterImpl.publishArr(XmlBlasterImpl.java:198)
 	at
org.xmlBlaster.client.protocol.local.LocalConnection.publishArr(LocalCon
nection.java:231)
 	at
org.xmlBlaster.client.dispatch.ClientDispatchConnection.publish(ClientDi
spatchConnection.java:233)
 	at
org.xmlBlaster.client.dispatch.ClientDispatchConnection.doSend(ClientDis
patchConnection.java:126)
 	at
org.xmlBlaster.util.dispatch.DispatchConnection.send(DispatchConnection.
java:231)
 	at
org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.send(DispatchCon
nectionsHandler.java:455)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:70)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.putPre(DispatchManager.java
:561)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.putPre(DispatchManager.java
:535)
 	at
org.xmlBlaster.util.queue.ram.RamQueuePlugin.put(RamQueuePlugin.java:723
)
 	at
org.xmlBlaster.client.XmlBlasterAccess.queueMessage(XmlBlasterAccess.jav
a:831)
 	at
org.xmlBlaster.client.XmlBlasterAccess.publish(XmlBlasterAccess.java:991
)
 	at
com.infovista.foundation.middleware.mom.xmlblaster.XBSession.publish(XBS
ession.java:333)
 	at
com.infovista.foundation.middleware.mom.Session.send(Session.java:107)
 	at
com.infovista.foundation.jmx.event.LoginAcknowledger.loginMessage(LoginA
cknowledger.java:223)
 	at
com.infovista.foundation.jmx.event.LoginAcknowledger.onMessage(LoginAckn
owledger.java:123)
 	at
com.infovista.foundation.middleware.mom.xmlblaster.XBSession.update(XBSe
ssion.java:257)
 	at
org.xmlBlaster.client.XmlBlasterAccess.update(XmlBlasterAccess.java:1082
)
 	at
org.xmlBlaster.client.protocol.AbstractCallbackExtended.update(AbstractC
allbackExtended.java:111)
 	at
org.xmlBlaster.client.protocol.AbstractCallbackExtended.update(AbstractC
allbackExtended.java:199)
 	at
org.xmlBlaster.protocol.local.CallbackLocalDriver.sendUpdate(CallbackLoc
alDriver.java:139)
 	at
org.xmlBlaster.engine.dispatch.CbDispatchConnection.doSend(CbDispatchCon
nection.java:292)
 	at
org.xmlBlaster.util.dispatch.DispatchConnection.send(DispatchConnection.
java:231)
 	at
org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.send(DispatchCon
nectionsHandler.java:455)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:114)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
 	at java.lang.Thread.run(Thread.java:619)
 
 "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-2" daemon
prio=10 tid=0x00c06000 nid=0xcb waiting for monitor entry
[0xd3bcf000..0xd3bcfb70]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:103)
 	- waiting to lock <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xdf576920> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp
atchManager.java:674)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D
ispatchManager.java:888)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
 	at java.lang.Thread.run(Thread.java:619)
 
 "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-1" daemon
prio=10 tid=0x00c05800 nid=0xca waiting for monitor entry
[0xd3bff000..0xd3bff8f0]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:103)
 	- waiting to lock <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xe052bdd8> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp
atchManager.java:674)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D
ispatchManager.java:888)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
 	at java.lang.Thread.run(Thread.java:619)
 
 "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-0" daemon
prio=10 tid=0x00c05400 nid=0xc9 waiting for monitor entry
[0xd3c2f000..0xd3c2f870]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo
ol.java:103)
 	- waiting to lock <0xdae67640> (a
org.xmlBlaster.util.dispatch.DispatchWorkerPool)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM
anager.java:722)
 	- locked <0xde036a68> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp
atchManager.java:674)
 	at
org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D
ispatchManager.java:888)
 	at
org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150)
 	at
EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
 	at java.lang.Thread.run(Thread.java:619)
 
 
 "XmlBlaster.MessageTimer" daemon prio=10 tid=0x0137cc00 nid=0xc6
waiting for monitor entry [0xd3cbf000..0xd3cbfaf0]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.engine.MsgUnitWrapper.toExpired(MsgUnitWrapper.java:521)
 	- locked <0xd83daa60> (a org.xmlBlaster.engine.MsgUnitWrapper)
 	at
org.xmlBlaster.engine.MsgUnitWrapper.timeout(MsgUnitWrapper.java:594)
 	at org.xmlBlaster.util.Timeout.run(Timeout.java:189)
 
 "XmlBlaster.TopicTimer" daemon prio=10 tid=0x00c01400 nid=0xc2 in
Object.wait() [0xd40ff000..0xd40ff8f0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at org.xmlBlaster.util.Timeout.run(Timeout.java:203)
 	- locked <0xdacbf2f8> (a org.xmlBlaster.util.Timeout)
 
 "XmlBlaster.PingTimer" daemon prio=10 tid=0x0137d800 nid=0xc1 waiting
for monitor entry [0xd48af000..0xd48af870]
    java.lang.Thread.State: BLOCKED (on object monitor)
 	at
org.xmlBlaster.util.dispatch.DispatchConnection.handleTransition(Dispatc
hConnection.java:523)
 	- waiting to lock <0xdff773b8> (a
org.xmlBlaster.util.dispatch.DispatchManager)
 	at
org.xmlBlaster.util.dispatch.DispatchConnection.ping(DispatchConnection.
java:350)
 	at
org.xmlBlaster.util.dispatch.DispatchConnection.timeout(DispatchConnecti
on.java:423)
 	at org.xmlBlaster.util.Timeout.run(Timeout.java:189)
  
 "XmlBlaster.PingTimer" daemon prio=10 tid=0x008d7400 nid=0xb4 in
Object.wait() [0xd3d6f000..0xd3d6fbf0]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at org.xmlBlaster.util.Timeout.run(Timeout.java:203)
 	- locked <0xdacbe8a8> (a org.xmlBlaster.util.Timeout)
 
 
 "XmlBlaster.HttpIORServer" daemon prio=10 tid=0x0130c000 nid=0x66
runnable [0xd4baf000..0xd4bafaf0]
    java.lang.Thread.State: RUNNABLE
 	at java.net.PlainSocketImpl.socketAccept(Native Method)
 	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 	- locked <0xda94f2d8> (a java.net.SocksSocketImpl)
 	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
 	at java.net.ServerSocket.accept(ServerSocket.java:421)
 	at
org.xmlBlaster.util.http.HttpIORServer.run(HttpIORServer.java:157)
 
 "XmlBlaster.SocketDriver" daemon prio=10 tid=0x01309c00 nid=0x65
runnable [0xd4bdf000..0xd4bdfa70]
    java.lang.Thread.State: RUNNABLE
 	at java.net.PlainSocketImpl.socketAccept(Native Method)
 	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 	- locked <0xda94f1e8> (a java.net.SocksSocketImpl)
 	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
 	at
com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketI
mpl.java:259)
 	at
org.xmlBlaster.protocol.socket.SocketDriver.run(SocketDriver.java:533)
 
 "XmlBlaster.SocketDriver" daemon prio=10 tid=0x013a7000 nid=0x64
runnable [0xd4c0f000..0xd4c0fbf0]
    java.lang.Thread.State: RUNNABLE
 	at java.net.PlainSocketImpl.socketAccept(Native Method)
 	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 	- locked <0xda94f260> (a java.net.SocksSocketImpl)
 	at java.net.ServerSocket.implAccept(ServerSocket.java:453)
 	at java.net.ServerSocket.accept(ServerSocket.java:421)
 	at
org.xmlBlaster.protocol.socket.SocketDriver.run(SocketDriver.java:533)
 
 "XmlBlaster.SessionTimer" daemon prio=10 tid=0x013a7800 nid=0x63 in
Object.wait() [0xd4c3f000..0xd4c3fb70]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	at org.xmlBlaster.util.Timeout.run(Timeout.java:203)
 	- locked <0xda949870> (a org.xmlBlaster.util.Timeout)
 
 "XmlBlaster.Consumer" daemon prio=10 tid=0x003fe400 nid=0x62 in
Object.wait() [0xd4c6f000..0xd4c6f8f0]
    java.lang.Thread.State: WAITING (on object monitor)
 	at java.lang.Object.wait(Native Method)
 	- waiting on <0xdac36698> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$Seria
lizableLock)
 	at java.lang.Object.wait(Object.java:485)
 	at
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(
LinkedBlockingQueue.java:316)
 	- locked <0xdac36698> (a
edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$Seria
lizableLock)
 	at
org.xmlBlaster.engine.TopicAccessor$Consumer.run(TopicAccessor.java:451)
 	at java.lang.Thread.run(Thread.java:619)
Follow-Ups:
- Re: [xmlblaster] server hangs when changing the size maximum of the DispathcWorkerPool
  - From: Marcel Ruff <mr at marcelruff.info>
References:
- Re: [xmlblaster] dead lock
  - From: "David Robison" <drrobison at openroadsconsulting.com>
- RE: [xmlblaster] dead lock
  - From: "Xavier Roques" <xRoques at infovista.com>
- Re: [xmlblaster] dead lock
  - From: Marcel Ruff <mr at marcelruff.info>
Prev by Date: Re: [xmlblaster] dead lock
Next by Date: Re: [xmlblaster] server hangs when changing the size maximum of the DispathcWorkerPool
Previous by thread: RE: [xmlblaster] dead lock
Next by thread: Re: [xmlblaster] server hangs when changing the size maximum of the DispathcWorkerPool
Index(es):
- Date
- Thread