Re: [xmlblaster-devel] Lock issue with corba protocoll

[Michele Laghi <laghi at swissinfo.org>]

A temporary solution would be to use sockets
Michele

Marcel Ruff wrote:
> Peter Antman wrote:
>
> > Hi,
> > I have stumbled on a lock/freeze issue with the corba driver, that I
> > have a hard time understanding.
> >
> > This seems to happen when I set up several connections at once.
> >
> > Here:s one scenario where I set up a pool of connections with
> > potentially several subscribers per connections. Once in a while the
> > first subscription will freeze. Here's a the thread stack dumps I can
> > find that affects this:
> >
> > On the client side, adding a subscriber:
> > "Thread Pool Worker-0" prio=1 tid=0x0x92f7b78 nid=0x2f8e in
> > Object.wait() [52a02000..52a03840]
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0x47ca8f50> (a org.jacorb.orb.ReplyReceiver)
> >     at java.lang.Object.wait(Object.java:426)
> >     at
> > org.jacorb.orb.connection.ReplyPlaceholder.getInputStream(ReplyPlaceholder.java:134) 
> >
> >     - locked <0x47ca8f50> (a org.jacorb.orb.ReplyReceiver)
> >     at org.jacorb.orb.ReplyReceiver.getReply(ReplyReceiver.java:230)
> >     - locked <0x47ca8f50> (a org.jacorb.orb.ReplyReceiver)
> >     at org.jacorb.orb.Delegate.invoke_internal(Delegate.java:851)
> >     at org.jacorb.orb.Delegate.invoke(Delegate.java:762)
> >     at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:457)
> >     at
> > org.xmlBlaster.protocol.corba.authenticateIdl._AuthServerStub.connect(_AuthServerStub.java:78) 
> >
> >     at
> > org.xmlBlaster.client.protocol.corba.CorbaConnection.connect(CorbaConnection.java:501) 
> >
> >     at
> > org.xmlBlaster.client.dispatch.ClientDeliveryConnection.connect(ClientDeliveryConnection.java:390) 
> >
> >     at
> > org.xmlBlaster.client.dispatch.ClientDeliveryConnection.doSend(ClientDeliveryConnection.java:144) 
> >
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryConnection.send(DeliveryConnection.java:195) 
> >
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryConnectionsHandler.send(DeliveryConnectionsHandler.java:357) 
> >
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryWorker.run(DeliveryWorker.java:76)
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryManager.putPre(DeliveryManager.java:446) 
> >
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryManager.putPre(DeliveryManager.java:425) 
> >
> >     at
> > org.xmlBlaster.util.queue.ram.RamQueuePlugin.put(RamQueuePlugin.java:642)
> >     at
> > org.xmlBlaster.client.XmlBlasterAccess.queueMessage(XmlBlasterAccess.java:552) 
> >
> >     at
> > org.xmlBlaster.client.XmlBlasterAccess.connect(XmlBlasterAccess.java:241)
> >     - locked <0x47b73048> (a org.xmlBlaster.client.XmlBlasterAccess)
> >     at org.backsource.in.blaster.Connection.setUp(Connection.java:93)
> >     at org.backsource.in.blaster.Connection.<init>(Connection.java:84)
> >     at
> > org.backsource.in.blaster.ConnectionPool.getConnection(ConnectionPool.java:149) 
> >
> >     at
> > org.backsource.in.blaster.ConnectionPool.addSubscriber(ConnectionPool.java:89) 
> >
> >     - locked <0x468877b0> (a org.backsource.in.blaster.ConnectionPool)
> >     at
> > org.backsource.in.blaster.BlasterSubscriberManagerJMX.addSubscriber(BlasterSubscriberManagerJMX.java:324) 
> >
> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >     at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> >
> >     at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
> >
> >     at java.lang.reflect.Method.invoke(Method.java:324)
> >     at
> > org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284) 
> >
> >     at 
> > org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
> >     at org.backsource.jmx.MBeanProxy.invoke(MBeanProxy.java:230)
> >     at $Proxy48.addSubscriber(Unknown Source)
> >     at
> > org.backsource.in.blaster.server.SubscriberManagerListener.onMessage(SubscriberManagerListener.java:99) 
> >
> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >     at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
> >
> >     at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
> >
> >     at java.lang.reflect.Method.invoke(Method.java:324)
> >     at
> > org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:391) 
> >
> >     at
> > org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186) 
> >
> >     at
> > org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:88) 
> >
> >     at
> > org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:107) 
> >
> >     at
> > org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:237) 
> >
> >     at
> > org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:98)
> >     at
> > org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:100) 
> >
> >     at 
> > org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:208)
> >     at
> > org.jboss.ejb.MessageDrivenContainer.invoke(MessageDrivenContainer.java:302) 
> >
> >     at
> > org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:962) 
> >
> >     at
> > org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1038) 
> >
> >     at
> > org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:243)
> >     at
> > org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:603) 
> >
> >     at
> > org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:417)
> >     - locked <0x468913e8> (a java.util.LinkedList)
> >     at org.jboss.mq.SpySession.run(SpySession.java:296)
> >     - locked <0x4688dd48> (a java.util.LinkedList)
> >     at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180)
> >     at
> > EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:727) 
> >
> >     at java.lang.Thread.run(Thread.java:536)
> >
> > "Connection Consumer for dest TOPIC.subscriberQueue" prio=1
> > tid=0x0x50352a20 nid=0x2f8c in Object.wait() [52982000..52982840]
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0x468a4200> (a java.util.LinkedList)
> >     at java.lang.Object.wait(Object.java:426)
> >     at
> > org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:202)
> >     - locked <0x468a4200> (a java.util.LinkedList)
> >     at java.lang.Thread.run(Thread.java:536)
> >
> > On the server side, loging the session in:
> >
> > "RequestProcessor-5" daemon prio=1 tid=0x0x921f930 nid=0x2fa1 in
> > Object.wait() [53315000..53315840]
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0x47ca8970> (a org.jacorb.orb.ReplyReceiver)
> >     at java.lang.Object.wait(Object.java:426)
> >     at
> > org.jacorb.orb.connection.ReplyPlaceholder.getInputStream(ReplyPlaceholder.java:134) 
> >
> >     - locked <0x47ca8970> (a org.jacorb.orb.ReplyReceiver)
> >     at org.jacorb.orb.ReplyReceiver.getReply(ReplyReceiver.java:230)
> >     - locked <0x47ca8970> (a org.jacorb.orb.ReplyReceiver)
> >     at org.jacorb.orb.Delegate.invoke_internal(Delegate.java:851)
> >     at org.jacorb.orb.Delegate.invoke(Delegate.java:762)
> >     at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:457)
> >     at
> > org.xmlBlaster.protocol.corba.clientIdl._BlasterCallbackStub.ping(_BlasterCallbackStub.java:31) 
> >
> >     at
> > org.xmlBlaster.protocol.corba.CallbackCorbaDriver.ping(CallbackCorbaDriver.java:186) 
> >
> >     at
> > org.xmlBlaster.engine.dispatch.CbDeliveryConnection.connectLowlevel(CbDeliveryConnection.java:92) 
> >
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryConnection.initialize(DeliveryConnection.java:107) 
> >
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryConnectionsHandler.initialize(DeliveryConnectionsHandler.java:143) 
> >
> >     - locked <0x47ca8b28> (a java.util.ArrayList)
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryManager.<init>(DeliveryManager.java:115) 
> >
> >     at
> > org.xmlBlaster.authentication.SessionInfo.<init>(SessionInfo.java:144)
> >     at
> > org.xmlBlaster.authentication.Authenticate.connect(Authenticate.java:319)
> >     - locked <0x47ca8ce8> (a org.xmlBlaster.authentication.SubjectInfo)
> >     at
> > org.xmlBlaster.protocol.corba.AuthServerImpl.connect(AuthServerImpl.java:225) 
> >
> >     at
> > org.xmlBlaster.protocol.corba.AuthServerImpl.connect(AuthServerImpl.java:185) 
> >
> >     at
> > org.xmlBlaster.protocol.corba.authenticateIdl.AuthServerPOATie.connect(AuthServerPOATie.java:48) 
> >
> >     at
> > org.xmlBlaster.protocol.corba.authenticateIdl.AuthServerPOA._invoke(AuthServerPOA.java:55) 
> >
> >     at
> > org.jacorb.poa.RequestProcessor.invokeOperation(RequestProcessor.java:247) 
> >
> >     at org.jacorb.poa.RequestProcessor.process(RequestProcessor.java:496)
> >     at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:635)
> >
> >
> > This also happens sometimes for the publisher, using the k2 adapter.
> > However I have not been able to find the server side equivalence:
> > "Thread Pool Worker-6" prio=1 tid=0x0x82eab48 nid=0x1ca7 in
> > Object.wait() [5d6b7000..5d6b8840]
> >     at java.lang.Object.wait(Native Method)
> >     - waiting on <0x474da268> (a org.jacorb.orb.ReplyReceiver)
> >     at java.lang.Object.wait(Object.java:426)
> >     at
> > org.jacorb.orb.connection.ReplyPlaceholder.getInputStream(ReplyPlaceholder.java:134) 
> >
> >     - locked <0x474da268> (a org.jacorb.orb.ReplyReceiver)
> >     at org.jacorb.orb.ReplyReceiver.getReply(ReplyReceiver.java:230)
> >     - locked <0x474da268> (a org.jacorb.orb.ReplyReceiver)
> >     at org.jacorb.orb.Delegate.invoke_internal(Delegate.java:851)
> >     at org.jacorb.orb.Delegate.invoke(Delegate.java:762)
> >     at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:457)
> >     at
> > org.xmlBlaster.protocol.corba.authenticateIdl._AuthServerStub.ping(_AuthServerStub.java:31) 
> >
> >     at
> > org.xmlBlaster.client.protocol.corba.CorbaConnection.ping(CorbaConnection.java:757) 
> >
> >     at
> > org.xmlBlaster.client.dispatch.ClientDeliveryConnection.connectLowlevel(ClientDeliveryConnection.java:95) 
> >
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryConnection.initialize(DeliveryConnection.java:107) 
> >
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryConnectionsHandler.initialize(DeliveryConnectionsHandler.java:143) 
> >
> >     - locked <0x474da428> (a java.util.ArrayList)
> >     at
> > org.xmlBlaster.util.dispatch.DeliveryManager.<init>(DeliveryManager.java:115) 
> >
> >     at
> > org.xmlBlaster.client.XmlBlasterAccess.connect(XmlBlasterAccess.java:227)
> >     - locked <0x47442db0> (a org.xmlBlaster.client.XmlBlasterAccess)
> >     at
> > org.xmlBlaster.j2ee.k2.BlasterManagedConnection.doLogin(BlasterManagedConnection.java:388) 
> >
> >     at
> > org.xmlBlaster.j2ee.k2.BlasterManagedConnection.<init>(BlasterManagedConnection.java:138) 
> >
> >     at
> > org.xmlBlaster.j2ee.k2.BlasterManagedConnectionFactory.createManagedConnection(BlasterManagedConnectionFactory.java:101) 
> >
> >     at
> > org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnection(InternalManagedConnectionPool.java:359) 
> >
> >     at
> > org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:150) 
> >
> >     at
> > org.jboss.resource.connectionmanager.JBossManagedConnectionPool$OnePool.getConnection(JBossManagedConnectionPool.java:548) 
> >
> >     at
> > org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:467) 
> >
> >     at
> > org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:532) 
> >
> >     at
> > org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:822) 
> >
> >     at
> > org.xmlBlaster.j2ee.k2.BlasterConnectionFactoryImpl.getConnection(BlasterConnectionFactoryImpl.java:70) 
> >
> >     at
> > se.tim.in.kernel.channel.server.JmsAdapter.onMessage(JmsAdapter.java:190)
> >     at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
> >     at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
> >
> >     at java.lang.reflect.Method.invoke(Method.java:324)
> >     at
> > org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:391) 
> >
> >     at
> > org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186) 
> >
> >     at
> > org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:88) 
> >
> >     at
> > org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:107) 
> >
> >     at
> > org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:237) 
> >
> >     at
> > org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:98)
> >     at
> > org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:100) 
> >
> >     at 
> > org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:208)
> >     at
> > org.jboss.ejb.MessageDrivenContainer.invoke(MessageDrivenContainer.java:302) 
> >
> >     at
> > org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:962) 
> >
> >     at
> > org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1038) 
> >
> >     at
> > org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:243)
> >     at
> > org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:603) 
> >
> >     at
> > org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:417)
> >     - locked <0x46ebdc58> (a java.util.LinkedList)
> >     at org.jboss.mq.SpySession.run(SpySession.java:296)
> >     - locked <0x46ebaea8> (a java.util.LinkedList)
> >     at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180)
> >     at
> > EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:727) 
> >
> >     at java.lang.Thread.run(Thread.java:536)
> >
> > Any ideas about this??
>
>
> Yes i know about it, i have tried to track it down or have
> a reproducable test but i didn't succeed.
> I have reported it to JacORB but they didn't know how
> to track it down, so it remained an open issue.
> See
>   http://www.jacorb.org/bugzilla/
>   Bug #:  380
>
> AFAIK it is a mutlithreading bug in JacORB around a notify/wait
> in ReplyReceiver.java, if your case is reproducable we could debug it 
> and find
> the offending code
>
> Marcel
>
>
>
> > //Peter