Re: [xmlblaster] locking bug

[Marcel Ruff <mr at marcelruff.info>]

Hi,

i have looked at your thread dump, but i couldn't come to a conclusion,
it is not possible to see the lock-taking thread ...

Could you please send me the logfiles as well, during the time the
threads are consumed
(probably with  -logging/org.xmlBlaster.engine FINE
but this will blow up your files).

Could you please add in TopicAccessor.java:376
the keyword 'volatile':
     private volatile TopicHandler topicHandler;
It is just i very blind guess.

Can you make a grep in your log files on the line
  log.warning("Trying again to get a TopicHandler ...
if it is logged somewhere.

Is it always the same topicId which fails?
Do you have many other topics witch work fine?

Thanks
Marcel



Marcel Ruff schrieb:
> Póka Balázs schrieb:
> > Hello to all XmlBlaster developers!
> >
> > I may have discovered a locking problem in
> > org.xmlblaster.engine.TopicAccessor. The symptom is xmlBlaster running
> > out of memory. I made a stack dump the last time when this happened
> > (and it happens a lot), and there were innumerable threads stuck here:
> >
> > "XmlBlaster.HandleClient" prio=10 tid=0x00002aaac12ce800 nid=0xf62 in
> > Object.wait() [0x00002aaae417e000..0x00002aaae417eca0]
> >    java.lang.Thread.State: WAITING (on object monitor)
> >         at java.lang.Object.wait(Native Method)
> >         at java.lang.Object.wait(Object.java:485)
> >         at 
> > edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:199) 
> >
> >         - locked <0x00002aaab42ec888> (a
> > edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync) 
> >
> >         at 
> > edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:481) 
> >
> >         at 
> > org.xmlBlaster.engine.TopicAccessor$TopicContainer.lock(TopicAccessor.java:403) 
> >
> >         at 
> > org.xmlBlaster.engine.TopicAccessor.findOrCreate(TopicAccessor.java:179)
> >         at 
> > org.xmlBlaster.engine.RequestBroker.publish(RequestBroker.java:1677)
> >         at 
> > org.xmlBlaster.engine.RequestBroker.publish(RequestBroker.java:1405)
> >         at 
> > org.xmlBlaster.engine.RequestBroker.publish(RequestBroker.java:1393)
> >         at 
> > org.xmlBlaster.engine.XmlBlasterImpl.publish(XmlBlasterImpl.java:180)
> >         at 
> > org.xmlBlaster.engine.XmlBlasterImpl.publishArr(XmlBlasterImpl.java:219)
> >         at 
> > org.xmlBlaster.util.protocol.RequestReplyExecutor.receiveReply(RequestReplyExecutor.java:447) 
> >
> >         at 
> > org.xmlBlaster.protocol.socket.HandleClient.handleMessage(HandleClient.java:194) 
> >
> >         at 
> > org.xmlBlaster.protocol.socket.HandleClient$1.run(HandleClient.java:369)
> >         at 
> > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:665) 
> >
> >         at 
> > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:690) 
> >
> >         at java.lang.Thread.run(Thread.java:619)
> >
> > There was no deadlock. It seems that some thread locked the
> > ReentrantLock instance contained in
> > org.xmlblaster.engine.TopicAccessor$TopicContainer, and forgot to
> > unlock it.
> > The clients subsequently disconnected and reconnected to another
> > session id since from their point of view, the server did not respond,
> > thus the huge number of stale threads and clients resulted.
> >
> > I noted that the implementation of
> > org.xmlblaster.engine.TopicAccessor.release(...) seems problematic
> > since it may return _without unlocking_ under erroneous conditions,
> > while not even logging any INFO or higher level message! Maybe the
> > tc.unlock() call would need to be placed in a finally block. There may
> > as well be other places which may hold instances of this lock without
> > unlocking it ever again. 
> If the map throws a Runtime exception, yes. But this should
> be logged somewhere up the stack.
>
> Could you please send us the complete stack trace?
> Since when did you notice this behaviour?
> Which version of xmlBlaster do you use?
>
> Thanks
> Marcel
>
> > Thanks for your help.
> >
> > regards,
> > Balázs Póka
> >
> >   


--
Marcel Ruff
http://www.xmlBlaster.org
http://watchee.net
Phone: +49 7551 309371