I am working on a project that is using xmlBlaster.
I have one client that stops receiving messages in the face of very high message rates and size. This client is observing all the messages in the system, so it is receiving both lot's of small messages (~2/sec) and some number of large messages (2-10 MB). At some peak moment in a run the client will stop receiving messages and doesn't start receiving them again. Using the jmx web page I can see that the session is still there and subscribed.
The logs I'm going to show you are from a test I ran in which I was trying to break it. I ran the system normally for a while and then threw a bunch of large files at it (~9 MB each). I got the following errors. These are typical of the errors I see in the logs when the client breaks.
I was hoping if I showed you excerpts from the logs, you could help me figure out what's going on with this client.
First I got some messages like this:
[Oct 10, 2005 4:23:42 PM [31;40mERROR[0m XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 MsgQueueUpdateEntry-callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/1128975795940000000/ComponentHeartbeat] No no meat found, incr=-1
Then I got a bunch of client side exceptions such as:
[Oct 10, 2005 4:26:36 PM [33;40mWARN [0m XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 RequestBroker] Generating dead message 'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/1128975991609000002/ComponentHeartbeat' from publisher=/node/xmlBlaster_195_1_1_41_3412/client/PD/-9 because delivery with queue 'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4' failed: XmlBlasterException errorCode=[user.update.internalError] serverSideException=false location=[AbstractCallbackExtended.update] message=[#exported Error in client code : java.lang.NullPointerException] [See URL http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.errorcodes.listing.html#user.update.internalError]
Then it got lots more of these errors. (These messages actually always show up with the problem):
[Oct 10, 2005 4:30:08 PM [31;40mERROR[0m XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 MsgQueueUpdateEntry-callback:/node/xmlBlaster_195_1_1_41 3412/client/jaguar/-4/NORM/1128976172075000002/ComponentHeartbeat] No no meat found, incr=-1
Followed by lots of these (I've often seen these null queue messages with this problem):
[Oct 10, 2005 4:44:45 PM [33;40mWARN [0m XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-1 RequestBroker] Generating dead message 'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/1128977008660000000/SystemHeartbeat' from publisher=/node/xmlBlaster_195_1_1_41_3412/client/MC/-6 because delivery with queue 'null' failed: XmlBlasterException serverSideException=true node=[xmlBlaster_195_1_1_41_3412] location=[MsgQueueUpdateEntry]
stackTrace=errorCode=internal.unknown message=#exported Message 1128977060215000002 not found
at org.xmlBlaster.engine.queuemsg.ReferenceEntry.getMsgUnit(ReferenceEntry.java:289)
at org.xmlBlaster.engine.dispatch.CbDispatchConnection.doSend(CbDispatchConnection.java:135)
at org.xmlBlaster.util.dispatch.DispatchConnection.send(DispatchConnection.java:221)
at org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.send(DispatchConnectionsHandler.java:420)
at org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:166)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)
at java.lang.Thread.run(Thread.java:536)
versionInfo=version=1.0.6,revision=exported,os.name=Linux,os.version=2.4.21-15.EL,java.vm.vendor=Blackdown Java-Linux Team,java.vm.version=Blackdown-1.4.1-01,os.arch=i386,build.timestamp=08/10/2005 01:45 PM,build.java.vendor=Sun Microsystems Inc.,build.java.version=1.4.2_06
errorCode description=http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.errorcodes.listing.html#internal.unknown
I'm running xmlBlaster version 1.0.6. The client is written in Java, compiled using Java 1.4.2.
Let me know if you need more information.
Thank you in advance for your help.
Marshall