Re: [xmlblaster] client stops receiving messages

[Michele <laghi at swissinfo.org>]

Hi Marshal,
could you send to me (laghi at swissinfo.org) and marcel 
(mr at marcelruff.info) the complete logs (if you have it possibily with 
traces) of xmlBlaster ?

Also the publishQos of the messages would be interesting. (to get them 
you could temporarly switch dump[socket] on).

Regards
Michele



Marshall Shapiro wrote:
> Marcel,
>
> The session is not persistent. Neither are the subscriptions. (I believe)
> the messages are also not persistent. Most have a very short lifetime. (The
> idea is to deliver to those listening and get rid of it). (I didn't
> implement all the clients so I can't be 100% sure that no client is
> mistakenly publishing persistent messages, but I don't think so).
>
> I'm not using a database, just flat files for storage. Is that a problem?
> (Btw, am I losing performance because of that? If I switch to a DB, do you
> recommend postgres or mysql?)
>
> The setting in xmlBlaster.properties related to cache size are:
>
> --------------------
> # How many history messages of same key oid (topic) to hold in xmlBlaster
> (holds references only)
> queue/history/maxEntriesCache = 10
> queue/history/maxEntries = ${queue/history/maxEntriesCache}
>
> # How big is a callback queue for each client (holds references only)
> queue/callback/maxEntriesCache = 1000
> queue/callback/maxEntries = ${queue/callback/maxEntriesCache}
>
> # How big is the topic cache for messages of same oid (here your message
> content occupies the memory)
> persistence/msgUnitStore/maxEntriesCache = 10000
> persistence/msgUnitStore/maxEntries = 100000000
> persistence/msgUnitStore/maxBytesCache = 2097152
> persistence/msgUnitStore/maxBytes = 2147483647
>
> # Tail back queue on client side (contains the real data of all message
> types)
> queue/connection/maxEntriesCache = 1000
> queue/connection/maxEntries = 10000000
> --------------------
>
> The callback queue is not full when the client stops getting messages, but
> it then fills up quickly. (Originally I thought the problem was that the
> callback queue was full, but after watching more carefully I saw that the
> client stops getting the messages first, and then, because of that, the
> queue fills up). I'm not sure what you mean by the callback queue swapping.
>
>
> Thanks again. Let me know if still need more information. Any insight into
> what is going on is appreciated.
>
>
> Marshall
>
>
> -----Original Message-----
> From: owner-xmlblaster at server.xmlblaster.org
> [mailto:owner-xmlblaster at server.xmlblaster.org] On Behalf Of Marcel Ruff
> Sent: Tuesday, October 11, 2005 7:13 AM
> To: xmlblaster at server.xmlblaster.org
> Subject: Re: [xmlblaster] client stops receiving messages
>
> Hi,
>
> which persistence configuration do you have, are the message persistent?
> Which database and pool sizes (like queue/callback/maxEntriesCache etc 
> from xmlBlaster.properties).
>
> Is the callback queue filled and swapping (checking with JMX)?
>
> thanks
> Marcel
>
> Marshall Shapiro wrote:
>
>
> > I am working on a project that is using xmlBlaster.
> > I have one client that stops receiving messages in the face of very 
> > high message rates and size. This client is observing all the messages 
> > in the system, so it is receiving both lot's of small messages 
> > (~2/sec) and some number of large messages (2-10 MB). At some peak 
> > moment in a run the client will stop receiving messages and doesn't 
> > start receiving them again. Using the jmx web page I can see that the 
> > session is still there and subscribed.
> >
> > The logs I'm going to show you are from a test I ran in which I was 
> > trying to break it. I ran the system normally for a while and then 
> > threw a bunch of large files at it (~9 MB each). I got the following 
> > errors. These are typical of the errors I see in the logs when the 
> > client breaks.
> >
> > I was hoping if I showed you excerpts from the logs, you could help me 
> > figure out what's going on with this client.
> >
> >
> > First I got some messages like this:
> >
> > [Oct 10, 2005 4:23:42 PM ERROR 
> > XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 
>
> MsgQueueUpdateEntry-callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/
> -4/NORM/1128975795940000000/ComponentHeartbeat] 
>
> > No no meat found, incr=-1
> >
> >
> >
> > Then I got a bunch of client side exceptions such as:
> >
> > [Oct 10, 2005 4:26:36 PM WARN  
> > XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 
> > RequestBroker] Generating dead message 
>
> 'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/11289759916
> 09000002/ComponentHeartbeat' 
>
> > from publisher=/node/xmlBlaster_195_1_1_41_3412/client/PD/-9 because 
> > delivery with queue 
> > 'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4' failed: 
> > XmlBlasterException errorCode=[user.update.internalError] 
> > serverSideException=false location=[AbstractCallbackExtended.update] 
> > message=[#exported Error in client code : 
> > java.lang.NullPointerException] [See URL 
>
> _http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.errorcodes.list
> ing.html#user.update.internalError_]
>
> > Then it got lots more of these errors. (These messages actually always 
> > show up with the problem):
> >
> > [Oct 10, 2005 4:30:08 PM ERROR 
> > XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-0 
> > MsgQueueUpdateEntry-callback:/node/xmlBlaster_195_1_1_41 
> > 3412/client/jaguar/-4/NORM/1128976172075000002/ComponentHeartbeat] No 
> > no meat found, incr=-1
> >
> >
> >
> > Followed by lots of these (I've often seen these null queue messages 
> > with this problem):
> >
> > [Oct 10, 2005 4:44:45 PM WARN  
> > XmlBlaster.DispatchWorkerPool.xmlBlaster_195_1_1_41_3412-1 
> > RequestBroker] Generating dead message 
>
> 'callback:/node/xmlBlaster_195_1_1_41_3412/client/jaguar/-4/NORM/11289770086
> 60000000/SystemHeartbeat' 
>
> > from publisher=/node/xmlBlaster_195_1_1_41_3412/client/MC/-6 because 
> > delivery with queue 'null' failed: XmlBlasterException 
> > serverSideException=true node=[xmlBlaster_195_1_1_41_3412] 
> > location=[MsgQueueUpdateEntry]
> >
> > stackTrace=errorCode=internal.unknown message=#exported Message 
> > 1128977060215000002 not found
> >        at 
>
> org.xmlBlaster.engine.queuemsg.ReferenceEntry.getMsgUnit(ReferenceEntry.java
> :289) 
>
> >        at 
>
> org.xmlBlaster.engine.dispatch.CbDispatchConnection.doSend(CbDispatchConnect
> ion.java:135) 
>
> >        at 
>
> org.xmlBlaster.util.dispatch.DispatchConnection.send(DispatchConnection.java
> :221) 
>
> >        at 
>
> org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.send(DispatchConnect
> ionsHandler.java:420) 
>
> >        at 
> > org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:166)
> >        at 
>
> EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.ja
> va:732) 
>
> >        at java.lang.Thread.run(Thread.java:536)
>
> versionInfo=version=1.0.6,revision=exported,os.name=Linux,os.version=2.4.21-
> 15.EL,java.vm.vendor=Blackdown 
>
> > Java-Linux 
>
> Team,java.vm.version=Blackdown-1.4.1-01,os.arch=i386,build.timestamp=08/10/2
> 005 
>
> > 01:45 PM,build.java.vendor=Sun Microsystems 
> > Inc.,build.java.version=1.4.2_06
> >
> > errorCode 
>
> description=http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.erro
> rcodes.listing.html#internal.unknown
>
> > I'm running xmlBlaster version 1.0.6. The client is written in Java, 
> > compiled using Java 1.4.2.
> >
> > Let me know if you need more information.
> >
> > Thank you in advance for your help.
> >
> > Marshall