Re: [xmlblaster] Callback message queue fills up

["David Robison" <drrobison at openroadsconsulting.com>]

I think part of the problem might be that the subscriptions, even when you specify a domain, are not domain specific. What I mean is that a user connected to B subscribes to messages for a domain that is mastered on A. However, when the subscription is forwarded to A, it matches messages from all domains, even those generated on B and sent to A. Does this make sense? Could this be part of the problem?

David

---

From: David R Robison [mailto:drrobison at openroadsconsulting.com]
To: xmlblaster at server.xmlBlaster.org
Sent: Wed, 21 Nov 2007 10:41:10 -0500
Subject: Re: [xmlblaster] Callback message queue fills up

Here is a dunp of one of the messages:

<MsgUnit index='0'>

<key oid='DomainHeartbeat-Albemarle911' contentMime='text/xml'
contentMimeExtended='1.0' domain='Albemarle911'/>
<content size='46'>Domain Albemarle911 ALIVE at 11/21/07
09:48:43</content>

<qos>
<subscribable/>
<sender>/node/Albemarle911/client/A-NATIVE-CLIENT-PLUGIN/-3</sender>
<priority>MAX</priority>
<subscribe id='__subId:StauntonSTC-XPATH1195628463329000000'/>
<expiration lifeTime='30000' remainingLife='22703' forceDestroy='true'/>
<rcvTimestamp nanos='1195659482613000002'/>
<queue index='0' size='1'/>
<persistent>false</persistent>
<isUpdate/>
</qos>
</MsgUnit>

The message was created on node B and sent to node A because of a
subscription on node A. But it is now in the callback queue on A to go
back to B. Also, I have never seen the route data in the messages. Is
there a way to turn this on?

David

Marcel Ruff wrote:
> David R Robison wrote:
>> One other thought. Heartbeat messages are published on node B and
>> subscribed to by clients on node A. Also, there are clients on node B
>> that subscribe to messages on node A. However, it appears that the
>> subscriptions the clients on node B are using are also matching the
>> heartbeat messages from node B that have been sent to node A. Could I
>> have some kind of circular queue? A message is posted on B then sent
>> to A because a subscription by a client on A. Then sent back to B
>> because of a subscription by a client on B for messages on A. Then
>> the message gets sent back to A and the whole cycle repeats?
> Could be, usually the cluster should prevent this ...
> The messages contain in their QoS the nodes traversed:
>
> <qos>
> <sender>joe</sender>
> <route>
> <node id='bilbo' stratum='2' timestamp='34460239640'/>
> <node id='frodo' stratum='1' timestamp='34460239661'/>
> <node id='heron' stratum='0' timestamp='34460239590'/>
> </route>
> </qos>
>
> it would be nice to see the dump of such messages,
> Use the jconsole or logging output from your receiving client or use the
> message sniffer, e.g.:
> java javaclients.simplereader.SimpleReaderGui -xpath "//key"
> -session.name simpleReader -passwd secret -protocol SOCKET
> -dispatch/connection/plugin/socket/hostname 192.168.1.25 -dumpToFile true
> or peek the callback queue with administrative messages as described
> in one of your last posts,
>
> thanks
> Marcel
>
>>
>> Could this be possible? David
>>
>> David R Robison wrote:
>>> Thanks, See in line...
>>>
>>> Marcel Ruff wrote:
>>>> Hi David,
>>>>
>>>> do you have a jconsole to observe the two nodes?
>>> I don't have a jconsole, but can I get the same using the admin
>>> messages?
>>>>
>>>> If yes, please check the number of subscriptions the node A has
>>>> forwarded to node B
>>>> (look into node B and check the number of subscriptions of client
>>>> A) during such a case.
>>>> In case the subscribeQos has set
>>> I will check.
>>>>
>>>> <multiSubscribe>true</multiSubscribe>
>>> I believe that we set all to false.
>>>>
>>>> (which is the default) it could be that the subscriptions multiplied
>>>> during small connection errors and reconnects.
>>>> This is just a guess.
>>>> If it is the case please set multiSubscribe to false.
>>>>
>>>> Is there a high CPU load during the 1001 message case?
>>> No
>>>> Are the hearbeat messages persistent messages?
>>> Yes, but the only live 30 seconds. At any given time there should
>>> only be at most 2 in the history queue
>>>> Was the client connected or offline during this message overflow?
>>> No, the client was online
>>>> Does your heartbeat have a unique id so that you can tell for sure
>>>> if the same
>>> No, but the content of the message has a timestamp so I knew they
>>> were duplicates
>>>> published message is cloned many times (try a peek on the callback
>>>> queue with jconsole)?
>>> Can this be done with the admin messages
>>>>
>>>> A final option is to use the current svn xmlBlaster and switch on
>>>> the checkpoint logging
>>>> to get a better idea what is going on.
>>> We will try this in house, unfortunately, the problem nodes are in a
>>> production environment.
>>>>
>>>> And finally it could be a problem with your client not taking the
>>>> callback messages.
>>> Could be, but what I don't see is the queue gradually growing.
>>> Instead, it "all-of-a-sudden" appears to be full.
>>>>
>>>> Another idea: The callback queue contains only a reference on the
>>>> message.
>>>> If it expires the message-'meat' is destroyed but the reference
>>>> remains in the queue
>>>> until it is looked at during delivery (and then thrown to garbage),
>>>> Michele, could this be?
>>>>
>>>> thanks
>>>> Marcel
>>>>
>>>>
>>>> David R Robison wrote:
>>>>> We are experiencing something strange in xmlBlaster 1.6.1. We have
>>>>> two nodes, node A subscribes to messages from node B. These are
>>>>> heartbeat messages and are generated every 15 seconds with a
>>>>> lifetime of 30 seconds. A client connects to node A and subscribes
>>>>> to the messages, node A then passes the subscription onto node B.
>>>>> Watching the callback message queue, everything seems to run well,
>>>>> at most 1 message in the queue waiting to be sent. It can run like
>>>>> this for days. Then, unexpectedly, the callback queue will show as
>>>>> being full (in this case 1001 messages). The queue contains many
>>>>> duplicated messages with different timestamps. From there, the
>>>>> server struggles to deliver the messages and keep the queue empty.
>>>>> The reader never seems to read enough messages to get the queue
>>>>> back down to zero. If I stop the client and reconnect, it will
>>>>> recreate its queue and be back to normal. I know this is a bit
>>>>> sketchy, but it is becoming a real problem for us.
>>>>>
>>>>> Any thoughts on what might be the problem? Any idea of where to
>>>>> start looking?
>>>>>
>>>>> One more note, when the client is subscribing to heartbeats that
>>>>> are generated on Node A, the client never fails in this manor,
>>>>> only when it is subscribing to node A for a message generated on
>>>>> node B.
>>>>>
>>>>> Thanks, in advance,
>>>>> David Robison
>>>>>
>>>>
>>>
>>
>
>

--

David R Robison
Open Roads Consulting, Inc.
708 S. Battlefield Blvd., Chesapeake, VA 23322
phone: (757) 546-3401
e-mail: drrobison at openroadsconsulting.com
web: http://openroadsconsulting.com
blog: http://therobe.blogspot.com
book: http://www.xulonpress.com/book_detail.php?id=2579