Re: [xmlblaster] Durable QoS performance (fwd)

[Heinrich Götzger <goetzger at gmx.net>]

Approved by maintainer.

Doug.Palmer at csiro.au wrote:

>>If the issue is urgent you would need to code it yourself.
>>
>>
>
>It's not urgent in the sense that there's an mission critical application
>waiting on it. But I've got an obvious desire to be able to say "it works
>reliably with enough throughput to be usable for a distributed sensor
>application, even if it is a pig to program and configure".
>
:-)

>(Just to give you an idea if what I'm thinking of. The basic aim is to build
>a middleware layer for a set of distributed environmental monitoring sensors
>so that land & water scientists can build an accurate picture of the state
>of an area. The data comes from various wireless links. The data needs to be
>delivered to databases, web pages, other applications ... So I think you can
>see why xmlBlaster looks attractive. To begin with, we're talking on the
>order of 1000 sensors, each generating a reading a minute. So that's on the
>order of 20 durable messages a second, with enough fat built in to cope with
>expansion and traffic bursts.)
>
>
>
>>The oid is one message, you can collect message to groups with XPath
>>subscriptions.
>>
>>
>
>I've replaced Oids with
>
>      this.publishKey =
>        "<key oid='' contentMime='application/octet-stream'>\n" +
>        "   <" + this.groupName + "></" + this.groupName + ">\n" +
>        "</key>";
>
>for the publisher and
>
>      this.subscribeKey =
>        "<key oid='' queryType='XPATH'>" +
>        "/xmlBlaster/key/" + this.groupName +
>        "</key>";
>
>for the subscriber. Is this correct?
>
>This seems to be working, up to a point. (I do have to remember to turn INFO
>level logging off, to stop it telling me that it's found a match for every
>message.)
>
We should remove the INFO ...

> But the overall performance seems to have taken a dive:
>
>Size	MPS	Sdrs	Rcvrs	Subj	Time		Sent	Rcvd	TS
>TR
>4096	10	10	1	1	119999	1200	1200	10	10
>4096	20	10	1	1	119984	2400	2400	20	20
>4096	50	10	1	1	119999	4541	4531	38	38
>4096	100	10	1	1	119984	7593	7583	63	63
>
If you use the same oid="sensor" for all messages, it should boost
performance.

>
>Part of the problem seems to be that when I stop a test and start another, I
>start getting messages of the form
>
>[Jul 8, 2002 11:14:21 AM WARN  RequestBroker-http://152.83.91.4:3412]
>Generating dead letter oid=http://A.B.C.D:3412-1026090587859-72 from
>publisher=guest because delivery to 'guest' cbQueue=session:23 failed. --
>I've removed my IP address.
>
>Eventually, the dead letter queue fills up and the whole thing dies in a
>welter of OutOfMemoryErrors. To shut down, I'm unSubscribing the receiers,
>using the Oid supplied by subcribe() method. I then disconnect() and then
>shutdown(). (I've tried assorted variations on this pattern.) Is this right?
>Do I have to explicitly erase every message I've sent?
>
The problem is that as a default the last message of every key-oid remains
in memory. So in your case just use the same oid for all messages.
Or erase them or set them to 'isVolatile'.
But the best choice for you is probably to use the same oid.

>
>The above is for non-durable messaging. With durable, I get:
>
>id=FileUtil reason=Can't write file java.io.FileNotFoundException:
>C:\Documents and
>Settings\dougp\tmp\http:\A.B.C.D:3412-1026091518765-11-XmlKey.xml (The
>filename, directory name, or volume label syntax is incorrect)
>
>Which is caused by the ':' being illegal in filenames under Win2k.
>(Incidentally, why am I getting http: to begin with? I thought that IIOP was
>the default. This could be an explanation for the difference in performance
>between your and my benchmarks. The clients are using the inbuilt
>xmlBlaster.properties file from the xmlBlaster jar.)
>
>Another thing is that I'm getting lots of messages of the form "Looping
>nanoCounter=1". The code in Timeout.java says that I should be only getting
>this to avoid two similar keys and it shouldn't happen very often. Since the
>oids are generated, and look pretty different, should this occur?
>
I will remove the message, as they are not important.

>
>Last thing. Doing pub/sub via raw XML seems awfully laborious. Particularly
>since the PublishKeyWrapper and SubscribeKeyWrapper effectively provide an
>API for pub/sub via oids. I would have thought that, since oids seem to be
>individually message-specific, the wrappers would provide an API for simple
>pub/sub using topic names, which is what most people would want to do.
>Instead, the Java demo examples and API seem to positively encourage a
>newbie like me to use oids as a quick'n'dirty subject name. (Don't get me
>wrong, I'm immensely impressed by the coolness of using XPath to allow
>subscriptions to almost anything under the sun. And it's got applications in
>the area I'm thinking of. It's just that I feel that the API conspires to
>discourage me from using it.)
>
The main thing is NOT to use too many different oid's as they are
performance hungry
and memory hungry.

>
>
>
>>This is strange, the default callback only gets PtP messages and  subcribed
>>
>>
>messages
>
>
>>which didn't specify an own Callback on subscription.
>>If i invoke 'java HelloWorld4' this seems to work fine.
>>
>>
>
>>>From HelloWorld4, I get:
>
>[8/07/2002 13:28:01 INFO  HelloWorld4] Reveiving asynchronous message
>'HelloWorld4' in HelloWorld4 handler
>[8/07/2002 13:28:01 INFO  HelloWorld4] Reveiving asynchronous message
>'SomeOtherMessage' in default handler
>
>Which I think is what's supposed to happen.
>
>It's the 'default handler' bit that bothers me. How do I stop a connection
>>from receiving 'SomeOtherMessage' and having it go to the default handler,
>if I don't want to see it at all? I would have thought that there's a
>default 'explicit subscriptions only' option, to avoid having a client
>drowning in extraneous messages.
>
You only get what you subscribed for.
Only PtP messages may arrive like a flood of spam ...

regards,

Marcel