Re: [xmlblaster] Durable QoS performance

[Marcel Ruff <ruff at swand.lake.de>]

Hi Doug,

>It's not urgent in the sense that there's an mission critical application
>waiting on it. But I've got an obvious desire to be able to say "it works
>reliably with enough throughput to be usable for a distributed sensor
>application, even if it is a pig to program and configure".
>

:-)
But it makes me reflective, how can we improve or simplify the
piggy programming and configuring tasks?


>(Just to give you an idea if what I'm thinking of. The basic aim is to 
build
>a middleware layer for a set of distributed environmental monitoring 
sensors
>so that land & water scientists can build an accurate picture of the state
>of an area. The data comes from various wireless links. The data needs 
to be
>delivered to databases, web pages, other applications ... So I think 
you can
>see why xmlBlaster looks attractive. To begin with, we're talking on the
>order of 1000 sensors, each generating a reading a minute. So that's 
on the
>order of 20 durable messages a second, with enough fat built in to 
cope with
>expansion and traffic bursts.)

Sounds to be a very interesting application you are planing.


>>The oid is one message, you can collect message to groups with XPath
>>subscriptions.
>
>I've replaced Oids with
>
>      this.publishKey =
>        "<key oid='' contentMime='application/octet-stream'>\n" +
>        "   <" + this.groupName + "></" + this.groupName + ">\n" +
>        "</key>";
>
>for the publisher and
>
>      this.subscribeKey =
>        "<key oid='' queryType='XPATH'>" +
>        "/xmlBlaster/key/" + this.groupName +
>        "</key>";
>
>for the subscriber. Is this correct?
>
>This seems to be working, up to a point. (I do have to remember to 
turn INFO
>level logging off, to stop it telling me that it's found a match for every
>message.)
>

I have removed the INFO in cvs


> But the overall performance seems to have taken a dive:
>
>Size	MPS 
Sdrs 
Rcvrs 
Subj 
Time 
	Sent	Rcvd	TS
>TR
>4096	10 
10 
1 
1 
119999 
1200 
1200 
10 
10
>4096	20 
10 
1 
1 
119984 
2400 
2400 
20 
20
>4096	50 
10 
1 
1 
119999 
4541 
4531 
38 
38
>4096	100 
10 
1 
1 
119984 
7593 
7583 
63 
63
>

If you use the same oid="sensor" for all messages, it should boost
performance.


>Part of the problem seems to be that when I stop a test and start 
another, I
>start getting messages of the form
>
>[Jul 8, 2002 11:14:21 AM WARN  RequestBroker-http://152.83.91.4:3412]
>Generating dead letter oid=http://A.B.C.D:3412-1026090587859-72 from
>publisher=guest because delivery to 'guest' cbQueue=session:23 failed. --
>I've removed my IP address.
>
>Eventually, the dead letter queue fills up and the whole thing dies in a
>welter of OutOfMemoryErrors.

Dead letters are volatile messages, this memory leak was fixed
some days ago.
Can you please confirm that the memory exhaust disappeared in the newest 
cvs?


> To shut down, I'm unSubscribing the receiers,
>using the Oid supplied by subcribe() method. I then disconnect() and then
>shutdown(). (I've tried assorted variations on this pattern.) Is this 
right?
>Do I have to explicitly erase every message I've sent?
>

The problem is that as a default the last message of every key-oid remains
in memory. So in your case just use the same oid for all messages.
Or erase them or set them to 'isVolatile'.
But the best choice for you is probably to use the same oid.



>The above is for non-durable messaging. With durable, I get:
>
>id=FileUtil reason=Can't write file java.io.FileNotFoundException:
>C:\Documents and
>Settings\dougp\tmp\http:\A.B.C.D:3412-1026091518765-11-XmlKey.xml (The
>filename, directory name, or volume label syntax is incorrect)
>
>Which is caused by the ':' being illegal in filenames under Win2k.

>(Incidentally, why am I getting http: to begin with? I thought that 
IIOP was

IIOP uses http to bootstrap (on port 3412)


>the default. This could be an explanation for the difference in 
performance
>between your and my benchmarks. The clients are using the inbuilt
>xmlBlaster.properties file from the xmlBlaster jar.)


>Another thing is that I'm getting lots of messages of the form "Looping
>nanoCounter=1". The code in Timeout.java says that I should be only 
getting
>this to avoid two similar keys and it shouldn't happen very often. 
Since the
>oids are generated, and look pretty different, should this occur?
>

I have removed the message, as it is not important.


>Last thing. Doing pub/sub via raw XML seems awfully laborious. 
Particularly
>since the PublishKeyWrapper and SubscribeKeyWrapper effectively provide an
>API for pub/sub via oids. I would have thought that, since oids seem to be
>individually message-specific, the wrappers would provide an API for 
simple
>pub/sub using topic names, which is what most people would want to do.
>Instead, the Java demo examples and API seem to positively encourage a
>newbie like me to use oids as a quick'n'dirty subject name. (Don't get

Try (see the new HelloWorld4.java):

...
pk = new PublishKeyWrapper("Banking", "text/plain", "1.0");
pk.wrap("<Account><withdraw/></Account>"); // Add banking specific meta data
...

>wrong, I'm immensely impressed by the coolness of using XPath to allow
>subscriptions to almost anything under the sun. And it's got 
applications in
>the area I'm thinking of. It's just that I feel that the API conspires to
>discourage me from using it.)
>

The main thing is NOT to use too many different oid's as they are
performance hungry and memory hungry.


>>This is strange, the default callback only gets PtP messages and 
subcribed
>>
>>
>messages
>
>
>>which didn't specify an own Callback on subscription.
>>If i invoke 'java HelloWorld4' this seems to work fine.
>>
>>
>
>>From HelloWorld4, I get:
>
>[8/07/2002 13:28:01 INFO  HelloWorld4] Reveiving asynchronous message
>'HelloWorld4' in HelloWorld4 handler
>[8/07/2002 13:28:01 INFO  HelloWorld4] Reveiving asynchronous message
>'SomeOtherMessage' in default handler
>
>Which I think is what's supposed to happen.
>
>It's the 'default handler' bit that bothers me. How do I stop a connection
>from receiving 'SomeOtherMessage' and having it go to the default handler,
>if I don't want to see it at all? I would have thought that there's a
>default 'explicit subscriptions only' option, to avoid having a client
>drowning in extraneous messages.
>

You only get what you subscribed for.
Only PtP messages arrive always in the default callback.


regards,

Marcel