[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [xmlblaster] Durable QoS performance



Hi Doug,

>It's not urgent in the sense that there's an mission critical application
>waiting on it. But I've got an obvious desire to be able to say "it works
>reliably with enough throughput to be usable for a distributed sensor
>application, even if it is a pig to program and configure".
>

:-)
But it makes me reflective, how can we improve or simplify the
piggy programming and configuring tasks?


>(Just to give you an idea if what I'm thinking of. The basic aim is to build
>a middleware layer for a set of distributed environmental monitoring sensors
>so that land & water scientists can build an accurate picture of the state
>of an area. The data comes from various wireless links. The data needs to be
>delivered to databases, web pages, other applications ... So I think you can
>see why xmlBlaster looks attractive. To begin with, we're talking on the
>order of 1000 sensors, each generating a reading a minute. So that's on the
>order of 20 durable messages a second, with enough fat built in to cope with
>expansion and traffic bursts.)


Sounds to be a very interesting application you are planing.


>>The oid is one message, you can collect message to groups with XPath
>>subscriptions.
>
>I've replaced Oids with
>
> this.publishKey =
> "<key oid='' contentMime='application/octet-stream'>\n" +
> " <" + this.groupName + "></" + this.groupName + ">\n" +
> "</key>";
>
>for the publisher and
>
> this.subscribeKey =
> "<key oid='' queryType='XPATH'>" +
> "/xmlBlaster/key/" + this.groupName +
> "</key>";
>
>for the subscriber. Is this correct?
>
>This seems to be working, up to a point. (I do have to remember to turn INFO
>level logging off, to stop it telling me that it's found a match for every
>message.)
>


I have removed the INFO in cvs


> But the overall performance seems to have taken a dive:
>
>Size MPS Sdrs Rcvrs Subj Time Sent Rcvd TS
>TR
>4096 10 10 1 1 119999 1200 1200 10 10
>4096 20 10 1 1 119984 2400 2400 20 20
>4096 50 10 1 1 119999 4541 4531 38 38
>4096 100 10 1 1 119984 7593 7583 63 63
>


If you use the same oid="sensor" for all messages, it should boost
performance.


>Part of the problem seems to be that when I stop a test and start another, I
>start getting messages of the form
>
>[Jul 8, 2002 11:14:21 AM WARN RequestBroker-http://152.83.91.4:3412]
>Generating dead letter oid=http://A.B.C.D:3412-1026090587859-72 from
>publisher=guest because delivery to 'guest' cbQueue=session:23 failed. --
>I've removed my IP address.
>
>Eventually, the dead letter queue fills up and the whole thing dies in a
>welter of OutOfMemoryErrors.


Dead letters are volatile messages, this memory leak was fixed
some days ago.
Can you please confirm that the memory exhaust disappeared in the newest cvs?



> To shut down, I'm unSubscribing the receiers,
>using the Oid supplied by subcribe() method. I then disconnect() and then
>shutdown(). (I've tried assorted variations on this pattern.) Is this right?
>Do I have to explicitly erase every message I've sent?
>


The problem is that as a default the last message of every key-oid remains
in memory. So in your case just use the same oid for all messages.
Or erase them or set them to 'isVolatile'.
But the best choice for you is probably to use the same oid.



>The above is for non-durable messaging. With durable, I get:
>
>id=FileUtil reason=Can't write file java.io.FileNotFoundException:
>C:\Documents and
>Settings\dougp\tmp\http:\A.B.C.D:3412-1026091518765-11-XmlKey.xml (The
>filename, directory name, or volume label syntax is incorrect)
>
>Which is caused by the ':' being illegal in filenames under Win2k.

>(Incidentally, why am I getting http: to begin with? I thought that IIOP was

IIOP uses http to bootstrap (on port 3412)


>the default. This could be an explanation for the difference in performance
>between your and my benchmarks. The clients are using the inbuilt
>xmlBlaster.properties file from the xmlBlaster jar.)



>Another thing is that I'm getting lots of messages of the form "Looping
>nanoCounter=1". The code in Timeout.java says that I should be only getting
>this to avoid two similar keys and it shouldn't happen very often. Since the
>oids are generated, and look pretty different, should this occur?
>


I have removed the message, as it is not important.


>Last thing. Doing pub/sub via raw XML seems awfully laborious. Particularly
>since the PublishKeyWrapper and SubscribeKeyWrapper effectively provide an
>API for pub/sub via oids. I would have thought that, since oids seem to be
>individually message-specific, the wrappers would provide an API for simple
>pub/sub using topic names, which is what most people would want to do.
>Instead, the Java demo examples and API seem to positively encourage a
>newbie like me to use oids as a quick'n'dirty subject name. (Don't get


Try (see the new HelloWorld4.java):

...
pk = new PublishKeyWrapper("Banking", "text/plain", "1.0");
pk.wrap("<Account><withdraw/></Account>"); // Add banking specific meta data
...

>wrong, I'm immensely impressed by the coolness of using XPath to allow
>subscriptions to almost anything under the sun. And it's got applications in
>the area I'm thinking of. It's just that I feel that the API conspires to
>discourage me from using it.)
>


The main thing is NOT to use too many different oid's as they are
performance hungry and memory hungry.


>>This is strange, the default callback only gets PtP messages and subcribed
>>
>>
>messages
>
>
>>which didn't specify an own Callback on subscription.
>>If i invoke 'java HelloWorld4' this seems to work fine.
>>
>>
>
>>From HelloWorld4, I get:
>
>[8/07/2002 13:28:01 INFO HelloWorld4] Reveiving asynchronous message
>'HelloWorld4' in HelloWorld4 handler
>[8/07/2002 13:28:01 INFO HelloWorld4] Reveiving asynchronous message
>'SomeOtherMessage' in default handler
>
>Which I think is what's supposed to happen.
>
>It's the 'default handler' bit that bothers me. How do I stop a connection
>from receiving 'SomeOtherMessage' and having it go to the default handler,
>if I don't want to see it at all? I would have thought that there's a
>default 'explicit subscriptions only' option, to avoid having a client
>drowning in extraneous messages.
>


You only get what you subscribed for.
Only PtP messages arrive always in the default callback.


regards,

Marcel