[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [xmlblaster] Durable QoS performance



Doug.Palmer at csiro.au wrote:

say "it works
>reliably with enough throughput to be usable for a distributed sensor
>application, even if it is a pig to program and configure".
>


:-)
But it makes me reflective, how can we improve or simplify the
piggy programming and configuring tasks?



Please accept what follows in the spirit of constructive critism. I don't want to make it sound like I'm sounding off with "xmlBlaster is a load of crap" when I think the opposite.

Actually, server configuration is pretty straightforward. Especially when
compared with some products that I've worked with. It was a bit of a scare
to realise that the config file is in the xmlBlaster jar and, therefore,
relatively inaccessable. But I can see the reasoning behind that.

It is just a default which is always found. Just specify
one in your home directory or on command line to overwrite
those properties you want.


The biggest problem I have is with sudden layering shifts. Keys, Qos, no problem! However, I seem to have, on one hand, something like PublishKeyWrapper, which looks like the kind of thing I need to set up a publication and, on the other hand, the need to write raw XML whenever things get a little complicated. The wrap() method is pretty obscure, and the only example is the HelloWorld3 program, which uses oids as keys anyway. The need to use toXml sometimes also suggests encapsulation leaks.

As a suggestion, I'd keep the Java API using objects all the time, instead
of raw XML and have a class hierarchy that allows raw XML. For example

                    PublishKeyWrapper (abstract)
                             ^
                             |
       +---------------------+--------------------------+
       |                     |                          |
PublishKeyOidWrapper   PublishKeyTopicWrapper   PublishKeyRawWrapper

PusblishKeyOidWrapper indicates publication though an explicit Oid,
PublishKeyRawWrapper through raw XML. The PublishKeyTopicWrapper publishes
under a "topic" in the ordinary pub/sub model. Topics could be strings with
a heirarchy of topic names. Each subclass has its own toXml() method that
generates the appropriate XML for lower layers. The same approach could be
used for subscribers, with SubscribeKeyTopicWrapper generating a suitable
XPath for the topic. (The class hierarchy here may be a bit off. Rather than
having separate {Publish/Subscribe/Get/Erase/Unsubscribe}KeyWrapper
heirarchies, it might be worth having a straight KeyWrapper heirarchy
separated by {Oid,Topic,Raw} approach. But that's neither here nor there.)

This is another possible view on it.


The mixture of paradigms also makes things tricky. Is an oid a message id or a topic id? It's both: it's a drain cleaner and a meat substitute! Is it a messaging system or a shared data repository? Getting caught by the default value of isVolatile has made me realise that it's not a straight MOM system. Nothing wrong with that, but it is a trap for the unwary. However, I can see the benefits of having the mixture, so I've got no concrete suggestions here.

Interesting thoughts, i'm learning daily ...

The problem is that as a default the last message of every key-oid remains
in memory. So in your case just use the same oid for all messages.
Or erase them or set them to 'isVolatile'.
But the best choice for you is probably to use the same oid.



OK. So the default behaviour is non-volatile

Yes.

>wrong, I'm immensely impressed by the coolness of using XPath to allow
>subscriptions to almost anything under the sun. And it's got applications in
>the area I'm thinking of. It's just that I feel that the API conspires to
>discourage me from using it.)
>


The main thing is NOT to use too many different oid's as they are
performance hungry and memory hungry.



This is going to be a little difficult if you want to use durable messages, since part of the point of using them is to get the MOM system to manage failures in service. If you've got volatile message producers (as I have, since the sensors can go offline quite happily at any time) I want to be able to put a message into the MOM system and then forget about it. If a message producer has to wait until the previous message with the same oid is delivered, then there's a good chance that the waiting producer will go offline before it can send its message. If you've got bursty traffic, then things get even worse.

Not that I'm too worried about memory footprints etc. I'm used to the idea
that, if you want serious performance, you slap in a few more processors and
an extra Gb of RAM. The main thing, for me, is to get sufficient
parallelism.


If you have 1000 different sensors, and every sensor has its own ID (key oid)
this should be no problem at all.
The first time a sensor publishes a message, it takes some milliseconds
more to initialize the message, all follower messages of this
sensor (with the same oid) are processed much faster.


Note that for a new subscriber coming, the MoM only holds the newest
value to deliver.
Subscribed clients have a callback queue to deliver all messages in sequence
(if no message priority is specified), so you have a virtual message history here
if a consuming client is slow in processing or unreachable for some time.





>It's the 'default handler' bit that bothers me. How do I stop a connection
>from receiving 'SomeOtherMessage' and having it go to the default handler,
>if I don't want to see it at all? I would have thought that there's a
>default 'explicit subscriptions only' option, to avoid having a client
>drowning in extraneous messages.
>


You only get what you subscribed for.
Only PtP messages arrive always in the default callback.



But HelloWorld4 doesn't set a destination. So it can't be a PtP message (right?). So why does SomeOtherMessage (now called Banking) turn up at the default handler?

Because we have subscribed to it with:

sk = new SubscribeKeyWrapper("HelloWorld4");
sq = new SubscribeQosWrapper();
SubscribeRetQos sr2 = con.subscribe(sk.toXml(), sq.toXml(), new I_Callback() {
.... // we specify a specific callback!!!
});



Here we don't specify a specific callback -> the default callback is used:

        SubscribeKeyWrapper sk = new SubscribeKeyWrapper("Banking");
        SubscribeQosWrapper sq = new SubscribeQosWrapper();
        SubscribeRetQos sr1 = con.subscribe(sk.toXml(), sq.toXml());


regards

Marcel