Re: [xmlblaster] Durable QoS performance

[Marcel Ruff <ruff at swand.lake.de>]

Doug.Palmer at csiro.au wrote:

> > say "it works
> > >reliably with enough throughput to be usable for a 
> > distributed sensor
> > >application, even if it is a pig to program and configure".
> > >
> >
> > :-)
> > But it makes me reflective, how can we improve or simplify the
> > piggy programming and configuring tasks?
> >    
>
> Please accept what follows in the spirit of constructive critism. I don't
> want to make it sound like I'm sounding off with "xmlBlaster is a load of
> crap" when I think the opposite.
>
> Actually, server configuration is pretty straightforward. Especially when
> compared with some products that I've worked with. It was a bit of a scare
> to realise that the config file is in the xmlBlaster jar and, therefore,
> relatively inaccessable. But I can see the reasoning behind that.
It is just a default which is always found. Just specify
one in your home directory or on command line to overwrite
those properties you want.

> The biggest problem I have is with sudden layering shifts. Keys, Qos, no
> problem! However, I seem to have, on one hand, something like
> PublishKeyWrapper, which looks like the kind of thing I need to set up a
> publication and, on the other hand, the need to write raw XML whenever
> things get a little complicated. The wrap() method is pretty obscure, and
> the only example is the HelloWorld3 program, which uses oids as keys anyway.
> The need to use toXml sometimes also suggests encapsulation leaks.
>
> As a suggestion, I'd keep the Java API using objects all the time, instead
> of raw XML and have a class hierarchy that allows raw XML. For example
>
>                     PublishKeyWrapper (abstract)
>                              ^
>                              |
>        +---------------------+--------------------------+
>        |                     |                          |
> PublishKeyOidWrapper   PublishKeyTopicWrapper   PublishKeyRawWrapper
>
> PusblishKeyOidWrapper indicates publication though an explicit Oid,
> PublishKeyRawWrapper through raw XML. The PublishKeyTopicWrapper publishes
> under a "topic" in the ordinary pub/sub model. Topics could be strings with
> a heirarchy of topic names. Each subclass has its own toXml() method that
> generates the appropriate XML for lower layers. The same approach could be
> used for subscribers, with SubscribeKeyTopicWrapper generating a suitable
> XPath for the topic. (The class hierarchy here may be a bit off. Rather than
> having separate {Publish/Subscribe/Get/Erase/Unsubscribe}KeyWrapper
> heirarchies, it might be worth having a straight KeyWrapper heirarchy
> separated by {Oid,Topic,Raw} approach. But that's neither here nor there.)
This is another possible view on it.

> The mixture of paradigms also makes things tricky. Is an oid a message id or
> a topic id? It's both: it's a drain cleaner and a meat substitute! Is it a
> messaging system or a shared data repository? Getting caught by the default
> value of isVolatile has made me realise that it's not a straight MOM system.
> Nothing wrong with that, but it is a trap for the unwary. However, I can see
> the benefits of having the mixture, so I've got no concrete suggestions
> here.
Interesting thoughts, i'm learning daily ...

> > The problem is that as a default the last message of every 
> > key-oid remains
> > in memory. So in your case just use the same oid for all messages.
> > Or erase them or set them to 'isVolatile'.
> > But the best choice for you is probably to use the same oid.
> >    
>
> OK. So the default behaviour is non-volatile
Yes.

> > >wrong, I'm immensely impressed by the coolness of using 
> > XPath to allow
> > >subscriptions to almost anything under the sun. And it's got 
> > applications in
> > >the area I'm thinking of. It's just that I feel that the 
> > API conspires to
> > >discourage me from using it.)
> > >
> >
> > The main thing is NOT to use too many different oid's as they are
> > performance hungry and memory hungry.
> >    
>
> This is going to be a little difficult if you want to use durable messages,
> since part of the point of using them is to get the MOM system to manage
> failures in service. If you've got volatile message producers (as I have,
> since the sensors can go offline quite happily at any time) I want to be
> able to put a message into the MOM system and then forget about it. If a
> message producer has to wait until the previous message with the same oid is
> delivered, then there's a good chance that the waiting producer will go
> offline before it can send its message. If you've got bursty traffic, then
> things get even worse.
>
> Not that I'm too worried about memory footprints etc. I'm used to the idea
> that, if you want serious performance, you slap in a few more processors and
> an extra Gb of RAM. The main thing, for me, is to get sufficient
> parallelism.

If you have 1000 different sensors, and every sensor has its own ID (key 
oid)
this should be no problem at all.
The first time a sensor publishes a message, it takes some milliseconds
more to initialize the message, all follower messages of this
sensor (with the same oid) are processed much faster.

Note that for a new subscriber coming, the MoM only holds the newest
value to deliver.
Subscribed clients have a callback queue to deliver all messages in sequence
(if no message priority is specified), so you have a virtual message 
history here
if a consuming client is slow in processing or unreachable for some time.

>  
>
> > >It's the 'default handler' bit that bothers me. How do I 
> > stop a connection
> > >from receiving 'SomeOtherMessage' and having it go to the 
> > default handler,
> > >if I don't want to see it at all? I would have thought that 
> > there's a
> > >default 'explicit subscriptions only' option, to avoid 
> > having a client
> > >drowning in extraneous messages.
> > >
> >
> > You only get what you subscribed for.
> > Only PtP messages arrive always in the default callback.
> >    
>
> But HelloWorld4 doesn't set a destination. So it can't be a PtP message
> (right?). So why does SomeOtherMessage (now called Banking) turn up at the default handler?
Because we have subscribed to it with:

       sk = new SubscribeKeyWrapper("HelloWorld4");
       sq = new SubscribeQosWrapper();
       SubscribeRetQos sr2 = con.subscribe(sk.toXml(), sq.toXml(), new 
I_Callback() {
                                             .... // we specify a 
specific callback!!!
                                             });


Here we don't specify a specific callback -> the default callback is used:

        SubscribeKeyWrapper sk = new SubscribeKeyWrapper("Banking");
        SubscribeQosWrapper sq = new SubscribeQosWrapper();
        SubscribeRetQos sr1 = con.subscribe(sk.toXml(), sq.toXml());


regards

Marcel


>