Re: [xmlblaster] xmlBlaster performance issues

[Michele Laghi <michele at laghi.eu>]

Hi Nicholas,
could you please before your tests delete any persitence (if you are
using the default hsqldb delete in ${HOME}/tmp/xmlBlaster* and
${HOME}/tmp/client*).

Then run the tests and verify (under and after the tests) that the
xmlBlaster* and client* files in that directory are not there). If they
are there and are big, then persitence is not completely disactivated.

My guess is that you are publishing much faster than messages are
delivered to the subscriber resulting in swapping which is time consuming.

I made some tests myself now. Took away the comments of the first 7
lines of xmlBlaster.properties and changed

queue/callback/maxEntriesCache = 100000
queue/callback/maxEntries = ${queue/callback/maxEntriesCache}

# How big is the topic cache for messages of same oid (here your message
content occupies the memory)
persistence/msgUnitStore/maxEntriesCache = 100000
persistence/msgUnitStore/maxEntries = 100000000
persistence/msgUnitStore/maxBytesCache = 2097152
persistence/msgUnitStore/maxBytes = 2147483647

# Tail back queue on client side (contains the real data of all message
types)
queue/connection/maxEntriesCache = 100000
queue/connection/maxEntries = 10000000

This way I will be able to fit 100000 msg in the queue (which is the
worst case). Otherwise you may get Dead Messages since the messages
would not fit in the queues (also verify that you did not get such dead
messages by looking at the server logs).

I then run the server with more memory (otherwise I may get an
OutOfMemoryException):

java -Xms64M -Xmx512M org.xmlBlaster.Main

and then run in other terminals:

java javaclients.HelloWorldSubscribe
(and push enter to subscribe)

and in a further terminal:

java javaclients.HelloWorldPublish -numPublish 100000 -interactive false
-sleep 0 -contentSize 1000 -logging WARNING

The publisher runs 100000 msg in 3':0" which is 555 mps (1kB size each)
The subscriber receives 8':40" which is 192 mps

So in my case you would achieve a nearly optimal balance by putting a
delay in the publisher of aprox.

5.2 ms - 1.8 = 3.4 ms let's say 4 ms

This way I would mimimize the risk of swapping even when using CACHE.
Could you please make these checks and let us know ?

A made my tests on a single machine, an AMD64-3000 Mhz with 1GB RAM.

Regards
Michele


Nicholas Piegdon wrote:
> Marcel,
> 
> Thanks for your feedback.  We have tried most of your suggestions.  We
> saw some improvement.
> 
> However, I think I mis-characterized our throughput before.  I mentioned
> the LoadTestSub test and a figure near 300 mps.  Those 300 mps were
> using your packaged test with Java client, running both the client and
> server on the same machine in two different terminals.
> 
> Our *actual* measured throughput in practice between all of our
> applications can rarely exceed about 30 mps.  More than that though, in
> a test where we had two simple clients (a "sender" and "receiver", both
> written using the C API), we maxed out at 30 mps at first -- but over
> time, the server slowed down.  Over just a few thousand messages (2 or 3
> minutes) it ran down to only 7-10 mps.  We could completely disconnect
> and shut down both clients, but after a reconnect (without restarting
> the server) the same (ever-slowing-down) rate continued.  This was also
> after we uncommented the 7 lines in the properties file, so
> theoretically nothing should be persistent.
> 
> It was this "slowing down over time" behavior that has really been
> causing us problems.  (I was so busy describing our environment last
> time and trying to enumerate all our issues that I let that one slip by.
> :) )  It appears to be on the sending side of the C client.  In
> particular, "Publish" has always seemed like it should be a non-blocking
> call, so in our main C app we didn't take any special precautions (such
> as a message dispatch thread), and just called our send routines
> wherever we needed to in our main program loop.  Over the lifetime of
> the application though, the rest of our interface begins to slow to a
> crawl.  We can alleviate this with something like a dispatch thread, but
> that just means we'd be pushing the problem somewhere else where a queue
> would just be getting backed up.
> 
> Does any of this "slowing down" behavior sound familiar to you?
> 
> Thanks again.
>