Re: [xmlblaster] xmlBlaster performance issues

[Michele Laghi <michele at laghi.eu>]

Hi again,
I forgot you also need to add to xmlBlaster.properties:

queue/callback/maxBytes = 2147483647

and in the publisher should be invoked as:
java javaclients.HelloWorldPublish -numPublish 100000 -interactive false
-sleep 0 -contentSize 1000 -logging WARNING -erase false

Regards
Michele




Michele Laghi wrote:
> Hi Nicholas,
> could you please before your tests delete any persitence (if you are
> using the default hsqldb delete in ${HOME}/tmp/xmlBlaster* and
> ${HOME}/tmp/client*).
> 
> Then run the tests and verify (under and after the tests) that the
> xmlBlaster* and client* files in that directory are not there). If they
> are there and are big, then persitence is not completely disactivated.
> 
> My guess is that you are publishing much faster than messages are
> delivered to the subscriber resulting in swapping which is time consuming.
> 
> I made some tests myself now. Took away the comments of the first 7
> lines of xmlBlaster.properties and changed
> 
> queue/callback/maxEntriesCache = 100000
> queue/callback/maxEntries = ${queue/callback/maxEntriesCache}
> 
> # How big is the topic cache for messages of same oid (here your message
> content occupies the memory)
> persistence/msgUnitStore/maxEntriesCache = 100000
> persistence/msgUnitStore/maxEntries = 100000000
> persistence/msgUnitStore/maxBytesCache = 2097152
> persistence/msgUnitStore/maxBytes = 2147483647
> 
> # Tail back queue on client side (contains the real data of all message
> types)
> queue/connection/maxEntriesCache = 100000
> queue/connection/maxEntries = 10000000
> 
> This way I will be able to fit 100000 msg in the queue (which is the
> worst case). Otherwise you may get Dead Messages since the messages
> would not fit in the queues (also verify that you did not get such dead
> messages by looking at the server logs).
> 
> I then run the server with more memory (otherwise I may get an
> OutOfMemoryException):
> 
> java -Xms64M -Xmx512M org.xmlBlaster.Main
> 
> and then run in other terminals:
> 
> java javaclients.HelloWorldSubscribe
> (and push enter to subscribe)
> 
> and in a further terminal:
> 
> java javaclients.HelloWorldPublish -numPublish 100000 -interactive false
> -sleep 0 -contentSize 1000 -logging WARNING
> 
> The publisher runs 100000 msg in 3':0" which is 555 mps (1kB size each)
> The subscriber receives 8':40" which is 192 mps
> 
> So in my case you would achieve a nearly optimal balance by putting a
> delay in the publisher of aprox.
> 
> 5.2 ms - 1.8 = 3.4 ms let's say 4 ms
> 
> This way I would mimimize the risk of swapping even when using CACHE.
> Could you please make these checks and let us know ?
> 
> A made my tests on a single machine, an AMD64-3000 Mhz with 1GB RAM.
> 
> Regards
> Michele
> 
> 
> Nicholas Piegdon wrote:
>> Marcel,
>>
>> Thanks for your feedback.  We have tried most of your suggestions.  We
>> saw some improvement.
>>
>> However, I think I mis-characterized our throughput before.  I mentioned
>> the LoadTestSub test and a figure near 300 mps.  Those 300 mps were
>> using your packaged test with Java client, running both the client and
>> server on the same machine in two different terminals.
>>
>> Our *actual* measured throughput in practice between all of our
>> applications can rarely exceed about 30 mps.  More than that though, in
>> a test where we had two simple clients (a "sender" and "receiver", both
>> written using the C API), we maxed out at 30 mps at first -- but over
>> time, the server slowed down.  Over just a few thousand messages (2 or 3
>> minutes) it ran down to only 7-10 mps.  We could completely disconnect
>> and shut down both clients, but after a reconnect (without restarting
>> the server) the same (ever-slowing-down) rate continued.  This was also
>> after we uncommented the 7 lines in the properties file, so
>> theoretically nothing should be persistent.
>>
>> It was this "slowing down over time" behavior that has really been
>> causing us problems.  (I was so busy describing our environment last
>> time and trying to enumerate all our issues that I let that one slip by.
>> :) )  It appears to be on the sending side of the C client.  In
>> particular, "Publish" has always seemed like it should be a non-blocking
>> call, so in our main C app we didn't take any special precautions (such
>> as a message dispatch thread), and just called our send routines
>> wherever we needed to in our main program loop.  Over the lifetime of
>> the application though, the rest of our interface begins to slow to a
>> crawl.  We can alleviate this with something like a dispatch thread, but
>> that just means we'd be pushing the problem somewhere else where a queue
>> would just be getting backed up.
>>
>> Does any of this "slowing down" behavior sound familiar to you?
>>
>> Thanks again.
>>
> 
> 
>