[xmlblaster] xmlBlaster performance issues

[Nicholas Piegdon <piegdon at soartech.com>]

Hello,

We've been using xmlBlaster for over a year now in one of our projects.  
We've slowly been adding connected clients, increasing the number of 
messages we send, and growing the size of the messages.  Just recently 
we seem to have hit the limits of the xmlBlaster server.  However we're 
still dramatically under the performance listed on the "performance" 
page of the xmlBlaster.org site.

Here's our usage model:
- 4-5 clients using XmlBlasterAccessUnparsed in the C API.
- 2-3 clients using the Java API.
- A couple of the C clients actually make more than one connection to 
the server.  (The system architecture for our project dictates a few 
"conceptual" components that don't match 1:1 with our actual running 
processes.)

We have 4 high-end machines (2-4 GB RAM, dual-core, dual-processor 
Xeon/64-bit Opteron depending) running both Linux and Windows.  All 4 
have gigabit Ethernet connected to a single switch.  We spread our 
applications between the machines relatively evenly.

The messages are all point-to-point save for a single "monitoring" 
component that just starts up and subscribes to '*' with a graphical 
front end to verify the correctness of the information we're passing 
around.  We believe we're using the Socket protocol in all of our 
components.

The messages have a minimal key but contain an average of maybe 1k-4k in 
their content section (maxing out around maybe 10k).

- We send messages in the C client using publish (vs. publishMsgArr).
- We altered xmlBlaster.properties by uncommenting the first 7 lines, 
and we saw a large performance gain.
- We defined XB_USE_PTHREADS and saw a performance gain.

- We tried to enable burst mode on a few of the C clients but didn't see 
much of a difference.  (The "Client features" page says that burst mode 
isn't implemented in the C client though.)

We ran the first test you have listed on the performance page 
(org.xmlBlaster.test.stress.LoadTestSub):
You: (600 MHz K7) 672 mps.
Us: (3.0 GHz dual-Xeon) 33 mps.
Us: (after uncommenting 7 lines in xmlBlaster.properties) 300 mps.

The 7 lines did help, but in our actual system, we're still not seeing 
anything close to that 300 mps.  In particular, the C client responsible 
for most of the messages in the system appears to sleep (CPU usage near 
0) whenever delivering a lot of messages back-to-back (even with 
XB_USE_PTHREADS).  This causes the GUI in that app to refresh at only a 
frame every second or two.

We suspected the single "subscribe '*'" client might have been the 
culprit for hitting the performance, but taking it out of the equation 
doesn't help much.

I was wondering if you have a list of tips for squeezing more 
performance out of the client and server:
- Is there a way to enable asynchronous send?
- Should we use "publish message array" in the C client?
- Should we use "publish one way"?
- Does compiling in release mode speed things along significantly?  (We 
tried on the C clients, but didn't see much difference.)
- Will compression help?  (I suspect the message-send overhead is worse 
than any kind of bandwidth limitations we might be running into.)
- Is there anything else we're missing?

Thanks for any assistance you can offer.

--
Nicholas Piegdon
Soar Technology
piegdon at soartech.com