Re: [xmlblaster] Hardware requirements for 10,000 messages/sec (1k msg size)

[Marcel Ruff <mr at marcelruff.info>]

Hi Richard,

the most important performance factor is if your messages
need to be persistent to survive a server crash.
If they need to be persistent xmlBlaster is certainly not capable
to solve the task, as there are many limiting disk I/O.

If your messages can be transported 'transient' (processing in RAM only)
xmlBlaster in its current shape can't handle 10,000 msg/sec.

XmlBlaster is in need to get a performance tuning cycle,
but this is not done in a few hours of work. Even after that
the message throughput won't exceed 1000-2000 msg/sec (guessed).

Collecting for example 50 messages and sending them in a bulk
could solve the bottleneck, and with a bit of tuning xmlBlaster will do 
the job
assuming a 1GB Ethernet environment.
( 1k * 10000 msgs == 50k * 200 msgs = 10MByte/sec = 100 Mbit/sec)
As we have a very optimized compression mode, this can increase
throughput again dramatically.
Does your use case allow collecting messages (burst mode)?

If you have different publishing apps, you can split the traffic on 
different
computers, but this is very problem domain depending.

Adding more subscribers has not a too high impact on the server
and can, as you mentioned, be addressed by a master/slave cluster approach.

Reusing the client/server interfaces of xmlBlaster and other proven parts
and implementing a slim pub/sub forwarding engine (for example in C)
could probably address your needs.

So all depends on your time and budget,
if your budget is high enough we will assist you in any way needed
to solve your needs :-)

regards
Marcel


Richard PublishThenFilter wrote:
> Hi,
> I've been reading up on MOMs and xmlblaster caught my eye :)
>
> I am in the planning stages of adding a message bus to tie together a
> handful of (mostly) java server components. I don't want to use JMS as
> i anticipate deploying components in other languages (C++). We don't
> currenly use any messaging middleware. There are currently around 500
> msgs/second we would want to send, the majority of which we would want
> in one publish/subscribe group. I anticipate adding another busy group
> in a few months, reaching a total or around 10,000 msgs/second by the
> end of the year. There would be 1 or 2 apps publishing, and 3 or 4
> subscribing to the bulk of the messages.
>
> It appears xmlblaster will scale nicely by chaining servers together
> in master-slave formation. What i'm especially interested in is the
> hardware requirements for suporting the message volume outlined above;
> namely 10,000 msgs/second where each msg is <=1KB in size.
>
> Anyone running xmlblaster with this sort of msg volume+size, or
> higher? Can you quote your hardware spec and message volume please? :)
>
> Regards,
> Richard
>
> PS: FWIW the other system i want to test is Spread.. but i dislike the
> advertising clause in their license agreement.