Re: New performance peak: 405 message/sec

[Jerry at Westrick.com]

Marcel...
It's me again (Jerry)...

If I work on XMLBlaster I'll be doing from home!

You mentioned the CVS version.  I'll Attempt to get it tomorrow, It's 
getting late...
wish me luck.


About the Flow-Based-System.  It's not that your view is wrong it's more 
that my view is a step farther....

Your view:
----------
The Pub/Sub system offers a perfect MOM, Allowing easy integration 
between diferent applications.  It defines an external "API" which is 
easy to use and understand, and allows you to add new interfaces without 
changing you backend application.  This allows for Level of flexibility 
unheard of before Pub/Sub.

A good (banking) example would be a check cashing interface for a bank's 
host.  This would allow different ATM applications (one for each machine 
type) to be easily written.  Additionally, any number of other uses can 
(and will) be found which could use a "Book a Withdrawl" such as "Teller 
Cashbox" application.

Schematic:

/--------\
¦ ATM    ¦
¦ Type 1 +----\
¦ xxx    ¦     \    
\--------/      \
                 \
/--------\        \     /------------\           /--------\
¦ ATM    ¦         \    ¦            ¦           ¦        ¦ 
¦ Type 1 +---------+----+ XMLBlaster +-----------¦ Host   ¦
¦ yyy    ¦         /    ¦            ¦           ¦ System ¦
\--------/        /     \------------/           \--------/
                 /
/---------\     /
¦ cashier ¦    /
¦ Box     +---/
¦ zzz     ¦        
\---------/    




My View:
--------
The PUB/SUB system offers Tremedous advantages, so why should I stop in 
the Middle? How about an entire Banking system (Multiple Applications) 
all written in PUB/SUB.  This would open up the internals of the banking 
applications to allow further usage of the events happening within the 
applications without risking changes to the mission critical applications 
them selves.  The messages then represent banking application events like 
"Book x $ from account x to account y", "Validate Order for 200 IBM on 
NY, for account y", "Order was Validated",
"order was placed to Broker XYZ", "Validate Trade for Order", "Trade was 
Validated" ....

I'm not talking about MOM, (Message Orientated Middleware) instead I 
envision MOAs (Message Orientated Applications).  In this view I have 
Multiple interfaces each one for as specific Banking Function.  Some of 
our Current Interface Definitions are:

   Broker Interface:	
   -----------------
   Pub: "Order Accepted for Proccesing",
        "Order InMarket",
        "Execution (Trade) for Order",
        "Order Expired",
        ...  

   Sub: "Place Order",
        "Cacel Order",
        "Orders info",
        ...

   BackOffice Interface (Host):
   ----------------------------
   Pub: "Validated Order",
        "Rejected order",
        "Booked Trade",
        "Rejected Trade",
        "Authorised Trade" (contains settlement info),
        "I/U/D Instrument" (Things that can be traded),
        "I/U/D Account",
        ...

   Sub: "Validate Order",
        "Execution for Order",
        ....


   ... Interface:
   --------------

etc


Schematic:
----------
The Schema looks like a bowl of spaghetti with meatballs.  It just can't 
be documentated that way, so we came up with a new documentation form, 
"Interface definitions" and "Flow Diagrams".

An Interface definition describes the messages Pub/Sub 'ed by an 
Interface, something along the lines as above.

A flow diagram document has a particular message as a starting point, and 
shows the messges that can/are produced as a result of the messages as 
it's processed though out the system. Something like:

/----\                             /----\
¦    ¦                             ¦    ¦
¦ II +---1 "Validate Order"------->+ BO ¦
¦    +<--2 "Order Accepted"--------+    ¦
\----/                       +-----+    ¦
                             ¦     \----/
/----\                       ¦
¦    ¦                       ¦     /----\
¦ IM +<--3 "Validated Order"-+     ¦    ¦
¦    +---4 "Place Order"---------->+ BI ¦
¦    +<--5 "Order In Market"-------+    ¦
\----/                             \----/

In such a system, you quickly loose the overview of all the messages 
flying around, so we needed additional tools to watch and inverstigate 
the Messages...

We built a "Flow Tree Window" where, when a new flow is started by a 
message, it is placed in the tree at the appropiate place, and any 
"Children" (such as "Validated Order" is a Child of "Validate Order") are 
placed under it.  In this way we've built a Several Applications which 
contain the advantages of the Pub/Sub MOM built directly into the system.

This where the name "Flow Based System" comes from.

One example of the advantages came when the bank wanted to send notices 
of Stock market Executions to it's customers in "Near Time" (anybody who 
plans to send Faxes, Emails, and SMS in REAL-TIME is in for a 
REAL-SURPISE).  This was implemented as follows:

A program was written which recieves "Notice messages" and fills forms 
with the field values in the Notices.  It then passes the Filled form to 
an E-Mail server, Fax-Server, or SMS server. 

A second program was written which Subscribes to "Execution for Order", 
and decides weither this client wants a Notice for a particular 
execution.  If so the Program Publishes a Notice Message (child of 
Execution).  Because we had the events as Msgs received in real time the 
2 programs took less than 2 weeks to implement (Analysis, Design, 
Programming, Testing and Rollout) done in 4 weeks elapsed time.  That's 
time to market!

Notice that we did NOT change any existing programs.  We added 
functionality to the basic system with out modifying any currently 
running applications (Touching any of the misson critical programs would 
have required at least three weeks on intensive testing, to be added to 
the project!)

Last Hype...
The system (as I attempted to described here) has been running at ABNAMRO 
zuerich for 4 years, (the first production version had only 2 
Applications).

Now that everything is clear as mud, I'll be going to bed!
Jerry Westrick