Re: New performance peak: 405 message/sec

[Marcel Ruff <ruff at swand.lake.de>]

Jerry at Westrick.com wrote:
> 
> Marcel...
> It's me again (Jerry)...

Jerry,

you present an impressive framework here.
I read it a couple of times, but i feel i did
not understand it completely yet.

Your approach remembers me strongly about JINI
which uses the term 'RemoteEvent'.
JINI has an excellent concept about failsave distributed
applications.
Only their attribute (topic) concept is weak imho,
an XML based approach would be much stronger.
You certainly should visit http://www.jini.org/
(you will know all that already).

 
> My View:
> --------
> The PUB/SUB system offers Tremedous advantages, so why should I stop in
> the Middle? How about an entire Banking system (Multiple Applications)
> all written in PUB/SUB.  This would open up the internals of the banking
> applications to allow further usage of the events happening within the
> applications without risking changes to the mission critical applications
> them selves.  The messages then represent banking application events like
> "Book x $ from account x to account y", "Validate Order for 200 IBM on
> NY, for account y", "Order was Validated",
> "order was placed to Broker XYZ", "Validate Trade for Order", "Trade was
> Validated" ....
> 
> I'm not talking about MOM, (Message Orientated Middleware) instead I
> envision MOAs (Message Orientated Applications).  In this view I have
> Multiple interfaces each one for as specific Banking Function.  Some of

Are the interfaces defined with CORBA IDL?
Are the interface generic or type save at compile time?

How is a subscriber connecting to a publisher,
how does he find the service?

Who is doing authentication/authorization, a central
service or every application itself?

On failure (which is 'normal' in distributed systems)
how is reconnected, or queued or time-out handled?

> our Current Interface Definitions are:
> 
>    Broker Interface:
>    -----------------
>    Pub: "Order Accepted for Proccesing",
>         "Order InMarket",
>         "Execution (Trade) for Order",
>         "Order Expired",
>         ...

With Pub do you mean that the broker offers for example
a method
  broker.addOrderInMarketListener()
which subscribers can use to subscribe?
Comparable to 'In-JVM' event subscriptions?
Similar to the distributed observer pattern or 
MVC concept?

> 
>    Sub: "Place Order",
>         "Cacel Order",
>         "Orders info",
>         ...
> 
>    BackOffice Interface (Host):
>    ----------------------------
>    Pub: "Validated Order",
>         "Rejected order",
>         "Booked Trade",
>         "Rejected Trade",
>         "Authorised Trade" (contains settlement info),
>         "I/U/D Instrument" (Things that can be traded),
>         "I/U/D Account",
>         ...
> 
>    Sub: "Validate Order",
>         "Execution for Order",
>         ....
> 
>    ... Interface:
>    --------------
> 
> etc
> 
> Schematic:
> ----------
> The Schema looks like a bowl of spaghetti with meatballs.  It just can't
> be documentated that way, so we came up with a new documentation form,
> "Interface definitions" and "Flow Diagrams".
> 
> An Interface definition describes the messages Pub/Sub 'ed by an
> Interface, something along the lines as above.
> 
> A flow diagram document has a particular message as a starting point, and
> shows the messges that can/are produced as a result of the messages as
> it's processed though out the system. Something like:
> 
> /----\                             /----\
> ¦    ¦                             ¦    ¦
> ¦ II +---1 "Validate Order"------->+ BO ¦
> ¦    +<--2 "Order Accepted"--------+    ¦
> \----/                       +-----+    ¦
>                              ¦     \----/
> /----\                       ¦
> ¦    ¦                       ¦     /----\
> ¦ IM +<--3 "Validated Order"-+     ¦    ¦
> ¦    +---4 "Place Order"---------->+ BI ¦
> ¦    +<--5 "Order In Market"-------+    ¦
> \----/                             \----/
> 
> In such a system, you quickly loose the overview of all the messages
> flying around, so we needed additional tools to watch and inverstigate
> the Messages...
> 
> We built a "Flow Tree Window" where, when a new flow is started by a
> message, it is placed in the tree at the appropiate place, and any
> "Children" (such as "Validated Order" is a Child of "Validate Order") are
> placed under it.  In this way we've built a Several Applications which
> contain the advantages of the Pub/Sub MOM built directly into the system.

Ahh, i had the same problem as well, how to keep control over
all these messages flying around.
Does your "Flow Tree Window" monitor the messages in a hot running
system?
We need something like this as well.

> 
> This where the name "Flow Based System" comes from.
> 
> One example of the advantages came when the bank wanted to send notices
> of Stock market Executions to it's customers in "Near Time" (anybody who
> plans to send Faxes, Emails, and SMS in REAL-TIME is in for a
> REAL-SURPISE).  This was implemented as follows:
> 
> A program was written which recieves "Notice messages" and fills forms
> with the field values in the Notices.  It then passes the Filled form to
> an E-Mail server, Fax-Server, or SMS server.
> 
> A second program was written which Subscribes to "Execution for Order",
> and decides weither this client wants a Notice for a particular
> execution.  If so the Program Publishes a Notice Message (child of
> Execution).  Because we had the events as Msgs received in real time the
> 2 programs took less than 2 weeks to implement (Analysis, Design,
> Programming, Testing and Rollout) done in 4 weeks elapsed time.  That's
> time to market!
> 
> Notice that we did NOT change any existing programs.  We added
> functionality to the basic system with out modifying any currently
> running applications (Touching any of the misson critical programs would
> have required at least three weeks on intensive testing, to be added to
> the project!)

Yes, this are the advantages of Pub/Sub (MOM/MOA), as i experience it as
well!

> 
> Last Hype...
> The system (as I attempted to described here) has been running at ABNAMRO
> zuerich for 4 years, (the first production version had only 2
> Applications).
> 
> Now that everything is clear as mud, I'll be going to bed!
> Jerry Westrick

How could we extend xmlBlaster, using its protocol independence
and XML based topic and query approach,
fail save and queuing abilities etc.
to use in a framework like yours?

Thanks for sharing your knowledge

Marcel

-- 
Marcel Ruff
mailto:ruff at swand.lake.de
http://www.lake.de/home/lake/swand/
http://www.xmlBlaster.org