[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: New performance peak: 405 message/sec
>>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
On 11.10.00, 11:18:47, Marcel Ruff <ruff at swand.lake.de> wrote regarding Re:
New performance peak: 405 message/sec:
> Jerry at Westrick.com wrote:
> >
> > Marcel...
> > It's me again (Jerry)...
> Jerry,
> you present an impressive framework here.
> I read it a couple of times, but i feel i did
> not understand it completely yet.
> Your approach remembers me strongly about JINI
> which uses the term 'RemoteEvent'.
> JINI has an excellent concept about failsave distributed
> applications.
> Only their attribute (topic) concept is weak imho,
> an XML based approach would be much stronger.
> You certainly should visit http://www.jini.org/
> (you will know all that already).
I don't know much about JINI, but it is PUB/SUB, here "Buisness Event"
are represented by PUB's.
> > My View:
> > --------
> > The PUB/SUB system offers Tremedous advantages, so why should I stop in
> > the Middle? How about an entire Banking system (Multiple Applications)
> > all written in PUB/SUB. This would open up the internals of the banking
> > applications to allow further usage of the events happening within the
> > applications without risking changes to the mission critical applications
> > them selves. The messages then represent banking application events like
> > "Book x $ from account x to account y", "Validate Order for 200 IBM on
> > NY, for account y", "Order was Validated",
> > "order was placed to Broker XYZ", "Validate Trade for Order", "Trade was
> > Validated" ....
> >
> > I'm not talking about MOM, (Message Orientated Middleware) instead I
> > envision MOAs (Message Orientated Applications). In this view I have
> > Multiple interfaces each one for as specific Banking Function. Some of
> Are the interfaces defined with CORBA IDL?
> Are the interface generic or type save at compile time?
The Interface is via a Proprietary OCX, the messages are simple ASCII
records,
we generate libraries (C++/VB) with perl to read/write the messages. The
OCX encapsulates the low level propietary protocoll built on TCP/IP.
> How is a subscriber connecting to a publisher,
> how does he find the service?
The OCX has 2 functions available, "sgcom.Connect(EnvName)", which uses
NT "Trusted connection" and
"sgcom.connectRemote(EnvName,Username,Password)" which sends the password
encrypted, and then PUB/SUB server uses the NT equivalent to su to
validate the username/password. The actual TCP/IP parameters (host:port)
are taken out the NT Registry via the EnvName.
> Who is doing authentication/authorization, a central
> service or every application itself?
There are 2 levels of security built in, Authentication as described
above, and authorization...
Basically, the PUB/SUB server will never send a message down a connection
if the USER logged in is not "AUTHORISED" to see the contents of the
message.
A user can be authorised to be able to Read and/or write a specific
message type
with or without key-checking.
The key checking is done by:
1- building a key containing a static portion(message-type specific),
concatenated with varios parts of the message contents.
2- The key must be found in the UserAccessKeys table (User -> allowed
accesskeys)
> On failure (which is 'normal' in distributed systems)
> how is reconnected, or queued or time-out handled?
Aaahhh. You got me. In the current system "recovery" is done by the
applications. This forces each application to write to the database that
it has processed each message. This makes our system Schysophrenic
(has 2 presonalities). One the one side, all messages are received in
Real
Time, and recovery is based on views which rebuild the messages.
But for recovery purposes the Application must write Directly the message
to the database
which is relational and does not match the message flows at all!
The following schema will help understand the problem:
/----\ /----\
¦ AP ¦ ¦ AP ¦
¦ XX +---2 "Message"-------------->+ YY ¦
¦ +------+ 1 Write message ¦ ¦
\----/ ¦ +--3 Mark as Proc---+ ¦
¦ ¦ \----/
Y Y
/--------\
¦ DB ¦
¦ ¦
¦ ¦
\--------/
1) The App XX must save the "message" to the database
with status "ToBeProcessed"
2) The App XX pubs the message
3) The App YY receives the message, and processes it,
and then updates the DB to "Procesed"
This is one of the Functions I would like to see (maybe build) in
XMLBlaster.
> > our Current Interface Definitions are:
> >
> > Broker Interface:
> > -----------------
> > Pub: "Order Accepted for Proccesing",
> > "Order InMarket",
> > "Execution (Trade) for Order",
> > "Order Expired",
> > ...
> With Pub do you mean that the broker offers for example
> a method
> broker.addOrderInMarketListener()
> which subscribers can use to subscribe?
> Comparable to 'In-JVM' event subscriptions?
> Similar to the distributed observer pattern or
> MVC concept?
No... By Pub I mean the a Brooker interface must pub (as in pub/sub) as
message of the
type described. In out system "Order Accepted for Processig" is a
MsgType: 22 stored in
the constant ACCEPTED_ORDER.
> >
> > Sub: "Place Order",
> > "Cacel Order",
> > "Orders info",
> > ...
> >
> > BackOffice Interface (Host):
> > ----------------------------
> > Pub: "Validated Order",
> > "Rejected order",
> > "Booked Trade",
> > "Rejected Trade",
> > "Authorised Trade" (contains settlement info),
> > "I/U/D Instrument" (Things that can be traded),
> > "I/U/D Account",
> > ...
> >
> > Sub: "Validate Order",
> > "Execution for Order",
> > ....
> >
> > ... Interface:
> > --------------
> >
> > etc
> >
> > Schematic:
> > ----------
> > The Schema looks like a bowl of spaghetti with meatballs. It just can't
> > be documentated that way, so we came up with a new documentation form,
> > "Interface definitions" and "Flow Diagrams".
> >
> > An Interface definition describes the messages Pub/Sub 'ed by an
> > Interface, something along the lines as above.
> >
> > A flow diagram document has a particular message as a starting point, and
> > shows the messges that can/are produced as a result of the messages as
> > it's processed though out the system. Something like:
> >
> > /----\ /----\
> > ¦ ¦ ¦ ¦
> > ¦ II +---1 "Validate Order"------->+ BO ¦
> > ¦ +<--2 "Order Accepted"--------+ ¦
> > \----/ +-----+ ¦
> > ¦ \----/
> > /----\ ¦
> > ¦ ¦ ¦ /----\
> > ¦ IM +<--3 "Validated Order"-+ ¦ ¦
> > ¦ +---4 "Place Order"---------->+ BI ¦
> > ¦ +<--5 "Order In Market"-------+ ¦
> > \----/ \----/
> >
> > In such a system, you quickly loose the overview of all the messages
> > flying around, so we needed additional tools to watch and inverstigate
> > the Messages...
> >
> > We built a "Flow Tree Window" where, when a new flow is started by a
> > message, it is placed in the tree at the appropiate place, and any
> > "Children" (such as "Validated Order" is a Child of "Validate Order") are
> > placed under it. In this way we've built a Several Applications which
> > contain the advantages of the Pub/Sub MOM built directly into the system.
> Ahh, i had the same problem as well, how to keep control over
> all these messages flying around.
> Does your "Flow Tree Window" monitor the messages in a hot running
> system?
> We need something like this as well.
Yes, the "Tree Flow Window" is part of our "Operators GUI", called
StarEye.
The Pub/Sub Server (SGService) stores most messages published in the NT
LOG
file. The list of messages not stored is kept in the NT Registry...
A Server Program (Unix Deamon?) called eventServer, periodically (30s)
reads
the new items in the Log, and Pub's a 922 message (Msg was published)
message.
Naturally to avoid eveternal loops, the 922 is not Logged!
The StarEye Program then receives a 922 for every message published.
Theese are the
basis for the "Flow Tree".
The Flow is built out of the First 4 fields of every message, (might be
part of header
in XMLBlaster), which contain FlowType, FlowKey, FatherFlowType,
FatherFlowKey.
We've managed to get most built with this simple system, although it's
over simplified!
> >
> > This where the name "Flow Based System" comes from.
> >
> > One example of the advantages came when the bank wanted to send notices
> > of Stock market Executions to it's customers in "Near Time" (anybody who
> > plans to send Faxes, Emails, and SMS in REAL-TIME is in for a
> > REAL-SURPISE). This was implemented as follows:
> >
> > A program was written which recieves "Notice messages" and fills forms
> > with the field values in the Notices. It then passes the Filled form to
> > an E-Mail server, Fax-Server, or SMS server.
> >
> > A second program was written which Subscribes to "Execution for Order",
> > and decides weither this client wants a Notice for a particular
> > execution. If so the Program Publishes a Notice Message (child of
> > Execution). Because we had the events as Msgs received in real time the
> > 2 programs took less than 2 weeks to implement (Analysis, Design,
> > Programming, Testing and Rollout) done in 4 weeks elapsed time. That's
> > time to market!
> >
> > Notice that we did NOT change any existing programs. We added
> > functionality to the basic system with out modifying any currently
> > running applications (Touching any of the misson critical programs would
> > have required at least three weeks on intensive testing, to be added to
> > the project!)
> Yes, this are the advantages of Pub/Sub (MOM/MOA), as i experience it as
> well!
> >
> > Last Hype...
> > The system (as I attempted to described here) has been running at ABNAMRO
> > zuerich for 4 years, (the first production version had only 2
> > Applications).
> >
> > Now that everything is clear as mud, I'll be going to bed!
> > Jerry Westrick
> How could we extend xmlBlaster, using its protocol independence
> and XML based topic and query approach,
> fail save and queuing abilities etc.
> to use in a framework like yours?
The Million Dollar Question:
Recovery:
---------
1) XMLBlaster would need Recoverable Messages. That is similar to a
Queue, but for any
application. What is needed is
"Subscribe(MsgType,LastProcessedMsgId)".
This will allows us to restart Clients easily... But Requires
XMLBlaster to save Msgs to
a database. (why database later!)
2) A simple way to store/recover the LastProcessedMsgId will be needed.
Something simple
similar to CICS/TS. The concept would be something like
GetRecoverableStorage("xyz",MyStorageRec) and
SaveRecoverableStorage("xyz",MyStorageRec).
This could be implemented as special message type which is stored to
the same DB as the
messages. (explanation follows!). And where "xyz" is the primary key
value, and the rest
of the message is free format storage. (maybe XML?)
One complication arises, when a program is in the middle of a flow. For
example consider
a fictitious program called PGM, which subscribes to msg TypeOne. When it
recieves a such a
message it processes it (by doing fictum to it), and then it publishes a
message of TypeTwo, and a message of TypeThree.
When planning for recovery/restart, PGM must garuntee that for each msg
of TypeOne it recieves, PGM publishes one and only one msg of TypeTwo. It
also must garuntee that for each msg of TypeOne it receives, PGM
publishes one and only one msg TypeThree! According to 2) it will also
publish (write to RecoverableStorage) a record noting that it has
processed this msg of TypeOne.
3) In Order to support this XMLBlaster would have to hold on to the msg's
TypeTwo,
TypeThree, and the RecoverableStorage message until PGM says to COMMIT
them. Once
XMLBlaster recieves the COMMIT, it must write all three in one DB
transaction. This
leaves only the last message in doubt. (which PGM can recover by
reading the RS, to get
the last messageId of the last msg TypeOne processed, checking the
that first message may
already be processed, and restarting there!).
The system I built before StarGate was a Conectionless style (UDP like)
MOA system. But it had such recovery built in. Unfortunately you had to
specify the name of the application to which to send the message, instead
of the Pub/Sub concept.
> Thanks for sharing your knowledge
> Marcel
Thank you for letting me...
Jerry
P.S. Once we get this going, we can start talking about
Analysis/Design/Documentation tools to support the "Flow Based System".
Got my Ideas on that too!