[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [xmlblaster] HA possible using session & subscription persistence?

Saju Abraham wrote:

Marcel,

Thanks again for the tip!  I was doing a few more tests particularly for a
scenario when there could be a database crash.

Suppose if the DB goes down, xmlBlaster goes into a polling mode and tries
to re-establish the JDBC connection pool to the database.  During this time,
if any new sessions or subscriptions (or unsubscriptions / disconnections)
are made - those requests are set to be 'transient' as there's no connection
to the DB.  When the DB comes back up and the connection pool gets
reestablished, those 'transient' requests are not synched up with the
database - they're still only in memory.  This could result in lost data
when the current xmlBlaster instance fails and switches over to the second
node that reads the persistent data from the common DB.

I just wanted to clarify if this is the correct behavior, or if I'm doing
something wrong...


Hi Saju,

i think you are right.
We had one point of failure in mind during implementation,
but failing the DB and xmlBlaster was not considered.
In your use case you will have a week point.

One simple solution could be to halt xmlBlaster immediately
on lost DB and avoid any transient 'emergency' operation.
But this is not yet implemented.
The other solution is to add syncing, as you mentioned.

About changing runlevels just one more note:
You could send a SIGTERM signal to xmlBlaster to fire up from
runlevel 3 -> 9 or down from 9 -> 3 (each new SIGTERM toggles).
To support this you need to change the code of (or replace)
xmlBlaster/src/java/org/xmlBlaster/util/SignalCatcher.java
If you go this way and donate it back we will add it to the xmlBlaster distribution.

regards,
Marcel

Thanks again!

Kind regards,
Saju


-----Original Message-----
From: owner-xmlblaster at server.xmlBlaster.org
[mailto:owner-xmlblaster at server.xmlBlaster.org] On Behalf Of Marcel Ruff
Sent: Sunday, October 30, 2005 11:22 PM
To: xmlblaster at server.xmlBlaster.org
Subject: Re: [xmlblaster] HA possible using session & subscription
persistence?

*This message was transferred with a trial version of CommuniGate(tm) Pro*
Saju Abraham wrote:



Marcel,

I did a few preliminary tests for HA using persistence and it works great!
Thanks a lot for the tip on optimizing the server startup, I've done


exactly


that.  Though I'd like to see what would be the different times needed for
the server to start up from runlevel 3 to runlevel 9 with various loads of
persistent sessions & subscriptions in the database.  But in any case, I
think this approach is going to work for us beautifully...




Yes, i forgot that as you reuse the same DB you should also change
the persistent session/subscription loading runlevel to be after runlevel 3,
for example:

<plugin id='subPersistence' className='org.xmlBlaster.engine.SessionPersistencePlugin'>
<action do='LOAD' onStartupRunlevel='4' sequence='1' />
<action do='STOP' onShutdownRunlevel='4' sequence='6'/>
</plugin>

regards,
Marcel



Thanks!
Saju

-----Original Message-----
From: owner-xmlblaster at server.xmlBlaster.org
[mailto:owner-xmlblaster at server.xmlBlaster.org] On Behalf Of Marcel Ruff
Sent: Friday, October 28, 2005 2:01 AM
To: xmlblaster at server.xmlBlaster.org
Subject: Re: [xmlblaster] HA possible using session & subscription
persistence?

Saju Abraham wrote:





Hi,

I had a few questions again on clustering/HA of xmlBlaster:

1) I wanted to know whether a HA setup could be achieved using the session
and subscription persistence features. I'll be having two xmlBlaster


nodes


that would be managed by HA software and the client would connect to only
one of the nodes using a virtual IP address. Sessions and subscriptions
would be stored in Oracle. When the active node goes down, we would start
up the second node, but can the second node can pick up the persistent


data


of the 1st node from the DB and load all the sessions and subscriptions?
I'm thinking there should be some common key or identifier for the
persistent data so both nodes would use the same data, perhaps something
like node id? From the client perspective, it would be entirely




transparent




as they would only be connecting to the virtual IP address.






Smart idea! From a first glance i think this could work, just try it.
As all persistent informations (like subscriptions and messages) are there on startup
of the backup xmlBlaster node no information will be lost even if clients reconnect
after failover with longer delays.

If you, for example, start xmlBlaster (on both nodes) like:

 java org.xmlBlaster.Main -cluster.node.id heron

 java javaclients.HelloWorldSubscribe -persistentSubscribe true

and check the XB_ENTRIES table in Oracle you will see the 'queueName' entries like:

"topicStore_heron"
"session_heronsubPersistence"
"subscribe_heronsubPersistence"

and they are neutral to any physical host instance.
You probably have to STONITH the other node to avoid simultaneous running
xmlBlasters (but this is your HA setup doing anyhow).

Note: If you need to optimize xmlBlaster startup time you can usually
optimize it from normal startup of ~3 sec down to ~100 milli seconds.
You would need to have the standby xmlBlaster running in runlevel 3
(using command line option -runlevel 3)
and on activation fire it up to runlevel 9.
Change in xmlBlasterPlugins.xml to startup/shutdown the SOCKET
protocol plugin (if you for example use SOCKET) to runlevel 4
to find the new virtual addres.
Further remove all not needed protocols and other services (like telnet etc.).
To control the runlevels from outside you could consider using a
little JMX application, telnet or a fast C client (with an administrative SOCKET
protocol plugin in runlevel 2).





2) If suppose I wanted to implement mirroring of session and subscription
info across nodes (which is being extremely optimistic :-) ), which would




be




a good place to start looking into (package, file, etc.)






The best approach needs to be discussed, her some first ideas:
- Solve the initial replication when an additional node comes up
- Sending those info as normal messages to other cluster nodes
where sync and async (with queueing) approaches can make sense
depending on the use cases
syn: The network must be very reliable as any error would block all sync'd nodes
async: Loosely coupled nodes which can be down temporary as the messages are queued
- Or: mirroring the backend database table XB_ENTRIES (many issues to solve here)

As xmlBlaster has all communication, subscribing and queueing issues in its heart
the implementation should be straight forward, once we have found the concept.





3) For persistence of session and subscription info, if I wanted to use


the


JBoss distributed cache instead of a DB, I assume I need to create my own
persistence plugin which would be an implementation of I_Map?






I_Map and I_Queue.
Has JBoss distributed cache a JDBC interface?
If yes, it is simple, probably a configuration issue only.
If no, you need to implement the persistence part of I_Queue and I_Map.
Note that the RAM and CACHE implementations of I_Queue and I_Map
will still work no matter how your persistence plugin is working.

Enjoy,
Marcel





Many thanks in advance!

Kind regards,
Saju