XmlBlaster Logo



XmlBlaster Logo

Type NEW
Priority HIGH
Topic Replication is a plugin framework used to replicate data between database (RDBMS) sources and sinks.

Replication Overview

The Replication framework makes use of the DbWatcher- and the DbWriter frameworks. Its goal is to offer the xmlBlaster user a way to replicate data from one or several source databases to one or more sink databases. The framework has been designed for a singlemaster multislave approach. What does that mean ? It means that it does not offer any conflict resolution mechanism. In other words it is safe for cases where you have one master database, also called the source for the replication, and one or more slaves, also called the sinks of the replication or replicas. It will not be safe where you have more than one master. There are situations where this framework is currently used with more than one source for one sink, but there are limitations, for example you must ensure that the same data (at least on a row granularity) always comes from one single source.

replication with xmlBlaster

Which features does this framework offer ?

  • Single Master / Multi Slave approach
  • Conventional one to one Replication
  • Replication with Data Manipulation on a per sink individual basis with no intervention on the master side
  • Replication with Data Filtration on a per sink individual basis with no intervention on the master side
  • Use of all Communication Protocols offered by XmlBlaster (even the most fancy ones)
  • Persistence of Messages, Failover and guaranteed sequence of messages
  • Transactions are handled atomically and sent in a single Message
  • Possibility to interchange this with other sources/sinks for for example Meta-Databases
  • Plugin approach to allow business-specific functionality
  • Possibility to intermix different database versions and vendors
  • Performance tuning for specific vendors (for example for initial update)
  • Highly Compressed communication
  • Encrypted and secure replication
  • Unlimited number of sinks
  • Efficient and reliable even on unreliable networks

Why yet another replication solution when there are so many out there already ?
Because it is difficult (or impossible) to find a replication software which can offer all of the following:

  • Replicate between different databases (different versions or different vendors)
  • Allow individual (specific for every sink) business specific manipulation or filtering
  • pass complex and high secure gateways or firewalls
  • open source
  • replicated is xml based and can be used directly on an application level
  • ...

Since the framework is highly flexible, it is also quite complex. Probably the simplest way to describe it is with an easy tutorial. I will start with a very simple use case. Parallely a description of what is going on behind the scenes will also be given. Prerequisistes for this tutorial is that you have an oracle database up and running.

The demo is also available as a Flash movie here.

Prepare, build and start the server

  • To avoid messy and strange behaviours make sure you don't have any old configuration around. Consider that as cleaning up the kitchen before starting to cook a good meal, and believe me, this meal is really good!
    You can clean up by changing to your home directory and delete xmlBlaster.properties and xmlBlasterPlugins.xml
  • Delete old peristent queues. These normally reside in your ${HOME}/tmp directory. It normally suffices to delete all xmlBlaster* and client* files.
  • open the ${HOME}/xmlBlasterPlugins.xml file and uncomment the MX4JAdaptorPlugin, this way it will become active.
  • Go to your ${XMLBLASTER_HOME} Directory and set the environment (in unixes with . .bashrc).
  • Start the xmlBlaster with JMX enabled:
          java org.xmlBlaster.Main -xmlBlaster/jmx/HtmlAdaptor true

Start the Demo Client for the Replication

  • Go to your ${XMLBLASTER_HOME} Directory and set the environment (in unixes with . .bashrc).
  • Go to the directory where the demo resides: ${XMLBLASTER_HOME}/demo/javaclients/contrib/replication.
  • Put in this directory the jdbc driver you want to use. Currently we only have tested with Oracle, so copy the latest driver (we used ojdbc14.jar) and put it in this directory.
  • Add this directory and the jar file to the CLASSPATH (unixes: export CLASSPATH=.:ojdbc14.jar:${CLASSPATH}).
  • Edit the first three lines of the files in the current directory readerDemo1.properties and writerDemo1.properties to make them correspond to your database. The settings are currently pointing to an oracle database on the localhost on the default port with a SID=TEST, and a user (or schema) XMLBLASTER
  • Now you will start the client. The client is composed by a master (the reader of the database or source), and a slave (which is the destination of the Replication). Internally xmlBlaster uses here a DbWatcher and a DbWriter. Invoke:
     java javaclients.contrib.replication.ReplDemo -master readerDemo1.properties -slave writerDemo1.properties -interactive true
    You will see the client starts and both the reader and the writer connect to the xmlBlaster. At the end you will see the text REPLICATION AGENT IS NOW READY and a prompt appears. If you press ENTER, then a new line appears. On the line you could now execute SQL Statements. The prompt tells you on which Database instance the SQL Statement will be executed. Here you see master telling you the statements will be executed on the master and therefore will be replicated. If you would have started only the slave, the prompt slave would have appeared. Of course for the sake of simplicity we are using one single database here.
  • In the configuration file you defined the tables to be replicated:
    Meaning DEMO_USERS and DEMO_ACCOUNTS will be replicated, of course if and only if they exist. You also can see the trigger name used to detect changes on the DEMO_USERS will be called DEMO_USERS_TRG. Not choosing anything here would let the application decide a name for it.
  • Now you can add the table DEMO_USERS by typing on the command line of the started client:
      create table demo_users (account varchar(100) not null, name varchar(100), firstname varchar(100), primary key(account))
    When completed press ENTER.
  • Once the table has been created it is time to populate it with some data:
      insert into demo_users values ('account1', 'Topo', 'Gigio')
      insert into demo_users values ('account2', 'Mister', 'Bean')

Start the JMX Monitor to initiate the Replication

Open an html browser of your choice, for example Firefox. Point it to http://localhost:9999. You will see something like the following:

start replication

The Source (here DEMO_) is the prefix identifying the source of the replication. The Destination is the name identifying the Destination of the replication. In this demo you only have one possible combination allowed. So click on the Initiate Repl. Button. This way the replication will start.

By clicking on the button you will be redirected to another page displaying all initiated and running replications. In this case you only have one single entry.

replication list

Note that the status is initially switching from blue to yellow. When it is in the blue phase (INITIAL), the source is prepared for the replication. During this phase the Active/Standby Status is in Standby. This means the replication is temporarly hold back, no message is sent to the destination.
After a very short time the status switches to blue (TRANSITION). During this phase the source is ready and the messages are delivered to the destination but the destination is not completely up to date yet. You will see that during this phase the Active/Standby flag is on Active meaning processed messages are delivered immediately.

When all initial messages have been delivered, the status is switching again, this time from yellow to green (RUNNING). From now on the destination is ready and the initial data have been replicated.

On the server logs you will see that the initial command has been executed and on the client logs you will see the PL/SQL Code of the trigger followed by a sequence of two XmlBlaster Messages each in xml format. The first of them is the CREATE statement itself and the second is the message containing all initial data of the replicated table.

You can now verify that the new table DEMO_USERS_REPLICA really has been created and that it contains the same entries as the source by invoking on the prompt:

   select * from demo_users_replica

This way you replicated data from one table to another.


NOTE: Configuration parameters are specified on command line (-someValue 17) or in the xmlBlaster.properties file (someValue=17). See requirement "util.property" for details.
Columns named Impl tells you if the feature is implemented.
Columns named Hot tells you if the configuration is changeable in hot operation.

See API org.xmlBlaster.contrib.dbwriter.DbWriter
See API org.xmlBlaster.contrib.dbwatcher.DbWatcher
See API org.xmlBlaster.contrib.replication.ReplicationAgent
See API org.xmlBlaster.contrib.replication.ReplSlave
See API org.xmlBlaster.contrib.replication.impl.ReplManagerPlugin
See Replication Documentation
See REQ contrib.dbwatcher
See REQ contrib.dbwriter
See TEST org.xmlBlaster.test.contrib.dbwriter.TestRecordParsing

This page is generated from the requirement XML file xmlBlaster/doc/requirements/contrib.replication.xml

Back to overview