XmlBlaster Logo

REQUIREMENT

queue.jdbc.postgres

XmlBlaster Logo


Type NEW
Priority HIGH
Status CLOSED
Topic XmlBlaster has an advanced queuing facility. Among others one tested persistence layer is offered by communicating with a postgres database.
Des
cription

The generic plugin for jdbc based queues has been tested against postgres (postgres 7.2.2 up to 7.4.5 and 8.0-RC2 on linux and 8.0-RC2 on Windows).

To make xmlBlaster run with a persistent queue stored on postgres it is necessary to first install postgres. In some of the linux distributions postgres comes along. For others, you can download it from the postgres homepage. Please follow the instructions given there on how to download, install and initially configure postgres.. Keep in mind that you also need to install on your machine the jdbc driver working with postgres. You can download it from http://jdbc.postgresql.org/download.html and put it somewhere in your classpath. To keep it simple you can put the jar file (for example pgjdbc2.jar) in the $XMLBLASTER_HOME/lib directory.

If you have not initialized postgres previously, you should do it now by invoking 

$PGRES_HOME/bin/initdb -D $PGRES_HOME/data
as the user postgres, where '$PGRES_HOME' is the base directory where you installed your postgres (for example in /usr/local/pgres). Once postgres has been initialized you can start the server. Since we are communicating via TCP/IP with the jdbc driver, we need the '-i' switch on, for example: 
postmaster -i -N100 -B200 -D /usr/local/pgres/data
the -N50 flag and -B100 are needed if you want to run the complete testsuite since it will consume quite a big amount of tcp/ip connections. If you did'nt create a db before you can do it now. Lets suppose we call the database 'test', then what you have to do is invoke 
createdb test
And now you still have to configure xmlBlaster in order to be able to find the db. Lets say you run it in the standard port on your localhost and under the user postgres (you don't need a password for that user). If you choosed these settings you should be able to start xmlBlaster and automatically have the persistence queues writing their data on postgres. If you specified someting diffenrent for the db name, the user, the hostname or the port, please read more under 'configuration'.

Administrative maintenance during operation

Postgres tends to slow down drastically after some 100 thousand messages. To gain performance again you should always again start vacuumdb to clean up Postgres. This is possible in hot operation of xmlBlaster. 

vacuumdb -f -v test

This example starts vacuumdb every 2 minutes for an xmlBlaster under heavy load with persistent messages: 

while [ 1 ] ; do
  echo "***** Starting vacuumdb *****"
  vacuumdb -f -v test
  echo "Sleeping until next vacuumdb"
  sleep 120
done

On Linux you should try to improve hard disk performance with the hdparm command which can in many cases increase hard disk and persistent message throughput significantly: 

hdparm -d1 -c3 /dev/hda
hdparm -t /dev/hda
Example
Java 		These are example QoS of subscribe() invocations: 
<qos>
   <queue maxEntriesCache='1000' maxBytesCache='4000'
          maxEntries='10000' maxBytes='1000000000'
          onOverflow='deadMessage'/>
</qos>
Configure

These parameters allow to configure a cache queue running with postgres

Property Default Description Hot Impl
QueuePlugin[JDBC][1.0] org.xmlBlaster.util.queue.jdbc.JdbcQueueCommonTablePlugin specifies the JDBC implementing class to be used to store queue entries yes yes
StoragePlugin[JDBC][1.0] org.xmlBlaster.util.queue.jdbc.JdbcQueueCommonTablePlugin specifies the JDBC implementing class to be used to store the message content yes yes
queue/maxEntries 1000 specifies the maximum number of entries allowed in the ram portion of this queue yes yes
queue/maxBytes 2097152 (2MB) specifies the maximum total size for the ram portion allowed in this queue yes yes
queue/persistent/url jdbc:postgresql://localhost/test the url which specifies where to find the DB. Here postgresql is the db vendor name, localhost the hostname and test the db name. yes yes
queue/persistent/user postgres The db user under which to run yes yes
queue/persistent/password The unencrypted password. For the user 'postgres' the password is an empty string yes yes
Additionally the configuration parameters for the ram queue and the jdbc queue must be defined.

How to switch to Postgres

Please edit xmlBlaster.properties following the example below, typically you need to change the url setting to suit your environment.

JdbcDriver.drivers=sun.jdbc.odbc.JdbcOdbcDriver:org.postgresql.Driver:...

JdbcStorage[postgres]=org.xmlBlaster.util.queue.jdbc.JdbcQueueCommonTablePlugin,\
                      url=jdbc:postgresql://localhost/test,\
                      user=xmlblast,\
                      password=,\
                      connectionPoolSize=1,\
                      connectionBusyTimeout=90000,\
                      maxWaitingThreads=300,\
                      tableNamePrefix=XB_,\
                      entriesTableName=ENTRIES,\
                      dbAdmin=true

StoragePlugin[JDBC][1.0]=${JdbcStorage[postgres]}
QueuePlugin[JDBC][1.0]=${JdbcStorage[postgres]}

JdbcDriver.mapping[postgres]=string=text,longint=bigint,int=integer,boolean=char(1)

NOTE: Configuration parameters are specified on command line (-someValue 17) or in the xmlBlaster.properties file (someValue=17). See requirement "util.property" for details.
Columns named Impl tells you if the feature is implemented.
Columns named Hot tells you if the configuration is changeable in hot operation.

See API org.xmlBlaster.util.queue.I_Queue
See API org.xmlBlaster.util.queue.jdbc.CacheQueuePlugin
See API org.xmlBlaster.util.queue.QueuePluginManager
See API org.xmlBlaster.util.queue.I_QueueEntry
See API org.xmlBlaster.util.qos.storage.QueuePropertyBase
See API org.xmlBlaster.util.I_Plugin
See API org.xmlBlaster.util.Global
See REQ queue
See TEST org.xmlBlaster.test.classtest.queue.I_QueueTest
See TEST org.xmlBlaster.test.classtest.queue.JdbcQueueTest
See TEST org.xmlBlaster.test.classtest.queue.CacheQueueTest

This page is generated from the requirement XML file xmlBlaster/doc/requirements/queue.jdbc.postgres.xml

Back to overview