XmlBlaster Logo

REQUIREMENT

client.cpp.queue

XmlBlaster Logo


Type NEW
Priority MEDIUM
Status CLOSED
Topic XmlBlaster provides a persistent queue implementation for C++ clients
Des
cription

In C++ we have three plugins implementing the I_Queue interface. One is called "RAM" and is an in memory implementation, one is called "SQLite" and reuses the xmlBlaster C implementation of a peristent queue using the relational database SQLite. Finally you can use the "CACHE" queue implementation which implements a simple combination of the former two.

This queue resides on the C++ client side and stores messages which can't be delivered to the xmlBlaster server during connection problems. The "RAM" queue is always available in the xmlBlaster client library, the "SQLite" persistence queue needs some setup which is described below.

CACHE limitation

The CACHE class CacheQueuePlugin implements a very simple cache around the RAM and SQLite queue. Note that there is no swapping support for transient or persistent messages all transient messages are hold in RAM, and all persistent messages are duplicated to harddisk. When time permits we will add swapping support similar to the Java CACHE implementation.

If you have mainly persistent messages and need to take care on your RAM consumption with many messages in queue consider to use the "SQLite" persistent queue directly (without any RAM or CACHE) with the option -connection/queue/type SQLite instead of the default -connection/queue/type CACHE.
On the other hand if you use only transient message consider using the RAM queue directly with the option -connection/queue/type RAM instead of the default -connection/queue/type CACHE. In this case if your C++ client crashes the messages in your client queue are lost.

Setup

To use the persistent queue you need to download SQLite and compile it for your platform, our C/C++ compilation needs sqlite.h for compilation and libsqlite.so for linking (on Windows its a dll). Choose a version newer SQLite 2.8.13 / March 2004 to have the blob encoding feature. For more details please read the client.c.queue requirement.

Compilation with ant

First add those lines to your HOME/build.properties (or add it on command line with a leading -D option) 

XMLBLASTER_PERSISTENT_QUEUE=1
sqlite.include.dir=/opt/sqlite-bin/include
sqlite.lib.dir=/opt/sqlite-bin/lib

and compile it: 

build cpp-delete cpp
or similiar (with verbose switched on): 
build -verbose -DXMLBLASTER_PERSISTENT_QUEUE=1 -Dsqlite.include.dir= ... cpp

This expects /opt/sqlite-bin/include/sqlite.h and /opt/sqlite-bin/lib/libsqlite.so* (on Windows it is a dll).

Example
CPP

Here is an example how to play on command line with the client side queue and test the behaviour: 

java org.xmlBlaster.Main -call[core] true

cd xmlBlaster/demo/c++/bin

PublishDemo -persistent true -numPublish 100
            -session.name publisher/1 -dispatch/connection/delay 5000
            -dispatch/connection/pingInterval 1000

SubscribeDemo -persistentSubscribe true -dispatch/callback/retries -1
              -dispatch/connection/delay 5000 -session.name subscriber/1

You can kill the above process, restart them and check that nothing is lost.

The SQLite database files resides as a default setting in your home directory, here is an example for UNIX how to look into the database with the sqlite executable delivered with the SQLite distribution: 

sqlite /home/joe/xmlBlasterClientCpp.db

sqlite> .mode line

sqlite> select * from xb_entries;
   dataId = 1083681933046933000
   nodeId = /node/xmlBlaster_192_168_1_4_3412/client/publisher/1
queueName = connection_nodexmlBlaster_192_168_1_4_3412clientpublisher1
     prio = 5
     flag = MSG_RAW|publish
  durable = T
 byteSize = 257
     blob = ...

sqlite> .exit
Configure

Tested configurations:

No OS Compiler xml Blaster Vers. Date Author Comment
1 Linux 2.6.4 g++ and gcc 3.3.3 0.901+ 2004-05-04 Marcel build cpp
2 Linux 2.6.4 Intel(R) C++ Compiler for 32-bit, Version 8.0 (icc) 0.901+ 2004-05-04 Marcel build -Duse-icc=true cpp
3 WindowsXP VC++ 7 (Jan.2003) and VC++ STL 0.903 2004-05-16 Marcel build -Duse-msvc=true cpp

These parameters allow to configure the C++-client on command line, over xmlBlaster.properties or over the environment (with lower priority):

Property Default / Example Description Impl
-queue/connection/type CACHE Choose one of "RAM", "CACHE" or "SQLite" yes
-queue/connection/url ${user.home}/ xmlBlasterClientCpp.db The location and file name of the database (for SQLite). ${user.home} is preset in the C++ client library and resolves to your home directory. Other preset variables are described in xmlBlaster/src/c++/ util/Property.h. yes
-queue/connection/tablePrefix XB_ The prefix used for database table names, usually you don't change this. yes
-queue/connection/maxEntries 1000 The maximum allowed number of messages yes
-queue/connection/maxBytes 10485760 The maximum allowed bytes of all messages in the queue yes

Testing

The testsuite resides in xmlBlaster/testsuite/src/c++ to compile it use ant (with the above build.properties settings): 

cd xmlBlaster
build cpp-test

To run the tests invoke in directory xmlBlaster/testsuite/src/c++/bin 

TestQueue

NOTE: Configuration parameters are specified on command line (-someValue 17) or in the xmlBlaster.properties file (someValue=17). See requirement "util.property" for details.
Columns named Impl tells you if the feature is implemented.
Columns named Hot tells you if the configuration is changeable in hot operation.

Todo

The persistent queue implementation is finished and tested, but we need to:

  • Extend the CACHE implementation to support swapping.
  • Deliver sqlite library (.dll,.so,.sl) in directory xmlBlaster/lib at least for Linux/Win for ease of use?

See CODE Queue.h
See TestQueue.cpp
See http://www.sqlite.org/
See SQLite compile
See REQ client.cpp.failsafe
See REQ client.c.queue
See REQ client.cpp
See REQ Queue dump formating identical to SOCKET serialization
See TEST TestQueue

This page is generated from the requirement XML file xmlBlaster/doc/requirements/client.cpp.queue.xml

Back to overview