Re: [xmlblaster] RE: Some issues with release .84

[Marcel Ruff <mr at marcelruff.info>]

Lance Thomas wrote:

> Hello XMLBlaster folks,
>
> I'd like to thank you for contribution with XMLBlaster. It is a wonderful
> product, and it seems to be on its way to becoming an industrial strength
> messaging server.
>
> We have been testing out XMLBlaster 0.84 and we ran into the following
> difficulties. We were wondering if you had seen these before and if you had
> any suggestions:
>
> 1) Difficulties with database persistence: we are using a Postgres database
> to act as persistent storage. Every so often (around 1/100 times during our
> tests), XMLBlaster will throw the following error:
>
> java.sql.SQLException: ERROR:  Cannot insert a duplicate key into unique
This is fixed.

> 2) Deadlock: XMLBlaster clients occasionally appear to enter into deadlock,
> under certain simultaneous usage patterns. Other non-deadlocked clients seem
> to be able to connect and continue processing. One culprit may be postgres,
> although I am unsure.
This should be fixed. (Please report if it should show again).

> 3) Difficulties with large messages: some of the messages we send are 2
> megabytes in size. If there are about five clients publishing messages of
> this size simultaneously, meaning a total of 10 MB in motion, the server may
> throw an out of memory exception, even when the JVM is set to use 256 MB of
> memory. If there are any property settings or other configuration settings
> that would help with this, please let us know. If XMLBlaster is not supposed
> to be used with such large messages, or if we must figure in a 26x memory
> factor in deployment, please let us know. 
I made some investigations and found out it is related to the used JDBC 
driver.
Running RAM based only we use exactly the RAM which is the size of the 
message.
Running persistent showed me:

1. Using the deliverd Postgres JDBC driverin in xmlBlaster/lib:
  I had up to x40 memory factor used, this is temporary during JDBC 
operation
 and there after freed.

2. I played with newer Postgres JDBC versions (latest devel, latest 
stable for different
  JDK versions):
 They behave all the same, but "only" use half the amount of memory (up 
to x20)
 So i have updated the delivered JDBC driver in xmlBlaster/lib.
 You can replace it yourself if you find a better driver.

3. Using Oracle 9i JDBC driver.
 Here i found a temporar factor of x5.
 Note that again after insert completion the memory consumption is down 
to the
 expected amount.

Note: With the default setting the max . size of the cache is 2 MB, so 
if you send
 messages bigger 1MB (until any value) only one message is kept in cache,
 the others are swapped to harddisk.  Every topic (key oid) has its own 
cache.
Those properties configure the settings (xmpBlaster.properties):
persistence/msgUnitStore/maxBytesCache= [defaults to 2 MB]
persistence/msgUnitStore/maxBytes=  [The max size on harddisk, defaults 
to Integer.MAX_VALUE]

> 4) Large amounts of data in persistent storage: Suppose we publish 500 MB
> worth of messages into XMLBlaster with persistence flagged. XMLBlaster
> accepts these messages and writes them to persistent storage with no
> problem. However, if we stop and restart XMLBlaster, it attempts to reload
> the messages from persistent storage and eventually runs into memory-related
> errors (on our 256 MB virtual machine). Again, if there are any property
> settings that will help with this, please let us know.
I think this is covered by 3)


Please do a
 cvs update
 build.sh delete
 build.sh all
to get the newest.

i hope this helps,

Marcel

> Again, we thank you and commend you for a fine product, and if you have any
> ideas about these issues, please let us know.
>
> Take care,
>
> Lance Thomas
>
>
>