[xmlblaster] WG: Help! OutOfMemory by publishing lot big topics

["Thomas, Johann" <Johann.Thomas at danet.de>]

(after 3h still not seen in mailing list, resending)

-----Original-----

Hi,
 
I have chosen xmlBlaster to send messages (out of a Java Servlet) with
content of often few 10 or 100 Kbyte reliably, but not to often.

But now I ran into OutOfMemoryError in the server, by doing kind of a
dump through this interface. The problem seems to be that the published
messages are not swapped out to persistence, filling up the heap. 

I start with the question: 
- Should I try to keep the RAM caches empty by some settings and patches
or will it probably not work at last?

What I have done in this area:
I read some mailing list messages saying that xmlBlaster is not that
good for persisting lot of data :-( I started my testing with publishing
some 10MByte in 512KByte pieces and found:
- it helps switching away from HSQLDB to Oracle (OT: I used a 10g1 JDBC
against the 9g2 database)
- the content dumped is cached "somewhere" in the heap, but it can not
be garbage collected (each message has it' unique oid).
- I have set some properties maxEntriesCache = 0, but it had no effect
at publishing time. I have found some comments in the some of the code,
saying first overload of queue is allowed. 
- I have patched CacheQueueInterceptorPlugin for me, so that it does not
overload the cache above the specified settings. I'm not sure the server
is behaving as stable as before with this settings, btw.
- after the restart of the xmlBlaster server the heap was mostly free.

So I suppose, there some other queues/caches were the first published
message of a given oid is held in memory, but not preloaded at a server
restart.


Here the overview of my little patch (between 2006-06-16 2101 JTh patch
comments):

// .................... (cut for keeping the email short) package
org.xmlBlaster.engine.msgstore.cache;
public class PersistenceCachePlugin implements I_StoragePlugin,
I_StorageProblemListener, I_Map, PersistenceCachePluginMBean {

// .................... (cut for keeping the email short)

   private void assureTransientSpace(I_MapEntry mapEntry) throws
XmlBlasterException {

      while (!spaceLeft(mapEntry, this.transientStore)) {

// .................... (cut for keeping the email short)

         if (oldest == null) {
            if (log.isLoggable(Level.FINE)) log.fine("The RAM queue is
full, new entry '" + mapEntry.getUniqueId() + "' seems to be the first
and only one, so we accept it");
            break;
         }
         // ++++ 2006-06-16 2101 JTh patch start
         if (oldest == mapEntry) {
            if (log.isLoggable(Level.FINE)) log.fine("2006-06-16 2101
JTh patch: The RAM queue will become full, new entry '" +
mapEntry.getUniqueId() + "' seems to be the first and only one, we still
removed it, transientStore.numOfEntries now " +
this.transientStore.getNumOfEntries());
         }
         // ---- 2006-06-16 2101 JTh patch end
         if (log.isLoggable(Level.FINER)) log.finer("Swapping '" +
oldest.getLogId() + "' to HD ...");

// .................... (cut for keeping the email short)

      }
   }

   public int put(I_MapEntry mapEntry) throws XmlBlasterException {

// .................... (cut for keeping the email short)

      synchronized(this) {

// .................... (cut for keeping the email short)

         // separate persistent from transient entries and store the
persistents in persistence // .................... (cut for keeping the
email short)


         assureTransientSpace(mapEntry);
         
         numTransientPut = this.transientStore.put(mapEntry);
         
         // ++++ 2006-06-16 2101 JTh patch start
         assureTransientSpace(mapEntry);
         
         // ---- 2006-06-16 2101 JTh patch end
      } // sync(this)
// .................... (cut for keeping the email short)
   }
}
 
Regards,
 
Johann Thomas

--
OpenPGP: 0x224225EA (7454 D7BB A75D EA8E 5DFF EA4E 778E A85B 2242 25EA)