Re: [xmlblaster-devel] OutOfMemory Exceptions

[Joshua Bowman <josh at eiderdown.gsfc.nasa.gov>]

Michele,

I gave the incremental collector a shot.  It seemed to work out fairly
well at first, but as I entered the third day of runtime, the heap had
ballooned to 100+ MB (as opposed to the 10's and 20's that I had been
seeing originally).

I'm perusing these docs from Sun:
http://java.sun.com/docs/hotspot/gc1.4.2/
and will let you know if I find a better solution for what I imagine to
be a rather unique problem.

To answer your question, I'm using JDK 1.5 plus the JMX 1.2.1 and
NetBeans 4.1 extensions.  The operating system is Fedora Core 3 with
version 2.6.10 of the kernel.

- Josh

On Wed, 2005-09-07 at 14:12, Michele wrote:
> Hi Joshua,
> mmhh, it sounds strange that the garbage collector does not clean up 
> without human intervention. I really don't know what to say but to give 
> a try to activate the incremental garbage collector (-Xincgc).
> 
> java -Xincgc org.xmlBlaster.Main
> 
> Btw what jvm and OS are you using ?
> 
> Regards
> Michele
> 
> 
> Joshua Bowman wrote:
> > Marcel,
> > 
> > Thank you very much for your help.  I was able to see that the clients
> > are properly disposing of topics and that nothing is really collecting
> > on the bus itself.  It appears also that after running the bus for a
> > couple days, using the garbage collection button in the jconsole returns
> > memory usage to initial levels.
> > 
> > Is there a way to tell the bus to do its own garbage collection
> > periodically so that I don't have to enable the jmxremote libraries all
> > the time?  They appear to slow the system down considerably and I would
> > like the bus to run without human interaction anyway.
> > 
> > Thanks again!
> > 
> > - Josh
> > 
> > On Tue, 2005-08-30 at 13:11, Marcel Ruff wrote:
> > 
> >>Hi Joshua,
> >>
> >>please check my last response i send on 2005 08 26,
> >>it should help you further.
> >>
> >>If the above doesn't help: You can send a dump of xmlBlaster (from 
> >>command line just type 'd mydump.xml')
> >>zip it and send it to michele and myself directly after xmlBlaster was
> >>running for a day.
> >>
> >>regards
> >>Marcel
> >>
> >>
> >>
> >>Joshua Bowman wrote:
> >>
> >>
> >>>Hello,
> >>>
> >>>First, my setup:
> >>>
> >>>I have been tinkering with version 1.0.2 of the bus along with the Perl
> >>>client API.  I've got several clients connecting, some of them
> >>>exclusively publish-ing, others exclusively get-ing.  The get clients
> >>>stay connected and issue a get command every few seconds.  The
> >>>publishing clients only connect intermittently--a typical push setup.
> >>>
> >>>Each message has an entirely unique oid, and when a client sees an oid
> >>>more than once, it issues an erase command.  Multiple clients may be
> >>>get-ing that message, because I'm using XPATH queries that only care
> >>>about higher-level taxa, i.e., the unique timestamp at the end of the
> >>>oid is ignored when making most queries.
> >>>
> >>>Also, I should point out that I am running the middleware without any
> >>>persistence.  (I uncommented all the RAM plugin lines in the
> >>>xmlBlaster.properties file)  I've also set the message/maxRemainingLife
> >>>to 1000 ms and the destroyDelay to 1 sec.  Typical throughput is
> >>>somewhere in the range of a hundred messages/minute.
> >>>
> >>>The server is running Fedora Core 3 on a 2 GHz processor with 1 GB of
> >>>RAM.
> >>>
> >>>Now, the problem:
> >>>
> >>>The middleware and all clients function perfectly for a couple days and
> >>>then I start getting exceptions tossed all over.  Especially by the
> >>>get-ing clients.  A few examples:
> >>>
> >>>Fault returned from XML RPC Server, fault code 0: java.lang.Exception:
> >>>errorCode=legacy message=#exported DOM tree is invalid
> >>>
> >>>Fault returned from XML RPC Server, fault code 0: java.lang.Exception:
> >>>errorCode=user.query.invalid message=#exported Sorry, can't access,
> >>>query syntax of '/xmlBlaster/key[contains ( at oid,'<unique id>')]' is
> >>>wrong : java.lang.NullPointerException
> >>>
> >>>The middleware gives these exceptions:
> >>>
> >>>[Aug 25, 2005 5:09:35 AM WARN  Thread-10 XmlKeyDom.InvalidQuery] Sorry,
> >>>can't access, query syntax is wrong for '/xmlBlaster/key[contains
> >>>( at oid,'<same unique id as above>')]' : java.lang.NullPointerException
> >>>
> >>>java.lang.OutOfMemoryError
> >>>
> >>>It appears that the clients are properly deleting messages from the bus,
> >>>because when I restart a get-ing client, it doesn't recieve any of the
> >>>messages that were sent while it was down.  Are these errors due to some
> >>>sort of race condition, or am I really running out of memory?
> >>>
> >>>I'm inclined to believe it is the former, because I'm quite new to
> >>>xmlBlaster, and I'm fairly certain that I'm abusing the API pretty
> >>>badly.
> >>>
> >>>Also, I tried getting server stats using the admin commands, but I can't
> >>>figure out how to do anything other than these four, which are provided
> >>>in the code example:
> >>>
> >>>__cmd:?totalMem
> >>>__cmd:?freeMem
> >>>__cmd:?usedMem
> >>>__cmd:?clientList
> >>>
> >>>I'd really like to be able to see some queue details to see how long
> >>>messages are hanging around after I think they're supposed to be
> >>>deleted.
> >>>
> >>>Any help will be much appreciated.
> >>>
> >>>- Josh
> >>> 
> >>>
-- 
 

-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
 Joshua Bowman
 Software Developer
 NASA EO-1/ST-5/SDO
 Phone: 301-286-6294
 Email: josh at eiderdown.gsfc.nasa.gov
-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-