[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [xmlblaster-devel] OutOfMemory Exceptions
Hello,
The saga continues:
My application definitely has a lot of objects with short lifetimes and
almost none with longer ones. Thus, after reading the Sun docs, I just
increased the size of the eden space using the "NewSize" parameter and
used the default garbage collector. It seems to work much better this
way.
However, because I'm using Perl, it doesn't appear that I can make use
of the subscribe asynchronous callback. Using get in a one-to-many
propagation structure obviously has some problems. How does one make
sure that all clients get the message before the message is removed?
I'm currently kludging my way around this by just running a seperate
client that handles all erase-ing. All get-ing clients check in within
some specified time quantum, so the erase-ing client simply waits until
a message has been around for two full quanta and then assumes that all
clients have checked in and removes it from the bus.
However, this seems to lead to concurrency issues. If one client is
trying to get while another is erase-ing, it seems to generate an
exception. Is there a way to pass an option to the bus that will ensure
messages are cleared out only after a certain time interval has
elapsed? I've tried fiddling with destroyDelay and maxRemainingLife,
but it seems that I still need to invoke the erase command.
Any suggestions?
- Josh
On Fri, 2005-09-09 at 10:07, Joshua Bowman wrote:
> Michele,
>
> I gave the incremental collector a shot. It seemed to work out fairly
> well at first, but as I entered the third day of runtime, the heap had
> ballooned to 100+ MB (as opposed to the 10's and 20's that I had been
> seeing originally).
>
> I'm perusing these docs from Sun:
> http://java.sun.com/docs/hotspot/gc1.4.2/
> and will let you know if I find a better solution for what I imagine to
> be a rather unique problem.
>
> To answer your question, I'm using JDK 1.5 plus the JMX 1.2.1 and
> NetBeans 4.1 extensions. The operating system is Fedora Core 3 with
> version 2.6.10 of the kernel.
>
> - Josh
>
> On Wed, 2005-09-07 at 14:12, Michele wrote:
> > Hi Joshua,
> > mmhh, it sounds strange that the garbage collector does not clean up
> > without human intervention. I really don't know what to say but to give
> > a try to activate the incremental garbage collector (-Xincgc).
> >
> > java -Xincgc org.xmlBlaster.Main
> >
> > Btw what jvm and OS are you using ?
> >
> > Regards
> > Michele
> >
> >
> > Joshua Bowman wrote:
> > > Marcel,
> > >
> > > Thank you very much for your help. I was able to see that the clients
> > > are properly disposing of topics and that nothing is really collecting
> > > on the bus itself. It appears also that after running the bus for a
> > > couple days, using the garbage collection button in the jconsole returns
> > > memory usage to initial levels.
> > >
> > > Is there a way to tell the bus to do its own garbage collection
> > > periodically so that I don't have to enable the jmxremote libraries all
> > > the time? They appear to slow the system down considerably and I would
> > > like the bus to run without human interaction anyway.
> > >
> > > Thanks again!
> > >
> > > - Josh
> > >
> > > On Tue, 2005-08-30 at 13:11, Marcel Ruff wrote:
> > >
> > >>Hi Joshua,
> > >>
> > >>please check my last response i send on 2005 08 26,
> > >>it should help you further.
> > >>
> > >>If the above doesn't help: You can send a dump of xmlBlaster (from
> > >>command line just type 'd mydump.xml')
> > >>zip it and send it to michele and myself directly after xmlBlaster was
> > >>running for a day.
> > >>
> > >>regards
> > >>Marcel
> > >>
> > >>
> > >>
> > >>Joshua Bowman wrote:
> > >>
> > >>
> > >>>Hello,
> > >>>
> > >>>First, my setup:
> > >>>
> > >>>I have been tinkering with version 1.0.2 of the bus along with the Perl
> > >>>client API. I've got several clients connecting, some of them
> > >>>exclusively publish-ing, others exclusively get-ing. The get clients
> > >>>stay connected and issue a get command every few seconds. The
> > >>>publishing clients only connect intermittently--a typical push setup.
> > >>>
> > >>>Each message has an entirely unique oid, and when a client sees an oid
> > >>>more than once, it issues an erase command. Multiple clients may be
> > >>>get-ing that message, because I'm using XPATH queries that only care
> > >>>about higher-level taxa, i.e., the unique timestamp at the end of the
> > >>>oid is ignored when making most queries.
> > >>>
> > >>>Also, I should point out that I am running the middleware without any
> > >>>persistence. (I uncommented all the RAM plugin lines in the
> > >>>xmlBlaster.properties file) I've also set the message/maxRemainingLife
> > >>>to 1000 ms and the destroyDelay to 1 sec. Typical throughput is
> > >>>somewhere in the range of a hundred messages/minute.
> > >>>
> > >>>The server is running Fedora Core 3 on a 2 GHz processor with 1 GB of
> > >>>RAM.
> > >>>
> > >>>Now, the problem:
> > >>>
> > >>>The middleware and all clients function perfectly for a couple days and
> > >>>then I start getting exceptions tossed all over. Especially by the
> > >>>get-ing clients. A few examples:
> > >>>
> > >>>Fault returned from XML RPC Server, fault code 0: java.lang.Exception:
> > >>>errorCode=legacy message=#exported DOM tree is invalid
> > >>>
> > >>>Fault returned from XML RPC Server, fault code 0: java.lang.Exception:
> > >>>errorCode=user.query.invalid message=#exported Sorry, can't access,
> > >>>query syntax of '/xmlBlaster/key[contains ( at oid,'<unique id>')]' is
> > >>>wrong : java.lang.NullPointerException
> > >>>
> > >>>The middleware gives these exceptions:
> > >>>
> > >>>[Aug 25, 2005 5:09:35 AM WARN Thread-10 XmlKeyDom.InvalidQuery] Sorry,
> > >>>can't access, query syntax is wrong for '/xmlBlaster/key[contains
> > >>>( at oid,'<same unique id as above>')]' : java.lang.NullPointerException
> > >>>
> > >>>java.lang.OutOfMemoryError
> > >>>
> > >>>It appears that the clients are properly deleting messages from the bus,
> > >>>because when I restart a get-ing client, it doesn't recieve any of the
> > >>>messages that were sent while it was down. Are these errors due to some
> > >>>sort of race condition, or am I really running out of memory?
> > >>>
> > >>>I'm inclined to believe it is the former, because I'm quite new to
> > >>>xmlBlaster, and I'm fairly certain that I'm abusing the API pretty
> > >>>badly.
> > >>>
> > >>>Also, I tried getting server stats using the admin commands, but I can't
> > >>>figure out how to do anything other than these four, which are provided
> > >>>in the code example:
> > >>>
> > >>>__cmd:?totalMem
> > >>>__cmd:?freeMem
> > >>>__cmd:?usedMem
> > >>>__cmd:?clientList
> > >>>
> > >>>I'd really like to be able to see some queue details to see how long
> > >>>messages are hanging around after I think they're supposed to be
> > >>>deleted.
> > >>>
> > >>>Any help will be much appreciated.
> > >>>
> > >>>- Josh
> > >>>
> > >>>
--
-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
Joshua Bowman
Software Developer
NASA EO-1/ST-5/SDO
Phone: 301-286-6294
Email: josh at eiderdown.gsfc.nasa.gov
-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-