Re: [xmlblaster] ssl and compression with SOCKET

[PBal <bpoka at axelero.hu>]

Hi Marcel!

> this is a useful extension to our basic SOCKET implementation.
> If you are willing to donate it under LGPL license we would be
> happy to add it to the xmlBlaster distribution.

I'm willing.

> If so, you can send the patch directly to my mail address.
> You would need to add a test case as well and some documentation in
> the SOCKET requirement (xmlBlaster/doc/requirements/protocol.socket.xml).

> Is there a compatible C compression library with a free license around
> which could be added to the C/C++ client library?
> Do you have a property for a minimum message size to switch on
> compressing?

Yes, there is a compatible C compression library. I mentioned that jzlib 
was used as the java library. As it is based on (actually, quite 
copy-paste work) the GNU zlib C library, the libs and their 
inputs/outputs have to be fully compatible. It's also stated on their 
website: http://www.jcraft.com/jzlib/

Let me explain the implementation details:
- Currently, compression is turned on if and only if SSL sockets are 
used. However, these features can be separated easily.
- The _whole_ tcp stream is compressed, because it was the easiest to 
implement it this way. This means that the "compression window" is not 
reinitialized on every message; The dictionary isn't flushed in the 
deflater, so repeating sequences are compressed even if they were in a 
previous message. I thought that would enable the deflater to compress 
the beginning of the message, too, and helps to achive better results 
when the same type of small messages are sent frequently (in my case, 
that is very true).
What I don't know is how all this behaves when random messages are sent 
on a connection. I mean, what if virtually every message would have some 
ugly binary content, for example? I still believe this would not make 
the messages bigger on average.

A deeper understanding (and more modification) of the SOCKET 
implementation would be required if I wanted to compress individual 
messages. This is not impossible, either. Moreover, if I put the 
compression filter in the right place, we would have compression for 
every protocol, wouldn't we?

Thinking about this, I already have another implementation which, in a 
way, supports compressing individual messages: it only compresses a 
block of bytes when the stream is flushed or when its buffer is full. 
When xmlBlaster sends a message, it writes the whole message to the 
OutputStream, then flushes it. When flush is invoked, the buffer state 
would enable it to decide whether to compress the current buffer (which 
should be a message, or a big part of it) or not. This way, we would 
lose the state of the deflater object and start with a new one on every 
flush, compress the buffer, and see if it's smaller than the original 
buffer. If it is, we'd send it compressed, otherwise not. Some simple 
protocol wrapper is also needed, but that is no problem.

I think these are our options.

I will assemble a patch for you as soon as I sepatated the SSL and 
compression layer and cleaned up some code. If you like the idea of 
deciding compression on a per-flush basis, then that will be included, too.

greetings,
Balázs