Re: How do I inflate compressed data from an asynchronous socket?

Again, keeping mind my lack of first-hand knowledge with the exact situation...

On Mon, 11 Jun 2007 02:25:29 -0700, Pat B <to_pat@xxxxxxxxxxx> wrote:

When I was receiving the raw data I was not using NetworkStream, I was
just using the socket itself. I was calling BeginReceive and
EndReceive because I don't want the read to block. It is much simpler
to make use of the asynchronous calls than to implement threads or
thread pools.

Yes, I agree that the asynchronous methods of the Socket class are simple and convenient. Do keep in mind that you are still using the thread pool though; you just do it implicitly in this case, rather than explicitly.

[...]
First I tried:

OnConnect
{
readStream = new SharpZipLib.InflaterInputStream(new
NetworkStream(socket))
readStream.BeginRead(tempbuffer[256], OnReceive)
}

OnReceive
{
read = readStream.EndRead()
readData += CopyLeftBytes(tempbuffer, read)
ProcessData(readData)
readStream.BeginRead(tempbuffer[256], OnReceive)
}

ProcessData
{
...
}

This works... when everything works. The problem is when the supernode
(server) terminates the connection while you are waiting for data, or
if some invalid data is sent by the supernode, the exception is thrown
on a different thread (initiated by windows?) and so I cannot catch it
an clean up. This is because the Begin and End are asynchronous and
the exception is being thrown in the middle.

I see. So the decompressor (SharpZipLib) exposes a stream from which you can read. IMHO, it seems to me that SharpZipLib ought to provide a mechanism for cleanly dealing with errors on the NetworkStream you pass to it. It could do this in a variety of ways, but I can't comment on whether or how it does, never having used it.

If it does in fact provide no mechanism, even though it should, then one thing you might consider is writing your own intermediate Stream class to handle the errors. You would have this class simply forward all the basic calls to the NetworkStream it contains, so that the SharpZipLib is calling your class rather than NetworkStream directly. That way if and when an exception or other error occurs, you have a chance to deal with it before propogating it back to the SharpZipLib class.

Using the dotNET DeflateStream in the place of
SharpZipLib.InflaterInputStream blew up with the error: "Block length
does not match with its complement."

So then I tried this:

OnConnect
{
inflater = new SharpZipLib.Inflater()
socket.BeginReceive(tempbuffer[256], OnReceive)
}

OnReceive
{
read = readStream.EndRead()
inflater.SetInput(CopyLeftBytes(tempbuffer, read))
read = inflater.Inflate(inflateBuffer[1024])
readData += CopyLeftBytes(inflateBuffer, read)
ProcessData(readData)
readStream.BeginRead(tempbuffer[256], OnReceive)
}

ProcessData
{
...
}

This works only when tempBuffer is large enough to receive the entire
packet before passing it into the inflater. If not it blows up on the
inflater.Inflate(...) line with the "broken uncompressed block" error.

This sounds to me like the same error you get using the "dotNET DeflateStream" mechanism: essentially, you've tried to decompress data in one shot without having the complete data available to decompress. No surprise then that doesn't work.

- So in each case I am catering for different size blocks of data
coming from the socket.

Yes, it seems to me that you're correctly handing the basic network receiving correctly.

- I have no way of knowing how much data to expect until after I have
decompressed it.

This seems to me to be a fundamental mistake in the Gnutella protocol, which requires you to make at least some extra effort beyond that which should be necessary. I suspect you can get SharpZipLib to do what you want, but in the worst case if you couldn't, the Gnutella protocol would put you in a position where you'd have to do all the decompressing yourself in your own code so that you can work around not knowing how much data you're expecting.

Whether this limitation really exists in Gnutella, I can't say. It may be that it does, or it may be that you're just overlooking something in the protocol that would address this.

- I don't want to have to implement using thread pools and blocking
calls.

Well, as I said, you are already using the thread pool implicitly. Still, it doesn't seem to me that you would wind up in a position where you have to implicitly do some threading, given the apparent design of the libraries you're trying touse.

- Using streams may be the solution, only if there is a good way of
catching the exceptions from the asynchronous reads.

See above: I suspect that if you simply write your own Stream-derived wrapper for the NetworkStream, you can intercept the exceptions before they get to the SharpZipLib class, allowing you to handle the exceptions however you see fit.

Maybe I am supposed to expect the error from the inflater and take
that as an indication to wait for more data before trying to
decompress again?

You could do that as well. I don't know why it wouldn't, even though you say it doesn't. When you tried this, are you sure that you retained the data at the beginning of your array for future attempts? It seems to me that if you keep trying to decompress the same data, except that new data is appended between each attempt, eventually you would have enough data to decompress and it would succeed. It doesn't seem like a very efficient approach to the problem though, so I think you should explore the Stream-derived wrapper solution first.

Pete
.

Relevant Pages

  • [bds 2006] Decompress .gz files with zlib unit?
    ... The files are not corrupt -- WinZip was unable decompress them. ... "The exception unknown software exception occurred in the ... bufferSize: Integer; ... bytesRead: Integer; ...
    (comp.lang.pascal.delphi.misc)
  • Re: Decompressing a ZIP File.
    ... I don't think System.IO.Compression namespace supports it. ... SharpZipLib ... How does one open and decompress a zip file using the ... Prev by Date: ...
    (microsoft.public.dotnet.framework)