🚀 go-pugleaf

"Mike Ruskai" <BUTthannydI@DONTearthlinkLIKE.netSPAM> wrote in message 
news:nko0t3h85jl7pm57md2te5ttc8b7m6glb7@4ax.com...

> The actual problem is that ReadBytes() is *NOT* calling ExtractToBytes()
> after the socket has been disconnected by the remote, following a
> completed transfer.

Yes, it does.  I have explained to you several times now why it can do so. 
NO EXCEPTION CAN BE RAISED if all of the required bytes have been received 
successfully and put into the InputBuffer.  It does not matter whether or 
not the remote party disconnected after sending the data.  ReadFromSource() 
will not get far enough to see that.

Think about it again.  Let's again assume the server sends 1024 bytes and 
then disconnects.  ReadBytes() is called with 1024 as the requested number 
of bytes.

Whenever ReadFromSource() is called, a single call to recv() is performed, 
passing in a buffer that is RecvBufferSize (32K by default) in size.  If 
even 1 byte is received into that buffer, the buffer is copied into the 
InputBuffer and ReadFromSource() exits without setting the ClosedGracefully 
property.  CheckForDisconnect() then exits without error, and ReadBytes() 
checks the InputBuffer size again, repeating the loop if needed.

Let's say, for the sake of argument, that the first call to ReadFromSource() 
received only 512 bytes.  ReadBytes() would keep looping, calling 
ReadFromSource() again.  Let's say that ReadFromSource() then receives the 
remaining 512 bytes the second time.  It puts them into into the 
InputBuffer, again ClosedGracefully is not set, CheckForDisconnect() again 
exits without error, the loop finally stops, and ExtractBytes() is called.

At no point can EIdConnClosedGracfully be raised by ReadBytes() unless the 
socket returned 0 bytes when ReadFromSource() was expecting at least 1 byte 
to arrive.  The ONLY other ways for the ClosedGracefully property to be set 
to true while ReadBytes() is running is if either 1) you are closing the 
connection from another thread at the same time, or 2) the socket reported a 
WSAESHUTDOWN or Id_WSAECONNABORTED error (in which case the data is lost 
anyway).

If you still think that is not the case in your situation, then please 
provide a stack trace and InputBuffer dump showing otherwise.  Because this 
logic is fundamental to Indy's core design, and has been running the way I 
have described, not the way you have desacribed, for years without problem.

> Asking for more bytes than are available is not a problem, according to
> the dictates of common sense, and the documentation for the function.

That is how the recv() function works.  Not the ReadBytes() method. 
ReadBytes() WILL NOT return less bytes than what you actually asked for. 
That is how ReadBytes() is designed to work.  That is how the documentation 
for ReadBytes() says it works.  Stop trying to treat ReadBytes() the same 
way as recv().  They have very different sementics, on purpose.  Using 
ReadBytes() like recv() is not the correct way to use Indy.

> It will either block until the requested number are available, or return 
> less
> if the socket is closed.

The recv() function does, yes.  The ReadBytes() method does not, no.  It 
raises an exception if the socket is closed before all bytes have been read.

> But it's returning ZERO when the socket is closed, not the total
> actually available.

recv() returns 0 when the disconnect is processed, yes.  At that point, 
there are no more bytes available in the socket to read.  That is when 
ReadBytes() raises the EIdConnClosedGracefully exception.

> But that still doesn't change the fact that ReadBytes() does not
> conform to its documented behavior, as stated above.

Yes, it does, and I have already demonstrated how it does.

> But it's not reading all the data when the socket is disconnected.

Yes, it is.  It keeps reading right up to the point when the socket returns 
0 bytes.  There is no more data available to read after that happens.

> I wrote a simple function to behave more like recv(), so I could
> plug it into a larger routine that loops through received data.

No wonder you are having so many problems.  You are using ReadBytes() in a 
manner that expects it to behave exactly like recv() does - but that is NOT 
how ReadBytes() works!!!  You are not even attempting to use the code I gave 
you earlier.

> The log output above shows conclusively that ReadBytes() does
> not return the last 253 bytes read from the socket.

That is because you asked it to read 7620 bytes, not 253 bytes.  It did 
exactly what it was supposed to do.  It tried to wait until all 7620 bytes 
were available, but after 253 bytes it detected the disconnect and so it 
raised the exception.  That is NORMAL BEHAVIOR for ReadBytes().

In order to use ReadBytes() in the manner you are attempting, you must use 
it more like the following instead:

    int __fastcall DataConstructor::recvData(TIdTCPClient *idDataSock, char 
*data, int dataLen)
    {
        int rlen;
        TIdBytes bdata;

        ofstream dbg("e:\\indyrecv.txt", ios::out | ios::app);

        dbg << "Request: " << dataLen << endl;

        try
        {
            // wait for at least 1 byte to be available for reading
            while (idDataSock->IOHandler->InputBufferIsEmpty())
            {
                idDataSock->IOHandler->CheckForDataOnSource(10);
                idDataSock->IOHandler->CheckForDisconnect(true, true);
            }

            idDataSock->IOHandler->ReadBytes(bdata, 
min(idDataSock->IOHandler->InputBuffer->Size, dataLen), false);

            rlen = bdata.Length;
            if (rlen > 0)
                BytesToRaw(bdata, data, rlen);
        }
        catch (const Exception &)
        {
            rlen = -1;
        }

        dbg << "Total received: " << rlen << endl;

        dbg.close();

        return rlen;
    }

> Let's look at IdIOHandler.ReadBytes() again:
<snip>
> Line 8 is executed, and does nothing, because the last call to
> ReadFromSource() did not set FClosedGracefully.
>
> Line 6 still evaluates to true, so line 7 is executed again.  This time, 
> it
> reads 0 bytes from the socket, and sets FClosedGracefully to True.

FINALLY!!!  You just confirmed what I have been telling you all along.  You 
told ReadBytes() to read too many bytes.  A disconnect was detected before 
all of the requested bytes were received, so an exception was raised.  That 
is what ReadBytes() is supposed to do!!!  You were using ReadBytes() in the 
wrong manner, and then had to write a workaround for your bad code.  If you 
had used ReadBytes() properly in the first place, which I even explained to 
you earlier how to do, you would not have had this problem.

> An exceptionally literal reading of that does suggest that when the socket 
> is
> disconnected, then it was not a "successful completion of the read 
> request".

That is correct.

> In other words, in agreement with what you've been saying, it appears
> to be broken by design.

You code was broken, not ReadByte().  ReadByte() did what it was designed to 
do.

> I stand by my assertion that it is, in fact, a broken design.

Because your mind is still focused on recv()'s behavior.  As I told you 
before, that is NOT how Indy is designed to be used.  Read the documentation 
again.  There is a whole chapter on Indy's design at the very beginning.

> It should not be necessary to know in advance exactly how many bytes to
> read, which is the only way that ReadBytes() can be used by itself to read
> everything from the socket.

That is how Indy is meant to be used in general.  You know what to expect 
ahead of time, so tell it what to get and it gets it from you in a simple, 
blocking operations.  That works fine for 99% of socket programming in 
general.  For the other 1%, which is clearly what you are in, there are 
other mechanisms available to allow you to read arbitrary byte counts more 
dynamically, to which I have already shown you how to do that.  ReadBytes() 
cannot do that by itself, nor is it meant to.


Gambit