🚀 go-pugleaf

On or about Wed, 5 Mar 2008 21:29:14 -0800 did "Remy Lebeau \(TeamB\)"
<no.spam@no.spam.com> dribble thusly:

>
>"Mike Ruskai" <BUTthannydI@DONTearthlinkLIKE.netSPAM> wrote in message 
>news:n9rus3198r3a36l12l301alsg8o2s7q11i@4ax.com...
>
>> The entire amount of data was sent over the socket
>
>That does not necessarily mean that the client will be able to read all of 
>it, though.  It is up to the OS whether the disconnect is reported before or 
>after all of the bytes have actually been read.

The disconnect doesn't happen until the receiving end sends an ACK packet for
the last data packet.  That means all of the data is on the client's stack,
able to be read.

>The only way for ReadBytes() to know that there is no more data to read from 
>the socket is if the socket reports a disconnect inside of ReadFromSource(). 
>ReadBytes() calls ReadFromSource() only when the size of the InputBuffer is 
>not sufficient.  Even if ReadFromSource() fills the InputBuffer to the right 
>size, the ClosedGracefully property WOULD NOT be set to True, 
>CheckForDisconnect() WOULD exit without an error, the loop WOULD stop, and 
>ReadBytes() would call ExtractBytes() and exit normally.
>
>So what is the actual problem???  I still do not see any.  This logic has 
>been running in Indy for YEARS without any problem.

The actual problem is that ReadBytes() is *NOT* calling ExtractToBytes() after
the socket has been disconnected by the remote, following a completed
transfer.

It is, instead, throwing an exception, making it necessary to extract the
bytes directly from the input buffer.

>> So it throws the exception, which leaves no recourse but to read from
>> the buffer directly for the last bit of data that was sent.
>
>It does not raise an exception while data is still being received into the 
>InputBuffer.  You have not proved otherwise yet.  The exception cannot be 
>raised until the socket stops providing data, which can only happen when you 
>are asking ReadBytes() for more bytes than are actually available.

Asking for more bytes than are available is not a problem, according to the
dictates of common sense, and the documentation for the function.  It will
either block until the requested number are available, or return less if the
socket is closed.

But it's returning ZERO when the socket is closed, not the total actually
available.

>> But I said that from the start - ReadBytes() wants to block until it
>> gets the total requested bytes
>
>That is how it is designed and meant to work.
>
>> when a proper function of the sort would read UP TO that number
>> of bytes, returning anywhere from 0 to the requested number.
>
>That is NOT what ReadBytes() is meant for, that is NOT how it is designed, 
>and that is NOT how Indy is designed to be used in general.  Indy is a 
>blocking library.  It waits for the requested number of bytes.  That is a 
>core fundamental design shift from standard socket API programming 
>practices.  That is how it is designed from the ground up to work.  It 
>intentionally handles all of the "I-asked-for-X-but-only-got-Y" logic 
>internally so the developer doesn't have to anymore.  Indy uses a different 
>paradigm to socket programming than most socket developers are used to.  If 
>this is not the kind of functionality you want, then Indy is not the right 
>choice for you.  Go use an asynchronous socket library instead.  That is the 
>kind of thing they are good at - making you work more to accomplish the same 
>tasks manually.

If I could find a more sensible library that handles SOCKS transparently, I
would.

But that still doesn't change the fact that ReadBytes() does not conform to
its documented behavior, as stated above.

>> Or at the least, it would only block until it receives at least 1 byte.
>
>That is what ReadFromSource() does, until a timeout elapses (if specified).
>
>> As it is, the function is documented as either waiting until that number
>> of bytes is read, or until the socket is disconnected.
>
>That is exactly what it does.

But it's not reading all the data when the socket is disconnected.  I've
demonstrated that clearly.

>> But when the socket is disconnected, it's not retrieving the last bytes
>> from the buffer
>
>Yes, it does, when used properly.

That statement is either false, or you're using the term "properly" in a
bizarro-world way.

>> necessitating the workaround I used.
>
>Then you are clearly still using it wrong.  Such workarounds are NOT needed 
>to use Indy effectively.  Please show a more complete snippet of how exactly 
>you are using it.
>
>> If you can show "proper" usage of ReadBytes() that doesn't require
>> going to the buffer for the last bit of data, I'm all eyes.
>
>Or, you could show what you are trying to do in the first place, and then 
>someone can tell whether you are using it right or wrong.

I wrote a simple function to behave more like recv(), so I could plug it into
a larger routine that loops through received data.

Here it is, with some logging added:

int __fastcall DataConstructor::recvData(TIdTCPClient *idDataSock, char *data,
    int dataLen)
{
    int rlen;
    TIdBytes bdata;

    ofstream dbg("e:\\indyrecv.txt", ios::out | ios::app);

    dbg << "Request: " << dataLen << endl;

    try
        {
        try
            {
            idDataSock->IOHandler->ReadBytes(bdata, dataLen, false);
            dbg << "ReadBytes() received: " << bdata.Length << endl;
            }
        catch (EIdConnClosedGracefully &exc)
            {
            dbg << "Exception with buffer at : " <<
idDataSock->IOHandler->InputBuffer->Size << endl;
            idDataSock->IOHandler->InputBuffer->ExtractToBytes(bdata);
            }

        rlen=bdata.Length;

        if (rlen>0)
            {
            BytesToRaw(bdata, data, rlen);
            }
        }
    catch (Exception &exc)
        {
        rlen=-1;
        }

    dbg << "Total received: " << rlen << endl;

    dbg.close();    

    return rlen;
}
//---------------------------------------------------------------------------

Here are the contents of the log after one data connection session:

Request: 8
ReadBytes() received: 8
Total received: 8
Request: 8400
ReadBytes() received: 8400
Total received: 8400
Request: 7620
ReadBytes() received: 7620
Total received: 7620
Request: 7620
ReadBytes() received: 7620
Total received: 7620
Request: 7620
Exception with buffer at : 253
Total received: 253

The above function contains all the code used with idDataSock, except for one
previous call to Write(), and its pointer being passed around a bit.  I don't
call anything but ReadBytes() until the exception is thrown.

The log output above shows conclusively that ReadBytes() does not return the
last 253 bytes read from the socket.  I have to call ExtractToBytes() directly
to get them.

Let's look at IdIOHandler.ReadBytes() again:

01) procedure TIdIOHandler.ReadBytes(var VBuffer: TIdBytes; AByteCount:
Integer; AAppend: Boolean = True);
02) begin
03)   Assert(FInputBuffer<>nil);
04)   if AByteCount > 0 then begin
05)     // Read from stack until we have enough data
06)     while FInputBuffer.Size < AByteCount do begin
07)       ReadFromSource(False);
08)       CheckForDisconnect(True, True);
09)     end;
10)     FInputBuffer.ExtractToBytes(VBuffer, AByteCount, AAppend);
11)   end else if AByteCount < 0 then begin
12)     ReadFromSource(False, ReadTimeout, False);
13)     CheckForDisconnect(True, True);
14)     FInputBuffer.ExtractToBytes(VBuffer, -1, AAppend);
15)   end;
16) end;

On the last call, line 6 evaluates to true, since the requested number of
bytes is greater than the number of bytes in the buffer.  Let's assume that
number is 0, with 253 bytes remaining in the TCP/IP stack's buffer to be read.

Line 7 is then executed.  It reads 253 bytes from the socket. 

Line 8 is executed, and does nothing, because the last call to
ReadFromSource() did not set FClosedGracefully.

Line 6 still evaluates to true, so line 7 is executed again.  This time, it
reads 0 bytes from the socket, and sets FClosedGracefully to True.

Line 8 is executed again.  Since a) the socket is disconnected, b) it's told
to throw an exception when that happens, and c) it's told to ignore the fact
that there's still data in the input buffer when deciding to throw the
exception, it throws EIdConnClosedGracefully.

Line 10 is not executed, so the last 253 bytes are still in the input buffer,
and ReadBytes() will forever throw an exception when called under the current
state of affairs.

Therefore, going straight to the input buffer is required.

Here's what the latest help file states:

"ReadBytes reads data from the IOHandler data source until the number of bytes
in AByteCount has been accumulated in InputBuffer, or CheckForDisconnect
interrupts the read request.

On successful completion of the read request, values are extracted from
InputBuffer into VBuffer."

An exceptionally literal reading of that does suggest that when the socket is
disconnected, then it was not a "successful completion of the read request".

In other words, in agreement with what you've been saying, it appears to be
broken by design.

I stand by my assertion that it is, in fact, a broken design.  It should not
be necessary to know in advance exactly how many bytes to read, which is the
only way that ReadBytes() can be used by itself to read everything from the
socket.