🚀 go-pugleaf

"Nicolae Fieraru" <nospam@please.com> wrote in message
news:40f53800@newsgroups.borland.com...

> I try to use Indy IdFTP to upload the content of
> a folder and it always fails after transfering a few
> files (read timeout).

Please show your actual code.


Gambit

Hi Remy,

The code I use is this below. In order to understand what is going on, I
added messages in a Memo for almose every event of the FTP component. I
notice that when it times out, the last message in the Memo is "Status".
You told me that the FTP client is waiting for some acknowledge message from
the server, which never arrives. That might be ok, but I can transfer the
same group of files with CuteFTP. I wonder how did they solve this problem?
And from my tests, the ftp transfer always stops at the same file. I tried
adding an extra file to be uploaded before that one, and it still stops
there. That isn't the only file which causes me problems, in previous tests
it failed with a couple of them.

Regards,
Nicolae

void __fastcall TForm1::btnConnectClick(TObject *Sender)
{
//        IdFTP1->Port = edPort->Text;  //I directly assigned them in the
IdFTP Component
//        IdFTP1->Host = edHost->Text;
//        IdFTP1->Username = edUsername->Text;
//        IdFTP1->Password = edPassword->Text;
        IdFTP->SendBufferSize = 16384; //I decreased this just to test if it
works any better, but it still times out
        IdFTP->Passive = true;
        try
         {
            IdFTP1->Connect( true, 5000 );
         }
        catch(Exception &E)
         {
            ShowMessage( E.Message ); //this show the message "Not
Connected" when Abort() or Quit()
         }
}
//--------------------------------------------------------------------------
-

void __fastcall TForm1::IdFTP1AfterClientLogin(TObject *Sender)
{
  TSearchRec sr;
  int iAttributes = 0;
  iAttributes = faAnyFile;  //here I should remove the faDirectory
  if (FindFirst(edFolder->Text + "\\*", iAttributes, sr) == 0)
  {
    do
    {
      if ((sr.Attr & iAttributes) == sr.Attr)
      {
       if(sr.Name != "." && sr.Name != "..")
        {
          Memo1->Lines->Add("Uploading " + sr.Name);
          IdFTP1->Put(edFolder->Text + "\\" + sr.Name, sr.Name, false );
          Application->ProcessMessages();
          Memo1->Lines->Add("Finished uploading " + sr.Name);
        }
      }
    } while (FindNext(sr) == 0);
    FindClose(sr);
  }
}
//--------------------------------------------------------------------------
-

void __fastcall TForm1::IdFTP1Status(TObject *ASender,
      const TIdStatus AStatus, const AnsiString AStatusText)
{
  Memo1->Lines->Add("Status");
}
//--------------------------------------------------------------------------
-

"Nicolae Fieraru" <nospam@please.com> wrote in message
news:40f61327@newsgroups.borland.com...

> void __fastcall TForm1::IdFTP1AfterClientLogin(TObject *Sender)

Don't put your upload code in the OnAfterLogin event like that.  Move it
into btnConnectClick() directly after Connect() returns.  The reason being
that TIdFTP triggers OnAfterLogin inside of Connect and does other
operations after OnAfterLogin returns.


Gambit

Shouldn't the line
 Memo1->Lines->Add("Status");
be changed to
 Memo1->Lines->Add(AStatusText);
?
-- 
Mark Jacobs
DK Computing
http://www.dkcomputing.co.uk
markj atty critical dotty co dotty uk

Nicolae Fieraru wrote:
> I
> notice that when it times out, the last message in the Memo is "Status".

As Mark already pointed out: you should Add() AStatusText.

What does it say ?

And what kind of info can be extracted from const TIdStatus AStatus ?

Hans.

Hi Mark,

That is only my own comment. I just specified that the program got in the
point where the On Status even has been reached. I checked now, and you are
right. It is more useful to write down the variable you mentioned. Thank you
for that

Regards,
Nicolae




"Mark Jacobs" <markj atty critical dotty co dotty uk> wrote in message
news:40f644bf@newsgroups.borland.com...
> Shouldn't the line
>  Memo1->Lines->Add("Status");
> be changed to
>  Memo1->Lines->Add(AStatusText);
> ?
> -- 
> Mark Jacobs
> DK Computing
> http://www.dkcomputing.co.uk
> markj atty critical dotty co dotty uk
>
>

Dear Hans,

Thank you for indicating me what to look for.
I did what you and Mark told me and now I have a clearer idea about the
point where the program fails.
The file is transferred completely, but for some reason, the message about
finishing transferring the file is never received and the IdFTP->Put() never
returns to the function which called it.

Here is a copy of the messages I display in the memo:

---------------------
Uploading 04.JPG
AStatus: 6
Status: Starting FTP transfer
Work Begin:1
On Work:16384
On Work:32768
On Work:49152
On Work:55767
Word ended
AStatus: 7
Status: Transfer complete
Finished uploading 04.JPG  <---------------------------- This line is
printed just after IdFTP->Put()
---------------------
Uploading 05.JPG
AStatus: 6
Status: Starting FTP transfer
Work Begin:1
On Work:16384
On Work:32768
On Work:49152
On Work:63140            <---- this is the real size of my jpg file,
transfer was OK
Word ended
AStatus: 7
Status: Transfer complete  <----- Although the transfer is completed, the
IdFTP->Put() doesn't return
                                        // anymore in the function which
called it. What it is waiting for, because the WorkEnd
//event has been triggered?

Regards,
Nicolae




"Hans Galema" <dontusethis@dontusethis.nl> wrote in message
news:40f684fe$1@newsgroups.borland.com...
> Nicolae Fieraru wrote:
> > I
> > notice that when it times out, the last message in the Memo is "Status".
>
> As Mark already pointed out: you should Add() AStatusText.
>
> What does it say ?
>
> And what kind of info can be extracted from const TIdStatus AStatus ?
>
> Hans.

It may be that your block upload size is too big. I am not well-versed in the
FTP protocol, but in TCP/IP you are best advised to keep blocks at around 2048
bytes tops. More preferable is a block size of 512 bytes. I think it takes
longer (and is more dodgy) to get an acknowledgement of the transfer of a
block as big as 16K, or the ACK is being lost at the end because the last
block smothered it. This is only a guess, but since it is so easy to change
the block size in Indy, can you let us know what happens with 512 bytes
instead of your 16K? TIA,
-- 
Mark Jacobs
DK Computing
http://www.dkcomputing.co.uk
markj atty critical dotty co dotty uk

Dear Mark,

Thank you very much for your advice. It seems it solved my problem. I tried
with a block of 2048 bytes and it passed easily the point where my program
was timing out. Almost every time before it was blocking at the end of the
same file.
I was thinking if that block isn't too big (32 KB) and I tried with 16KB.
And just before your message I decided to reduce it to 8KB, but it still
timed out. Eventually maybe I would have decreased it even lower, but I
didn't think this will solve the problem...

And now the program just finished transferring all the files for the very
first time :-)

Best wishes to everybody,
Nicolae




"Mark Jacobs" <markj atty critical dotty co dotty uk> wrote in message
news:40f6a51a@newsgroups.borland.com...
> It may be that your block upload size is too big. I am not well-versed in
the
> FTP protocol, but in TCP/IP you are best advised to keep blocks at around
2048
> bytes tops. More preferable is a block size of 512 bytes. I think it takes
> longer (and is more dodgy) to get an acknowledgement of the transfer of a
> block as big as 16K, or the ACK is being lost at the end because the last
> block smothered it. This is only a guess, but since it is so easy to
change
> the block size in Indy, can you let us know what happens with 512 bytes
> instead of your 16K? TIA,
> -- 
> Mark Jacobs
> DK Computing
> http://www.dkcomputing.co.uk
> markj atty critical dotty co dotty uk
>
>

"Nicolae Fieraru" <nospam@please.com> wrote in message
news:40f681db$1@newsgroups.borland.com...

> Is there a way I can debug the upload, to have a
> look what is transmitted between my computer
> and the server?

Yes.  Attach an Intercept to the connection, such as TIdLogFile or
TIdLogEvent.  Although, that will only intercept the traffic on the command
connection and not the data connection (keep in mind that FTP uses 2 socket
connections).

BTW, why does your ADSL modem require you to use Passive mode only?  Does
your ISP block incoming traffice or something?


Gambit

"Nicolae Fieraru" <nospam@please.com> wrote in message
news:40f6a339@newsgroups.borland.com...

> AStatus: 7
> Status: Transfer complete  <----- Although the transfer is completed,
> the IdFTP->Put() doesn't return anymore in the function which
> called it. What it is waiting for, because the WorkEnd
> event has been triggered?

It is waiting for the server to acknowledge the end of the transfer.  The
statuses you see are TIdFTP's own local statuses.


Gambit

"Mark Jacobs" <markj atty critical dotty co dotty uk> wrote in message
news:40f6a51a@newsgroups.borland.com...

> It may be that your block upload size is too big. I am not
> well-versed in the FTP protocol, but in TCP/IP you are
> best advised to keep blocks at around 2048 bytes tops.

That has nothing to do with it.  Block size does not matter to Indy at all,
since every byte received is buffered in memory regardless of the block size
used.  Indy reads however many bytes are actually available in the socket.
If Put() is waiting for a response from the server that never comes, then
the server is not sending the response to begin with, or it is somehow
getting lost during transmission altogether.


Gambit

Hi Remy,

The reason I didn't try before decreasing the size of the blocks. I thought
that by sending tinier blocks will make the program to wait for more
responses from the server, and it could be even worse.
But after Mark adviced me to decrease the size to 2048 it worked perfectly.
Yesterday I didn't have time to do extensive testing, but I managed to
upload all the images from the folder for the first time.
Even if you say that the size of the blocks has nothing to do with it, I am
not so sure about that. I think IdFTP uploads the file to the server in
blocks of a preset size, it doesn't have to do with how the bytes are stored
in the memory. (I apologise for not agreeing with you on this matter,
considering I know nothing about how IdFTP works and you know so much about
everything :-) )

I will continue doing tests tonight (with different block sizes) and I will
let you know what is going on. But I think the size of the blocks counts.

Regards,
Nicolae



"Remy Lebeau (TeamB)" <gambit47.no.spam@no.spam.yahoo.com> wrote in message
news:40f6cbe5$1@newsgroups.borland.com...
>
> "Mark Jacobs" <markj atty critical dotty co dotty uk> wrote in message
> news:40f6a51a@newsgroups.borland.com...
>
> > It may be that your block upload size is too big. I am not
> > well-versed in the FTP protocol, but in TCP/IP you are
> > best advised to keep blocks at around 2048 bytes tops.
>
> That has nothing to do with it.  Block size does not matter to Indy at
all,
> since every byte received is buffered in memory regardless of the block
size
> used.  Indy reads however many bytes are actually available in the socket.
> If Put() is waiting for a response from the server that never comes, then
> the server is not sending the response to begin with, or it is somehow
> getting lost during transmission altogether.
>
>
> Gambit
>
>

Hi Remy,

You described what happens when Indy receives a file, but isn't the problem
different when sending a file?

"Remy Lebeau (TeamB)" <gambit47.no.spam@no.spam.yahoo.com> wrote in message
news:40f6cbe5$1@newsgroups.borland.com...
>
> "Mark Jacobs" <markj atty critical dotty co dotty uk> wrote in message
> news:40f6a51a@newsgroups.borland.com...
>
> > It may be that your block upload size is too big. I am not
> > well-versed in the FTP protocol, but in TCP/IP you are
> > best advised to keep blocks at around 2048 bytes tops.
>
> That has nothing to do with it.  Block size does not matter to Indy at
all,
> since every byte received is buffered in memory regardless of the block
size
> used.  Indy reads however many bytes are actually available in the socket.
> If Put() is waiting for a response from the server that never comes, then
> the server is not sending the response to begin with, or it is somehow
> getting lost during transmission altogether.
>
>
> Gambit
>
>

"Nicolae Fieraru" <nospam@please.cxm> wrote in message
news:40f7172f$1@newsgroups.borland.com...

> I think IdFTP uploads the file to the server
> in blocks of a preset size

The size of the buffer used is specified by the connection's SendBufferSize
property.

> it doesn't have to do with how the bytes are stored in the memory.

That is not what I was implying earlier.


Gambit

"Nicolae Fieraru" <nospam@please.cxm> wrote in message
news:40f7177f$1@newsgroups.borland.com...

> You described what happens when Indy receives a file,
> but isn't the problem different when sending a file?

The buffer sizes should never matter.  TCP guarantees data delivery (or
delivery errors) regardless of the data size, whether you are sending 10
bytes or 1,000,000 bytes.  The socket stack returns how many bytes were
actually accepted everytime send() is called, and then that many bytes are
guaranteed to be delivered.  It is the caller's responsibility to adjust its
buffer sending accordingly.  Indy does exactly that, regardless of the
buffer sizes used (chances are the socket can't except full buffers at a
time anyway).


Gambit

So are you saying that no matter what SneBufferSize I choose in IdFTP, the
program will run in identical manner, therefore it is pointless to adjust
the predefined value of 32768?

"Remy Lebeau (TeamB)" <gambit47.no.spam@no.spam.yahoo.com> wrote in message
news:40f7220a$1@newsgroups.borland.com...
>
> "Nicolae Fieraru" <nospam@please.cxm> wrote in message
> news:40f7177f$1@newsgroups.borland.com...
>
> > You described what happens when Indy receives a file,
> > but isn't the problem different when sending a file?
>
> The buffer sizes should never matter.  TCP guarantees data delivery (or
> delivery errors) regardless of the data size, whether you are sending 10
> bytes or 1,000,000 bytes.  The socket stack returns how many bytes were
> actually accepted everytime send() is called, and then that many bytes are
> guaranteed to be delivered.  It is the caller's responsibility to adjust
its
> buffer sending accordingly.  Indy does exactly that, regardless of the
> buffer sizes used (chances are the socket can't except full buffers at a
> time anyway).
>
>
> Gambit
>
>

"Nicolae Fieraru" <nospam@please.cxm> wrote in message
news:40f7295f$1@newsgroups.borland.com...

> So are you saying that no matter what SneBufferSize I
> choose in IdFTP, the program will run in identical manner,
> therefore it is pointless to adjust the predefined value of 32768?

The SendBufferSize property only specifies the size of the buffer that Indy
used to read from the stream.  The entire buffer, whatever it is, is then
passed to WriteBuffer(), which loops through the buffer sending the data to
the socket as much as it will accept at a time.


Gambit

I understand. Thank you for support

Regards,
Nicolae

"Remy Lebeau (TeamB)" <gambit47.no.spam@no.spam.yahoo.com> wrote in message
news:40f6cb6e$1@newsgroups.borland.com...
>
> "Nicolae Fieraru" <nospam@please.com> wrote in message
> news:40f6a339@newsgroups.borland.com...
>
> > AStatus: 7
> > Status: Transfer complete  <----- Although the transfer is completed,
> > the IdFTP->Put() doesn't return anymore in the function which
> > called it. What it is waiting for, because the WorkEnd
> > event has been triggered?
>
> It is waiting for the server to acknowledge the end of the transfer.  The
> statuses you see are TIdFTP's own local statuses.
>
>
> Gambit

May I just clarify why I gave that advice. I have been streaming sound over
TCP/IP recently, and using Client and Server Sockets (the ones that come with
BCB5 on the Internet tab). I found that the larger I made the packets I was
sending through ServerSocket1, the longer the delays and the worse the
reliability of delivery. I agree with Remy that this should *NOT* be
happening, and that TCP/IP is supposed to guarantee delivery or an error,
whatever the packet size. But there is a substrate that underlies the TCP/IP
protocol, and that is the network layer. Perhaps under Windows OS, this layer
is limited to a certain packet size before reliability is compromised. Who
knows?!? In my experiments, I found that 2048 is fine for transfers across
LANS and 512 is great for Internet transfers. Indy probably does chop the
output buffer up to send it out, but the minimum packet size is wrongly
calculated. Sure TCP/IP will report it can handle 16K chunks, but it won't
reliably in reality. Setting a smaller ftp buffer size in Indy means that the
socket send buffer Indy uses can be only as big as the specified buffer size.
and this, I think, has cured the problem.

Glad it sorted your problem out, Nicolae. I reckon that it shouldn't slow your
transfers down at all - it may even speed them up, because there are less
packets having to be re-sent.
-- 
Mark Jacobs
DK Computing
http://www.dkcomputing.co.uk
markj atty critical dotty co dotty uk


"Remy Lebeau (TeamB)" <gambit47.no.spam@no.spam.yahoo.com> wrote in message
news:40f6cbe5$1@newsgroups.borland.com...
|
| "Mark Jacobs" <markj atty critical dotty co dotty uk> wrote in message
| news:40f6a51a@newsgroups.borland.com...
|
| > It may be that your block upload size is too big. I am not
| > well-versed in the FTP protocol, but in TCP/IP you are
| > best advised to keep blocks at around 2048 bytes tops.
|
| That has nothing to do with it.  Block size does not matter to Indy at all,
| since every byte received is buffered in memory regardless of the block size
| used.  Indy reads however many bytes are actually available in the socket.
| If Put() is waiting for a response from the server that never comes, then
| the server is not sending the response to begin with, or it is somehow
| getting lost during transmission altogether.

Hi Mark,

Thank you for your explanations. Once I decreased the size of the packets I
noticed an improvement in speed and now I can transfer all the files, which
wasn't the case before. Occasionally I still get some read timeout, but
overall your advice helped a lot

Best regards,
Nicolae

Hi Remy,

When I send a file, once the packet arrives at the server, it should send
back an acknowledge message (whatever is the name of that message). Does
that arrive on the command connection or on the socket connection?

My ISP accept both active and passive modes.
I use my router with NAT (router always connected as a gateway) and if I
don't use passive mode, the traffic socket won't establish. In active mode,
I can connect the IdFTP, but I can't put any files there. I don't know
exactly how the router works, but because it servers more computers, it has
to do the NAT for my network, and because of that, passive mode has to be
used (I think)

How can I catch read timeout errors?

I was trying something like

try
 {
   IdFTP->Put(......)
 }
catch(...)
 {
    do something
 }

and I want to catch only read timeout errors. I suppose that when I get such
an error, I have to disconnect and to reconnect the IdFTP, because otherwise
(although IdFTP is connected) it will never manage to send the file.

Regards,
Nicolae








"Remy Lebeau (TeamB)" <gambit47.no.spam@no.spam.yahoo.com> wrote in message
news:40f6cad0$1@newsgroups.borland.com...
>
> "Nicolae Fieraru" <nospam@please.com> wrote in message
> news:40f681db$1@newsgroups.borland.com...
>
> > Is there a way I can debug the upload, to have a
> > look what is transmitted between my computer
> > and the server?
>
> Yes.  Attach an Intercept to the connection, such as TIdLogFile or
> TIdLogEvent.  Although, that will only intercept the traffic on the
command
> connection and not the data connection (keep in mind that FTP uses 2
socket
> connections).
>
> BTW, why does your ADSL modem require you to use Passive mode only?  Does
> your ISP block incoming traffice or something?
>
>
> Gambit
>
>

"Nicolae Fieraru" <nospam@please.cxm> wrote in message
news:41004ff4$1@newsgroups.borland.com...

> once the packet arrives at the server

I assume that by "packet", you are referring to the file as a whole?  The
word "packet" used in this context can be very confusing when used
improperly, since the word "packet" has its own meaning.

> it should send back an acknowledge message

The FTP server does send back an acknowledgement after the file is received
(or an error occurs), yes.

> Does that arrive on the command connection or on the socket connection?

They are both socket connections.  One is used for commands and the other is
used for data.  You should refer to them as "commands connections" and "data
connection", or "commands channel" and "data channel", etc.

As for which connection the acknowledgement comes it, it is the commands
connection.  The data connection is always one-way, in the direction the
data is transferring.

> I use my router with NAT (router always connected as
> a gateway) and if I don't use passive mode, the traffic
> socket won't establish. In active mode, I can connect
> the IdFTP, but I can't put any files there.

Your router needs to be configured to use port forwarding in order to make
active transfers work.  It still won't work, though, because TIdFTP would
have to provide the router's IP as the address the FTP server connects to -
something TIdFTP does not currently support.

> How can I catch read timeout errors?

You already know the answer to that - use a try...catch block.

> catch(...)

"..." does not work very well with VCL exceptions.  You should catch them
directly:

    catch(const Exception &)

> and I want to catch only read timeout errors.

Why only read timeout errors?

> I suppose that when I get such an error, I have to disconnect
> and to reconnect the IdFTP, because otherwise (although IdFTP
> is connected) it will never manage to send the file.

Not necessarily.  A read timeout can occur for any number of reasons, not
all of them harmful to the transfer.

In any case, Indy uses infinite timeouts by default, so there would be no
read timeout error.  That would only occur if you had set the ReadTimeout
property to a non-zero value.  In which case, EIdReadTimeout is the
exception that is thrown.


Gambit