Problem with HTTP download locking up.

#2836

Author: Tom
Date: Tue, 24 Jan 2006 00:18

239 lines
7961 bytes

I'm having a problem with a customers computer locking up as soon as a 
request
to download a file via http is made. To my knowledge this is the only 
customer
with this problem. He's using XP SP2, I'm still waiting for a response to 
make
sure a Proxy server isn't needed, but even then I would think it would 
simply
throw an exception.

I've tried both TNMHttp and TIdHTTP and it seems to happen at the same
point with both controls..

Because TNMHttp controsl are supposed to be buggy what is below is using
the TId component. I'm using Indy 9.0.50.

The first call I make is to a web page and return the result as a string.
Then I download a small file (22 bytes) this seems to work fine. The next
file that gets downloaded is 24MB and as soon as the Get is called the 
program
freezes solid. I do have the AntiFreeze component on the form, and on my 
machine
that doesn't lock up it functions just fine with being able to move the 
window, click
buttons etc without it appearing to be frozen. On his, it's locked up solid, 
can't
move the windows, click the cancel button, nothing.

I have the OnWork event setup to calculate the current progress of the 
transfer
using the Http1->Response->ContentStream->Position and the ContentLength
variables. With a Application->ProcessMessages() thrown in to update the 
display.

Initially I had it setup in a thread for the downloading with synchronize
calls for updating the display, but rather than lockup, the thread was just 
dying without
any errors being thrown.

I'm including the code below for downloading:

<excerpt from function of the code that is not locking up>

   if(!Download("http://www.companyname.com/SmallFile.nfo", InstallDir + 
"\\SmallFile.nfo"))
   {
     Msg = "Unable to download Information File.";
     Write_Debug(Msg);
     UpdateStatus();  // Adds a line to a TMemo and 
Application->ProcessMessages();
     DownloadFailed = true;
     AbortUpdate = true;
     return;
   }

<end excerpt>


<excerpt from function of the code that IS locking up>

Msg.sprintf("Downloading Update number %d.", iCtr);
Write_Debug(Msg);
UpdateStatus(Msg);

 if(!FileExists(FileName5) && !Download("http://www.companyname.com/" + 
FileName, InstallDir + "\\" + FileName))
{
   if(!AbortUpdate)
   {
      Msg.sprintf("Unable to download update number %d.", iCtr);
      Write_Debug(Msg);
      UpdateStatus();
   }
  DownloadFailed = true;
  AbortUpdate = true;
  return;
}

<end excerpt>

<Code from the Download Function>

bool Download(AnsiString const FileName, AnsiString const FullPath)
{
   Retry = 10;
   IsDownloading = true;
   DLResult = false;

   while(IsDownloading && !DLResult && Retry != 0)
   {
     frmUpdate->StartTime = Now();
     try
     {
       TMemoryStream *ms = new TMemoryStream;
       frmUpdate->Http1->Get(FileName, ms);
       if(!AbortUpdate)
       {
         if(FileExists(FullPath))
           DeleteFile(FullPath);
         ms->SaveToFile(FullPath);
         DLResult = true;
       }
       delete ms;
       IsDownloading = false;
     }
     catch(const EIdSocketError &e)
     {
       AnsiString Message = "EIdSocketError: "  + AnsiString(e.LastError) + 
" - " + e.Message;
       Write_Debug(Message);
       --Retry;
     }
     catch(const EIdHTTPProtocolException &e)
     {
       AnsiString Message = "EIdHTTPProtocolException: " + 
AnsiString(e.ReplyErrorCode) + " - " + e.ErrorMessage;
       Write_Debug(Message);
       --Retry;
     }
     catch(const EIdException &e)
     {
       AnsiString Message = "EIdException: " + e.Message;
       Write_Debug(Message);
       --Retry;
     }
     catch(const Exception &e)
     {
       AnsiString Message = "Exception: " + e.ClassName() + " - " + 
e.Message;
       Write_Debug(Message);
       --Retry;
     }
   }

   return DLResult;
}

<Code from OnWork Event>

void __fastcall TfrmUpdate::Http1Work(TObject *Sender,
      TWorkMode AWorkMode, const int AWorkCount)
{
 if(IsDownloading)
 {
  if(AbortUpdate)
  {
    Http1->Disconnect();
    DLResult = false;
    Retry = 0;
    Msg = "Download was aborted by the user.";
    UpdateStatus();
    Write_Debug(Msg);
    return;
  }
  if(Http1->Response->ContentLength >= 1048576)
  {
    lblFileSize->Caption = lblFileSize->Caption.FormatFloat("0.00", 
float(Http1->Response->ContentLength) / 1048576);
    Label5->Caption = "MB";
  }
  else if(Http1->Response->ContentLength >= 1024)
  {
    lblFileSize->Caption = lblFileSize->Caption.FormatFloat("0.00", 
float(Http1->Response->ContentLength) / 1024);
    Label5->Caption = "KB";
  }
  else
  {
    lblFileSize->Caption = Http1->Response->ContentLength;
    Label5->Caption = "Bytes";
  }

  if(Http1->Response->ContentStream->Position >= 1048576)
  {
    lblBytesReceived->Caption = 
lblBytesReceived->Caption.FormatFloat("0.00", 
float(Http1->Response->ContentStream->Position) / 1048576);
    Label6->Caption = "MB";
  }
  else if(Http1->Response->ContentStream->Position >= 1024)
  {
    lblBytesReceived->Caption = 
lblBytesReceived->Caption.FormatFloat("0.00", 
float(Http1->Response->ContentStream->Position) / 1024);
    Label6->Caption = "KB";
  }
  else
  {
    lblBytesReceived->Caption = Http1->Response->ContentStream->Position;
    Label6->Caption = "Bytes";
  }
  TDateTime CurrentTime = Now();
  double NumSecs = SecondSpan(CurrentTime, frmUpdate->StartTime);
  double BytesLeft = Http1->Response->ContentLength - 
Http1->Response->ContentStream->Position;
  float TimeLeft;
  if(Http1->Response->ContentStream->Position != 0)
    TimeLeft = BytesLeft / (Http1->Response->ContentStream->Position / 
NumSecs);
  else
    TimeLeft = BytesLeft / (1 / NumSecs);

  if(TimeLeft >= 3600)
  {
    int Hours = (int)(TimeLeft / 3600);
    int Minutes = (int)((TimeLeft - (Hours * 3600)) / 60);
    int Seconds = (int)((TimeLeft - (Hours * 3600) - (Minutes * 60)));
    lblTimeRemaining->Caption.sprintf("%d hrs %d mins %d secs", Hours, 
Minutes, Seconds);
  }
  else if(TimeLeft >= 60)
  {
    int Minutes = (int)(TimeLeft/ 60);
    int Seconds = (int)((TimeLeft - (Minutes * 60)));
    lblTimeRemaining->Caption.sprintf("%d mins %d secs", Minutes, Seconds);
  }
  else
     frmUpdate->lblTimeRemaining->Caption.sprintf("%d secs", (int)TimeLeft);

  Application->ProcessMessages();
 }
}

Here is the contents of the Log file that Write Debug writes to.

2006-01-23 7:54:27 AM : Downloading Update Infomation File
2006-01-23 7:54:27 AM : Information file downloaded sucessfully.
2006-01-23 7:54:27 AM : Old Information file opened. Old AM Version: 77. Old 
Updater Version: 6. Last Update Number: 5.
2006-01-23 7:54:27 AM : New Information file opened. New AM Version: 80. New 
Updater Version: 7. New Update Number: 8.
2006-01-23 7:54:27 AM : New version of Updater.exe found, downloading it!
2006-01-23 7:54:30 AM : Downloading Update number 6.

If anymore information needs to be provided please let me know.

Thanks,

Tom

Re: Problem with HTTP download locking up.

#2839

Author: Remy Lebeau (TeamB)
Date: Tue, 24 Jan 2006 01:18

58 lines
3498 bytes

"Tom" <thomasf1021@hotmail.com> wrote in message
news:43d571a6$1@newsgroups.borland.com...

> The next file that gets downloaded is 24MB and as soon as the
> Get is called the program freezes solid.

Are you calling Get() in the context of the main thread?  If so, then the
freeze is to be expected.

> I do have the AntiFreeze component on the form, and on my machine
> that doesn't lock up it functions just fine with being able to move the
> window, click buttons etc without it appearing to be frozen.

TIdAntiFreeze is not perfect.  A better solution would be to move the
downloading into a separate worker thread instead.

> On his, it's locked up solid, can't move the windows, click the cancel
button, nothing.

Then TIdHTTP (or something else in your code) has been completely blocked
> to the point where TIdHTTP cannot even call into TIdAntiFreeze at all.
Again, you should move the download into a separate thread.  At least that
way, the program will not be frozen even if such a blockage occurs again.

> I have the OnWork event setup to calculate the current progress
> of the transfer using the Http1->Response->ContentStream->Position
> and the ContentLength variables.

The ContentStream->Position is always at the end of the stream  You should
be using the AWorkCount value that is provided by the OnWork event instead.
That is the total number of bytes that have been transferred since the
beginning of the transfer.

> With a Application->ProcessMessages() thrown in to update
> the display.

TIdAntiFreeze calls ProcessMessages() internally.  Or, if you move the code
into its own thread, then ProcessMessages() is not needed at all.

> Initially I had it setup in a thread for the downloading with
> synchronize calls for updating the display, but rather than lockup,
> the thread was just dying without any errors being thrown.

Then your thread code was likely wrong to begin with.  Please show what you
were doing.  Also keep in mind that TThread swallows uncaught exceptions, so
if you want an exception to be reported, you have to catch the exceptions
yourself so that you can respond to them accordingly.

> bool Download(AnsiString const FileName, AnsiString const FullPath)

That code has a memory leak if either TIdHTTP::Get() or
TMemoryStream::SaveToFile() fail.  You are also doing a lot of processing
inside the OnWork event handler.  You might consider streamlining the code
better to reduce the numbr of placesd where errors can occur.


Gambit

Re: Problem with HTTP download locking up.

#2840

Author: Tom
Date: Tue, 24 Jan 2006 01:19

28 lines
2164 bytes

I got some more information from the user after adding
some more logging in the Connect,Disconnect and WorkBegin
events.

It's calling all of the above before locking up.
Here's the latest text from the log file:

2006-01-23 8:11:24 PM : Downloading Update Infomation File
2006-01-23 8:11:24 PM : Connect to server.
2006-01-23 8:11:24 PM : Beginning Download
2006-01-23 8:11:24 PM : Download Completed
2006-01-23 8:11:24 PM : Disconnected from server.
2006-01-23 8:11:24 PM : Information file downloaded sucessfully.
2006-01-23 8:11:24 PM : Old Information file opened. Old AM Version: 77. Old 
Updater Version: 6. Last Update Number: 5.
2006-01-23 8:11:24 PM : New Information file opened. New AM Version: 80. New 
Updater Version: 7. New Update Number: 8.
2006-01-23 8:11:24 PM : New version of Updater.exe found, downloading it!
2006-01-23 8:11:24 PM : Connect to server.
2006-01-23 8:11:24 PM : Beginning Download
2006-01-23 8:11:39 PM : Download Completed
2006-01-23 8:11:39 PM : Disconnected from server.
2006-01-23 8:11:39 PM : Downloading Update number 6.
2006-01-23 8:11:39 PM : Connect to server.
2006-01-23 8:11:40 PM : Beginning Download


-Tom

Re: Problem with HTTP download locking up.

#2841

Author: Tom
Date: Tue, 24 Jan 2006 02:29

100 lines
5007 bytes

>> Are you calling Get() in the context of the main thread?  If so, then the
>> freeze is to be expected.
It is if I wasn't calling ProcessMessages() in the OnWork event.

I plan on moving all of that work out of the event, it was just a quick
conversion from TNMHttp over to TIdHTTP to see if that could correct
the problem with it freezing.


>> The ContentStream->Position is always at the end of the stream
Not true, it updates as the data is received. I'm using this
method because of another post by you that says AWorkCount
is not accurate if the data is sent in different chunks. From my
testing the ContentStream->Size (which is what you said
to use in that other post) is set to the same size as the ContentLength.
when the transfer is started. The Position updates as that data is received.

>> Then your thread code was likely wrong to begin with.
It was the same code except it was inside a Tthread. The UpdateStatus
function was synchronized for access to the main forms visual controls.
DLResult, IsDownloading and Retry all were public variables of the thread.

>> That code has a memory leak if either TIdHTTP::Get() or
>> TMemoryStream::SaveToFile() fail.
Would an auto_ptr take care of that? I'm still looking for a good place
to learn about them.

Thanks,

Tom




"Remy Lebeau (TeamB)" <no.spam@no.spam.com> wrote in message 
news:43d57fba$1@newsgroups.borland.com...
>
> "Tom" <thomasf1021@hotmail.com> wrote in message
> news:43d571a6$1@newsgroups.borland.com...
>
>> The next file that gets downloaded is 24MB and as soon as the
>> Get is called the program freezes solid.
>
> Are you calling Get() in the context of the main thread?  If so, then the
> freeze is to be expected.
>
>> I do have the AntiFreeze component on the form, and on my machine
>> that doesn't lock up it functions just fine with being able to move the
>> window, click buttons etc without it appearing to be frozen.
>
> TIdAntiFreeze is not perfect.  A better solution would be to move the
> downloading into a separate worker thread instead.
>
>> On his, it's locked up solid, can't move the windows, click the cancel
> button, nothing.
>
> Then TIdHTTP (or something else in your code) has been completely blocked
>> to the point where TIdHTTP cannot even call into TIdAntiFreeze at all.
> Again, you should move the download into a separate thread.  At least that
> way, the program will not be frozen even if such a blockage occurs again.
>
>> I have the OnWork event setup to calculate the current progress
>> of the transfer using the Http1->Response->ContentStream->Position
>> and the ContentLength variables.
>
> The ContentStream->Position is always at the end of the stream  You should
> be using the AWorkCount value that is provided by the OnWork event 
> instead.
> That is the total number of bytes that have been transferred since the
> beginning of the transfer.
>
>> With a Application->ProcessMessages() thrown in to update
>> the display.
>
> TIdAntiFreeze calls ProcessMessages() internally.  Or, if you move the 
> code
> into its own thread, then ProcessMessages() is not needed at all.
>
>> Initially I had it setup in a thread for the downloading with
>> synchronize calls for updating the display, but rather than lockup,
>> the thread was just dying without any errors being thrown.
>
> Then your thread code was likely wrong to begin with.  Please show what 
> you
> were doing.  Also keep in mind that TThread swallows uncaught exceptions, 
> so
> if you want an exception to be reported, you have to catch the exceptions
> yourself so that you can respond to them accordingly.
>
>> bool Download(AnsiString const FileName, AnsiString const FullPath)
>
> That code has a memory leak if either TIdHTTP::Get() or
> TMemoryStream::SaveToFile() fail.  You are also doing a lot of processing
> inside the OnWork event handler.  You might consider streamlining the code
> better to reduce the numbr of placesd where errors can occur.
>
>
> Gambit
>
>

Re: Problem with HTTP download locking up.

#2842

Author: Tom
Date: Tue, 24 Jan 2006 02:33

3 lines
1002 bytes

Sorry, I meant to cut out the extra text at the bottom.

-Tom

Re: Problem with HTTP download locking up.

#2844

Author: Tom
Date: Tue, 24 Jan 2006 05:14

6 lines
1148 bytes

To be more specific I guess. What would cause
it to lock up like that for 1 person and not the
rest including me?


-Tom

Re: Problem with HTTP download locking up.

#2861

Author: Remy Lebeau (TeamB)
Date: Thu, 26 Jan 2006 07:58

15 lines
1656 bytes

"Tom" <thomasf1021@hotmail.com> wrote in message
news:43d57fdb@newsgroups.borland.com...

> Here's the latest text from the log file:

Logs are useless without seeing the actual code that is producing them.
Even more useful would be a log of the raw socket activity, not the
application's higher-level behaviors.  If you attach one of the TIdLog...
components to TIdHTTP, or alternatively use an external packet sniffer such
as Ethereal (http://www.ethereal.com), then youcan view the actual HTTP
commands and responses that are being transmitted over the socket.


Gambit

Re: Problem with HTTP download locking up.

#2866

Author: Tom
Date: Thu, 26 Jan 2006 13:19

26 lines
1910 bytes

I'll try the TIdLog and see if I can get the customer
to give it another go.

-Tom

"Remy Lebeau (TeamB)" <no.spam@no.spam.com> wrote in message 
news:43d880b2$3@newsgroups.borland.com...
>
> "Tom" <thomasf1021@hotmail.com> wrote in message
> news:43d57fdb@newsgroups.borland.com...
>
>> Here's the latest text from the log file:
>
> Logs are useless without seeing the actual code that is producing them.
> Even more useful would be a log of the raw socket activity, not the
> application's higher-level behaviors.  If you attach one of the TIdLog...
> components to TIdHTTP, or alternatively use an external packet sniffer 
> such
> as Ethereal (http://www.ethereal.com), then youcan view the actual HTTP
> commands and responses that are being transmitted over the socket.
>
>
> Gambit
>
>
>

🚀 go-pugleaf

Thread View: borland.public.cppbuilder.internet.socket

Problem with HTTP download locking up.

Re: Problem with HTTP download locking up.

Re: Problem with HTTP download locking up.

Re: Problem with HTTP download locking up.

Re: Problem with HTTP download locking up.

Re: Problem with HTTP download locking up.

Re: Problem with HTTP download locking up.

Re: Problem with HTTP download locking up.

Thread Navigation