🚀 go-pugleaf

Muhammad Momin Rashid,

> 1. This approach is significantly slower than the windows file copy
> operation itself.  I need to speed up my file coping operation.

Is this for a compressed file? The Windows shell copies those without 
decompressing them first.

> 2. The operation consumes a lot of processing power while reading from media
> such as CDROM etc.  Is there any way to bring down the processor usage?

I wouldn't have thought so but what size buffer are you using? You should use 
at least 16kB but less than 64kB. Also check if the CDROM reader is using PIO 
or DMA. Windows tends to use PIO if it's at all unsure about the CDROM 
reader's ability.

> 3. If there is a problem in file reading (bad or corrupted media) the system
> locks up for a fair amount of time.  How to avoid this? and how can I

Check your buffer. If you are reading too much in one go it might have this 
effect.
--
Andrue Cope
[Bicester, UK]

If your way is significantly slower than the CopyFile function and speed is
an issue then why not change the code to use the CopyFile function.

.  Ed

> Muhammad Momin Rashid wrote in message
> news:3ec35dd4@newsgroups.borland.com...
> ...

I am not using CopyFile function because I need to have control over the
data being copied, in some cases I need to clean the corrupted data (which
occurs due to faulty media).  This the reason I am reading file in chunks
and writing these chunks to a new file.

Given the above scenario, can you think of an alternate i can use, which is
both speedier and gives me this sort of control?

Regards,
Muhammad Momin Rashid

"Ed Mulroy [TeamB]" <ed@mulroyREMOVETHESECHARS.org> wrote in message
news:3ec3944e$1@newsgroups.borland.com...
> If your way is significantly slower than the CopyFile function and speed
is
> an issue then why not change the code to use the CopyFile function.
>
> .  Ed
>
> > Muhammad Momin Rashid wrote in message
> > news:3ec35dd4@newsgroups.borland.com...
> > ...
>
>
>

"Andrue Cope" <not.a.valid@email.address.sorry> wrote in message
news:VA.00001cc9.007db8e9@email.address.sorry...
> Muhammad Momin Rashid,
>
> > 1. This approach is significantly slower than the windows file copy
> > operation itself.  I need to speed up my file coping operation.
>
> Is this for a compressed file? The Windows shell copies those without
> decompressing them first.

No, this is a normal file copy operation, which attempts to read chunks of
data from a file and write it to another file without modification unless
there is a media error.

> > 2. The operation consumes a lot of processing power while reading from
media
> > such as CDROM etc.  Is there any way to bring down the processor usage?
>
> I wouldn't have thought so but what size buffer are you using? You should
use
> at least 16kB but less than 64kB. Also check if the CDROM reader is using
PIO
> or DMA. Windows tends to use PIO if it's at all unsure about the CDROM
> reader's ability.

I am using a 512K buffer at the moment, let me try reducing the buffer size
and see if it results in any significat speed improvement.  My devices are
set to use DMA, so theoratically such copying operation should not use CPU
Cycles at all.

> > 3. If there is a problem in file reading (bad or corrupted media) the
system
> > locks up for a fair amount of time.  How to avoid this? and how can I
>
> Check your buffer. If you are reading too much in one go it might have
this
> effect.

Are you sure about this?  I thought when you try to read a part of a file
and error occurs, the read function just quits and returns the number of
bytes read.  Whereas, your response gives an impression that even if an
error occurs the read function continues to read (byte by byte or in some
preset chunks?) until it reaches the desired number of bytes to read.  Maybe
I need a bit more elaboration on this issue.

Thanks for your time and help
Regards,
Muhammad Momin Rashid.

> Andrue Cope
> [Bicester, UK]
>

If you are reading, examining and possibly altering the file's data prior to
writing a new file then what you are doing is not a copy and a comparison to
the speed to that of Windows' CopyFile operation is not valid.

If I had that task and were concerned about speed, I would apportion most of
my efforts to two items:

-I would use the native API for the file operations
-I would try to optimize my activities with respect
 to data examination and alteration.

I would probably use CreateFile, ReadFile, WriteFile and CloseHandle and
specificly open with FILE_FLAG_NO_BUFFERING and allocate aligned data
buffers to facilitate that mode.

I would probably use one of several different functions for examining and
altering a given file and its data depending upon which variation of
alteration is desired for a given file and what file size is to be handled.

If many files were to be handled, it may be profitable to investigate
apportioning the files among various threads and investigate the utility of
requesting a higher priority from Windows.

Note that without careful programming, no changes, including none of what I
have said, are likely to provide significant benefit.  You must try code,
evaluate its performance and compare to alternate versions of the code.
Typically you will find that the bulk of any differences in speed reside in
20% or less (probably much less) of the code but without evaluation and
performance comparisons you will end up spending time to optimize portions
where the effects of the optimization are trivial.  For instance, it could
turn out that performance using TFileStream is within 2% of that using
other, lower level methods, because the bulk of the time is spent elsewhere.

.  Ed

> Muhammad Momin Rashid wrote in message
> news:3ec3be2e@newsgroups.borland.com...
>
> I am not using CopyFile function because I need to have
> control over the data being copied, in some cases I need
> to clean the corrupted data (which occurs due to faulty
> media).  This the reason I am reading file in chunks
> and writing these chunks to a new file.
>
> Given the above scenario, can you think of an alternate i can
> use, which is both speedier and gives me this sort of control?

"Muhammad Momin Rashid" <momin@hotmail.com> wrote in message
news:3ec3bff7@newsgroups.borland.com...

> No, this is a normal file copy operation

That wasn't the question.  Which OS are you using?  When using NTFS under
NT/2K/XP, files can be compressed inside the file system itself.
Application code never has to deal with that directly, as the OS handles the
compression/decompression internally.  However, if you are actually reading
the file data, and the file is compressed, then of course the OS will have
to decompress the data before giving it to you.  That might be a costly
operation depending on what you're actually doing.

> I thought when you try to read a part of a file and error
> occurs, the read function just quits and returns the number
> of bytes read.

Only if you are using overlapped I/O, which the VCL does not.  Otherwise,
ReadFile() waits for all of the requested bytes to be read in full, or else
fails the entire operation.


Gambit

On Thu, 15 May 2003 14:28:50 +0500, "Muhammad Momin Rashid"
<momin@hotmail.com> wrote:

>1. This approach is significantly slower than the windows file copy
>operation itself.  I need to speed up my file coping operation.

This operation should not be "significantly" slower, however, you must
certainly abandon TFileStream and use Win API (CreateFile, ReadFile,
WriteFile). You gain a little bit of power by bypassing the VCL layer,
but most importantly you are able to use the appropriate flags in
CreateFile such as - for example - the FILE_FLAG_SEQUENTIAL_SCAN flag
in dwFlagsAndAttributes.

The buffer size used in copying is critical. I use a size of 32K as
the best for all purposes.

>2. The operation consumes a lot of processing power while reading from media
>such as CDROM etc.  Is there any way to bring down the processor usage?

The only way is 1) to use a thread, and 2) to take a Sleep() for 0 or
1 ms, after each copyed chunk. This will give you a usage of about 4%
even on old hardware.

>3. If there is a problem in file reading (bad or corrupted media) the system
>locks up for a fair amount of time.  How to avoid this? and how can I
>control the amount of time system has to keep trying before it quits trying
>to read?

In most cases Windows recover. The behaviour varies between versions.
You can't do much about this except for using SetErrorMode with
SEM_FAILCRITICALERRORS especially in case of dealing with removable
media. In any case, the copy is handled by a thread which can send a
periodic signal to your main (UI) thread.


-- 

Nick B. Cassos

"Ed Mulroy [TeamB]" <ed@mulroyREMOVETHESECHARS.org> wrote in message
news:3ec3c37a@newsgroups.borland.com...

> I would probably use CreateFile, ReadFile,
> WriteFile and CloseHandle and specificly open
> with FILE_FLAG_NO_BUFFERING and
> allocate aligned data buffers to facilitate that mode.

If you're conserned about speed, then memory mapping the soource file will
be much faster then using ReadFile() for it.  By mapping the file into
memory (CreateFileMapping() and MapViewOfFile()), you can access the file
data directly as a simple BYTE* array, hardly any disk I/O is used at all
(except for maybe occasional page swapping if the file is very large).  Disk
I/O is always a bottleneck when performing data processing, so the more
processing you can do in memory directly, the better the performance will
be.


Gambit

I have not found memory mapping to be faster or even as fast as unbuffered
reading.  However I've not compared them since Win95 (maybe 98).  Perhaps
it's faster under WIn2K or XP.

.  Ed

> Remy Lebeau wrote in message
> news:3ec41b29$1@newsgroups.borland.com...
>
> If you're conserned about speed, then memory
> mapping the soource file will be much faster
> then using ReadFile() for it.

Muhammad Momin Rashid,

> No, this is a normal file copy operation, which attempts to read chunks of
> data from a file and write it to another file without modification unless
> there is a media error.
>

What I meant was is the file compressed by the FS (as in NTFS compression). 
When Explorer copies such files it copies them without decompressing them 
and this vastly reduces I/O calls.
--
Andrue Cope
[Bicester, UK]

"Remy Lebeau \(TeamB\)" <gambit47@yahoo.com> wrote:
>If you're conserned about speed, then memory mapping the soource file will
>be much faster then using ReadFile() for it.  By mapping the file into
>memory (CreateFileMapping() and MapViewOfFile()), you can access the file
>data directly as a simple BYTE* array, hardly any disk I/O is used at all
>(except for maybe occasional page swapping if the file is very large).  Disk
>I/O is always a bottleneck when performing data processing, so the more
>processing you can do in memory directly, the better the performance will
>be.
>

I think that file mapping is not much faster then Win32 API 
in this case. Most data on CDROM and even those that is 
located on HDD CDROM could be changes - it is source file.
So PAGE_WRITECOPY should be used like follow:

hFile = CreateFile("D:\\Software\\Drivers\\TWSWTCP\\TWSWTCP.exe", GENERIC_READ , FILE_SHARE_READ, NULL, OPEN_EXISTING,
        FILE_ATTRIBUTE_NORMAL, 0);

if(hFile == INVALID_HANDLE_VALUE)
        {
         DWORD ddd=GetLastError();
         return;
        }

hFileMapping = CreateFileMapping(hFile, NULL, PAGE_WRITECOPY, 0, 0, NULL);
if(hFileMapping == 0)
        {
         CloseHandle(hFile);

         return;
        }

lpFileBase = MapViewOfFile(hFileMapping, FILE_MAP_COPY, 0, 0, 0);
if(lpFileBase == 0)
        {
         CloseHandle(hFileMapping);
         CloseHandle(hFile);

         return;
        }

Microsoft said that for PAGE_WRITECOPY are both GENERIC_READ 
and GENERIC_WRITE are needed but it is incorrect for CDROM 
(only GENERIC_READ). When PAGE_WRITECOPY will be used 
then the pages will automatically swappable after changes for 
big files (files that can't be loading as a whole). 
If file can be loadind entirely it means that it can be 
loading entirely via ReadFile API.

Vladimir.