🚀 go-pugleaf

"Deniel" <DenielMail@Internet.com> wrote:

>that i'll fill with data from a file, and while the program is
>running i continuously have to read data from this buffer, do
>some calculations and return the results to the buffer, and i
>have to do this process again and again and again for a VERY
>long time (several mounths i guess) until getting the final 
>results, if i could i would use even 2 GB of memory, but again,
>i can't tell you for sure at this point, i just know that i 
>want to use ONLY 'real' memory, not hard-drive memory (paging).

No, you do _NOT_ know this. In fact, you are doing your own memory
paging. If your data size is larger than your memory, then either you or
the operating system is going to have to 'page' it in and out. Which is
better at it depends.

>Does CreateFileMapping let me allocate 150 MB buffer in the
>'real' memory, and make sure it will not use the hard-disk?

Entirely the opposite - it allows you to ask the operating system to
pretend the disk is RAM - but at the same time, to use its best caching
algorithms to ensure disk is only read or written when it can't avoid
it.

What matters is the algorithm you will be using. This is vastly more
important than anything else you're asking, because until you know what
you're doing, you can't determine the locality of the operations.

Let's suppose that you've got three big buffers. You read buffer1[0],
add it to buffer2[0], and write the result to buffer3[0]. You then
advance your index, and so on.

This is pretty linear, you're going to have to read or write each value
once, and you're also going to have to read or write each section of the
buffers to/from disk once. It's a reasonable place to do file mapping.

Now let's suppose you have the three buffers, but this time you're
adding the products of buffer1[0] and buffer2[0], buffer1[1] and
buffer2[1], all the way to index 100, and storing the result in
buffer3[0]. The next loop, you do [1..101], and so on.

Here, your values are still close together, but you're reading the
values in buffers 1 and 2 multiple times. Again, because you staying
close to the previous index each time, the chances are that the value is
still not only in RAM, but still in the processor cache (and processor
cache is vastly faster than normal memory).

Again, file mapping will work beautifully here.

Now let's change your algorithm a bit. Instead of buffer1[0..100], we'll
use buffer1[0] to buffer1[100000] in steps of 1000. The same number of
values, but they're not near each other. The second loop, it's
[1..100001], and so on. In this case, it's a large number of loop
iterations before the same values are accessed again. This is disastrous
for the processor cache. It's also disastrous for the file mapping
approach.

It's part of the reason why, when choosing a sorting algorithm, quite
what you are sorting (in the sense of disk or RAM) does actually matter,
since some algorithms which are pretty good entirely in RAM become
catastrophically bad when used on database records.

So, before you start, work out the 'locality' qualities of you
algorithm. If you aren't darting all over your data sets, then file
mapping is (a) easy, and (b) solid.

(Frankly, it's also most likely to be recoverable when the system
crashes. IIRC, Windows 95 actually always crashed at something like 39.5
days of running due to a timer overflow. They just never knew this until
they'd stopped it crashing due to random reasons before that point.)

>Yes, i'm VERY concerned about the speed here, and i'm also 
>concerned about wear and tear my hard drive, and if i'll run
>my application for several mounths continuesly that's what can
>happen, that's also why i want to use ONLY real memory, becouse
>it can't be wear and tear by lot of reading and writing.

You will still need to read from and write to the disk anyway. For this,
you may want to buy server-grade drives (i.e. ones rated to be able to
run continuously).

Alan Bellingham
-- 
Me <url:mailto:alanb@episys.com> <url:http://www.doughnut.demon.co.uk/>
ACCU - C, C++ and Java programming               <url:http://accu.org/>
The 2004 Discworld Convention                   <url:http://dwcon.org/>