🚀 go-pugleaf

Deniel wrote:

>>That's still pretty vague.
>>How big is the data set?
>>Assume you had a MegaTerra Byte of RAM.
>>How much would you need?
>
>It's realy a hard question for me at this stage, i still have
>to think about it and do some calculatings to know that, all

Then you don't really know at all yet, what the best approach might
be.
You say you are reading data from a file, but you have no idea how big
that file is.
You say you need to write the results back to file, but you don't want
to read or write to file during the process.
You need to decide if you are going to artificially limit your program
to running _ONE_ set of data that will fit into memory (however small
that may be), or writting it so you can use larger data sets, and read
and write as needed to bring your small window of memory to bear on
the larger dataset.

In the first case, where you do nothing with the hard drive between
startup and shutdown, VirtualAlloc could be the way to go.

In the larger case, where you need to access more data than you have
memory, the MappedFile would be by far the simplest, as it will
read/write as needed (and only as needed), and will page itself out
when you run out of RAM.  It is supposed to be pretty efficient about
that.


>i know at this point is that i'll have to use several buffers
>that i'll fill with data from a file, and while the program is
>running i continuously have to read data from this buffer, do
>some calculations and return the results to the buffer, and i
>have to do this process again and again and again for a VERY
>long time (several mounths i guess) until getting the final 
>results,

I'll save you the months...The final result is 47.

>>If your calculating proceeds in an orderly manner through the data
>>(sequential, or nearly so rather than truely random), then you might
>>have 2 threads. 1 thread calculating and updating the data, the other
>>thread anticipating the data about-to-be-needed, and reading that off
>>disk (Reading in chunks is, of course, better that per-double)
>>Double-buffering, in that thread 1 is reading/calculating/updating one
>>buffer while thread 2 is saving/retrieving the next bit.
>
>I'm not sure that it's fits to my situation, i have to read 
>several data values from the buffer, do some calculates with
>them, and then change the data in that buffer accordingly, it's
>not 2 processes that can be done on parallel.

Sure it is.
Triple buffer:
Buffer One: data has been updated
Buffer Two: data is being processed
Buffer Three: data will be proccessed next.

Thread One works with buffer 2
Thread Two writes Buffer 1 and reads Buffer 3.
Sync the threads.
Repeat
Maybe the Buffers are 10MB each so the read/write is less frequent.

>Does CreateFileMapping let me allocate 150 MB buffer in the
>'real' memory, and make sure it will not use the hard-disk?

Doesn't matter.
If your data runs larger than available contiguouse RAM, you will need
to swap it out eventually. The good thing about using FileMapping is
if the OS needs to swap out a page, it doesn't write it if it is
already up-to-date. And if it isn't up-to-date, then it will be,
because the swap-out will be to the underlying data file.


>How much memory can i allocate using FileMapping? and does it
>force the system to allocate ONLY 'real' memory without paging?

No. The paging is the beauty of the method.

>>If I was still concerned about the speed
>
>Yes, i'm VERY concerned about the speed here, and i'm also 

I meant *AFTER* initial testing.
If the initial program seemed too slow..then, and only then, would I
look for more complicated "solutions".

>concerned about wear and tear my hard drive, and if i'll run
>my application for several mounths continuesly that's what can
>happen, that's also why i want to use ONLY real memory, becouse
>it can't be wear and tear by lot of reading and writing.

I can only say "Try it and see".
If you write it to load/save large, aligned buffers, then there will
be less load on the drive.
 
>, I'd add a visible progress
>>device that updates less than 1 per second (visual updates are slow)
>>so I could time it.
>>Then add VirtualAlloc code (more complicated) and retest.
>>As for disk reads...if you can get a minute of calculation out of a 1
>>second disk read, i wouldn't sweat it as you are talking 1/2 day of
>>disk delay per month of calculation.
>
>I did not understood what you where saying here. 

For computationals, a Console app will often suffice.
Every several iterations, output a period. or a counter.
Lets you know that things are still progressing.
Also lets you use a stopwatch to time the loop.
But, even putting a period on the screen is increadibly slow compared
to the loop, so you don't want to update the screen very often.

In one app, I updated a GUI ProgressBar every iteration. It took 10-15
minutes to run. On a whim, I modified it up update every 1000
iterations, and the time fell to 5 seconds.

Or were you asking about the second (VitualAlloc)  part?
I was saying  I would try the easy method first (FileMapping), with a
visual speed indicator. Then, if I felt the disk was going crazy, or
it was too sluggish, I'd revamp to VirtualAlloc, and start it again to
see if there were any noticeable improvement.

Or were you asking about the 1/2 day?
I was just saying that if you could read enough data in 1 second to
keep the computation going for 1 minute, then the over-all delay on a
one-month run would amount to 1/2 a day. It also wouldn't be a burden
on the drive to run for 1 second per minute.

>No, i'm using a desktop PC.

Then you shouldn't have disk speed issues.