🚀 go-pugleaf

To the operating system files have no structure other than simply being 
written from beginning of file to the end.  If you wish to insert something 
into the beginning of a file you could use SetFilePosition to move the 
read/write pointer to the beginning and write to the file.  However the data 
written will over-write what is already in that part of the file.

The way to insert something at a place other than the end of a file is to 
create a new file, stream in data from the old and out to the new until the 
insert point is reached, write the new data, then stream data in and out for 
the remaining portion of the file.

By designing your system to be last written at the beginning of the file you 
have arranged to not achieve the maximum speed that you desire.  A simpler 
method might be to save the data "upside down", with the last written at the 
end.  When reading back it is easy to seek to the end-of-file minus a record 
size and read that header from there.

A system with a strict need for minimum time to execute has conflicting 
requirements with the need to reconfigure the event log.  Better to include 
an 'deleted' bit in the record's header and to start another event file when 
the number of records reaches some predetermined number.  When the system 
reaches a state where a little time is available it is a simple task to then 
pass through the event files, compacting to a new, trimmed file.  The 
'deleted' bit also allows you to know enough about the event files' contents 
to get correct information even before compaction.

No comments on how to use for TFileStream as I see no value in using it for 
this task.  AnsiString and TStrings could be handy if you are saving data as 
characters but is conversion from raw data to characters consistent with 
your desire for very fast operation?

.  Ed

> Mike Collins wrote in message
> news:479b78fd@newsgroups.borland.com...
>
> Hi all, feel a little embarrassed to as such a seemingly easy question 
> but...
>
> I'm trying to create an event logging system.  I have a main thread that 
> monitors certain things and I need to log these in a text based event log. 
> Because the file i/o make take some time, I've created a new thread to do 
> the actual writing operations.  The main thread pushes events onto the 
> secondary threads stl stack.  The secondary thread should then process 
> these events in a FIFO order and write them to the event log.  So far i 
> think my logic is sound :-).
>
> However, I cannot find any reference or explanation on how to use the 
> WriteFile() api to insert a new piece of data at the start of the file.  I 
> want my events to be stored in ascending order of when they occurred. 
> I've used WriteFile a million times to create files or write data to the 
> end of them i.e. appending them. but i can't find anyway to write to the 
> start of the file.  Any suggestions.
>
> Secondly, I'm looking to achieve the best performance possible so with 
> this in mind, is it more resource intensive to write new events to the 
> start of a file as opposed to appending them to the end?
>
> Thirdly, I need to ensure that the event log does not exceed a certain 
> size. In either of the above cases (writing at the start or writing at the 
> end) how would i approach the problem of removing an entry from my event 
> log.  I would ideally like to perform a check to see if the log has 
> exceeded a certain size and then delete the oldest entry.
>
> Finally, are there any issues with performing file I/O from within a 
> thread? What happens if the thread's CPU time elapses before the write 
> operation has finalised?
>
> Sorry for all the questions, just really can't get my head around this.  I 
> had thought of maybe using one of the VCL wrappers i.e. TFileStream or the 
> TStrings but i need to ensure that each entry is written as it is 
> processed and not simple keep writing the whole stream or strings object 
> each time.
>
> Any help or advice would be gratefully received.

"Mike Collins" <its@TheBottomOfThePost> wrote in message 
news:479b78fd@newsgroups.borland.com...

> However, I cannot find any reference or explanation on how to
> use the WriteFile() api to insert a new piece of data at the start
> of the file.

There is no way to insert data at the beginning of a file while preserving 
the data that was already present.  Writing to a file always overwrites 
whatever was previously present at the position you are writing to.  To do 
what you are attempting, you must create a new file, write the new record 
first, then copy all of the existing records from the old file to the new 
file, then replace the old file with the new file, and then repeat 
everything again for each new record you want to insert.

Otherwise, you would be better off simply writing the records to a single 
file in descending order normally, and then write your own log viewer 
program that can display the records in reverse order.

> Secondly, I'm looking to achieve the best performance possible so with
> this in mind, is it more resource intensive to write new events to the 
> start
> of a file as opposed to appending them to the end?

Yes, a lot more.

> Thirdly, I need to ensure that the event log does not exceed a certain
> size. In either of the above cases (writing at the start or writing at the
> end) how would i approach the problem of removing an entry from
> my event log.

If you write records to the front of the file, then removing old records is 
simply a matter of truncating the existing file to the appropriate size via 
SetFilePointer() and SetEndOfFile().

If you write records to the end of the file, then removing old records 
requires creating a new file, seeking the old file to the starting position 
of the first record you want to keep, copying the remaining records from the 
old file to the new file, then replacing the old file with the new file.

> I would ideally like to perform a check to see if the log has exceeded a 
> certain size and then delete the oldest entry.

You can query the file size before you write any data to it, and you can 
calculate the size of the record you want to write.


Gambit