Sign in to follow this  
Followers 0
KaFu

How to... trim a log-file to maximum defined size?

13 posts in this topic

Hiho Community,

I wonder how you trim your log-files to a maximum defined size? In general no problem, but how to do it efficiently for large logs with lots of changes?

Currently I'm trimming the log-file by moving it to a _TempFile(), reading the first (up to maximum) bytes with fileread($file,bytes) and writing those bytes to the original log-file location, purging the tempfile afterwards.

I also took a deeper look into the _WinAPI functions. _WinAPI_SetEndOfFile() looked promising to me, but the problem with that is that new log-entries then have to be written to the beginning of the file... what also takes reading the file and pushing the content 1 line down (at least thats how it's done in the _FileWriteLog($file,$line,1) UDF function if you set flag = 1). But this is imho also not a really efficient way :)...

Any thoughts on this one?

Best Regards

Share this post


Link to post
Share on other sites



Hi,

why do you want to do that? What app writes the log? Maybe you can change the settings there.

Mega


Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites

why do you want to do that? What app writes the log? Maybe you can change the settings there.

My own AutoIt app writes the log, an extensive one. And I want to set a maximum size limit for the log, always keeping the latest log-results only.

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Ah okay, and what about using the generation principle? Writing e.g. 3 files after a size has reached switching to the next and so on ...

Edited by Xenobiologist

Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites

Ah okay, and what about using the generation principle? Writing e.g. 3 files after a size has reached switching to the next and so on ...

Hmm, that's for sure a solution with a good performance, always only keeping the actual log and the last one, purging all older ones... but on the other hand I would be nice to have all log-results in a single file only :). Thanks for this idea, I'll keep it in mind, but maybe someone else got another idea?

Share this post


Link to post
Share on other sites

depending on the contents of the log file, it can probably be done fairly easily with StringRegExpReplace()


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites

Hmm, each row contains a unique description of a work-step performed, either successful or failed. I have an option (combo) to limit size to 4MB, 16MB, 64MB. Parsing that amount of data (often) will for sure be a burden on the performance.

Most performant way should be to write new entries to the start of the file and just truncate it. But I don't know a method to write to the start of a file (without reading it's content first and pushing appending it to the new line). Guess that just doesn't exist :)...

Share this post


Link to post
Share on other sites

can you post a sample line? Is there anything variable that can be checked?


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites

can you post a sample line? Is there anything variable that can be checked?

GEOSoft, thanks for the offer. I could any variable to the log-lines. What I'm seraching for is a general principle to often write lines to large log files, limiting their overall size without any huge impact on the performance. Hmmm, switching to SQLite and looping through only 100k rows would be a way. But I also want to keep the file a readable with a text-editor.

Share this post


Link to post
Share on other sites

This is one method that I use to keep a log file under control. I write new entries to the start of the file and trim the file to a fixed size. If the log file is being updated very frequently then I trim it a smaller size than the maximum I want for the file. This avoids the file being trimmed every time it is written to.

#include <file.au3>
#Include <WinAPI.au3>

$logdata = ''
For $i = 1 To 180
    ;Write new line at begining of file
    _FileWriteLog(@ScriptDir & "\test.log","###############################This is an entry in the log file.#############################",0)
    Sleep(1000)
    If FileGetSize(@ScriptDir & "\test.log") > 4096 Then
        $hFile = _WinAPI_CreateFile(@ScriptDir & "\test.log", 2, 6,6)
        _WinAPI_SetFilePointer($hFile, 4096)  ; make the value a bit smaller than the max size if writing to log very frequently
        _WinAPI_SetEndOfFile($hFile)
        _WinAPI_CloseHandle($hFile)
    EndIf
Next


;~ For $i = 1 To 1000
;~ _FileWriteLog("test.log","###############################This is an entry in the log file.#############################",0)
;~ Sleep(1000)
;~ If FileGetSize("test.log") > 2048 Then
;~  FileWrite("test.log" ,FileRead ( "test.log" , 1536 ))
;~  EndIf
;~ Next

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites

This is one method that I use to keep a log file under control. I write new entries to the start of the file and trim the file to a fixed size. If the log file is being updated very frequently then I trim it a smaller size than the maximum I want for the file. This avoids the file being trimmed every time it is written to.

Nice code :party:, the method with _WinAPI_SetEndOfFile($hFile) should work rather quick :). But how do you write entries to the beginning of the log-file? I assume by first reading the log-file, then append this to the new line and replace the old log-file content, correct? Truncate this way is fine, but writing will be still a performance issue once the file get big and the update's are frequent (I assume ~5 entries a second, with filelimits 4, 16 and 64mb).

Share this post


Link to post
Share on other sites

Nice code :party:, the method with _WinAPI_SetEndOfFile($hFile) should work rather quick :). But how do you write entries to the beginning of the log-file? I assume by first reading the log-file, then append this to the new line and replace the old log-file content, correct? Truncate this way is fine, but writing will be still a performance issue once the file get big and the update's are frequent (I assume ~5 entries a second, with filelimits 4, 16 and 64mb).

The _FileWriteLog() function takes a 3rd parameter and if set to anything other than -1 writes to the start rather than the end of the file. I use _FileWriteLog() all the time for writing logs due to the convenience of automatically having a date/time stamp added to every line.


"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites

The _FileWriteLog() function takes a 3rd parameter and if set to anything other than -1 writes to the start rather than the end of the file. I use _FileWriteLog() all the time for writing logs due to the convenience of automatically having a date/time stamp added to every line.

Yeah, but I looked into the UDF, using flag=1 it reads the file into memory and apends it to the newline before writing it with fileopen($file,2) (2 = Write mode (erase previous contents)). Hmmm, but I think that's the best way to go.

- Read entries to buffer, and flush to file only once per cycle, not every log-entry by itself

- Use _FileWriteLog() to write

- Truncate file once per cycle (_WinAPI_SetEndOfFile($hFile) should truncate rather quick, I guess it just changes the allocated filesize in the MFT).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0