Reading LogFiles

[mattw112]

Does anyone know why I wouldn't be able to read a log file using Autoit and the TailRw function?

This log file is a bit weird. Even after its been updated with many lines the modified date never changes? Not sure why.

But I have a script that ever 10ms looks at the log file for changes and I'm not gettign anything back? I know the script works because I can use it against a static log file just fine, just not this specific real log file?

My guess is it has been opened exclusively by the process. But it is just a text file, can it do this?

T

[tAKTelapis]

.. are you opening the file and closing it every 10ms ? or are you opening it, and doing the file read every 10 ms? something like as follows:

(this is not code, more logic of code layout)

ie:

wait 10ms then

open file

read tail

close file

go back to waiting

OR

open file

wait 10 ms then

read tail

go back to waiting

Close file on exit

some example code would be nice.

Depending on what the log file is for, i see no reason it should not be readable like this.

Edited May 11, 2007 by tAKTelapis

[mattw112]

.. are you opening the file and closing it every 10ms ? or are you opening it, and doing the file read every 10 ms? something like as follows:

(this is not code, more logic of code layout)

ie:

wait 10ms then

open file

read tail

close file

go back to waiting

OR

open file

wait 10 ms then

read tail

go back to waiting

Close file on exit

some example code would be nice.

Depending on what the log file is for, I see no reason it should not be readable like this.

You are right, I am opening the file at the begining, doing my while, then doing a close at the very end. I'll bet I need to open and close it with every check...

Currently I was doing a wait of just 10ms, because sometimes this log gets updated very quickly. Does anyone have any idea what a good amount of time is to wait and not miss anything? For example could I bump that up to 250 ms?

Thanks,

Terry

[eltorro]

You could try using FileGetTime("log.txt",0) to check the modified time.

[flyingboz]

First, a file can be opened for exclusive write access; what data the file contains is immaterial.

Second, if the file is being kept open by the writing process, then the timestamp won't change until the process closes the filehandle.

Third: Whether or not you "miss" something is not dependent on how often you read the file, rather by the robustness of the logic performing the read.

For example, if you store something about the last line read (a timestamp, for example) , and you attempt to read the file again, if you don't have the last line read in the "window" there is the potential for missing data. Have your code re-read the file, grabbing a bigger slice until you know you haven't missed anything.

[mattw112]

First, a file can be opened for exclusive write access; what data the file contains is immaterial.

Second, if the file is being kept open by the writing process, then the timestamp won't change until the process closes the filehandle.

Third: Whether or not you "miss" something is not dependent on how often you read the file, rather by the robustness of the logic performing the read.

For example, if you store something about the last line read (a timestamp, for example) , and you attempt to read the file again, if you don't have the last line read in the "window" there is the potential for missing data. Have your code re-read the file, grabbing a bigger slice until you know you haven't missed anything.

Thanks.

Questions:

1.) If a log file is opened exclusively, is it possible to still read it? I know that may sound dumb, but is there some other way I can read it using Autoit?

2.) This must be what I'm seeing.

3.) I'm using the TailRW.au3 script for the reading the last line. As far as I know about the logic of it, it just reads the last line. So I guess I'll stay with the 10ms for now.

As for my last post I was wrong, I'm not opening at the begining and closing at the end, I'm really not doing any of the opening and closing at all (besides reading an ini files that's not related), the tailrw include script is doing that on the log file. Which is doing an open and close at the end of what it is doing.

[eltorro]

Are you trying to read it at too quick an interval? You could try FileGetTime to check the modified time before trying to read it.

Just talking out my A** here.

Another possiblility is to make a copy, read then delete the copy.

[mattw112]

Are you trying to read it at too quick an interval? You could try FileGetTime to check the modified time before trying to read it.

Just talking out my A** here.

Another possiblility is to make a copy, read then delete the copy.

Thanks,

The modified time isn't changing so that doesn't work (see flyingboz post above). That was originally how I was trying to do it, but have changed now so that it actually just reads the last line of the file to see if it has changed.

The problem is that for some reason I can't read this log file, even though during testing I can read a static log file...

Edited May 14, 2007 by mattw112

[eltorro]

Well, here is one more thing you can try,

Local $FILE_SHARE_READ = 1, $FILE_SHARE_WRITE = 2, $FILE_SHARE_RW
$FILE_SHARE_RW = BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE)

$hFile = _APIFileOpenMod("logfile.log",$FILE_SHARE_RW) ;
... <snip> ...
Func _APIFileOpenMod($sFile,$iShareMode=0)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080
    Local $AFO_h
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long", BitOR($GENERIC_READ, $GENERIC_WRITE), "long", $iShareMode, "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
EndFunc   ;==>_APIFileOpen

You could also be experiencing a problem with security attribs.

Edit: Changed variable name.

Edited May 14, 2007 by eltorro

[randallc]

Hi,

This could be TailRW problem;

which version?

which AutoIt?

Can you post a scrip?

Best, Randall

[enaiman]

I think that maybe randallc could be right - you can create an old and secure way to get the file read and see what's happening. A big part of my work is about reading a log and based on the content to perform different actions (I read Tera Term log - a Terminal like program - to configure network switches and during the program execution the log is read hundred-thousand times)

I'm not using any UDF for this but a plain old method - I store the last line read in a variable and I read only starting from that line and this is working for me.

[mattw112]

Hi,

This could be TailRW problem;

which version?

which AutoIt?

Can you post a scrip?

Best, Randall

8/24/2006 is the date on the TailRW.au3 include file.

I'm using the latest version of AutoIt and Latest SciEditor

It will be really hard for anyone to test this script if you dont have SMS installed and can watch how it handles the execmgr.log

but the script is shown below, any help or comments are welcome. I have verified it works if I manually create the log file and write the right log entries, but I just can't get it to work in production:

expandcollapse popup

;=====================================
; OPTIONS
;=====================================
Opt("TrayIconDebug", 1)
Opt("WinTitleMatchMode", 2)
Opt("WinDetectHiddenText",1)
Opt("ExpandEnvStrings", 1)

; ========================
; INCLUDE HELPER FUNCTIONS
; ========================
#include <constants.au3>
#include <GUIConstants.au3>
#include <String.au3>
#include <Array.au3>
#include <file.au3>
#include <Date.au3>
#include <TailRW.au3>

; ===========================
; VARIABLES AND DECLARATIONS
; ===========================
Dim $AvailableArray [1]
Dim $ProgramsToInstall
Dim $PLabel1
Dim $ItemStart
Dim $s_ReadLine
Dim $PLabel[30]
Dim $LABELS
Dim $NAME

$ProgressArray = ""
$INIFILE = @SystemDir & "\ServerBuild.ini"
$LOG = @SystemDir & "\CCM\Logs\execmgr.log"
$Pause_Time = "10"
$LASTCHECK = ""
$x = "1"

; ================
; OTHER
; ================
; Install ServerBuild.ini for this specific install
; Change the source path to match your location.
; Remember the "PROGRAM=" in the ini must equal the exact name that will be shown in the execmgr.log which is the SMS Program Name.
; "SERVERNAME=" Can be set to whatever you want to appear on the GUI.  Normally should be descriptive of the Server Type.

FileInstall("F:\Server Build\SCCM Integration\Files\TESTGUI\ServerBuild.ini", @SystemDir & "\ServerBuild.ini", 1)

; Install Logo for GUI to use
FileInstall("F:\Server Build\SourceFiles\LOGO.bmp", "c:\temp\LOGO.bmp", 1)

; ================
; MAIN
; ================
; Create To Do list from ini file found in systemdir named ServerBuild.ini, I'll create an INI file with sections related to
; what apps to expect and then put it down each time, overlapping the previous one.
_FileToArray ($AvailableArray, $INIFILE)

; Install Check
 If $ProgramsToInstall = 0 Then
      msgbox(0, "Error", "No Programs Found, Config File may be missing or incorrectly formatted")
      RunWait('eventcreate /t ERROR /SO WATCH /L Application /id 140 /d "No Programs Found, Config File may be missing or incorrectly formatted"', "", @SW_HIDE)
      Exit
EndIf

; Create the GUI and make it visible
_CreateGUI()

; Monitor File and update GUI as needed
While $x = "1"
    Sleep($Pause_Time)
    
    ; If someone presses the close button then exit
    If GUIGetMsg() = $GUI_EVENT_CLOSE Then
        Exit
    EndIf
    
    $RETURN = _Read_last($log)
        
    ; If there's a difference then changes have been made, check to see if the last line of the log contains one of the following and do some action.
    If $LASTCHECK <> $s_ReadLine Then
        
        ; If the last log line contains "<![LOG[Execution Request for program ", parse everything out so we get the program name.  Find that label and 
        ; make it bold.
        
        ; Example: "<![LOG[Execution Request for program TERRY TEST - GUI Remove state change from WaitingContent to NotifyExecution]LOG]!>
        ;<time="16:14:14.423+420" date="05-10-2007" component="execmgr" context="" type="1" thread="3132" file="executionrequest.cpp:487">"
        
        $tPosS = StringInStr($s_ReadLine, "<![LOG[Execution Request for program ") ; 37 Characters

        If $tPosS <> 0 Then
            $tLineS = StringTrimLeft ($s_ReadLine, $tPosS + 36) ;37 Characters - 1
            $len = StringLen($tLineS)
            $tPosB = StringInStr($tLineS, " state change from WaitingContent to NotifyExecution") ; 52 Characters
            $tLineB = StringTrimRight ($tLineS, $len - $tPosB + 1) ; Contains Program Name

            For $k=0 to $LABELS
                $READ = GUICtrlRead($PLabel[$k])
                If $READ = $tLineB Then
                    GUICtrlSetFont ($PLabel[$k],9, 800, 1, "Arial") ; Makes it Bold
                EndIf
            Next
            
        EndIf
        
        ; If the last log line has "<![LOG[Raised Program Started Event for Ad:" then parse out everything to get the program name.
        ; find it on to do list and make it bold with underline.
        
        ; Example: "<![LOG[Raised Program Started Event for Ad:D1A20029, Package:D1A0003D, Program: TERRY TEST - GUI Remove]LOG]!>
        ;<time="16:14:15.417+420" date="05-10-2007" component="execmgr" context="" type="1" thread="3132" file="executioncontext.cpp:512">"
        
        $tPosS = StringInStr($s_ReadLine, "<![LOG[Raised Program Started Event for Ad:") ;43 Characters

        If $tPosS <> 0 Then
            $tLineS = StringTrimLeft ($s_ReadLine, $tPosS + 79) ;43 - 1 + 8(for advert id) + 10 for ", Package:" + 8 for Program Id + 11 for ", Program: "
            $len = StringLen($tLineS)
            $tPosB = StringInStr($tLineS, ']LOG]!><time="') ; 13 Characters
            $tLineB = StringTrimRight ($tLineS, $len - $tPosB + 1) ; Contains Program Name

            For $k=0 to $LABELS
                $READ = GUICtrlRead($PLabel[$k])
                If $READ = $tLineB Then
                    GUICtrlSetFont ($PLabel[$k],9, 800, 4, "Arial")  ; Make it Underlined and Bold
                EndIf
            Next
            
        EndIf   
        
        ; If the last log line has "<![LOG[Execution is complete for program " and contains ". The exit code is ", parse the stuff to get the 
        ; program name and the exit code.  And if this is equal to the last program to install then exit altogether.
        ; find on to do list and strikethrough, unbold.  
        
        ; Example: "<![LOG[Execution is complete for program TERRY TEST - GUI Remove. The exit code is 0, the execution status is SuccessRebootRequired]LOG]!>
        ;<time="16:14:17.132+420" date="05-10-2007" component="execmgr" context="" type="1" thread="3132" file="execreqmgr.cpp:3820">"
        
        $tPosS = StringInStr($s_ReadLine, "<![LOG[Execution is complete for program ") ; 41 Characters
        
        If $tPosS <> 0 Then
            $tLineS = StringTrimLeft ($s_ReadLine, $tPosS + 40) ; 41 Characters - 1
            $len = StringLen($tLineS)
            $tPosC = StringInStr($tLineS, ". ") ; 2 Characters
            $tLineC = StringTrimRight ($tLineS, $len - $tPosC + 1) ; Contains Program Name
            ;MsgBox(0, "Test", "Program Name is: " & $tLineC)
            $tPosD = StringInStr($tLineS, ",")
            $tLineD = StringTrimRight ($tLineS, $len - $tPosD + 1)
            $tPosE = StringInStr($tLineD, "The exit code is ") ; 17 Characters
            $tLineE = StringTrimLeft($tLineD, $tPosE + 16) ; 17 Characters - 1, contains exit code
            ;MsgBox(0, "Test", "ExitCode Was: " & $tLineE)
            ; Create Event log for exit code just to have in another place
            Run('eventcreate /t INFORMATION /SO EXECMGR /L Application /id 140 /d "Exit Code for ' & $tLineC & ' was: ' & $tLineE & '"', "", @SW_HIDE)
            
            For $k=0 to $LABELS
                $READ = GUICtrlRead($PLabel[$k])
                If $READ = $tLineC Then
                    GUICtrlSetFont ($PLabel[$k],9, 400, 8, "Arial") ; Makes it Strikethrough with Normal Font Size
                    ; If this is the last item on the list exit
                    If $k = $LABELS Then
                        ; Sleep for a couple seconds so operator can see the last strikethrough
                        Sleep(2000)
                        Exit
                    EndIf
                EndIf
            Next
            
        EndIf
        
    EndIf
    
    ; Set the last line read to lastcheck to compare with the next one.
    $LASTCHECK = $s_ReadLine
    
WEnd

; Close GUI
GUIDelete()

Exit


; ================
; FUNCTIONS
; ================

; Creates GUI
Func _CreateGUI()
    Dim $tSize
    $LABELS = $ProgramsToInstall - 1
    $j = 50
    $tArrayItem = "0"
    $k = "0"
    
    ; Figuring out size for GUI
    If $LABELS < 30 Then
        $tSize = 20 * $LABELS
    Else
        $tSize = 270
    EndIf
    
    ; Create GUI
    GUICreate( "Application Progress From Execmgr.log", 400, $tSize + 250, 20, 25, -1)
    GUISetFont ( 14 , 400 , "", "Arial")
    GUISetBkColor(0xFFFFFF)
     
    ; Labels on GUI
    $LabelContext = GUICtrlCreateLabel( "Installing Applications for " & $NAME, 10, 10, 260 , 20 )
    GUICtrlSetFont (-1, 11, 646, "", "Arial")
    
    ; Create Labels and Populate them with the contents from the ini file
    For $k=0 to $LABELS
        ;$PLabelNew = $PLabel & $k
        $TEXT = $AvailableArray[$tArrayItem + $k]
        $PLabel[$k] = GUICtrlCreateLabel( $TEXT , 30, $j , 260 , 60 )
        GUICtrlSetFont ( -1,  9, 400, "", "Arial" )
        $j+=20
    Next
    
    ; Logo at Bottom
    $Pic1 = GUICtrlCreatePic("C:\temp\WAMULOGO.bmp", 30, $tSize + 150, 226, 69)

    ; Make GUI Visible
    GUISetState ()
    
EndFunc

;>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

; Reads last line of logfile
Func _Read_last ($log)
    $LineNumber = -1
    $s_ReadLine = __FileReadLine ($Log, $LineNumber, 1)
EndFunc

;>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

; Converts INI file to array
Func _FileToArray (ByRef $tArray, $INIFILE) 
   $i = 0
   $tOpenFile = FileOpen ($INIFILE, 0 )
   $ProgramsToInstall = "0"
    
    While 1
        $tLine = FileReadLine ($tOpenFile)
      
        If @error = -1 Then ExitLoop
      
        ; Removes white space from begining and end if any
        $tLine = StringStripWS ($tLine, 3 )
        
        ; Check if the line contains "PROGRAM" if it does add it to the array and remove '=' sign.
        ; if it contains "SERVERN" then add it to a different variable
        If StringLeft ( $tLine, 7)  = "PROGRAM" Then
            ReDim $tArray [$i+1]
            $tPos  = StringInStr ($tLine, "=")
            $tLine = StringTrimLeft ($tLine, $tPos)
            $tLine = StringStripWS ($tLine, 3)
            $tArray [$i] = $tLine
            $i = $i + 1
            ;Count programs to install
            $ProgramsToInstall = $ProgramsToInstall + 1
        ElseIf StringLeft ( $tLine, 7)  = "SERVERN" Then
            $tPos  = StringInStr ($tLine, "=")
            $tLine = StringTrimLeft ($tLine, $tPos)
            $tLine = StringStripWS ($tLine, 3)
            $NAME = $tLine
        EndIf
        
    Wend
  
   FileClose ($INIFILE)
   
EndFunc

The ini file just is simple there's:

SERVERNAME=<yourserver name for the GUI>

PROGRAM=<programname1>

PROGRAM=<programname2>

PROGRAM=<programname3>

PROGRAM=<programname4>

etc...

The program name must be exactly how SMS will be sending it out.

I have put examples in the script of the log entries I'm looking for.

Terry

[mattw112]

Well, here is one more thing you can try,

Local $FILE_SHARE_READ = 1, $FILE_SHARE_WRITE = 2, $FILE_SHARE_RW
$FILE_SHARE_RW = BitOr($FILE_SHARE_READ,$FILE_SHARE_WRITE)

$hFile = _APIFileOpenMod("logfile.log",$FILE_SHARE_RW) ;
... <snip> ...
Func _APIFileOpenMod($sFile,$iShareMode=0)
    Local $GENERIC_READ = 0x80000000, $GENERIC_WRITE = 0x40000000, $OPEN_ALWAYS = 4, $FILE_ATTRIBUTE_NORMAL = 0x00000080
    Local $AFO_h
    $AFO_h = DllCall("kernel32.dll", "hwnd", "CreateFile", "str", $sFile, "long", BitOR($GENERIC_READ, $GENERIC_WRITE), "long", $iShareMode, "ptr", 0, "long", $OPEN_ALWAYS, "long", $FILE_ATTRIBUTE_NORMAL, "long", 0)
    Return $AFO_h[0]
EndFunc   ;==>_APIFileOpen

You could also be experiencing a problem with security attribs.

Edit: Changed variable name.

wow thanks, can you tell me what this does and why it might work?

Thanks,

Terry

[flyingboz]

1.) If a log file is opened exclusively, is it possible to still read it?

See the definition of exclusive. -- without expounding too heavily, can you "type" the file and see new data? The program holding the file open likely has some process that periodically 'flushes' its write cache, you may be able to "pipe" the data , checking to see when you've got a new flush. If you can't type it , or open the file and see new data in notepad, then you likely are out of luck -- unless you can periodically 'bounce' the process holding the file open.

2.) This must be what I'm seeing.

While not definitive, indicative.

3.) I'm using the TailRW.au3 script for the reading the last line.

Make sure you know what it's supposed to do , then learn how to alter it if need be. Or change approaches, and use tools that you understand. ( I don't mean to be rude w/ this, just pointing out the truism that if you don't know what something is supposed to do, and how it does it, fixing it is difficult.) Edited May 15, 2007 by flyingboz

[eltorro]

wow thanks, can you tell me what this does and why it might work?

Thanks,

Terry

Your asking for shared access instead of exclusive.

[mattw112]

Your asking for shared access instead of exclusive.

OK, So I've updated TailRW.au3 with your suggestions. I also made changes throughout the au3 where needed.

I'm going to test it now.

Thanks,

Terry