Jump to content

_FileWriteToLine() - optimized: 2x faster


Zedna
 Share

Recommended Posts

Original uses FileReadToArray(), operates with this array and finally converts whole array to string (with delimiter CRLF).

My optimization which is approx. 2x faster uses StringInStr() to search position of CRLF at start/end of desired line and extracts desired line by StringMid() and after changes to desired line it merge final content of file by StringLeft() + "new line" + StringMid()

To get number of lines in original file I use:

Local $sData = FileRead($sFilePath)
StringReplace($sData,@CRLF, '', 0, $STR_CASESENSE) 
Local $iLinesOrig = @extended + 1

original:

Func _FileWriteToLine($sFilePath, $iLine, $sText, $bOverWrite = False, $bFill = False)
    If $bOverWrite = Default Then $bOverWrite = False
    If $bFill = Default Then $bFill = False
    If Not FileExists($sFilePath) Then Return SetError(2, 0, 0)
    If $iLine <= 0 Then Return SetError(4, 0, 0)
    If Not (IsBool($bOverWrite) Or $bOverWrite = 0 Or $bOverWrite = 1) Then Return SetError(5, 0, 0)
    If Not IsString($sText) Then
        $sText = String($sText)
        If $sText = "" Then Return SetError(6, 0, 0)
    EndIf
    If Not IsBool($bFill) Then Return SetError(7, 0, 0)
    ; Read current file into array
    Local $aArray = FileReadToArray($sFilePath)
    ; Create empty array if empty file
    If @error Then Local $aArray[0]
    Local $iUBound = UBound($aArray) - 1
    ; If Fill option set
    If $bFill Then
        ; If required resize array to allow line to be written
        If $iUBound < $iLine Then
            ReDim $aArray[$iLine]
            $iUBound = $iLine - 1
        EndIf
    Else
        If ($iUBound + 1) < $iLine Then Return SetError(1, 0, 0)
    EndIf
    ; Write specific line - array is 0-based so reduce by 1 - and either replace or insert
    $aArray[$iLine - 1] = ($bOverWrite ? $sText : $sText & @CRLF & $aArray[$iLine - 1])
    ; Concatenate array elements
    Local $sData = ""
    For $i = 0 To $iUBound
        $sData &= $aArray[$i] & @CRLF
    Next
    $sData = StringTrimRight($sData, StringLen(@CRLF)) ; Required to strip trailing EOL
    ; Write data to file
    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, 0)
    FileWrite($hFileOpen, $sData)
    FileClose($hFileOpen)
    Return 1
EndFunc   ;==>_FileWriteToLine

optimized:

Func _FileWriteToLine_Optim($sFilePath, $iLine, $sText, $bOverWrite = False, $bFill = False)
    If $bOverWrite = Default Then $bOverWrite = False
    If $bFill = Default Then $bFill = False
    If Not FileExists($sFilePath) Then Return SetError(2, 0, 0)
    If $iLine <= 0 Then Return SetError(4, 0, 0)
    If Not (IsBool($bOverWrite) Or $bOverWrite = 0 Or $bOverWrite = 1) Then Return SetError(5, 0, 0)
    If Not IsString($sText) Then
        $sText = String($sText)
        If $sText = "" Then Return SetError(6, 0, 0)
    EndIf
    If Not IsBool($bFill) Then Return SetError(7, 0, 0)
    ; Read current file
    Local $sData = FileRead($sFilePath)
    StringReplace($sData,@CRLF, '', 0, $STR_CASESENSE) ; get number of lines
    Local $iLinesOrig = @extended + 1
    ; If Fill option set
    If $bFill Then
        ; If required resize number of lines to allow line to be written
        If $iLinesOrig < $iLine Then
            Local $sAddedLines = ""
            For $i = 1 To $iLine - $iLinesOrig
                $sAddedLines &= @CRLF
            Next
            $sData &= $sAddedLines
            $iLinesOrig = $iLine
        EndIf
    Else
        If ($iLinesOrig) < $iLine Then Return SetError(1, 0, 0)
    EndIf

    $poz2 = StringInStr($sData, @CRLF, $STR_CASESENSE, $iLine)
    If $iLine > 1 Then
        $poz1 = StringInStr($sData, @CRLF, $STR_CASESENSE, -1, $poz2 - 1)
        $sLine_orig = StringMid($sData, $poz1 + 2, $poz2 - ($poz1 + 2))
    Else
        $sLine_orig = StringLeft($sData, $poz2 - 1)
    EndIf

    ; Write specific line - and either replace or insert
    $sLine = ($bOverWrite ? $sText : $sText & @CRLF & $sLine_orig)
    ; insert new line to original content
    If $iLine > 1 Then
        $sData = StringLeft($sData, $poz1 + 2 - 1) & $sLine & StringMid($sData, $poz2)
    Else
        $sData = $sLine & StringMid($sData, $poz2)
    EndIf
    ; Write data to file
    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, 0)
    FileWrite($hFileOpen, $sData)
    FileClose($hFileOpen)
    Return 1
EndFunc   ;==>_FileWriteToLine

example of use and compare of speed:

#include <File.au3>
#include <MsgBoxConstants.au3>
#include <WinAPIFiles.au3>

Example()
Example2()

Func Example()
    ; Create a constant variable in Local scope of the filepath that will be read/written to.
    Local Const $sFilePath = _WinAPI_GetTempFileName(@TempDir)

    ; Create data to be written to the file.
    Local $sData = "Line 1: This is an example of using _FileWriteToLine()" & @CRLF & _
            "Line 2: This is an example of using _FileWriteToLine()" & @CRLF & _
            "Line 3: This is an example of using _FileWriteToLine()" & @CRLF & _
            "Line 4: This is an example of using _FileWriteToLine()" & @CRLF & _
            "Line 5: This is an example of using _FileWriteToLine()" & @CRLF

    ; Create a temporary file to read data from.
    If Not FileWrite($sFilePath, $sData) Then
        MsgBox($MB_SYSTEMMODAL, "", "An error occurred whilst writing the temporary file.")
        Return False
    EndIf

    ; Write to line 3 with overwriting set to true.
    _FileWriteToLine_Optim($sFilePath, 3, "Line 3: THIS HAS BEEN REPLACED", True)

    ; Read the contents of the file using the filepath.
    Local $sFileRead = FileRead($sFilePath)

    ; Display the contents of the file.
    MsgBox($MB_SYSTEMMODAL, "", "Contents of the file:" & @CRLF & $sFileRead)

    ; Write to line 3 with overwriting set to false.
    _FileWriteToLine_Optim($sFilePath, 3, "Line 3: THIS HAS BEEN INSERTED", False)

    ; Read the contents of the file using the filepath.
    $sFileRead = FileRead($sFilePath)

    ; Display the contents of the file.
    MsgBox($MB_SYSTEMMODAL, "", "Contents of the file:" & @CRLF & $sFileRead)

    ; Delete the temporary file.
    FileDelete($sFilePath)
EndFunc   ;==>Example

Func Example2()
    Local Const $sFilePath = _WinAPI_GetTempFileName(@TempDir)

    ; ***** original *****

    Local $sData = ""
    For $i = 1 To 10000
        $sData &= "Line " & $i & ": This is an example of using _FileWriteToLine()" & @CRLF
    Next

    If Not FileWrite($sFilePath, $sData) Then
        MsgBox($MB_SYSTEMMODAL, "", "An error occurred whilst writing the temporary file.")
        Return False
    EndIf

    $start = TimerInit()
    For $i = 10 To 1 Step -1
        $j = $i*1000
        _FileWriteToLine($sFilePath, $j, "Line " & $j & ": THIS HAS BEEN REPLACED", True)
        _FileWriteToLine($sFilePath, $j, "Line " & $j & ": THIS HAS BEEN INSERTED", False)
    Next
    ConsoleWrite('original: ' & TimerDiff($start) & @CRLF)

    FileDelete($sFilePath)

    ; ***** optimized *****

    Local $sData = ""
    For $i = 1 To 10000
        $sData &= "Line " & $i & ": This is an example of using _FileWriteToLine()" & @CRLF
    Next

    If Not FileWrite($sFilePath, $sData) Then
        MsgBox($MB_SYSTEMMODAL, "", "An error occurred whilst writing the temporary file.")
        Return False
    EndIf

    $start = TimerInit()
    For $i = 10 To 1 Step -1
        $j = $i*1000
        _FileWriteToLine_Optim($sFilePath, $j, "Line " & $j & ": THIS HAS BEEN REPLACED", True)
        _FileWriteToLine_Optim($sFilePath, $j, "Line " & $j & ": THIS HAS BEEN INSERTED", False)
    Next
    ConsoleWrite('optimized: ' & TimerDiff($start) & @CRLF)

    FileDelete($sFilePath)
EndFunc   ;==>Example2

 

EDIT: This source is from AutoIt 3.3.14.4

Edited by Zedna
Link to comment
Share on other sites

  1. seems to be depending on number of lines in the file
    100 lines then original is faster
    1000 lines then about 30% faster
    10000 lines then twice as fast
  2. original also deals only with @CRLF so should not be an issue
    one way of getting the newline type to see which character(s) it is
     
    Local $tArray = StringRegExp($sData, "(.*\R)", $STR_REGEXPARRAYMATCH)
        If (@error = 0) Then
    ;~  Determine linetype to be CRLF of LF or CR
            Local $newLine = (StringRight($tArray[0], 2) = @CRLF) ? @CRLF : StringRight($tArray[0], 1)
        Else
            Return SetError(8, 0, 0)
        EndIf

     
  3. We are still in range of milliseconds so although good to know when you have to deal with text files frequently
     
  4. I tried with stringregex but then after dealing with files > 5000 lines memory problems occur / errors
    anyway its with regex only marginally faster then original (about 20%)
    Local $sRegExPattern = "((.*\R){" & ($iLine - 1) & "})(.*)((.*\R){1,})(.*)"

     

 

 

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...