Re-write of _FileWriteToLine()

[jaberwacky]

Durrr, didn't check to see if it actually wrote to a file.  Yeah, does what it says on the tin.

[guinness]

This doesn't write to line 9999 with @extended returning 25 when I place under SRER, also choosing 8000 or 9000 creates a hard crash?!

#include <FileConstants.au3>

Local $hFileOpen = FileOpen('Test.txt', $FO_OVERWRITE)
For $i = 1 To 100000
    FileWriteLine($hFileOpen, $i)
Next
FileClose($hFileOpen)
Local $sData = FileRead('Test.txt')
$sData = _WriteToLine($sData, 9999, 'Line_10', 1)
ConsoleWrite(@error & @CRLF)
ClipPut($sData)

[jaberwacky]

Even though it doesn't write to the line I don't get the hard crash or the @extended. 

Correction, it does write to the line but without the @extended or the crash.  Was looking  in the wrong place.  Am useless sometimes.

Edited January 8, 2014 by jaberwocky6669

[guinness]

I am on 8.1 with v3.3.10.2 x64.

[jaberwacky]

good idea: win7 x64   v3.3.10.2

[Mat]

Ok, well I'm hitting a limit other than the repeating quantifier limit, and I thought it was the length of a matched group but that's not the case either.

expandcollapse popup

#include <FileConstants.au3>

Local $sData = ""

For $i = 1 To 100000
    $sData &= "Line " & $i & @CRLF
Next

Local $hFile = FileOpen("Test.txt", $FO_OVERWRITE)
FileWrite($hFile, $sData)
FileClose($hFile)

$sData = _FileWriteToLine("Test.txt", 9300, "$1", True)
ConsoleWrite(@error & @LF)

$sData = _FileWriteToLine("Test.txt", 9400, "$1", True)
ConsoleWrite(@error & @LF)


Func _FileWriteToLine($sFilePath, $iLine, $sText, $fOverwrite = False)
    If $iLine <= 0 Then Return SetError(1, 0, 0) ; Negative line
    If Not FileExists($sFilePath) Then Return SetError(2, 0, 0) ; File does not exist

    Local $sData = FileRead($sFilePath)
    If @error Then Return SetError(3, 0, 0) ; Could not read the file

    ; Escape matched group references in replacement text
    $sText = StringReplace($sText, "\", "\\")
    $sText = StringReplace($sText, "$", "\$")

; TODO:
;  * If line = 0 Then insert at first line?

    $iLine -= 1

    Local $sPattern = "\A"
    Local $iGroups = 0

    ; 9359
    While $iLine > 2000
        $iLine -= 2000
        $sPattern &= "((?:.*?\R){2000})"
        $iGroups += 1
    WEnd

    If $iLine Then
        $sPattern &= "((?:.*?\R){" & $iLine & "})"
        $iGroups += 1
    EndIf

    $sPattern &= "(.*?\R)"

    Local $sReplPattern = ""

    For $i = 1 To $iGroups
        $sReplPattern &= "$(" & $i & ")"
    Next

    $sReplPattern &= (($fOverwrite And $sText = "") ? "" : (($fOverwrite ? "" : "${" & ($iGroups + 1) & "}") & $sText & @CRLF))

    $sData = StringRegExpReplace($sData, $sPattern, $sReplPattern, 1)
    ConsoleWrite("Regex " & $sPattern & ": " & @error & " " & @extended & @LF)


    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(4, 0, 0) ; Could not open file for writing

    FileWrite($hFileOpen, $sData)
    FileClose($hFileOpen)

    Return $sData
EndFunc   ;==>_WriteToLine

Is there a memory limit on the total length of matched groups? Or am I hitting a memory constraint? Consistently hit a limit at replacing line 9360. 

A quick calculation is that it fails trying to match about 101844 characters. But it's not a character issue because changing the code to use LF rather than CRLF makes no difference.

[guinness]

That example just crashes when it tries to replace line 9300.

[trancexx]

Replace pattern uses round instead of curly brackets in that example.

The main problem is that the size of patterns is too big, you are exceeding value of MAX_PATTERN_SIZE. Last time I checked AutoIt's PCRE this value was 65536 (1 << 16), but Jon did lots of work afterwars, so that could have been changed, even though I think not. There is another option or value, called LINK_SIZE. Changing this option MAX_PATTERN_SIZE also changes, so maybe that's the solution for situations like this. I think Jon was saying something about the sizes of patterns some time ago, search around if you are interested about what he said.

[guinness]

I just remember this >> 

[jchd]

link_size is just for pattern compilation. Values > 2 are needed only for exceptionally complex patterns. This has nothing to do with captured groups.

There are other issues with the last code posted

[trancexx]

Trust me on this, I know what I'm talking about. Pattern size here is too big.

If AutoIt's StringRegExp error code would be set to real regexp error code you would see that too.

[trancexx]

But that doesn't mean there aren't other problems.

I don't get crashes, guinness does, that's something to think about.

[jchd]

Just what I said: compiled pattern too big. BTW, is the pattern studied and would switching to JIT solve the issue with this precise script?

[DXRW4E]

Hi All, in these cases the string function are very much faster than regexp, for this reason i asked StringReplace & StringRegExpReplace Add OffsetThe starting position of the search here is an old function which is 2 times faster and which is more compatible with FileReadLine -1 last line or use the -2 penultimate line ect ect

expandcollapse popup

#include <FileConstants.au3>


; #CONSTANTS# ====================================================
#Region ;**** Global vars for _FileWriteToLineEx
Global Const $FRL_APPEND     = 0
Global Const $FRL_OVERWRITE  = 1
Global Const $FRL_DELETE     = 2
Global Const $FRL_OVERWRITE_DELETE  = BitOR($FRL_OVERWRITE, $FRL_DELETE) ;INTERNALLY USE ONLY
#EndRegion ;**** Global vars for _FileWriteToLineEx
; ================================================================

;;$fTimerDiff = TimerInit()
_FileWriteToLineEx(@DesktopDir & '\Test.txt', 'TestLine')
If @Error Then ConsoleWrite(@error & @CRLF)
;;$fTimerDiff = TimerDiff($fTimerDiff)
;;ConsoleWrite($fTimerDiff & @CRLF)


; #FUNCTION# ====================================================================================================================
; Name...........: _FileWriteToLineEx
; Description ...: Edit a specific line in a file.
; Syntax.........: _FileWriteToLineEx($sFilePath, $sText[, $iLine[, $iFlags]] )
; Parameters ....: $sFilePath - The File Path
;                  $sText     - The text to write
;                  $iLine     - The line number to edit to (Default edit last line), Use a negative occurrence to search line from the right side
;                  $iFlags    - Optional, Flag to indicate the type of stripping that should be performed
;                  |$FRL_APPEND (0)    - (Default) Append mod
;                  |$FRL_OVERWRITE (1) - overwrite the old line
;                  |$FRL_DELETE (2)    - delete\remove the line
; Return values .: If the function succeeds, it returns TRUE; otherwise, FALSE.
;                  @error  - 0 = No error
;                  |1 = File does not exist
;                  |2 = File has less lines than $iLine
;                  |3 = Error when opening file, should be ERROR_ACCESS_DENIED = 5 (0x5) Access is denied. ($sFilePath is ReadOnly or has not Security Access Rights)
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileWriteToLineEx($sFilePath, $sText, $iLine = Default, $iFlags = 0)
    If Not FileExists($sFilePath) Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $iOffSet, $sData = StringRegExpReplace(FileRead($sFilePath), '\r(?!\n)', @CRLF)
    If $iLine = 1 Then
        $iOffSet = BitAND($iFlags, $FRL_OVERWRITE_DELETE) ? 0 : StringInStr($sData, @LF, 1, 1)
    Else
        $iOffSet = StringInStr($sData, @LF, 1, ($iLine < 0 ? $iLine : $iLine - 1))
        If Not $iOffSet Then Return SetError(2, 0, False)
    EndIf
    $_iOffSet = StringInStr($sData, @LF, 1, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sData)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= (1 + Number(StringMid($sData, $_iOffSet -1, 1) = @CR))
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, False)
    FileWrite($hFileOpen, StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet))
    FileClose($hFileOpen)
    Return True
EndFunc   ;==>_FileWriteToLineEx

Ciao.

Edited January 9, 2014 by DXRW4E

[guinness]

Certainly a lot quicker DXRW4E. I actually prefer your function to the current UDF one.

I know people will disagree, but I just feel inclined to go with _FileWriteToLine($sFilePath, $sText, $iLine, $iFlags = 0) making $iLine mandatory but swapping both text and line around. I know it would be a script breaking change, but it feels right and similar to FIleWriteLine() IMHO. I was also thinking that (similar to FileWriteLine() if no @LF, @CR or @CRLF is at the end of $sText then append, otherwise leave as is.

Suggestions? Comments? Dislikes?

If you have nothing to say verbally, then like this post will give me a hint you're happy with this choice. Thanks.

Edited January 9, 2014 by guinness

[guinness]

The only issue with the version above is r and converted to rn, which could be a problem for a user who has file that use @CR as the EOL character.

[DXRW4E]

I know people will disagree, but I just feel inclined to go with _FileWriteToLine($sFilePath, $sText, $iLine, $iFlags = 0) making $iLine mandatory but swapping both text and line around. I know it would be a script breaking change, but it feels right and similar to FIleWriteLine() IMHO.

because by default _FileWriteToLine will have to write always last line i think is logical, however you can also pass it _FileWriteToLine($sFilePath, $iLine = Default, $sText = "", $iFlags = 0) in the end $iLine = Default, but i think is better the _FileWriteToLine($sFilePath, $sText, $iLine = Default, $iFlags = 0)

 

I was also thinking that (similar to FileWriteLine() if no @LF, @CR or @CRLF is at the end of $sText then append, otherwise leave as is.

it is so?

after _FileWriteToLineEx(@DesktopDir & 'Test.inf', ';Comment Line')

The only issue with the version above is r and converted to rn, which could be a problem for a user who has file that use @CR as the EOL character.

in that case I think should be so (I hate the regexp in these cases "StringRegExp($sData, 'A[^rn]*+r(?!n)')", even if it only reads a  first line, it takes time as if he is reading the entire file hmmmmm)

expandcollapse popup

#include <FileConstants.au3>
#include <StringConstants.au3>
 
; #CONSTANTS# ====================================================
#Region ;**** Global vars for _FileWriteToLineEx
Global Const $FRL_APPEND     = 0
Global Const $FRL_OVERWRITE  = 1
Global Const $FRL_DELETE     = 2
Global Const $FRL_OVERWRITE_DELETE  = BitOR($FRL_OVERWRITE, $FRL_DELETE) ;INTERNALLY USE ONLY
#EndRegion ;**** Global vars for _FileWriteToLineEx
; ================================================================
 
$fTimerDiff = TimerInit()
_FileWriteToLineEx(@DesktopDir & '\Test.inf', ';Comment Line')
If @Error Then ConsoleWrite(@error & @CRLF)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @CRLF)
 
 
; #FUNCTION# ====================================================================================================================
; Name...........: _FileWriteToLineEx
; Description ...: Edit a specific line in a file.
; Syntax.........: _FileWriteToLineEx($sFilePath, $sText[, $iLine[, $iFlags]] )
; Parameters ....: $sFilePath - The File Path
;                  $sText     - The text to write
;                  $iLine     - The line number to edit to (Default edit last line), Use a negative occurrence to search line from the right side
;                   $iFlags    - Optional, Flag to indicate the type of stripping that should be performed
;                  |$FRL_APPEND (0)    - (Default) Append mod
;                  |$FRL_OVERWRITE (1) - overwrite the old line
;                  |$FRL_DELETE (2)    - delete\remove the line
; Return values .: If the function succeeds, it returns TRUE; otherwise, FALSE.
;                  @error  - 0 = No error
;                  |1 = File does not exist
;                  |2 = File has less lines than $iLine
;                  |3 = Error when opening file, should be ERROR_ACCESS_DENIED = 5 (0x5) Access is denied. ($sFilePath is ReadOnly or has not Security Access Rights)
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileWriteToLineEx($sFilePath, $sText, $iLine = Default, $iFlags = 0)
    If Not FileExists($sFilePath) Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $EOL, $iOffSet, $sData = FileRead($sFilePath)
    $EOL = StringRegExp($sData, '\A[^\r\n]*+\r(?!\n)') ? @CR : @LF
    If $iLine = 1 Then
        $iOffSet = BitAND($iFlags, $FRL_OVERWRITE_DELETE) ? 0 : StringInStr($sData, $EOL, $STR_CASESENSE, 1)
    Else
        $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, ($iLine < 0 ? $iLine : $iLine - 1))
        If Not $iOffSet Then Return SetError(2, 0, False)
    EndIf
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sData)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
        ;; or more understandable
        ;;$_iOffSet -= 1
        ;;If $EOL == @LF And StringMid($sData, $_iOffSet -1, 1) == @CR Then $_iOffSet -= 1
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, False)
    FileWrite($hFileOpen, StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet))
    FileClose($hFileOpen)
    Return True
EndFunc   ;==>_FileWriteToLineEx

Ciao.

Edited January 9, 2014 by DXRW4E

[DXRW4E]

here, i think it is better to use "StringMid($sData, StringInStr($sData, @CR, $STR_CASESENSE, 1) + 1, 1)" should be much more faster than RegExp "StringRegExp($sData, 'A[^rn]*+r(?!n)')" in large files

 

expandcollapse popup

#include <FileConstants.au3>
#include <StringConstants.au3>

; #CONSTANTS# ====================================================
#Region ;**** Global vars for _FileWriteToLineEx
Global Const $FRL_APPEND     = 0
Global Const $FRL_OVERWRITE  = 1
Global Const $FRL_DELETE     = 2
Global Const $FRL_OVERWRITE_DELETE  = BitOR($FRL_OVERWRITE, $FRL_DELETE) ;INTERNALLY USE ONLY
#EndRegion ;**** Global vars for _FileWriteToLineEx
; ================================================================

$fTimerDiff = TimerInit()
_FileWriteToLineEx(@DesktopDir & '\Test.inf', ';Comment Line')
If @Error Then ConsoleWrite(@error & @CRLF)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @CRLF)


; #FUNCTION# ====================================================================================================================
; Name...........: _FileWriteToLineEx
; Description ...: Edit a specific line in a file.
; Syntax.........: _FileWriteToLineEx($sFilePath, $sText[, $iLine[, $iFlags]] )
; Parameters ....: $sFilePath - The File Path
;                  $sText     - The text to write
;                  $iLine     - The line number to edit to (Default edit last line), Use a negative occurrence to search line from the right side
;                   $iFlags    - Optional, Flag to indicate the type of stripping that should be performed
;                  |$FRL_APPEND (0)    - (Default) Append mod
;                  |$FRL_OVERWRITE (1) - overwrite the old line
;                  |$FRL_DELETE (2)    - delete\remove the line
; Return values .: If the function succeeds, it returns TRUE; otherwise, FALSE.
;                  @error  - 0 = No error
;                  |1 = File does not exist
;                  |2 = File has less lines than $iLine
;                  |3 = Error when opening file, should be ERROR_ACCESS_DENIED = 5 (0x5) Access is denied. ($sFilePath is ReadOnly or has not Security Access Rights)
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileWriteToLineEx($sFilePath, $sText, $iLine = Default, $iFlags = 0)
    If Not FileExists($sFilePath) Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $EOL, $iOffSet, $sData = FileRead($sFilePath)
    ;;$EOL = StringRegExp($sData, '\A[^\r\n]*+\r(?!\n)') ? @CR : @LF    ;is Ok
    $EOL = StringMid($sData, StringInStr($sData, @CR, $STR_CASESENSE, 1) + 1, 1) = @LF ? @LF : @CR ;should be much more faster in large files
    If $iLine = 1 Then
        $iOffSet = BitAND($iFlags, $FRL_OVERWRITE_DELETE) ? 0 : StringInStr($sData, $EOL, $STR_CASESENSE, 1)
    Else
        $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, ($iLine < 0 ? $iLine : $iLine - 1))
        If Not $iOffSet Then Return SetError(2, 0, False)
    EndIf
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sData)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
        ;; or more understandable
        ;;$_iOffSet -= 1
        ;;If $EOL == @LF And StringMid($sData, $_iOffSet -1, 1) == @CR Then $_iOffSet -= 1
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, False)
    FileWrite($hFileOpen, StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet))
    FileClose($hFileOpen)
    Return True
EndFunc   ;==>_FileWriteToLineEx

Ciao. Edited January 9, 2014 by DXRW4E

[guinness]

Try...

When $sText is ';Comment Line' & @CRLF.

Workaround...

$sText = ';Comment Line' & @CRLF
If Not StringRegExp($sText, '\R$') Then $sText &= @CRLF ; CRLF by default.

[DXRW4E]

Try...

When $sText is ';Comment Line' & @CRLF.

Workaround...

$sText = ';Comment Line' & @CRLF
If Not StringRegExp($sText, '\R$') Then $sText &= @CRLF ; CRLF by default.

I do not understand, you want to add it ?, that is not correct, because if you choose not to cancel but to OVERWRITE last line, if the line and empty after always adds another line (_FileWriteToLine can be executed in a loop, if you will write default last line, imagine what will happen after them, could try to write several times the same line), however in every possible scenario it is not correct that is a serious BUG, last line will not have to add never @CRLF

 

When $sText is ';Comment Line' & @CRLF.

if not after is not more last line, like this, this is not the last line

_FileWriteToLine does not penalize any kind of datatext, remains the user to choose whether or not to add @CRLF ect ect

Ciao.

Edited January 9, 2014 by DXRW4E