Re-write of _FileWriteToLine()

[Tlem]

You are not just with me. You talk always about XP, but what I talk about is not for XP only.

Like I show you, on Windows Seven and Windows 8 the problem is still present.

Install a x32 version in a VM on your PC and see it.

You probably go to tell me not to use any more x32 system, but for the moment and probably a long time, there are lot of cases where a x32 system must be kept because of material or software incompatibilities.

 

and use it also in Windows 95 with 32 MB RAM if you want

Stop laughing at me and speaking always about Windows XP or older OS's.

THIS PROBLEM IS NOT ONLY ON WINDOWS XP IT'S A WIN32 PROBLEM!!!!

Making this function using buffer is not for satisfied me, it's just that writing a function in a right way to make it compatible for everybody is more professional.

 

You're talking about losing 30% of speed, but did you compare with the actual version?

Like I have read on other topic :

 

Leave out your opinion on how you think it should work because your opinion probably won't matter to the devs. Second leave out your little bits about how the performance is increased by 60-70%. Thats just a bonus. For the most part they don't care about performance enhancements. Anything that even starts to get close to the realm of speed is the root of all evil seems to to get shot down.

So I'm not sure that the great work which you made will be considered...

It's like _FileCountLine(). Someone (jchd) have made a very good work but nobody consider it to be the best choice.

I hope sincerely that certain developers will read attentively this subject, because I do not certainly intend to argue so for every external function create with feet. The shortest route is not still the best, yes, it is faster and less complicated, but not necessarily the best solution. ^^

I am really sorry to have pollued this subject, but I think sincerely that there are efforts which are not vain.

It has been a long time since I speak about this problem of file size and things move forward little by little but they move. ^^

Edited January 12, 2014 by Tlem

[DXRW4E]

You are not just with me. You talk always about XP, but what I talk about is not for XP only.

@Tlem

there is nothing wrong with the function I think there are limits, it seems more than that is not possible, I doubt that other function can never do better, the StringInStr works byref I think logically serve them only a few Kb of memory to read also files of 1 GB, no complications reads character by character

still do not understand why this function, will have to have a special behavior regarding Memory allocation (I repeat this is the function that takes less memory than all other "removing file size, that will have to be loaded into memory", if we take other functions that use regexp or stringreplace or other, are 10000000 times heavier than this ehhhhhh), Memory allocation is a general problem

in all my posts I was referring to _FileWriteToLineEx() & StringInStr, I have not ever said that regexp is not invasive, or that could create problems with memory (is already reported in Autoit Help, that need to be careful with regexp)

Making this function using buffer is not for satisfied me

that was a joke

Ciao.

PS: Window XP is not old, but it is very very old, 13 years have passed ehhhh

Edited January 12, 2014 by DXRW4E

[Tlem]

You are pulling my leg with Windows XP. Maybe I should certainly have begun by speaking about Windows 7 x32.

In all my posts I was referring to memory allocation error when using FileRead($data) with a big quantity of datas !

I have nothing against you or this function in particular. It's just that guinness has launched the fishhook and that I bit!

[Tlem]

@DXRW4E

Sorry not to have made a report earlier.

I have tested functions on a 167066.69 Kb and 148466.69 file.

If I test original _FileWriteToLine before your _FileWriteToLineEx function, then the memory allocation error appear.

If I do the test with your last work that I have called _FileWriteToLineExWithBuffer (for convenience) then all run well.

If I decrease the size of the file for doing test on all 3 functions, then this is what I have in my Scite console for result testing :

14:02:56 : Starting _FileWriteToLine testing.
End creating test file of 111266.69 Kb
14:03:06 : Start testing original _FileWriteToLine
14:03:35 : End testing original _FileWriteToLine
28.96 seconds for original _FileWriteToLine
Pausing for 2 seconds.
14:03:37 : Start testing _FileWriteToLineEx from DXRW4E
14:03:46 : End testing _FileWriteToLineEx from DXRW4E
8.41 seconds for _FileWriteToLineEx from DXRW4E
Pausing for 2 seconds.
14:03:48 : Start testing _FileWriteToLineExWithBuffer from DXRW4E
14:03:58 : End testing _FileWriteToLineExWithBuffer from DXRW4E
10.18 seconds for _FileWriteToLineExWithBuffer from DXRW4E

If I increase the size for playing with limits and compare only your 2 last functions, this is what I have :

14:06:10 : Starting _FileWriteToLine testing.
End creating test file of 167066.69 Kb
14:06:22 : Start testing _FileWriteToLineEx from DXRW4E
14:06:35 : End testing _FileWriteToLineEx from DXRW4E
13.05 seconds for _FileWriteToLineEx from DXRW4E
Pausing for 2 seconds.
14:06:37 : Start testing _FileWriteToLineExWithBuffer from DXRW4E
14:06:51 : End testing _FileWriteToLineExWithBuffer from DXRW4E
14.64 seconds for _FileWriteToLineExWithBuffer from DXRW4E
Pausing for 2 seconds.

About 1.6 seconds of difference for a 167Mb file !!!

But now, even if it's crazy, we can do that :

14:12:08 : Starting _FileWriteToLine testing.
End creating test file of 618888.96 Kb
14:12:49 : Start testing _FileWriteToLineExWithBuffer from DXRW4E
14:13:44 : End testing _FileWriteToLineExWithBuffer from DXRW4E
54.47 seconds for _FileWriteToLineExWithBuffer from DXRW4E

If I decrease the buffer size, this is what I get :

15:19:24 : Starting _FileWriteToLine testing.
End creating test file of 167066.69 Kb
Pausing for 10 seconds.
15:19:46 : Start testing _FileWriteToLineEx from DXRW4E
15:19:59 : End testing _FileWriteToLineEx from DXRW4E
13.38 seconds for _FileWriteToLineEx from DXRW4E
Pausing for 10 seconds.
15:20:09 : Start testing _FileWriteToLineExWithBuffer from DXRW4E (Buffer 10Mb).
15:20:23 : End testing _FileWriteToLineExWithBuffer from DXRW4E (Buffer 10Mb).
13.45 seconds for _FileWriteToLineExWithBuffer from DXRW4E (Buffer 10Mb).
Pausing for 10 seconds.

0.07 seconds of difference !!!

That's what I call a professional coding.

I think that a lot of people could thank you, beginning by me.

Hope that it will be doing for all external functions. ^^

Just for those how want to know more information about cpu and memory usage for each function, this is a test capture :

Pink : Testing file creation

Blue : Actual _FileWriteToLine function

Yellow : _FileWriteToLine function ameliorated by DXRW4E (increase speed).

Green : _FileWriteToLine function ameliorated by DXRW4E too (increase speed and taking care of large file).

In conclusion, I would say that if Dev chooses to keep the old processing scheme, it's that something not running on the good way!

[DXRW4E]

Hi Tlem, there's nothing genius them, everything is based on simplicity, I mean is a code that can write (really) anyone, even users with little experience

in the end the logic used is simple, read & check & write, before everything was done in one step, but now it is done in more steps, but the FOR NEXT loop is fast so the differences are really small

Ciao.

Edited January 12, 2014 by DXRW4E

[Tlem]

I never said that's genius. I only said that it's the professional would only sais that it's the more universal and the most conscientious way to write this type of function (cause using FileRead).

Edited January 12, 2014 by Tlem

[Tlem]

Hello again.

I have playing with the last function on multiple OS's : XP, Vista x64, Seven x32 and x64, 8 x32 and x64

I have testing on different file size : 300Ko, 3Mo, 30 Mo and 300Mo with different buffer size 1Mo, 8Mo, 10Mo, 50Mo, 100Mo

This is what I can said about all of that :

1st : Playing with buffer size greater than what old OS's can do isn't interesting. So buffer size 200, 500, 1000, and 2000 can be omitted.

2nd : Buffer size 50 and 100 increase treatment time. So you can omitted too.

3rd : Between buffer size of 1Mb, 8Mb and 10Mb, there's no significant differences. It seems that 1Mb buffer size use more CPU, but not all the time.8Mb and 10Mb are pretty similar. Maybe 8Mb is more powerful for standard files, but I repeat that the gap is really insignificant (about 1 to 4 seconds for a 300Mo file). I have read somewhere that for reading file with buffering, 8Mb is a good compromise. So why not.

This is what I recommend for finalize this function :

- Use by default 8Mb for buffering and delete all value greater than 10Mb (no gain). Maybe not using flag for $iBuffer and $iLineSize (not sure that this can offer a sensible gain).

- Re-attribute value of $iFlags like older version (0 add to line - 1 Overwrite line) to keep compatibility.

@DXRW4E

This is my thin contribution :

expandcollapse popup

#include <FileConstants.au3>
#include <StringConstants.au3>

; #CONSTANTS# ====================================================
#Region ;**** Global vars for _FileWriteToLine
Global Const $FRL_OVERWRITE = 1
Global Const $FRL_NOEOL = 2
Global Const $FRL_DELETE = 4
Global Const $FRL_BUFFER_0 = 1024 ; 1 Kb
Global Const $FRL_BUFFER_1 = 1048576 ; 1 MB
Global Const $FRL_BUFFER_8 = 8388608 ; 8 MB
Global Const $FRL_BUFFER_10 = 10485760 ; 10 MB
Global Const $FRL_DEFBUFFER = $FRL_BUFFER_8
Global Const $FRL_USE_LINEFEED = 8
Global Const $FRL_SZTEMPPATH = @TempDir & "\TMP_" & Random(11111, 99999, 1) & "_.tmp"
Global Const $FRL_OVERWRITE_DELETE = BitOR($FRL_OVERWRITE, $FRL_DELETE) ;INTERNALLY USE ONLY
Global Const $FRL_NOEOL_DELETE = BitOR($FRL_NOEOL, $FRL_DELETE) ;INTERNALLY USE ONLY
#EndRegion ;**** Global vars for _FileWriteToLine
; ================================================================


; #FUNCTION# ====================================================================================================================
; Name...........: _FileWriteToLine
; Description ...: Edit a specific line in a file.
; Syntax.........: _FileWriteToLine($sFilePath, $iLine, $sText[, $iFlags] )
; Parameters ....: $sFilePath - The File Path
;                  $iLine     - The line number to edit to
;                  $sText     - The text to write
;                  $iFlags    - Optional, Flag to indicate the type of stripping that should be performed
;                  |  Default FileWriteLine Mod, The line of text to write to the text file. If the line does NOT end in @CR or @LF
;                  |   then a DOS linefeed (@CRLF) will be automatically added.
;                  |$FRL_OVERWRITE (1)    - overwrite the old line
;                  |$FRL_NOEOL     (2)    - Disable check for @CR or @LF (do not add @CR or @LF if there are not present)
;                  |$FRL_DELETE    (4)    - delete\remove the line
;                  $iBuffer    - Optional, Buffer Size used, Default 8 MB
;                  $iLineSize  - Optional, Line Size to check for Default EOL, Default 1 MB
; Return values .: If the function succeeds, it returns TRUE (Check @extended for iLineOffset); otherwise, FALSE.
;                  @error  - 0 = No error
;                  |-n = Error when FileClose (will not have to ever happen)
;                  |1 = Line Number invalid
;                  |2 = File does not exist
;                  |3 = Error when opening file, should be ERROR_ACCESS_DENIED = 5 (0x5) Access is denied. ($sFilePath is ReadOnly or has not Security Access Rights)
;                  |4 = File has less lines than $iLine
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileWriteToLine($sFilePath, $iLine, $sText, $iFlags = 0, $iBuffer = $FRL_DEFBUFFER, $iLineSize = $FRL_BUFFER_1)
    If $iLine < 1 Then Return SetError(1, 0, False)
    If Not FileExists($sFilePath) Then Return SetError(2, 0, False)
    Local $hFileOpen = FileOpen($sFilePath, $FO_READ), $htFileOpen = FileOpen($FRL_SZTEMPPATH, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Or $htFileOpen = -1 Then Return SetError(3, FileClose($hFileOpen) + FileClose($htFileOpen), False)
    If $iBuffer < $FRL_BUFFER_1 Then $iBuffer = $FRL_BUFFER_1
    If $iLineSize < 3 Then $iLineSize = $FRL_BUFFER_0
    Local $sData, $iOffSet = 0, $_iOffSet, $EOL = @LF, $_iBuffer = Ceiling(FileGetSize($sFilePath) / $iBuffer)
    $EOL = StringInStr(FileRead($sFilePath, $iLineSize), @LF, $STR_CASESENSE, 1) ? @LF : @CR ;;@LF has priority
    If $iLine > 1 Then
        $iLine -= 1
        For $i = 1 To $_iBuffer
            $sData = FileRead($hFileOpen, $iBuffer)
            For $y = 1 To $iLine
                $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
                If Not $iOffSet Then ExitLoop
                $iLine -= 1
            Next
            If $iOffSet Then
                FileWrite($htFileOpen, StringLeft($sData, $iOffSet))
                $sData = StringTrimLeft($sData, $iOffSet)
                $iOffSet += ($i - 1) * $iBuffer
                ExitLoop
            Else
                FileWrite($htFileOpen, $sData)
            EndIf
        Next
        If Not $iOffSet Then Return SetError(4, FileClose($hFileOpen) + FileClose($htFileOpen) + FileDelete($FRL_SZTEMPPATH), False)
    EndIf
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE)
    If Not $_iOffSet And StringLen($sData) < $iLineSize Then
        $sData &= FileRead($hFileOpen, $iLineSize)
        $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE)
    EndIf
    If Not $_iOffSet Then
        If Not BitAND($iFlags, $FRL_DELETE) Then
            If Not BitAND($iFlags, $FRL_OVERWRITE) Then FileWrite($htFileOpen, $sData)
            If Not BitAND($iFlags, $FRL_NOEOL) And StringRight($sText, 1) <> $EOL Then $sText &= ($EOL = @CR ? @CR : @CRLF)
            FileWrite($htFileOpen, $sText)
        EndIf
        Return SetError(FileClose($hFileOpen) + FileClose($htFileOpen) - 2, $iOffSet, FileMove($FRL_SZTEMPPATH, $sFilePath, $FC_OVERWRITE) = 1)
    ElseIf BitAND($iFlags, $FRL_DELETE) Then
        FileSetPos($hFileOpen, $_iOffSet, $FILE_CURRENT)
        FileWrite($htFileOpen, StringTrimLeft($sData, $_iOffSet))
    Else
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
        If Not BitAND($iFlags, $FRL_OVERWRITE) Then FileWrite($htFileOpen, StringLeft($sData, $_iOffSet))
        FileWrite($htFileOpen, $sText & StringTrimLeft($sData, $_iOffSet))
    EndIf
    While 1
        $sData = FileRead($hFileOpen, $iBuffer)
        If Not $sData Then ExitLoop
        FileWrite($htFileOpen, $sData)
    WEnd
    Return SetError(FileClose($hFileOpen) + FileClose($htFileOpen) - 2, $iOffSet, FileMove($FRL_SZTEMPPATH, $sFilePath, $FC_OVERWRITE) = 1)
EndFunc   ;==>_FileWriteToLine

[DXRW4E]

Hi Tlem, Thanks for reporting, you're right about $FRL_OVERWRITE & $FRL_NOEOL i had not paid attention to those, the post #98 already updated

$FRL_BUFFER_* have to give more info or give an idea to the users, i do not think anyone will use $FRL_BUFFER_2000 or $FRL_BUFFER_1000 or $FRL_BUFFER_500 ehhh, however see the

;;Global Const $FRL_DEFBUFFER    = (MemGetStats())[1] < 3404884 ? $FRL_BUFFER_100 : $FRL_BUFFER_500 ;;3404884 = 4 GB RAM
;;Global Const $FRL_DEFBUFFER    = (MemGetStats())[1] < 256 MB ram ? $FRL_BUFFER_8 : $FRL_BUFFER_100

is the one to use by default (certainly serves updateedit it, is just to give an example), based on the real system where function will work, that's is perfect , however remains guinness to see them and decide what is best

$iLineSize flag is to keep options open, I think no one will use it, but better say so, if users want to  will be able to use it

Ciao.

Edited January 13, 2014 by DXRW4E

[guinness]

I have only seen post 85 as potential. Which other one should I consider?

[DXRW4E]

Hi guinness, i recommend (not to say that it seems that by force will have to be the one) that the default function be the one in post #98

Ciao.

Edited January 13, 2014 by DXRW4E

[Tlem]

Hem, hem, DXRW4E sorry to disturb this topic again, but I would like to add some informations about the code you put in post #98.

You show testing between 3 functions, but one of them does not work in the same way !

_FileWriteToLineEx() does not write directly in file. So testing like it is showing is not impartial!

This function is incredibly faster for multiple treatments in a single pass (if you have to change/add/delete multiple lines in a file) because you read data in memory and do treatments directly before writing file. So it's a good thing to keep it, but be careful of the amount of data stored in $Data ... ^^

On the other hand, the syntax of the first two functions is still not identical to the current one : FonctionName($sFile, $iLine, $sText)

Edited January 13, 2014 by Tlem

[DXRW4E]

Hi Tlem, idea is that (have the opportunity) to work from memory, is that the real difference from FileWriteLine

Ciao.

[Tlem]

I had perfectly understood, but guinness have plan to rewrite original function, not to change its functioning. So if you do like that, it will be a script breaking and I doubt that guinness will be agree.

[guinness]

Though I applaud the discussion that has taken place here and some amazing work has happened as a result, I feel that bloating code for the "just in case" issue of large files is perhaps the wrong thing to do. I am all for efficiency, but no one has posted a Trac Ticket that "...using _FileWriteToLine() doesn't work on files of xxxMB" since it was added to the UDF library in 2006. So this leads me to believe people are using it on documents of a respectable size and as was intended. Therefore I will go with the code in post #85 and amend accordingly.

Thank you all.