Jump to content
Sign in to follow this  
guinness

Re-write of _FileWriteToLine()

Recommended Posts

my English does not allow me to explain myself better.

Same thing for me.

I point just 2 things :

1 - Memory allocation error for large files (size depending on the system you are) that can be solved if we add extra code.

2 - Deletion of original file if crash append on function treatment (this part can be optimized).

If that 2 points are not critical, the new proposal made is welcom. ;)

@guinness

Yes, actual function has its issues. But you plan to re-wrote it, so we can keep it's issue .. Or not. ^^

From there, things are not any more within my competence.


Best Regards.Thierry

Share this post


Link to post
Share on other sites

still do not understand why this function, will have to have a special behavior regarding Memory allocation (I repeat this is the function that takes less memory than all other "removing file size, that will have to be loaded into memory", if we take other functions that use regexp or stringreplace or other, are 10000000 times heavier than this ehhhhhh), Memory allocation is a general problem, or say more good if I'm on windows XP with 256 Mb RAM memory, I obviously can not work with programs that load most of 300 Mb RAM ehhh, of course i go to the error ehhhh

however is easy just need to use the http://www.autoitscript.com/autoit3/docs/functions/MemGetStats.htm and after calculating file size and totalfree memory, so you'll know when that file is supported or not

certainly the professional software before installation perform certain controls, if a system that support & CPU & RAM & ect ect, man i do not think this function ehhhhh you know ehhhh

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

after not only the memory but also the system itself, I see that archive (large files) with 7zip (in the same PC) is almost 2 times faster in Windows 7 than in Windows XP, Windows XP (EVEN IF HE WILL BE THE BEST SYSTEM EVER) is old ehhh instead Windows NT6.x 7/8/Next interpret more better many things (that affect the performance and many other things)

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

however I believe there is always a way to work around certain limitations, such as using a code like this, I do not know written on the run, need to test it (not tested yet, maybe it has bugs inside), but idea is that

Func _FileWriteToLineEx($sFilePath, $sText, $iLine = Default, $iFlags = 0)
    If Not FileExists($sFilePath) Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $EOL, $iOffSet = 0, $_iOffSet, $_sData, $sData = FileRead($sFilePath)
    ;;$EOL = StringMid($sData, StringInStr($sData, @CR, $STR_CASESENSE, 1) + 1, 1) = @LF ? @LF : @CR    ;;@CR has priority
    $EOL = StringInStr($sData, @LF, $STR_CASESENSE, 1) ? @LF : @CR  ;;@LF has priority
    If $iLine > 100001 Then
        For $i = 100000 To $iLine Step 100000
            $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 100000)
            If Not $_iOffSet Then Return SetError(2, 0, False)
            $iLine -= 100000
            MsgBox(0, $iLine, $_iOffSet)
            $_sData &= StringLeft($sData, $_iOffSet)
            $sData = StringTrimLeft($sData, $_iOffSet)
        Next
        $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, $iLine - 1)
        $sData = $_sData & $sData
        $_sData = ""
    ElseIf $iLine <> 1 Then
        $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, ($iLine < 0 ? $iLine : $iLine - 1))
    EndIf
    If Not $iOffSet Then Return SetError(2, 0, False)
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sData)
        If Not BitAND($iFlags, $FRL_NOEOL_DELETE) And StringRight($sText, 1) <> $EOL Then $sText &= ($EOL = @CR ? @CR : @CRLF)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, False)
    FileWrite($hFileOpen, StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet))
    FileClose($hFileOpen)
    Return True
EndFunc   ;==>_FileWriteToLineEx

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

@All

leaving the problem of RAM installed in the system (which is a problem that does not regards this function or autoit, but regards all the software), everything else is OKK, the StringInStr never stops, has no limit, here is a simple test, everyone will have to do a test like this
 

#include <FileConstants.au3>
#include <StringConstants.au3>

; #CONSTANTS# ====================================================
#Region ;**** Global vars for _FileWriteToLineEx
Global Const $FRL_OVERWRITE  = 1
Global Const $FRL_NOEOL      = 2
Global Const $FRL_DELETE     = 4
Global Const $FRL_OVERWRITE_DELETE  = BitOR($FRL_OVERWRITE, $FRL_DELETE) ;INTERNALLY USE ONLY
Global Const $FRL_NOEOL_DELETE  = BitOR($FRL_NOEOL, $FRL_DELETE) ;INTERNALLY USE ONLY
#EndRegion ;**** Global vars for _FileWriteToLineEx
; ================================================================

FileDelete(@DesktopDir & '\Test.txt')
Local $sData, $Line = " = This is Test for _FileWriteToLineEx Function" & @CRLF
For $i = 1 to 6
    For $y = 1 To 1000000
        $sData &= "Line - " & $y & $Line
    Next
    FileWrite(@DesktopDir & '\Test.txt', $sData) ; 57 MB
    $sData = ""
Next
;; @DesktopDir & '\Test.txt' - 354 MB 6.000.000 Line
$fTimerDiff = TimerInit()
_FileWriteToLineEx(@DesktopDir & '\Test.txt', ';This is Comment Line 1',5999997)
_FileWriteToLineEx(@DesktopDir & '\Test.txt', ';This is Comment Line 2',5999998, $FRL_OVERWRITE)
_FileWriteToLineEx(@DesktopDir & '\Test.txt', '', 5999999, $FRL_DELETE)
ConsoleWrite("iLineOffSet - " & @Extended & @CRLF)
If @Error Then ConsoleWrite("!@Error - " & @error & @CRLF)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @CRLF)
Exit

; #FUNCTION# ====================================================================================================================
; Name...........: _FileWriteToLineEx
; Description ...: Edit a specific line in a file.
; Syntax.........: _FileWriteToLineEx($sFilePath, $sText[, $iLine[, $iFlags]] )
; Parameters ....: $sFilePath - The File Path
;                  $sText     - The text to write
;                  $iLine     - The line number to edit to (Default edit last line), Use a negative occurrence to search line from the right side
;                  $iFlags    - Optional, Flag to indicate the type of stripping that should be performed
;                  |  Default FileWriteLine Mod, The line of text to write to the text file. If the line does NOT end in @CR or @LF
;                  |   then a DOS linefeed (@CRLF) will be automatically added.
;                  |$FRL_OVERWRITE (1) - overwrite the old line
;                  |$FRL_NOEOL     (2) - Disable check for @CR or @LF (do not add @CR or @LF if there are not present)
;                  |$FRL_DELETE    (4) - delete\remove the line
; Return values .: If the function succeeds, it returns TRUE (Check @extended for iLineOffset); otherwise, FALSE.
;                  @error  - 0 = No error
;                  |1 = File does not exist
;                  |2 = File has less lines than $iLine
;                  |3 = Error when opening file, should be ERROR_ACCESS_DENIED = 5 (0x5) Access is denied. ($sFilePath is ReadOnly or has not Security Access Rights)
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileWriteToLineEx($sFilePath, $sText, $iLine = Default, $iFlags = 0)
    If Not FileExists($sFilePath) Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $EOL, $iOffSet = 0, $_iOffSet, $sData = FileRead($sFilePath)
    ;;$EOL = StringMid($sData, StringInStr($sData, @CR, $STR_CASESENSE, 1) + 1, 1) = @LF ? @LF : @CR    ;;@CR has priority
    $EOL = StringInStr($sData, @LF, $STR_CASESENSE, 1) ? @LF : @CR  ;;@LF has priority
    If $iLine <> 1 Then
        $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, ($iLine < 0 ? $iLine : $iLine - 1))
        If Not $iOffSet Then Return SetError(2, 0, False)
    EndIf
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sData)
        If Not BitAND($iFlags, $FRL_NOEOL_DELETE) And StringRight($sText, 1) <> $EOL Then $sText &= ($EOL = @CR ? @CR : @CRLF)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, False)
    FileWrite($hFileOpen, StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet))
    FileClose($hFileOpen)
    Return SetError(0, $iOffSet, True)
EndFunc   ;==>_FileWriteToLineEx


Func _FileWriteToLineB(ByRef $sData, $sText, $iLine = Default, $iFlags = 0)
    If Not $sData Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $EOL, $iOffSet = 0, $_iOffSet
    $EOL = StringInStr($sData, @LF, $STR_CASESENSE, 1) ? @LF : @CR  ;;@LF has priority
    If $iLine <> 1 Then
        $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, ($iLine < 0 ? $iLine : $iLine - 1))
        If Not $iOffSet Then Return SetError(2, 0, False)
    EndIf
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sData)
        If Not BitAND($iFlags, $FRL_NOEOL_DELETE) And StringRight($sText, 1) <> $EOL Then $sText &= ($EOL = @CR ? @CR : @CRLF)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    $sData = StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet)
    Return SetError(0, $iOffSet, True)
EndFunc   ;==>_FileWriteToLineB
;~ >"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\DXRW4E\Desktop\_FileWriteToLineEx.au3" /UserParams    
;~ +>14:44:11 Starting AutoIt3Wrapper v.2.1.4.0 SciTE v.3.3.7.0 ;  Keyboard:00000409  OS:WIN_81/  CPU:X64 OS:X64    Environment(Language:0409  Keyboard:00000409  OS:WIN_81/  CPU:X64 OS:X64)
;~ >Running AU3Check (3.3.10.2)  from:C:\Program Files (x86)\AutoIt3
;~ +>14:44:12 AU3Check ended.rc:0
;~ >Running:(3.3.10.2):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\DXRW4E\Desktop\_FileWriteToLineEx.au3"    
;~ --> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
;~ iLineOffSet - 371333237
;~ 99448.3650223751
;~ +>14:45:51 AutoIt3.exe ended.rc:0
;~ >Exit code: 0    Time: 100.507

 

look imagine
the Line - 5999997 was written - Line - 5999997 = This is Test for _FileWriteToLineEx Function;This is Comment Line 1
the Line - 5999998 was overwrite - ;This is Comment Line 2
the Line - 5999999 was deleted - Line - 5999999 = This is Test for _FileWriteToLineEx Function Does not exist
 
 

jztn.png

 
 
@guinness
there just a detail, you have to improve it, perhaps need to ask Jon, who's the best way for large files

FileWrite($hFileOpen, StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet))
;or
FileWrite($hFileOpen, StringLeft($sData, $iOffSet))
FileWrite($hFileOpen, (BitAND($iFlags, $FRL_DELETE) ? "" : $sText))
FileWrite($hFileOpen, StringTrimLeft($sData, $_iOffSet))
;or
FileWrite($hFileOpen, StringLeft($sData, $iOffSet))
$sData = StringTrimLeft($sData, $_iOffSet)
If Not BitAND($iFlags, $FRL_DELETE) Then FileWrite($hFileOpen, $sText)
FileWrite($hFileOpen, $sData)

because she gives me the impression that the first mode, when attempting to write normally have to create another variable in memory, the variable $sVar1 + $svar2 + $svar3 and this is a problem multiply memory used, perhaps better to use several times the firewrite as in the second mode, my impression is the 3rd mode take less memory, but I do not know, I have no idea, see yourself what better to do

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

Tested on 2 other XP machines.

See informations attached.

These machine are professional machine installed where I work and I'm sure that they don't change it tomorow for newer machines ...

post-19411-0-05966000-1389376930_thumb.j

post-19411-0-82125700-1389376934_thumb.j

post-19411-0-59546200-1389376941_thumb.j


Best Regards.Thierry

Share this post


Link to post
Share on other sites

Hi Tlem, what is this a joke ?? you can not open a file of 170 MB and after try to open a file of 350 MB ??, the answer you have already in the post above, more than one post ;), is a problem of your system (HW) and not of _FileWriteToLine(), its a problem for all programs, look what happens when I try to open the same file with notepad2, notepad2 that works with stream directly from the file ehhhh (we have served more than 3 minutes to stabilize)

4adr.png

 

slgp.png

 

however here in Windows XP with file 180 Mb (3.000.000 line) is everything Ok

yt7r.png

bye4.png

 

is very very simple you can not expectpretend to work (with file 300-400 Mb) in an old system in 2014 with 1 GB Ram, and expectpretend to have a good preformane (can not expectpretend to open Autocad 2014 or Visual Studio 2013 or do not know, and pretend that open in 1-2 seconds ehhhh), is needless to say more now

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

So on that note, which version DXRW4E are you suggesting for the UDFs? Post #85?


UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites
Hi guinness, Function are the same (Post #43 & Post #85) nothing has changed, just added to return offset in extended, i think it may be useful (to know the exact point where the line was added) to the user (for example how does regex), however i believe that in Post #85 is more better

Ciao.
Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

OK great. Thanks for all your hard work. I will be back with a proposal shortly.


UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Share this post


Link to post
Share on other sites

Hi guinness, I know that's is an old story and I'm sure which will return again @LF or @CR if the line is "aaa @CR & bbb @LF" ect ect, i think is better (since it does not cost anything) to add the flag $FRL_USE_LINEFEED (or $FRL_FORCE_LINEFEED do not know how can have more sense, since they will repair the line is the case if they are not OK)

#include <FileConstants.au3>
#include <StringConstants.au3>

; #CONSTANTS# ====================================================
#Region ;**** Global vars for _FileWriteToLineEx
Global Const $FRL_NOEOL        = 1
Global Const $FRL_OVERWRITE    = 2
Global Const $FRL_DELETE       = 4
Global Const $FRL_USE_LINEFEED = 8
Global Const $FRL_OVERWRITE_DELETE  = BitOR($FRL_OVERWRITE, $FRL_DELETE) ;INTERNALLY USE ONLY
Global Const $FRL_NOEOL_DELETE  = BitOR($FRL_NOEOL, $FRL_DELETE) ;INTERNALLY USE ONLY
#EndRegion ;**** Global vars for _FileWriteToLineEx
; ================================================================

;~ FileDelete(@DesktopDir & '\Test.txt')
;~ Local $sData, $Line = " = This is Test for _FileWriteToLineEx Function" & @CRLF
;~ For $i = 1 to 6
;~     For $y = 1 To 1000000
;~         $sData &= "Line - " & $y & $Line
;~     Next
;~     FileWrite(@DesktopDir & '\Test.txt', $sData) ; 57 MB
;~     $sData = ""
;~ Next
;; @DesktopDir & '\Test.txt' - 354 MB 6.000.000 Line
$fTimerDiff = TimerInit()
_FileWriteToLineEx(@DesktopDir & '\Test.txt', ';This is Comment Line 1',5999997)
_FileWriteToLineEx(@DesktopDir & '\Test.txt', ';This is Comment Line 2',5999998, $FRL_OVERWRITE)
_FileWriteToLineEx(@DesktopDir & '\Test.txt', '', 5999999, $FRL_DELETE)
ConsoleWrite("iLineOffSet - " & @Extended & @CRLF)
If @Error Then ConsoleWrite("!@Error - " & @error & @CRLF)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @CRLF)
Exit

; #FUNCTION# ====================================================================================================================
; Name...........: _FileWriteToLineEx
; Description ...: Edit a specific line in a file.
; Syntax.........: _FileWriteToLineEx($sFilePath, $sText[, $iLine[, $iFlags]] )
; Parameters ....: $sFilePath - The File Path
;                  $sText     - The text to write
;                  $iLine     - The line number to edit to (Default edit last line), Use a negative occurrence to search line from the right side
;                  $iFlags    - Optional, Flag to indicate the type of stripping that should be performed
;                  |  Default FileWriteLine Mod, The line of text to write to the text file. If the line does NOT end in @CR or @LF
;                  |   then a DOS linefeed (@CRLF) will be automatically added.
;                  |$FRL_NOEOL     (1)    - Disable check for @CR or @LF (do not add @CR or @LF if there are not present)
;                  |$FRL_OVERWRITE (2)    - overwrite the old line
;                  |$FRL_DELETE    (4)    - delete\remove the line
;                  |$FRL_USE_LINEFEED (8) - forceuse of Line Feed
; Return values .: If the function succeeds, it returns TRUE (Check @extended for iLineOffset); otherwise, FALSE.
;                  @error  - 0 = No error
;                  |1 = File does not exist
;                  |2 = File has less lines than $iLine
;                  |3 = Error when opening file, should be ERROR_ACCESS_DENIED = 5 (0x5) Access is denied. ($sFilePath is ReadOnly or has not Security Access Rights)
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileWriteToLineEx($sFilePath, $sText, $iLine = Default, $iFlags = 0)
    If Not FileExists($sFilePath) Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $EOL = @LF, $iOffSet = 0, $_iOffSet, $sData = FileRead($sFilePath)
    If BitAND($iFlags, $FRL_USE_LINEFEED) Then
        $sData = StringRegExpReplace($sData, "\r(?!\n)", @CRLF)
    Else
        ;;$EOL = StringMid($sData, StringInStr($sData, @CR, $STR_CASESENSE, 1) + 1, 1) = @LF ? @LF : @CR    ;;@CR has priority
        $EOL = StringInStr($sData, @LF, $STR_CASESENSE, 1) ? @LF : @CR    ;;@LF has priority
    EndIf
    If $iLine <> 1 Then
        $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, ($iLine < 0 ? $iLine : $iLine - 1))
        If Not $iOffSet Then Return SetError(2, 0, False)
    EndIf
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sData)
        If Not BitAND($iFlags, $FRL_NOEOL_DELETE) And StringRight($sText, 1) <> $EOL Then $sText &= ($EOL = @CR ? @CR : @CRLF)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, False)
    FileWrite($hFileOpen, StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet))
    FileClose($hFileOpen)
    Return SetError(0, $iOffSet, True)
EndFunc   ;==>_FileWriteToLineEx


Func _FileWriteToLineB(ByRef $sData, $sText, $iLine = Default, $iFlags = 0)
    If Not $sData Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $EOL = @LF, $iOffSet = 0, $_iOffSet
    If BitAND($iFlags, $FRL_USE_LINEFEED) Then
        $sData = StringRegExpReplace($sData, "\r(?!\n)", @CRLF)
    Else
        $EOL = StringInStr($sData, @LF, $STR_CASESENSE, 1) ? @LF : @CR    ;;@LF has priority
    EndIf
    If $iLine <> 1 Then
        $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, ($iLine < 0 ? $iLine : $iLine - 1))
        If Not $iOffSet Then Return SetError(2, 0, False)
    EndIf
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sData)
        If Not BitAND($iFlags, $FRL_NOEOL_DELETE) And StringRight($sText, 1) <> $EOL Then $sText &= ($EOL = @CR ? @CR : @CRLF)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    $sData = StringLeft($sData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sData, $_iOffSet)
    Return SetError(0, $iOffSet, True)
EndFunc   ;==>_FileWriteToLineB

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

Hello, I know that my intervention is not welcome, but well that I know that I should not continue, I have the duty to clarify certain things.

Hi Tlem, what is this a joke ??

No it is not a joke !!

is needless to say more now

That's your point of view. For the community and in all honesty, I can't be agree with you.

is very very simple you can not expectpretend to work (with file 300-400 Mb) in an old system in 2014 with 1 GB Ram

I think THAT is a joke !!  ^^

You can exaggerate things in your advantage, the fact remains that you can see on captures that one of the machine have a C2D processor and 3GB of Ram and for information, the memory allocation error begin after 180 Mb (not 300 Mb like you exaggerate) !!!
 

You can consider that a machine of 4/5 years old is an old machine, but everybody haven't the possibility to change their machine all 3/4 years !!!

The problem with certain "developers", it's because they often use recent machines. For them, all which in more than 4 years should not exist any more and if their code does not work on a "very old" machine, then it belongs to the user to update it.

Conditions of use are the ones that they (the Dev) defined and not those of the user!!!

Nevertheless, we speak about a "community product", but if the community is reduced to a team of developer and some fortunate initiated, I am certainly going to revise the definition of "community".

I can agree to a certain extent, but in this case you should see revise the list of announced compatibility. Yes Windows XP is a very old OS and his support will stop in few month but it is not a reason to forget it. Yes, reading a 180MB text file is not common, but it is not because you do not need that others not one not!

I am now going to try to end with others elements which are going to show you that the error of memory allocation error during the reading of a big file in a variable is not bound to Windows XP, the size of the memory, or the age of the machine. You can see it on joint images (this is the result of testing your code on W7 and W8 on a recent machine).

post-19411-0-05163700-1389438721_thumb.j

post-19411-0-05778500-1389438736_thumb.j

 

Yes I try it on Win32 OS arch and since the beginning, it's what I try to make understand, the way of coding certain functions is not adapted to Win32 arch. If certain developers decided not to make any more their code compatible on this arch, perhaps it's time to precise it in documentation function(s) or maybe it's time to tell to Microsoft to stop to produce Win32 OS's and finally to tell to everybody to change their machine for Win64 arch !!!


Best Regards.Thierry

Share this post


Link to post
Share on other sites

Hi Tlem

Hello, I know that my intervention is not welcome,

you are wrong a lot, you're super welcome for me

is really useless to continue the discussion, it seems that you have taken personally, I'm sorry but there's nothing personal, I say what I think you say what you think (really i do not pretend to know more than you, or smarter than you), said this because you mentioned advantages ect ect

You can exaggerate things in your advantage

I think everyone here do all for funhobby, here do not give trophies at the end of the year so ehhhh

 

the fact remains that you can see on captures that one of the machine have a C2D processor and 3GB of Ram and for information, the memory allocation error begin after 180 Mb (not 300 Mb like you exaggerate) !!!

you believe that i lied about my test in windows xp?, do you believe that i have edited with photoshop, the images above ???

I do not think you are lying about your test, simply do not agree with what you expect them

however is more important, I do not know what say more (As you can see we are already OffTopic ehhh), I hope you solve your problems there

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

As you can see we are already OffTopic ehhh

 

I don't think so.

The title of this topic is the re-write of _FileWriteToLine() function.

guinness have ask to test to see if he haven't messed anything up.

This function (and all others that use FileRead() directly in a variable) are only usable on file not greater than ~180 Mb on 32 Bit OS's. If this point is precised in documentation, I have nothing more to say.

 

 

simply do not agree with what you expect them

I just expect elimination of limitations that some people have made by coding functions without thinking of exceptional situations or take care about what users have to said.

AutoIt is an extraordinary simple langage, but putting limitations of its use is not that can be better for it.

As I can notice it once again, it is extremely difficult to make change things to return them simpler and universal.

I still wonder why I continue to intervene on this forum because every remark or attempt of improvement is subject in of multiple controversies or demolition.

Edit :

One example among others : '?do=embed' frameborder='0' data-embedContent>>

On actual version of AutoIt (3.3.10.2) the file size limitation still exist and the performances are still not improved in spite of /#entry958820'>the proposal of a MVP !!! The proposal improve >50% of speed and have no file size limitation but nevertheless, the old function is kept.

And more recently, the controversy on the >ternary operator.

Edited by Tlem

Best Regards.Thierry

Share this post


Link to post
Share on other sites

Hi Tlem, need to take it with more peace of mind ;)  you will need to understand that you're wrong, I told you a thousand times but it seems that you do not understand or do not want to admit it, I repeat what this function center?, response is nothing is a function like all others

you know where the error happens???

Have you ever tried to debug, to understand what happens ???

here create a file test.au3

Local $sData = FileRead(@DesktopDir & '\Test.txt')
MsgBox(0, "Test", "Test")

see here with large files 300 MB (in Windows XP with 1 GB Ram installed) if not responding the native FileRead Function, the function _FileWriteToLine* is no longer in question OKKK (I hope that now you understand that you're wrong ehhh)

so I think this discussion is not about the _FileWriteToLine* better close here this discussion

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

 

you will need to understand that you're wrong, I told you a thousand times but it seems that you do not understand or do not want to admit it

It's your point of view. Personnaly, I think the same thing of you.

 

see here with large files 300 MB (in Windows XP with 1 GB Ram installed) if not responding the native FileRead Function, the function _FileWriteToLine* is no longer in question

Not only on Windows XP (see capture about Windows Seven and Windows 8 with much more than 1Gb of Ram). But if you read attentively my explanations, you will see that _FileCountLine (and many other) have actually this same problem. But for _FileCountLine there is a solution. A solution that increase the speed and eliminate this bug.

It's easy to use the internal functions for everything and anything. Still it is necessary to use them correctly!!! In FileRead documentation, you can read :

 

value that is too large can lead to AutoIt stopping with a memory allocation failure

So why those who writes the external functions doesn't take care about that!

Buffering data is not reserved to the other languages.

 

 

so I think this discussion is not about the _FileWriteToLine

You are not completely wrong, because it concerns not only this particular function.

But because it's in this topic question of its rewriting, why not coding correctly?

Anyway, this is not grave, nobody's perfect, and I even less than the others. ^^

 

 

better close here this discussion

Good idea. Close our eyes, keep this behavior and continue just like that. Can be that another day, somebody else will take back this subject. With this, keep your peace of mind and have a good Week-End. ;)

Ciao.


Best Regards.Thierry

Share this post


Link to post
Share on other sites

Good idea. Close our eyes

I'm really sorry but I'm not going to write again here 100 other posts for nothing, simply open a topic in forum and ask about FileRead ect ect, nobody forbids you to do it OK

 

keep this behavior and continue just like that.

in this point of view is the same behavior with IE11 (or almost all official releases of Microsoft as DotNet45 or LiveMessenger v16.x ect ect ect ect) which does not work in Windows XP Right? ;) (I love Windows XP Ok, in my signature you can find more than 10 GB of data regarding Windows XP OKK, but in the end we need to understand that XP is dead ehhhhh, if you want to use it again okay, but simply not pretend much there, all tests say that even in old PCs Windows 7 has more good performance)

for what I know, no one prevented you use FileRead function directly, use the FileOpen first if you want to and FileReadLine & FileWriteLine after Ok

Finally, please understand, I'm not the one who decides here OKK, I think the developers have read your posts, and cosider that what you've written, I do not know now

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

@guinness
Please satisfied Tlem :) 

However leaving the jokes, I think the _FileWriteToLine will be this, certainly (need to test it well) is never Ok enough there is always something to improve, but already seems OKK (at this point in _FileWriteToLine() use the _FileWriteToLine($sFilePath, $iLine, $sText, $iFlags), to not lose compatibility with the previous version)

#include <FileConstants.au3>
#include <StringConstants.au3>

; #CONSTANTS# ====================================================
#Region ;**** Global vars for _FileWriteToLineEx
Global Const $FRL_OVERWRITE    = 1
Global Const $FRL_NOEOL        = 2
Global Const $FRL_DELETE       = 4
Global Const $FRL_USE_LINEFEED = 8
Global Const $FRL_BUFFER_0     = 1024       ; 1 Kb
Global Const $FRL_BUFFER_1     = 1048576    ; 1 MB
Global Const $FRL_BUFFER_10    = 10485760   ; 10 MB
Global Const $FRL_BUFFER_50    = 52428800   ; 50 MB
Global Const $FRL_BUFFER_100   = 104857600  ; 100 MB
Global Const $FRL_BUFFER_250   = 262144000  ; 250 MB
Global Const $FRL_BUFFER_500   = 524288000  ; 500 MB
Global Const $FRL_BUFFER_1000  = 1048576000 ; 1GB
Global Const $FRL_BUFFER_2000  = 2147483647 ; 2GB _ Windows MAX SIZE Limit
Global Const $FRL_DEFBUFFER    = $FRL_BUFFER_100
;;Global Const $FRL_DEFBUFFER    = (MemGetStats())[1] < 3404884 ? $FRL_BUFFER_100 : $FRL_BUFFER_500    ;;3404884 = 4 GB RAM
Global Const $FRL_SZTEMPPATH   = @TempDir & "\TMP_" & Random(11111,99999, 1) & "_.tmp"
Global Const $FRL_OVERWRITE_DELETE  = BitOR($FRL_OVERWRITE, $FRL_DELETE) ;INTERNALLY USE ONLY
Global Const $FRL_NOEOL_DELETE  = BitOR($FRL_NOEOL, $FRL_DELETE) ;INTERNALLY USE ONLY
#EndRegion ;**** Global vars for _FileWriteToLineEx
; ================================================================

Local $fTimerDiff, $sData, $Line = " = This is Test for _FileWriteToLineEx Function" & @CRLF
FileDelete(@DesktopDir & '\Test.txt')
For $i = 1 to 6
    For $y = 1 To 1000000
        $sData &= "Line - " & $y & $Line
    Next
    FileWrite(@DesktopDir & '\Test.txt', $sData) ; 57 MB
    $sData = ""
Next
;; @DesktopDir & '\Test.txt' - 354 MB 6.000.000 Line
$fTimerDiff = TimerInit()
_FileWriteToLine(@DesktopDir & '\Test.txt', 5999997, ';This is Comment Line 1')
_FileWriteToLine(@DesktopDir & '\Test.txt', 5999998, ';This is Comment Line 2', $FRL_OVERWRITE)
_FileWriteToLine(@DesktopDir & '\Test.txt', 5999999, '', $FRL_DELETE)
ConsoleWrite("iLineOffSet - " & @Extended & @CRLF)
If @Error Then ConsoleWrite("!@Error - " & @error & @CRLF)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @CRLF)

$sData = FileRead(@DesktopDir & '\Test2.txt')
$fTimerDiff = TimerInit()
_FileWriteToLineEx($sData, ';This is Comment Line 1',5999997)
_FileWriteToLineEx($sData, ';This is Comment Line 2',5999998, $FRL_OVERWRITE)
_FileWriteToLineEx($sData, '', 5999999, $FRL_DELETE)
ConsoleWrite("iLineOffSet - " & @Extended & @CRLF)
If @Error Then ConsoleWrite("!@Error - " & @error & @CRLF)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @CRLF)
FileDelete(@DesktopDir & '\Test2.txt')
FileWrite(@DesktopDir & '\Test2.txt', $sData)
$sData = ""

$fTimerDiff = TimerInit()
_FileWriteToLineOLD(@DesktopDir & '\Test3.txt',5999997, ';This is Comment Line 1')
_FileWriteToLineOLD(@DesktopDir & '\Test3.txt',5999998, ';This is Comment Line 2', 1)
_FileWriteToLineOLD(@DesktopDir & '\Test3.txt',5999999, ';This is Comment Line 1')
ConsoleWrite("iLineOffSet - " & @Extended & @CRLF)
If @Error Then ConsoleWrite("!@Error - " & @error & @CRLF)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite($fTimerDiff & @CRLF)

Exit

; #FUNCTION# ====================================================================================================================
; Name...........: _FileWriteToLine
; Description ...: Edit a specific line in a file.
; Syntax.........: _FileWriteToLineEx($sFilePath, $iLine, $sText[, $iFlags] )
; Parameters ....: $sFilePath - The File Path
;                  $iLine     - The line number to edit to
;                  $sText     - The text to write
;                  $iFlags    - Optional, Flag to indicate the type of stripping that should be performed
;                  |  Default FileWriteLine Mod, The line of text to write to the text file. If the line does NOT end in @CR or @LF
;                  |   then a DOS linefeed (@CRLF) will be automatically added.
;                  |$FRL_OVERWRITE (1)    - overwrite the old line
;                  |$FRL_NOEOL     (2)    - Disable check for @CR or @LF (do not add @CR or @LF if there are not present)
;                  |$FRL_DELETE    (4)    - delete\remove the line
;                  $iBuffer    - Optional, Buffer Size used, Default 100 MB
;                  $iLineSize  - Optional, Line Size to check for Default EOL, Default 1 MB
; Return values .: If the function succeeds, it returns TRUE (Check @extended for iLineOffset); otherwise, FALSE.
;                  @error  - 0 = No error
;                  |-n = Error when FileClose (will not have to ever happen)
;                  |1 = Line Number invalid
;                  |2 = File does not exist
;                  |3 = Error when opening file, should be ERROR_ACCESS_DENIED = 5 (0x5) Access is denied. ($sFilePath is ReadOnly or has not Security Access Rights)
;                  |4 = File has less lines than $iLine
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileWriteToLine($sFilePath, $iLine, $sText, $iFlags = 0, $iBuffer = $FRL_DEFBUFFER, $iLineSize = $FRL_BUFFER_1)
    If $iLine < 1 Then Return SetError(1, 0, False)
    If Not FileExists($sFilePath) Then Return SetError(2, 0, False)
    Local $hFileOpen = FileOpen($sFilePath, $FO_READ), $htFileOpen = FileOpen($FRL_SZTEMPPATH, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Or $htFileOpen = -1 Then Return SetError(3, FileClose($hFileOpen) + FileClose($htFileOpen), False)
    If $iBuffer < $FRL_BUFFER_1 Then $iBuffer = $FRL_BUFFER_1
    If $iLineSize < 3 Then $iLineSize = $FRL_BUFFER_0
    Local $sData, $iOffSet = 0, $_iOffSet, $EOL = @LF, $_iBuffer = Ceiling(FileGetSize($sFilePath) / $iBuffer)
    $EOL = StringInStr(FileRead($sFilePath, $iLineSize), @LF, $STR_CASESENSE, 1) ? @LF : @CR    ;;@LF has priority
    If $iLine > 1 Then
        $iLine -= 1
        For $i = 1 To $_iBuffer
            $sData = FileRead($hFileOpen, $iBuffer)
            For $y = 1 To $iLine
                $iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
                If Not $iOffSet Then ExitLoop
                $iLine -= 1
            Next
            If $iOffSet Then
                FileWrite($htFileOpen, StringLeft($sData, $iOffSet))
                $sData = StringTrimLeft($sData, $iOffSet)
                $iOffSet += ($i - 1) * $iBuffer
                ExitLoop
            Else
                FileWrite($htFileOpen, $sData)
            EndIf
        Next
        If Not $iOffSet Then Return SetError(4, FileClose($hFileOpen) + FileClose($htFileOpen) + FileDelete($FRL_SZTEMPPATH), False)
    EndIf
    $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE)
    If Not $_iOffSet And StringLen($sData) < $iLineSize Then
        $sData &= FileRead($hFileOpen, $iLineSize)
        $_iOffSet = StringInStr($sData, $EOL, $STR_CASESENSE)
    EndIf
    If Not $_iOffSet Then
        If Not BitAND($iFlags, $FRL_DELETE) Then
            If Not BitAND($iFlags, $FRL_OVERWRITE) Then FileWrite($htFileOpen, $sData)
            If Not BitAND($iFlags, $FRL_NOEOL) And StringRight($sText, 1) <> $EOL Then $sText &= ($EOL = @CR ? @CR : @CRLF)
            FileWrite($htFileOpen, $sText)
        EndIf
        Return  SetError(FileClose($hFileOpen) + FileClose($htFileOpen) - 2, $iOffSet, FileMove($FRL_SZTEMPPATH, $sFilePath, $FC_OVERWRITE) = 1)
    ElseIf BitAND($iFlags, $FRL_DELETE) Then
        FileSetPos($hFileOpen, $_iOffSet, $FILE_CURRENT)
        FileWrite($htFileOpen, StringTrimLeft($sData, $_iOffSet))
    Else
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sData, $_iOffSet - 1, 1) == @CR))
        If Not BitAND($iFlags, $FRL_OVERWRITE) Then FileWrite($htFileOpen, StringLeft($sData, $_iOffSet))
        FileWrite($htFileOpen, $sText & StringTrimLeft($sData, $_iOffSet))
    EndIf
    While 1
        $sData = FileRead($hFileOpen, $iBuffer)
        If Not $sData Then ExitLoop
        FileWrite($htFileOpen, $sData)
    WEnd
    Return  SetError(FileClose($hFileOpen) + FileClose($htFileOpen) - 2, $iOffSet, FileMove($FRL_SZTEMPPATH, $sFilePath, $FC_OVERWRITE) = 1)
EndFunc   ;==>_FileWriteToLine


; #FUNCTION# ====================================================================================================================
; Name...........: _FileWriteToLineEx
; Description ...: Edit a specific line in a file.
; Syntax.........: _FileWriteToLineEx(ByRef $sFileData, $sText[, $iLine[, $iFlags]] )
; Parameters ....: $sFileData - String\Text\Data
;                  $sText     - The text to write
;                  $iLine     - Optional, The line number to edit to (Default edit last line), Use a negative occurrence to search line from the right side
;                  $iFlags    - Optional, Flag to indicate the type of stripping that should be performed
;                  |  Default FileWriteLine Mod, The line of text to write to the text file. If the line does NOT end in @CR or @LF
;                  |   then a DOS linefeed (@CRLF) will be automatically added.
;                  |$FRL_OVERWRITE (1)    - overwrite the old line
;                  |$FRL_NOEOL     (2)    - Disable check for @CR or @LF (do not add @CR or @LF if there are not present)
;                  |$FRL_DELETE    (4)    - delete\remove the line
;                  |$FRL_USE_LINEFEED (8) - forceuse of Line Feed
; Return values .: If the function succeeds, it returns TRUE (Check @extended for iLineOffset); otherwise, FALSE.
;                  @error  - 0 = No error
;                  |1 = File does not exist
;                  |2 = File has less lines than $iLine
;                  |3 = Error when opening file, should be ERROR_ACCESS_DENIED = 5 (0x5) Access is denied. ($sFilePath is ReadOnly or has not Security Access Rights)
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileWriteToLineEx(ByRef $sFileData, $sText, $iLine = Default, $iFlags = 0)
    If Not $sFileData Then Return SetError(1, 0, False)
    If Not $iLine Then $iLine = -1
    Local $EOL = @LF, $iOffSet = 0, $_iOffSet
    If BitAND($iFlags, $FRL_USE_LINEFEED) Then
        $sFileData = StringRegExpReplace($sFileData, "\r(?!\n)", @CRLF)
    Else
        ;;$EOL = StringMid($sFileData, StringInStr($sFileData, @CR, $STR_CASESENSE, 1) + 1, 1) = @LF ? @LF : @CR    ;;@CR has priority
        $EOL = StringInStr($sFileData, @LF, $STR_CASESENSE, 1) ? @LF : @CR    ;;@LF has priority
    EndIf
    If $iLine <> 1 Then
        $iOffSet = StringInStr($sFileData, $EOL, $STR_CASESENSE, ($iLine < 0 ? $iLine : $iLine - 1))
        If Not $iOffSet Then Return SetError(2, 0, False)
    EndIf
    $_iOffSet = StringInStr($sFileData, $EOL, $STR_CASESENSE, 1, $iOffSet + 1)
    If Not $_iOffSet Then
        $_iOffSet = StringLen($sFileData)
        If Not BitAND($iFlags, $FRL_NOEOL_DELETE) And StringRight($sText, 1) <> $EOL Then $sText &= ($EOL = @CR ? @CR : @CRLF)
    ElseIf Not BitAND($iFlags, $FRL_DELETE) Then
        $_iOffSet -= ($EOL = @CR ? 1 : 1 + Number(StringMid($sFileData, $_iOffSet - 1, 1) == @CR))
    EndIf
    If Not BitAND($iFlags, $FRL_OVERWRITE_DELETE) Then $iOffSet = $_iOffSet
    $sFileData = StringLeft($sFileData, $iOffSet) & (BitAND($iFlags, $FRL_DELETE) ? "" : $sText) & StringTrimLeft($sFileData, $_iOffSet)
    Return SetError(0, $iOffSet, True)
EndFunc   ;==>_FileWriteToLineEx



; #FUNCTION# ====================================================================================================================
; Author ........: cdkid
; Modified.......: partypooper, MrCreatoR, guinness
; ===============================================================================================================================
Func _FileWriteToLineOLD($sFilePath, $iLine, $sText, $iOverWrite = 0)
    If $iLine <= 0 Then Return SetError(4, 0, 0)
    If Not IsString($sText) Then
        $sText = String($sText)
        If $sText = "" Then Return SetError(6, 0, 0)
    EndIf
    If $iOverWrite <> 0 And $iOverWrite <> 1 Then Return SetError(5, 0, 0)
    If FileExists($sFilePath) = 0 Then Return SetError(2, 0, 0)

    Local $aArray = FileReadToArray($sFilePath)
    Local $iUBound = UBound($aArray) - 1
    If ($iUBound + 1) < $iLine Then Return SetError(1, 0, 0)

    Local $hFileOpen = FileOpen($sFilePath, FileGetEncoding($sFilePath) + $FO_OVERWRITE)
    If $hFileOpen = -1 Then Return SetError(3, 0, 0)

    Local $sData = ""
    $iLine -= 1 ; Now the array is 0-based, so reduce the line number by 1.
    For $i = 0 To $iUBound
        If $i = $iLine Then
            If $iOverWrite Then
                If $sText Then $sData &= $sText & @CRLF
            Else
                $sData &= $sText & @CRLF & $aArray[$i] & @CRLF
            EndIf
        ElseIf $i < $iUBound Then
            $sData &= $aArray[$i] & @CRLF
        ElseIf $i = $iUBound Then
            $sData &= $aArray[$i]
        EndIf
    Next

    FileWrite($hFileOpen, $sData)
    FileClose($hFileOpen)
    Return 1
EndFunc   ;==>_FileWriteToLine
;~ >Running:(3.3.10.2):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\DXRW4E\Desktop\AU3 TMP\_FileWriteToLineEx.au3"
;~ --> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
;~ _FileWriteToLine - iLineOffSet - 371333237
;~ _FileWriteToLine - 131180.182894921
;~ _FileWriteToLineEx - iLineOffSet - 371333237
;~ _FileWriteToLineEx - 13915.2032637522
;~ _FileWriteToLineOLD - Error - Memory allocation error
;~ ->09:38:28 AutoIt3.exe ended.rc:1
;~ >Exit code: 1    Time: 963.975

 
I'm not sure, but in my tests it appears that the new _FileWriteToLine() is almost 30% slower than in Post #85 I believe is acceptable, so what I suggest is to use 2 _FileWriteToLine, _FileWriteToLine() and also the _FileWriteToLineEx() that will work byref which is very much faster, I do not know see for yourself what is better to do

 

@Tlem

seriously for real, notepad it took 5 minutes to open the file, instead _FileWriteToLine() writeedit it (3 times) in 3 minute OKKK

iuva.png


9cd9.png

 

we hope that you will be happy, now put in $iBuffer the $FRL_BUFFER_1, and use it also in Windows 95 with 32 MB RAM if you want, so now if you want use also file 1.9 GB in Windows XP normally should be all OK

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

Folks,

You need a good breathe!

Also never forget that a 180 Mb ANSI text file loaded in one single shot will take up 180x2 = 360 Mb or memory. Indeed AutoIt uses UTF16, hence every character of every native string takes a 16_bit halfword.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Folks,

You need a good breathe!

Also never forget that a 180 Mb ANSI text file loaded in one single shot will take up 180x2 = 360 Mb or memory. Indeed AutoIt uses UTF16, hence every character of every native string takes a 16_bit halfword.

that does not matter much, since it's easy to control it (are very few functions that need filepath, in 99% of cases users load everything to the local or global variable), the point is need to be careful with the use of the regexp (regexp multiplies many times the memory used, I myself sometimes use overly regex in my script ehhh, though is already reported in Autoit Help, that need to be careful with regexp), as the script in the first post of the topic fails (Memory allocation error) with file 350 Mb, even if they are in Win8 x64 - 4 GB RAM

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...