num1g8rfan

Writing to numerous files ultimately slows to a crawl

3 posts in this topic

#1 ·  Posted (edited)

Hi All,

I have a script whose sole purpose is to walk through about 15,000 - 20,000 files in a single directory structure, making modifications to 5 or 6 lines within each file.  Simple.  I have a small splash screen that is simply counting how many files have been completed out of the overall total.  The script is functioning fine and the splash screams through the initial files, but through time begins to slow to a crawl (It becomes noticeable after about 1,000 to 1,500 files).  

I have used a combination of methods to write to the file, but none of which appear to make much of any difference:

Local $FileOpen = FileOpen($FileInList,1)
FileWriteLine($FileOpen,@CRLF)
FileClose($FileOpen)

OR

FileWriteLine($FileHandle, $NewLineString & @CRLF)

OR

Local $FileHandle = FileOpen($FileInList,2)
_FileWriteFromArray($FileHandle,$FileContentArray,1)
FileClose($FileHandle)

I have checked and the CPU and memory are fine.  My current work-around is to create 5-10 separate directory structures from the original one that contains 20,000 files, run the script and then combine them again in the end.  It works, but isn't very portable.  Any ideas?

Cleanup_AutoITPost.au3

Edited by num1g8rfan
Posting the code

Share this post


Link to post
Share on other sites



#2 ·  Posted

num1g8rfan,

Posting the whole script would probably be a good idea.

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

#3 ·  Posted

Posted the code per Melba23's suggestion. Thanks

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • Jibberish
      By Jibberish
      Junior Programmer here... 
      Not much experience with opening, changing and closing files.
      I am trying to replace strings in a Text file except StringReplace does not actually replace the text.
      Here is a sample of my code...
      #include <File.au3> #include <MsgBoxConstants.au3> #include <WinAPIFiles.au3> Local $iStrReturn = 0 Local $hFile Local $sText Local $sNewText ; Location of File to be read $sFileName = "C:\Temp\MyPlayer.exe.config" ; The default is FALSE. We want to change this to TRUE $bLoopChecked = True CheckBox() Func Checkbox() $hFile = FileOpen($sFileName,$FO_READ) ; Open file in read mode to get text If $hFile = -1 Then MsgBox($MB_SYSTEMMODAL, "", "An error occurred when Opening the file.") Exit EndIf FileSetPos($hFile, 0, 0) ; No idea if I need to do this, grasping at straws $sText = FileRead($hFile) ; Read the file into $sText If $sText = 1 Then MsgBox($MB_SYSTEMMODAL, "", "An error occurred when reading/writing the file.") Exit Else FileClose($hFile) ; Finished reading the file into $sText, so close the file. FileFlush($hFile) ; Manual says to use FileFlush between File Close and Open so here it is EndIf MsgBox(0,"Before Replacement",$sText) ; Displays the text read from the file to make sure something is there. ; Loop Check If $bLoopChecked = True Then ; Find the string return > 0 for success $iStrReturn = StringInStr('"<add key="LoopCheckbox" value=""False" />"', "False") ;MsgBox(0,"", "LoopCheckBox is " & $iStrReturn) If $iStrReturn > 0 Then ; If StringInStr returned > 0 the it found the string! ; The Meat of the code. This is where we have to replace "False" with "True" $sNewText = StringReplace($sText, '"<add key="LoopCheckbox" value="False" />"', '"<add key="LoopCheckbox" value="True" />"') MsgBox(0,"After Replacement",$sNewText) ; Display the text to see if it worked. $hFile = FileOpen($sFileName,$FO_OVERWRITE) ; Reopen the file to write to it, overwriting everything. FileWrite($hFile,$sNewText) ; Write the text to the file FileClose($hFile) ; Close the file EndIf EndIf EndFunc This is the file I am reading...
      <?xml version="1.0" encoding="utf-8"?> <configuration> <startup> <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6"/> </startup> <appSettings> <add key="LoopCheckbox" value="false"/> </appSettings> </configuration> I tried opening the file with $FO_UTF8 and $FO_UTF8_NOBOM but got errors opening the file.
      The MsgBox "After Replacement" shows the value is still false.
    • MrSparty1995
      By MrSparty1995
      I have a good handle on how to read a file line by line, and search for a given string.
      Basically using, FileOpen, FileReadLine, and StringInStr.
      I have been trying to figure out why my search keeps coming back with no match.
      For example, if my string line was "Where is Waldy", and I was searching for Waldy that comes back fine.
      But in my file, Let's say it is
      Where is "Waldy"
      So i read that line in and do a search for Waldy, but i comes back with no match since there is quotes around it.
      Is there a way to resolve this?
      Here is my actual example
      The line comes back like this
          <JposEntry logicalName="LineDisp_iSC480">
      I'm searching for LineDisp_iSC480, but its coming back with no match due to the quotes.
      Thanks in advance, James
    • fmlk
      By fmlk
      Hello,
      In a main script I have added logging and that is working. Now I created another small script where I need also logging. So I used the same code:
      What is wrong? Why is the logfile not created?

      $default_delay = 2000
      $logFilePath = ("C:\AutoIT_TestLogs")
      DirCreate($logFilePath)
      $Logfile = FileOpen($logFilePath & "\GMS_install" & @MON & "/" & @MDAY & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC & ".log",1)
      FileWriteLine($logFile,@CR & @MON & "/" & @MDAY & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC & "_Install_licenses")
      Consolewrite("Install_licenses" & @CR)
      Run("C:\Users\factory\Desktop\LicenseInstaller\LicenseInstaller.exe","C:\Users\factory\Desktop\LicenseInstaller")
      Sleep(1000)
      WinWait("Gatan License Installer")
      WinActivate("Gatan License Installer")
      WinWaitActive("Gatan License Installer")
      ControlClick("[ACTIVE]","","[NAME:CMD_Install]")
      Sleep($default_delay)
       If WinExists("LicenseInstaller") Then
           WinActivate("LicenseInstaller")
            WinWaitActive("LicenseInstaller")
            ControlClick("[ACTIVE]","","[CLASS:Button; INSTANCE:1]")
              sleep($default_delay)
         EndIf
      FileWriteLine($logFile,@CR & @MON & "/" & @MDAY & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC & "_Licenses_Installed")
      Consolewrite("Licenses are installed" & @CR)
       
    • oasis375
      By oasis375
      This doesn't work, it deletes (overwrites) file contents:
      $hFile = FileOpen("file.txt", 2) $content1 = FileRead($hFile) $content2 = StringReplace($content1, "old", "new") FileWrite($hFile, $content2) FileClose($hFile) Why? Instead, I have to use this workaround:
      $hFile = FileOpen("file.txt") $content1 = FileRead($hFile) $content2 = StringReplace($content1, "old", "new") FileClose($hFile) $hFile = FileOpen("file.txt",2) FileWrite($hFile, $content2) FileClose($hFile) That's it, opening, closing and reopening the file.
      Function reference says: FileWrite. "Write text/data to the end of a previously opened file." (Does it mean "append"?).
      Remarks: "The file must be opened in write mode or the FileWrite() command will fail."
      But what write mode? $FO_APPEND (1) = Write mode (append to end of file). $FO_OVERWRITE (2) = Write mode (erase previous contents).
       
    • souldjer777
      By souldjer777
      Howdy Everyone!
      I've been scouring the forum for something that works... and no dice yet... hoping to stop the frustration - hope someone in the forum knows what to do.
      Basically, I've saved .msg files out of MS Outlook and want to scan each on for email addresses but all I get with fileopen and fileread are symbols... ran through every iteration.
      I've seen people do some soft of binary extract and binarytostring but I haven't gotten that to work either... not sure what's going on.
      Neither of these have worked for me... yes I know they are old and FileOpen has changed... but like I said I tried everything that I know of.
      Please help
       
      $sFile = "C:\temp\messages\Test.msg" $hFile = FileOpen($sFile, 16) ; 16 = binary $binData = FileRead($hFile, 1024) ; read first 1KB FileClose($hFile) ConsoleWrite("Debug: First 1KB = " & $binData & @LF) or...
      $sFile = "C:\temp\messages\Test.msg" $hFile = FileOpen($sFile, 16) $binData = FileRead($hFile, 20480) FileClose($hFile) $startdef=StringTrimLeft(Binary('thetextIamlookingfor'), 20) $enddef=StringTrimLeft(Binary('moretextIamlookingfor'), 20) $binData=StringTrimLeft($binData, StringInStr($binData, $startdef)+StringLen($startdef)-1) $binData=StringLeft($binData, StringInStr($binData, $enddef)-1) ConsoleWrite(BinaryToString('0x'&$binData) & @LF) Thanks!