Jump to content
num1g8rfan

Writing to numerous files ultimately slows to a crawl

Recommended Posts

num1g8rfan

Hi All,

I have a script whose sole purpose is to walk through about 15,000 - 20,000 files in a single directory structure, making modifications to 5 or 6 lines within each file.  Simple.  I have a small splash screen that is simply counting how many files have been completed out of the overall total.  The script is functioning fine and the splash screams through the initial files, but through time begins to slow to a crawl (It becomes noticeable after about 1,000 to 1,500 files).  

I have used a combination of methods to write to the file, but none of which appear to make much of any difference:

Local $FileOpen = FileOpen($FileInList,1)
FileWriteLine($FileOpen,@CRLF)
FileClose($FileOpen)

OR

FileWriteLine($FileHandle, $NewLineString & @CRLF)

OR

Local $FileHandle = FileOpen($FileInList,2)
_FileWriteFromArray($FileHandle,$FileContentArray,1)
FileClose($FileHandle)

I have checked and the CPU and memory are fine.  My current work-around is to create 5-10 separate directory structures from the original one that contains 20,000 files, run the script and then combine them again in the end.  It works, but isn't very portable.  Any ideas?

Cleanup_AutoITPost.au3

Edited by num1g8rfan
Posting the code

Share this post


Link to post
Share on other sites
Melba23

num1g8rfan,

Posting the whole script would probably be a good idea.

M23


Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites
num1g8rfan

Posted the code per Melba23's suggestion. Thanks

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • AnonymousX
      By AnonymousX
      Hello,
      I need to pull data from company intranet website. I created a script that I can give a list of project numbers and it will open up the the related webpage for each project, save the html comments for that project, then move on to the next. 
      However my problem comes in that each time I open up a project it locks it for other users, and if you just exit the page it keeps it locked, you have to actually press the "close" button on the page. So I'm not sure how to deal with this as I end up just locking every project in my name when I run my program as I don't understand the nature of how the website is built. I don't understand how to close with just using my automation code without manually having to press the stupid button. 
      My Function:
      Func Getscript($ProjectAddress) ;getting the page source code and storing it into text file for easy reading it $file = FileOpen(@ScriptDir & "\source.txt", 10) $IE = _IECreate($ProjectAddress & ".html", 0, 0) $source = _IEDocReadHTML($IE) FileWrite($file, $source) $target_source = _StringBetween($source, "<BODY>", "</BODY>") ; only take content between the specified tags _IEQuit($IE) FileClose($file) return $target_source[0] EndFunc ;==>Getscript  
      Source Code I got from website that I think is relevant:
      function closeForm(){ if(topButtonEnabled("closeButton")){ parent.mainFrame.closeForm(); } } <button id="closeButton" class="saveCloseButton" style="LEFT: 77px;" onclick=closeForm()>&nbsp;Close&nbsp;</button>  
      Any Ideas of how to tell website to either activate the closeform function or just give it the command to close?
       
      Thanks
    • Jibberish
      By Jibberish
      Junior Programmer here... 
      Not much experience with opening, changing and closing files.
      I am trying to replace strings in a Text file except StringReplace does not actually replace the text.
      Here is a sample of my code...
      #include <File.au3> #include <MsgBoxConstants.au3> #include <WinAPIFiles.au3> Local $iStrReturn = 0 Local $hFile Local $sText Local $sNewText ; Location of File to be read $sFileName = "C:\Temp\MyPlayer.exe.config" ; The default is FALSE. We want to change this to TRUE $bLoopChecked = True CheckBox() Func Checkbox() $hFile = FileOpen($sFileName,$FO_READ) ; Open file in read mode to get text If $hFile = -1 Then MsgBox($MB_SYSTEMMODAL, "", "An error occurred when Opening the file.") Exit EndIf FileSetPos($hFile, 0, 0) ; No idea if I need to do this, grasping at straws $sText = FileRead($hFile) ; Read the file into $sText If $sText = 1 Then MsgBox($MB_SYSTEMMODAL, "", "An error occurred when reading/writing the file.") Exit Else FileClose($hFile) ; Finished reading the file into $sText, so close the file. FileFlush($hFile) ; Manual says to use FileFlush between File Close and Open so here it is EndIf MsgBox(0,"Before Replacement",$sText) ; Displays the text read from the file to make sure something is there. ; Loop Check If $bLoopChecked = True Then ; Find the string return > 0 for success $iStrReturn = StringInStr('"<add key="LoopCheckbox" value=""False" />"', "False") ;MsgBox(0,"", "LoopCheckBox is " & $iStrReturn) If $iStrReturn > 0 Then ; If StringInStr returned > 0 the it found the string! ; The Meat of the code. This is where we have to replace "False" with "True" $sNewText = StringReplace($sText, '"<add key="LoopCheckbox" value="False" />"', '"<add key="LoopCheckbox" value="True" />"') MsgBox(0,"After Replacement",$sNewText) ; Display the text to see if it worked. $hFile = FileOpen($sFileName,$FO_OVERWRITE) ; Reopen the file to write to it, overwriting everything. FileWrite($hFile,$sNewText) ; Write the text to the file FileClose($hFile) ; Close the file EndIf EndIf EndFunc This is the file I am reading...
      <?xml version="1.0" encoding="utf-8"?> <configuration> <startup> <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6"/> </startup> <appSettings> <add key="LoopCheckbox" value="false"/> </appSettings> </configuration> I tried opening the file with $FO_UTF8 and $FO_UTF8_NOBOM but got errors opening the file.
      The MsgBox "After Replacement" shows the value is still false.
    • MrSparty1995
      By MrSparty1995
      I have a good handle on how to read a file line by line, and search for a given string.
      Basically using, FileOpen, FileReadLine, and StringInStr.
      I have been trying to figure out why my search keeps coming back with no match.
      For example, if my string line was "Where is Waldy", and I was searching for Waldy that comes back fine.
      But in my file, Let's say it is
      Where is "Waldy"
      So i read that line in and do a search for Waldy, but i comes back with no match since there is quotes around it.
      Is there a way to resolve this?
      Here is my actual example
      The line comes back like this
          <JposEntry logicalName="LineDisp_iSC480">
      I'm searching for LineDisp_iSC480, but its coming back with no match due to the quotes.
      Thanks in advance, James
    • fmlk
      By fmlk
      Hello,
      In a main script I have added logging and that is working. Now I created another small script where I need also logging. So I used the same code:
      What is wrong? Why is the logfile not created?

      $default_delay = 2000
      $logFilePath = ("C:\AutoIT_TestLogs")
      DirCreate($logFilePath)
      $Logfile = FileOpen($logFilePath & "\GMS_install" & @MON & "/" & @MDAY & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC & ".log",1)
      FileWriteLine($logFile,@CR & @MON & "/" & @MDAY & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC & "_Install_licenses")
      Consolewrite("Install_licenses" & @CR)
      Run("C:\Users\factory\Desktop\LicenseInstaller\LicenseInstaller.exe","C:\Users\factory\Desktop\LicenseInstaller")
      Sleep(1000)
      WinWait("Gatan License Installer")
      WinActivate("Gatan License Installer")
      WinWaitActive("Gatan License Installer")
      ControlClick("[ACTIVE]","","[NAME:CMD_Install]")
      Sleep($default_delay)
       If WinExists("LicenseInstaller") Then
           WinActivate("LicenseInstaller")
            WinWaitActive("LicenseInstaller")
            ControlClick("[ACTIVE]","","[CLASS:Button; INSTANCE:1]")
              sleep($default_delay)
         EndIf
      FileWriteLine($logFile,@CR & @MON & "/" & @MDAY & "/" & @YEAR & " " & @HOUR & ":" & @MIN & ":" & @SEC & "_Licenses_Installed")
      Consolewrite("Licenses are installed" & @CR)
       
    • oasis375
      By oasis375
      This doesn't work, it deletes (overwrites) file contents:
      $hFile = FileOpen("file.txt", 2) $content1 = FileRead($hFile) $content2 = StringReplace($content1, "old", "new") FileWrite($hFile, $content2) FileClose($hFile) Why? Instead, I have to use this workaround:
      $hFile = FileOpen("file.txt") $content1 = FileRead($hFile) $content2 = StringReplace($content1, "old", "new") FileClose($hFile) $hFile = FileOpen("file.txt",2) FileWrite($hFile, $content2) FileClose($hFile) That's it, opening, closing and reopening the file.
      Function reference says: FileWrite. "Write text/data to the end of a previously opened file." (Does it mean "append"?).
      Remarks: "The file must be opened in write mode or the FileWrite() command will fail."
      But what write mode? $FO_APPEND (1) = Write mode (append to end of file). $FO_OVERWRITE (2) = Write mode (erase previous contents).
       
×