Jump to content

Writing to numerous files ultimately slows to a crawl


Recommended Posts

Hi All,

I have a script whose sole purpose is to walk through about 15,000 - 20,000 files in a single directory structure, making modifications to 5 or 6 lines within each file.  Simple.  I have a small splash screen that is simply counting how many files have been completed out of the overall total.  The script is functioning fine and the splash screams through the initial files, but through time begins to slow to a crawl (It becomes noticeable after about 1,000 to 1,500 files).  

I have used a combination of methods to write to the file, but none of which appear to make much of any difference:

Local $FileOpen = FileOpen($FileInList,1)
FileWriteLine($FileOpen,@CRLF)
FileClose($FileOpen)

OR

FileWriteLine($FileHandle, $NewLineString & @CRLF)

OR

Local $FileHandle = FileOpen($FileInList,2)
_FileWriteFromArray($FileHandle,$FileContentArray,1)
FileClose($FileHandle)

I have checked and the CPU and memory are fine.  My current work-around is to create 5-10 separate directory structures from the original one that contains 20,000 files, run the script and then combine them again in the end.  It works, but isn't very portable.  Any ideas?

Cleanup_AutoITPost.au3

Edited by num1g8rfan
Posting the code
Link to comment
Share on other sites

  • Moderators

num1g8rfan,

Posting the whole script would probably be a good idea.

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...