Sign in to follow this  
Followers 0
AmbiguousJoe

Problems writing to a text file quickly

7 posts in this topic

I created the following function to back-up rows from a 2D-array as my script runs through the data. I write to an Excel file at the end, so this provided a way for me to terminate the script early, without losing all my progress. It's works fine until my script starts writing to the file multiple times per second. My text files become corrupted. What should be a 125-500kb file with tab-delimited data, turns into an 80-123mb file with several thousand lines of nothing but blank characters/spaces.

Func _BackupArray($nRow) ;Receives the row number that will be backed up.
ConsoleWrite("Function: BackupArray"&@CRLF)
Local $nCol = 0 ;column counter
Local $sLineOutput = $aExcelList[$nRow][0] ;Collects the first array cell into a string

While($nCol <= $nTotalExcelCols-1) ;cycles through columns to collect Array data.
     $sLineOutput = $sLineOutput&@TAB&$aExcelList[$nRow][$nCol] ;Adds each array's cell to the line, TAB delimited.
     $nCol = $nCol+1 ;Increases the column counter
WEnd

_FileWriteToLine($sBackupFile, $nRow+1, $sLineOutput,1) ;Sends the string to the backup file
EndFunc ;==>BackupArray

I suspect that the the problem is that _FileWriteToLine is a slow or inefficient way to accomplish a backup, but perhaps someone more knowledgable could help me out. When there are several seconds between calling this function, I don't have any problems; it's only when the function starts getting called several times per second.

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

First try: Use a filehandle instead of a filename for _FileWriteToLine().

From your snippet I can't see whether you're using a filename or a file handle. In at least 99% of the time using a filehandle is faster.

Edited by hannes08

Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

Looks like he is using a filename cause of the way the variable $sBackupFile is named.

I wouldn't use _FileWriteToLine to write to a specific line number.

Use FileOpen to open the file and retrieve a handle and then use FileWriteLine to write the line to the end of the file.

To close the file use FileClose.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

The first thing that strikes me is that it is not necessary to write to the file every time. I would be inclined to write multiline to TSV after however many entries have been added. If you wanted to, you could save it as XLS later. That would be my prefered approach.

Share this post


Link to post
Share on other sites

hannes08:

FileWriteToLine doesn't use file handles, but I assume you are suggesting the same method as water.

water:

You're correct: the $sBackupFile variable is a filename selected in an earlier part of the script.

I write to specific lines so that I don't have to delete and re-create backup files every time the script runs. A single file is used and updated, which matches the format of the excel file (so I can just copy and paste it in). That being said, that's not a good reason to use a less efficient function under these circumstances.

czardas:

I write to file every single time because some rows can take several minutes to complete (based on the actions performed on that row). Saving after every 10 entries or so might solve the problem I first described, but it could mean that I lose around 45 minutes of progress if I stopped the script during a set of slower actions.

I replaced FileWriteToLine() with

$FileHandle = FileOpen($sBackupFile, 1)
FileWriteLine($FileHandle, $sLineOutput)
FileClose($FileHandle)

I let the script run through about 10,000 rows and it didn't have any problems writing to the file. Thanks for your help guys.

Share this post


Link to post
Share on other sites

Was 500 kb file only a test case or a typical data file size? If it's that small I would tend to accumulate the data in memory and write the file in one go in binary mode,, renaming the old file as backup. You could use

OnAutoItExitRegister

if necessary.

Share this post


Link to post
Share on other sites

I have to admit that you're right with _FileWriteToLine() is not able to handle a filehandle.

For gaining speed you could do as MilesAhead suggested, or use _FileReadToArray() in combination with _FileWriteFromarray(). Or use a SQLite Database.


Regards,Hannes[spoiler]If you can't convince them, confuse them![/spoiler]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0