Fastest way to write lots of data to file?

[corrado33]

Hi, I have a script where I have to deal with a couple thousand pairs of strings. Currently, I'm storing the two strings in two arrays (because 2d arrays are a pain). However, I've noticed a few things.

The script slows down once I've stored ~500 pairs of strings in the array. At least, I THINK that's why the script slows down. So I'm debating offloading the strings to a file every 200 strings or so, but I want to find the fastest way to write them to a file.

 

Would it be quicker to use "FileWriteLine" or "_FileWriteFromArray"? Obviously with "FileWriteLine" I'd have to loop through the array. 

 

I debated doing "FileWriteLine" EVERY iteration of my main loop, but that REALLY slowed things down.

 

I know this seems petty, but I'm using this script to analyze a 1,000,000 line file... so efficiency is key. 

[BigDaddyO]

How long are these strings?

You really should look into 2D arrays more as they are extremely useful. I usually try to remember 2D arrays as they would show in a spreadsheet:  $Array2D[row][column]

best way to speed up the FileWite is to make sure you do a FileOpen instead of specifying the filename in the FileWrite.

[Earthshine]

 

oops

Edited June 4, 2018 by Earthshine

[corrado33]

10 minutes ago, BigDaddyO said:

How long are these strings?

You really should look into 2D arrays more as they are extremely useful. I usually try to remember 2D arrays as they would show in a spreadsheet:  $Array2D[row][column]

best way to speed up the FileWite is to make sure you do a FileOpen instead of specifying the filename in the FileWrite.

I suppose you're right. I just tend to avoid 2d arrays since most languages handle them slightly differently and I don't feel like dealing with syntax. 

As for string length, the first is ~5 chars, the 2nd is a full file path on a unix system about... 10 folders deep. So ~100+ chars

And yes, I learned the FileOpen trick!

[Melba23]

corrado33,

Are you using ReDim on every pass to resize the arrays? If so then I am not surprised that your script slows down after 500 or so entries.

The trick is to initially declare the array at a some intermediate size and then double it whenever it gets full - look at the final example in the Recursion tutorial in the Wiki to see it in action.

M23

 

[corrado33]

55 minutes ago, Melba23 said:

corrado33,

Are you using ReDim on every pass to resize the arrays? If so then I am not surprised that your script slows down after 500 or so entries.

The trick is to initially declare the array at a some intermediate size and then double it whenever it gets full - look at the final example in the Recursion tutorial in the Wiki to see it in action.

M23

 

Interesting. Is it significantly slower simply to dimension the array to something greater than I'll ever have? Currently the script is only working on ~2000 pairs of data, so I dimensioned the array to 3000 (more than it'll ever need) then at the end I remove the null entries. Would it be quicker to do what you're doing? 

 

Also I did just change the script to read in the massive file (1 million lines) to an array and that exponentially sped up the processing speed of the script. (To the point where I don't even have to worry about this question anymore.)

[Melba23]

corrado33,

Every ReDim slows down the script, so having none at all is bound to be faster. However, unless you are absolutely sure that you will never exceed the initial array size, I would suggest not hard coding the maximum size when declaring the array and using a dynamic resize as indicated in the example I showed you.

M23