Jump to content
corrado33

Fastest way to write lots of data to file?

Recommended Posts

corrado33

Hi, I have a script where I have to deal with a couple thousand pairs of strings. Currently, I'm storing the two strings in two arrays (because 2d arrays are a pain). However, I've noticed a few things.

The script slows down once I've stored ~500 pairs of strings in the array. At least, I THINK that's why the script slows down. So I'm debating offloading the strings to a file every 200 strings or so, but I want to find the fastest way to write them to a file.

 

Would it be quicker to use "FileWriteLine" or "_FileWriteFromArray"? Obviously with "FileWriteLine" I'd have to loop through the array. 

 

I debated doing "FileWriteLine" EVERY iteration of my main loop, but that REALLY slowed things down.

 

I know this seems petty, but I'm using this script to analyze a 1,000,000 line file... so efficiency is key. 

Share this post


Link to post
Share on other sites
BigDaddyO

How long are these strings?

You really should look into 2D arrays more as they are extremely useful. I usually try to remember 2D arrays as they would show in a spreadsheet:  $Array2D[row][column]

best way to speed up the FileWite is to make sure you do a FileOpen instead of specifying the filename in the FileWrite.


Share this post


Link to post
Share on other sites
Earthshine
Posted (edited)

 

oops

Edited by Earthshine

My resources are limited. You must ask the right questions

 

Share this post


Link to post
Share on other sites
corrado33
10 minutes ago, BigDaddyO said:

How long are these strings?

You really should look into 2D arrays more as they are extremely useful. I usually try to remember 2D arrays as they would show in a spreadsheet:  $Array2D[row][column]

best way to speed up the FileWite is to make sure you do a FileOpen instead of specifying the filename in the FileWrite.

I suppose you're right. I just tend to avoid 2d arrays since most languages handle them slightly differently and I don't feel like dealing with syntax. 

As for string length, the first is ~5 chars, the 2nd is a full file path on a unix system about... 10 folders deep. So ~100+ chars

And yes, I learned the FileOpen trick!

Share this post


Link to post
Share on other sites
Melba23

corrado33,

Are you using ReDim on every pass to resize the arrays? If so then I am not surprised that your script slows down after 500 or so entries.

The trick is to initially declare the array at a some intermediate size and then double it whenever it gets full - look at the final example in the Recursion tutorial in the Wiki to see it in action.

M23

 


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites
corrado33
55 minutes ago, Melba23 said:

corrado33,

Are you using ReDim on every pass to resize the arrays? If so then I am not surprised that your script slows down after 500 or so entries.

The trick is to initially declare the array at a some intermediate size and then double it whenever it gets full - look at the final example in the Recursion tutorial in the Wiki to see it in action.

M23

 

Interesting. Is it significantly slower simply to dimension the array to something greater than I'll ever have? Currently the script is only working on ~2000 pairs of data, so I dimensioned the array to 3000 (more than it'll ever need) then at the end I remove the null entries. Would it be quicker to do what you're doing? 

 

Also I did just change the script to read in the massive file (1 million lines) to an array and that exponentially sped up the processing speed of the script. (To the point where I don't even have to worry about this question anymore.)

Share this post


Link to post
Share on other sites
Melba23

corrado33,

Every ReDim slows down the script, so having none at all is bound to be faster. However, unless you are absolutely sure that you will never exceed the initial array size, I would suggest not hard coding the maximum size when declaring the array and using a dynamic resize as indicated in the example I showed you.

M23

  • Like 1

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×