Jump to content
Skysnake

(solved) write CSV from Array efficiently

Recommended Posts

Skysnake

Hi there

I have searched and found many posts on READING CSV.  What I need to do is WRITE CSV.

Typically I have a Listview, I use @guinness's excellent  _GUICtrlListView_CreateArray to read that LV to Array. Currently I use _FileWriteFromArray to write the output.  It is fast and it is easy.

The benefits of using these Array functions are their inherit flexibility.  The code is portable.  Plug it into LV report and you can write to file instantly.  No formatting issues.

To be sure, my problem is not with the Listview or Array.  My problem is with the file write to CSV: it should be fast, accurate and efficient.

Part of the problem is Microsoft Excel's insistence that my CSVs are not what they appear to be... :(

 

I am inspired by the simplicity of SQLITE3.EXE's dot command options:

.headers on
.mode csv
.once output.csv

 

Problem is that I want to export the text packed in characters, such as ' " ' with fields delimited by another character, such as ' , '.

Current method using _FileWriteFromArray outputs data like this

12-04-2016|           0.00|         131.00|131|Description
12-04-2016|           0.00|         132.00|132|Description 132 rece
12-04-2016|         998.00|           0.00|998|Description Receipt
12-04-2016|           0.00|         900.00|900|Description

What I want to achieve is this:

"12-04-2016","           0.00","         131.00","131","Description"
"12-04-2016","           0.00","         132.00","132","Description 132 rece"
"12-04-2016","         998.00","           0.00","998","Description Receipt"
"12-04-2016","           0.00","         900.00","900","Description"

I have done this. The long way. I have written the output one $aResult[$i][1] item at a time, spaced with the $text & $field markers.  I have placed this in an array with additional columns just to fit in the formatting. I have also played with the idea of inserting the array into a sqlite db, then run sqlite3's dot commands on that.

Is there a different method?  Is there an efficient method to specify FileWriteFromArraytoCSV?  Does anyone have ideas?  I am sure I will know how to code this, I just need advice on the most efficient way of going about it. If I missed any threads, please enlighten  me.

Thanks for reading

Edited by Skysnake

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites
AutoBert

i think this

_ReplaceStringInFile('Test.csv','|','","')
_ReplaceStringInFile('Test.csv',@CRLF,'"'&@CRLF&'"')

does most of the job. Just inserting a '"' at begin and deliting last line and whole job is done.

Share this post


Link to post
Share on other sites
Skysnake

@czardas you deserve 5 stars.  Excellent. 

Thank you kindly

Edited by Skysnake
  • Like 1

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites
czardas

It's old code, but it's also robust and has not yet let me down. I'm happy if it helps.

Share this post


Link to post
Share on other sites
Skysnake

I should perhaps post that the default code works faultlessly on Win7 with OpenOffice, and on Win10 with MS Office Excel 2013. :) 

Even Excel opened the file without complaints ;)

 

  • Like 1

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • rm4453
      By rm4453
      Hello,
       
      I am currently writing a program that parses a massive table from a website, and need a way to add a progress bar while parsing.
      I am currently using the function _IETableWriteToArray($oObj, True) to parse the array. I need the progress bar to update as the table is parsed, not just at the end of the parsing.
      Any help at all would be very much appreciated!
       
      *EDIT --> The array I am left with after parsing is $array[0-50000][16]
    • TrashBoat
      By TrashBoat
      So Im trying to make a simple 2d game and make some sort of collision detection so why not to make a 2 dimensional array but i have no clue how  to write it in multiple lines
      Global $map[5,5] = [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0] something like this but it doesn't work
    • Zein
      By Zein
      #include "..\Include\Array.au3" #include "..\Include\File.au3" #include "..\Include\AutoItConstants.au3" Local $aRetArray Local $sFilePath = "n.csv" _FileReadToArray($sFilePath, $aRetArray, ",") ; _FileReadToArray($sFilePath, $aRetArray, $FRTA_COUNT, ",") _ArrayDisplay($aRetArray, "Original", Default, 8) The above code shows two versions of _FileReadToArray and both don't work as expected.
      The first one doesn't use the comma as a delimiter. (so I get a single column array)  I tried adding "Default" between $aRetArray and "," then it told me it had an incorrect number of parameters. 

      I looked again at the documentation:
       
      #include <File.au3> _FileReadToArray ( $sFilePath, ByRef $vReturn [, $iFlags = $FRTA_COUNT [, $sDelimiter = ""]] )
      And I with or without the flags params I should be getting a 2D array due to my file being a csv. 
      I then tried a regular flag, $FRTA_COUNT, and it tells me that I'm using a variable $FRTA_COUNT while it's not declared. Tried putting in 1 instead and it told me again, incorrect number of params. 

       
    • ternal
      By ternal
      Hi,
      Recently I have had the need to do a sort and then do a second sort while the item of the first sort stays the same ( double sorting , first on column x then while column x is the same sort column y).
      I did not put much efffort into error checking but so far I did not need it.
      For my applications so far it works perfectly however if someone is willing I want to test this extensivly.
      If anyone has big lists of random stuff to sort could you try this out please?
      #include <Array.au3> ; #FUNCTION# ==================================================================================================================== ; Name ..........: _ArraySort_Double ; Description ...: ; Syntax ........: _ArraySort_Double (Byref $array[, $first_index = Default[, $second_index = Default[, $ascending = Default]]]) ; Parameters ....: $array - 2d array to sort. ; $first_index - [optional] first column to sort. Default is 0. ; $second_index - [optional] second column to sort. Default is 1. ; $ascending - [optional] ascending/descending. Default is 1. ; Return values .: 1 if no errors occured , -1 if errors occured ; Author ........: Ternal ; Remarks .......: Needs excessive testing. ; Related .......: _arraysort() ; =============================================================================================================================== Func _ArraySort_Double (byref $array, $first_index = Default, $second_index = Default, $ascending = Default) Local $temp_value Local $counter = 1 If UBound($array, $UBOUND_DIMENSIONS) <> 2 Then MsgBox(0, "error", "error") return -1 EndIf If $first_index = Default Then $first_index = 0 If $second_index = Default Then $second_index = 1 If $ascending = Default Then $ascending = 1 _ArraySort($array, $ascending, 0, 0, $first_index); you can alter settings of primary sort here If @error Then MsgBox(0, "error", @error) return -1 EndIf $temp_value = $array[0][$first_index] For $x = 1 to UBound($array, 1) - 1 If Mod( $x, 10000) = 0 Then ConsoleWrite("at " & $x & " of a total : " & UBound($array, 1) & @CRLF) If $array[$x][$first_index] = $temp_value Then $counter+= 1 If $x = UBound($array, 1) - 1 Then; do last line here(if last line is not a new item) _ArraySort($array, $ascending, $x - $counter, $x, $second_index);you can alter settings of secondary sort here(don't forget to place line 34 the exact same) If @error Then MsgBox(0, "error", @error) return -1 EndIf EndIf Else If $counter > 0 Then ;at least 2 of the same _ArraySort($array, $ascending, $x - $counter, $x - 1, $second_index);you can alter settings of secondary sort here(don't forget to place line 29 the exact same) If @error Then MsgBox(0, "error", @error) return -1 EndIf $counter = 1 EndIf EndIf $temp_value = $array[$x][$first_index] Next Return 1 EndFunc Kind regards, Ternal
    • TrashBoat
      By TrashBoat
      So I've made this script that detects how long i have held down my left mouse button for and stores the information in an array and then sorts its using _ArraySort but the output is half sorted half broken.
      Here's my script:
      HotKeySet("{F1}","_exit") #include <Misc.au3> #include <Timers.au3> #include <Array.au3> Local $dll = DllOpen("user32.dll") $on = False Global $array[0] While(1) If _IsPressed(01,$dll) Then $timer = _Timer_Init() While _IsPressed(01,$dll) Sleep(1) WEnd $time = _Timer_Diff($timer) _ArrayAdd($array,"Time: " & Floor($time) & " ms") ;~ ConsoleWrite("Time: " & Floor($time) & " ms" & @CRLF) EndIf Sleep(50) WEnd Func _exit() _ArraySort($array) _ArrayDisplay($array) Exit EndFunc And the output:

      See how its not sorted?  What is the problem here?
×