Skysnake

(solved) write CSV from Array efficiently

6 posts in this topic

#1 ·  Posted (edited)

Hi there

I have searched and found many posts on READING CSV.  What I need to do is WRITE CSV.

Typically I have a Listview, I use @guinness's excellent  _GUICtrlListView_CreateArray to read that LV to Array. Currently I use _FileWriteFromArray to write the output.  It is fast and it is easy.

The benefits of using these Array functions are their inherit flexibility.  The code is portable.  Plug it into LV report and you can write to file instantly.  No formatting issues.

To be sure, my problem is not with the Listview or Array.  My problem is with the file write to CSV: it should be fast, accurate and efficient.

Part of the problem is Microsoft Excel's insistence that my CSVs are not what they appear to be... :(

 

I am inspired by the simplicity of SQLITE3.EXE's dot command options:

.headers on
.mode csv
.once output.csv

 

Problem is that I want to export the text packed in characters, such as ' " ' with fields delimited by another character, such as ' , '.

Current method using _FileWriteFromArray outputs data like this

12-04-2016|           0.00|         131.00|131|Description
12-04-2016|           0.00|         132.00|132|Description 132 rece
12-04-2016|         998.00|           0.00|998|Description Receipt
12-04-2016|           0.00|         900.00|900|Description

What I want to achieve is this:

"12-04-2016","           0.00","         131.00","131","Description"
"12-04-2016","           0.00","         132.00","132","Description 132 rece"
"12-04-2016","         998.00","           0.00","998","Description Receipt"
"12-04-2016","           0.00","         900.00","900","Description"

I have done this. The long way. I have written the output one $aResult[$i][1] item at a time, spaced with the $text & $field markers.  I have placed this in an array with additional columns just to fit in the formatting. I have also played with the idea of inserting the array into a sqlite db, then run sqlite3's dot commands on that.

Is there a different method?  Is there an efficient method to specify FileWriteFromArraytoCSV?  Does anyone have ideas?  I am sure I will know how to code this, I just need advice on the most efficient way of going about it. If I missed any threads, please enlighten  me.

Thanks for reading

Edited by Skysnake

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites



i think this

_ReplaceStringInFile('Test.csv','|','","')
_ReplaceStringInFile('Test.csv',@CRLF,'"'&@CRLF&'"')

does most of the job. Just inserting a '"' at begin and deliting last line and whole job is done.

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

@czardas you deserve 5 stars.  Excellent. 

Thank you kindly

Edited by Skysnake
1 person likes this

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites

It's old code, but it's also robust and has not yet let me down. I'm happy if it helps.

Share this post


Link to post
Share on other sites

I should perhaps post that the default code works faultlessly on Win7 with OpenOffice, and on Win10 with MS Office Excel 2013. :) 

Even Excel opened the file without complaints ;)

 

1 person likes this

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • czardas
      By czardas
      Haven't had much time to code recently. However the following thread inspired me.
      The debate about linear, parallel and binary search methods was rather interesting and, in an attempt to be diplomatic, I decided to combine @jchd's suggestion with @LarsJ's binary search example. I decided that the binary search algorithm required modification to make it more linear. As usual, 'if you invent something, it probably already exists and if it already exists, it exists for a reason'. My first attempt was not all that good. The code worked but was really a mess. I blame peer pressure (to post an example of a parallel search method).  I will delete that old code in due course.
      With a little memory jogging and a glance at the help file, the solution turned out to be quite easy: I just needed a better understanding of Euler. Further modification will be needed to work with more complicated unicode strings. The output could be returned as an array or a delimitered string. I'm not so interested in those details. I'm just going to post the algorithm for now and anyone, who wants to, can modify it to suit their needs. Both arrays must contain at least 1 element.
      Local $aFoo = [0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,19,20,23,24,26,30,35,39,40,41] Local $aBar = [0,1,5,6,7,8,9,10,11,12,13,14,17,18,19,21,24,25,26,27,34,35,38,40] ParallelExponetialSearch($aFoo, $aBar) ; Compares two lists - returning positive matches. Each input array must be unique (individually) and in alphabetical order. Func ParallelExponetialSearch($aFoo, $aBar) Local $sFind, _ $iMin_F = -1, $iMax_F = UBound($aFoo) -1, $Lo_F = $iMin_F, $Hi_F, _ $iMin_B = -1, $iMax_B = UBound($aBar) -1, $Lo_B = $iMin_B, $Hi_B While $iMin_F < $iMax_F And $iMin_B < $iMax_B ; Toggle Arrays - Which array has most untested elements? This is the one we want to search next, ; so we can bypass more comparisons because (in theory) mismatches have a greater chance of being skipped. If $iMax_F - $iMin_F >= $iMax_B - $iMin_B Then ; $aFoo has more (or an equal number of) untested elements $Hi_F = $iMax_F $iMin_B += 1 $sFind = $aBar[$iMin_B] While $Lo_F < $Hi_F ; search $aFoo For $i = 0 To Floor(Log($Hi_F - $Lo_F) / Log(2)) $Lo_F = $iMin_F + 2^$i If $aFoo[$Lo_F] = $sFind Then $iMin_F = $Lo_F ; each match should be added to the output [perhaps an array] ConsoleWrite($sFind & " found at $aFoo[" & $Lo_F & "] = $aBar[" & $iMin_B & "]" & @LF) ExitLoop 2 ElseIf $aFoo[$Lo_F] > $sFind Then $Hi_F = $Lo_F -1 $iMin_F += Floor(2^($i -1)) $Lo_F = $iMin_F ContinueLoop 2 EndIf Next $iMin_F = $Lo_F ; minimum increment is one WEnd Else ; $aBar has more untested elements $Hi_B = $iMax_B $iMin_F += 1 $sFind = $aFoo[$iMin_F] While $Lo_B < $Hi_B ; search $aBar For $i = 0 To Floor(Log($Hi_B - $Lo_B) / Log(2)) $Lo_B = $iMin_B + 2^$i If $aBar[$Lo_B] = $sFind Then $iMin_B = $Lo_B ; each match should be added to the output [perhaps an array] ConsoleWrite($sFind & " found at $aFoo[" & $iMin_F & "] = $aBar[" & $Lo_B & "]" & @LF) ExitLoop 2 ElseIf $aBar[$Lo_B] > $sFind Then $Hi_B = $Lo_B -1 $iMin_B += Floor(2^($i -1)) $Lo_B = $iMin_B ContinueLoop 2 EndIf Next $iMin_B = $Lo_B ; minimum increment is one WEnd EndIf WEnd EndFunc ;==> ParallelExponetialSearch I hope this will be useful to someone. I believe it deserved a thread of its own!
    • RC86
      By RC86
      Afternoon!
      I have a requirement to return both a string and an array from a function so as a result I put them both into an array and returned that.  I can access them in their entirety after returning them but then I can't seem to access the array elements after this.  Should I be able to or is there a prettier way?
      #include <Array.au3> ;Memory info returned as a string and an array $memoryInfo = _getMemoryInfo() msgbox(0,"Memory Info",$memoryInfo[0]) _ArrayDisplay($memoryInfo[1],"Memory as an Array") Local $newArray[7] $memoryInfo[1] = $newArray msgbox(0,"Test element",$newArray[0]) _ArrayDisplay($newArray) Func _getMemoryInfo() Local $newArray[7] Local $array = MemGetStats() $newArray[0] = $array[0] ;% of memory in use $newArray[1] = Round($array[1]/1024 * 0.001,2) ;Total physical RAM $newArray[2] = Round($array[2]/1024 * 0.001,2) ;Availaible physical RAM $newArray[3] = Round($array[3]/1024 * 0.001,2) ;Total pagefile $newArray[4] = Round($array[4]/1024 * 0.001,2) ;Available pagefile $newArray[5] = Round($array[5]/1024 * 0.001,2) ;Total virtual $newArray[6] = Round($array[6]/1024 * 0.001,2) ;Available virtual $memoryUsage = $newArray[1] - $newarray[2] $pagefileUsage = $newArray[3] - $newarray[4] ;Output/Return Local $returnArray[2] $returnArray[0] = "Memory: " & $memoryUsage & " GB/" & $newArray[1] & " GB " & @CRLF & "Pagefile: " & $pagefileUsage & " GB/" & $newArray[3] & " GB " $returnArray[1] = $newArray return $returnArray EndFunc A bit messy but hopefully it's understandable what I'm trying to achieve.
      Thanks!
    • Jibberish
      By Jibberish
      I am having difficulties getting updated results from GUICtrlCreateInput. I wrote a small script to demonstrate what I am trying to do.
      This script reads a text file into an array and displays the Name and Number from the text file.
      The user can check a name and change the number, and the results are displayed in _ArrayDisplay.
      Only the Checked names are displayed in their original position. This is necessary for future
      plumbing.
      The problem with this script is the original number is returned, not the changed number. I cannot
      figure out how to get the updated number to be displayed.
      ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ; ; This script reads a text file into an array and displays the Name and Number from the text file. ; The user can check a name and change the number, and the results are displayed in _ArrayDisplay. ; Only the Checked names are displayed in their original position. This is necessary for future ; plumbing. ; ; The problem with this script is the original number is returned, not the changed number. I cannot ; figure out how to get the updated number to be displayed. ; ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #include-once #include <MsgBoxConstants.au3> #include <array.au3> #include <Date.au3> #include <WinAPIFiles.au3> #include <AutoItConstants.au3> #include <Misc.au3> #include <File.au3> #include <GUIConstantsEx.au3> #include <StaticConstants.au3> Local $iMax = 6 ;Default number of Videos - 1 for the array counter Dim $aTextFile[$iMax][2] ; [start with 5 entries][Name, Number] Don't get confused! [Row][Column] Local $i = 0 Local $iLeft = 30 Local $iTop = 30 Local $sName Local $sNumber Local $aArray Local $iMaxCol = 5 Local $iRow = 0 Local $iCol = 0 Local $aNumberCount[$iMaxCol][2] $aArray1 = ReadFile() $aNamesNumbers = DisplayNames($aArray1) _ArrayDisplay($aNamesNumbers) ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ; ; ReadFile() places the contents of the NameNumber.txt file in an array ; ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Func ReadFile() Local $sFileData = @ScriptDir & "\NameNumber.txt" Local $iStrReturn = 0 Local $aArrayFile Local $aTextFile[$iMaxCol][2] Local $aName Local $sCheckBox1 Const $iL = 10 ; Count from Left for GUI so all buttons line up Local $sFill Local $iA = 0, $iN = 0 ; Put the Names into an Array _FileReadToArray($sFileData, $aArrayFile) For $iA = 0 To $aArrayFile[0] ; Step through the array looking for Names If StringInStr($aArrayFile[$iA], "Name:") Then $aName = StringSplit($aArrayFile[$iA],":") $sFill = $aName[2] If UBound($aTextFile) <= $iRow Then ; Resize the array when $iRow is equal to the element count in the array to prevent subscript error ReDim $aTextFile[UBound($aTextFile) + 1][$iMaxCol] EndIf $aTextFile[$iRow][$iCol] = $sFill $iRow += 1 EndIf Next $iCol = 1 $iRow = 0 $sFill = "" For $iA = 1 To $aArrayFile[0] ; Step through the array looking for Numbers If StringInStr($aArrayFile[$iA], "Number:") Then $aName = StringSplit($aArrayFile[$iA],":") $sFill = $aName[2] If UBound($aTextFile) <= $iRow Then ; Resize the array when $iRow is equal to the element count in the array to prevent subscript error ReDim $aTextFile[UBound($aTextFile) + 1][$iMaxCol] EndIf $aTextFile[$iRow][$iCol] = $sFill $iRow += 1 EndIf Next Return $aTextFile EndFunc ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ; ; Display the Name and Number ; ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Func DisplayNames($aArray1) Local $aName[$iMax] Local $aNumbers[$iMax] $iMMCount = UBound($aArray1) $iMMCount -=1 Local $iWidth = 300 Local $iLength = 300;$iMMCount * 30 GUISetFont(12) $hGUI = GUICreate("Edit / Update Number Test", $iWidth, $iLength, -1, -1) GUICtrlCreateLabel(" Name Number",1,5) ;$iTop += 30 ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ; ; Display Name with a Checkbox. Only Checked Names should be saved. ; ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For $i = 0 to $iMMCount Step 1 $sNameText = $aArray1[$i][0] $iNameLength = StringLen($sNameText) $aName[$i] = GUICtrlCreateCheckbox($sNameText,$iLeft, $iTop) $iTop += 30 Next ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ; ; Display the number from the text file below each name. Allow the user to change the number and display the ; new number. ; ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $iLeft = 60 GUIStartGroup() $iTop = 28 For $i = 0 to $iMMCount Step 1 $sNumberText = $aArray1[$i][1] $aNumberCount[$i][1] = $sNumberText $aNumberCount[$i][0] = GUICtrlCreateInput($sNumberText,$iLeft, $iTop, 50,18, $GUI_DOCKAUTO) GUICtrlSetPos($aNumberCount[$i][0],200) $iTop += 30 Next ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ; ; Read the checked names and (possibly updated) number ; ;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ $iMMCount = UBound($aNumberCount) $iMMNewCount = $iMMCount - 1 Local $aGUICheckbox[$iMMCount] Local $aCheckedNameNumber[$iMMCount][2] $iLeft = 30 Local $idCloseGUI = GUICtrlCreateButton("Close",$iLeft, $iTop) GUISetState(@SW_SHOW) While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit Case $idCloseGUI For $i = 0 To $iMMNewCount Step 1 Switch GUICtrlRead($aName[$i]) Case $GUI_CHECKED $aCheckedNameNumber[$i][0] = $aArray1[$i][0] Case $GUI_UNCHECKED EndSwitch Switch GUICtrlRead($aNumberCount[$i][0]) Case $aCheckedNameNumber[$i][1] = GUICtrlRead($aNumberCount[$i][1]) EndSwitch Next ExitLoop EndSwitch WEnd For $i = 0 to $iMMNewCount step 1 GUICtrlRead($aNumberCount[$i][0]) If $aCheckedNameNumber[$i][0] <> "" Then $aCheckedNameNumber[$i][1] = $aNumberCount[$i][1] EndIf Next GUIDelete($hGUI) Return $aCheckedNameNumber EndFunc This is the text file I am reading. If you want to try this out put the NameNumber.txt file in your script directory. It's attached to the post.
      * This is a dummy file with a Name and Number * The only purpose of this file is to read the updated Number. Name:Taggart Number:916 Name:Mongo Number:90 Name:Hedley Lamarr Number:22 Name:Bart Number:9999 Name:The Waco Kid Number:2244 If I change the number, the original number is displayed at the end, not the updated/modified number. I need the modified number to be displayed.
      Thanks in advance for any assistance!
      Jibberish
      NameNumber.txt
    • Jibberish
      By Jibberish
      I am working on a Video Player Test Script. I am reading a text file into a 3d array to be displayed in a GUI.
      Array Content:
      [x][0]FileName.mp4
      [x][1]UsageCount -> Up to a 4 digit number
      [x][2]EnableUsageCount (True/False)
      In the GUI the user can check the filename box, edit the number of usages and check EnableUsageCount to turn on the UsageCount control using the UsageCount number for the maximum number of plays.
      To make this easier to deal with I have removed the EnableUsageCount section, and am just concentrating on getting a method to put the edited (or unchanged) UsageCount in the array. Currently nothing is read into the array, due to my using the wrong method in the Case statement.
      Here are code snippets of what I am trying to do:
      ; Snippets from script ; I read the text file at the bottom and put the .mp4 filenames in $aManifest[x][0] and UsageCount in $aManifest[x][1] ; Then I create a GUI to display the .mp4 filenames with checkboxes and the UsageCount to the right. UsageCount is editable by the user. ; If the filename is checked, I want to read the filename into $aCheckedVideos[x][0] and the updated UsageCount in $aCheckedVideos[x][1] ; The GUICtrlRead($aVideoName[$i]) with Case $GUI_CHECKED & UNCHECKED works for the checkboxes ; This section puts the filenames in the GUI with a checkbox For $i = 0 to $iMMCount Step 1 $sMP4Text = $aManifest[$i][0] $iMP4Length = StringLen($sMP4Text) $aVideoName[$i] = GUICtrlCreateCheckbox($sMP4Text,$iLeft, $iTop) $iTop += 30 Next ; This section reads numbers from the Manifest array, and I want to be able to change the number and have them saved. ; So the Case $GUI_CHECKED & UNCHECKED won't work here, and I can't figure out what I should be doing here. For $i = 0 to $iMMCount Step 1 $sUsageText = $aManifest[$i][1] $aUsageCount[$i] = GUICtrlCreateInput($sUsageText,$iLeft, $iTop, 50,18, $GUI_DOCKAUTO) GUICtrlSetPos($aUsageCount[$i],200) $iTop += 30 Next While 1 $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit Case $idCloseGUI For $i = 0 To $iMMNewCount Step 1 Switch GUICtrlRead($aVideoName[$i]) Case $GUI_CHECKED $aCheckedVideos[$i][0] = $aManifest[$i][0] ; If checked, I put the results into a two D array Case $GUI_UNCHECKED ; where Video Name is $aCheckedVideos[x][0] EndSwitch Switch GUICtrlRead($aUsageCounter[$i]) Case $GUI_CHECKED $aCheckedVideos[$i][1] = $aUsageCount[$i] ;I want to put the text (numbers) in $aCheckedVideos[x][1] Case $GUI_UNCHECKED ;but what is returned is blank, probably due to EndSwitch ; $GUI_CHECKED being the wrong thing. Next ExitLoop EndSwitch WEnd ;The txt file I'm reading has the following: ;~ /** Title #1: Big Buck Bunny 1080p **/ ;~ "Name": "Big Buck Bunny", ;~ "URI": "..\\MediaFiles\\bbb_1080_60s.mp4", ;~ "UsageCount": 9999, ;~ "URI": "..\\MediaFiles\\bbb_1080_60s_enc1.mp4", ;~ "UsageCount": 45, ;~ "URI": "..\\MediaFiles\\bbb_1080_60s_enc1.mp4", ;~ "UsageCount": 2, ;~ /** Title #2: Tears of Steel 4K **/ ;~ "Name": "Tears of Steel 4K", ;~ "URI": "..\\MediaFiles\\tos_4K_60s_HEVC.mp4", ;~ "UsageCount": 9876, ;~ "URI": "..\\MediaFiles\\tos_4K_60s_HEVC_enc2.mp4", ;~ "UsageCount": 0, ;~ "URI": "..\\MediaFiles\\tos_4K_60s_HEVC_enc2.mp4", ;~ "UsageCount": 5, I am certain that the section
                          Switch GUICtrlRead($aUsageCounter[$i])
                              Case $GUI_CHECKED
                                  $aCheckedVideos[$i][1] = $aUsageCount[$i]    ;I want to put the text (numbers) in $aCheckedVideos[x][1]
                              Case $GUI_UNCHECKED                                ;but what is returned is blank, probably due to
                          EndSwitch                                            ; $GUI_CHECKED being the wrong thing.

      is wrong, and this is what I am looking for help with. Instead of $GUI_CHECKED what should I be looking for?
      The worst part of this is I had this working late last night, and then lost my changes and cannot for the life of me remember how I had this working.
      Help is truly appreciated!
      Jibberish
       
    • algiuxas
      By algiuxas
      Hello everybody, so I might found a bug in _ArraySearch:
       
      #include <Array.au3> Local $abc[2] = ["b", "b"] $h = "+---------------------------------+" & @CRLF ConsoleWrite($h) For $i = 0 To 1 $abc[0] = 0 out() $abc[0] = 1 out() $abc[0] = -1 out() $abc[0] = "Abc" out() $abc[1] = "Hello" Next Func txt() Return "$abc = [" & $abc[0] & ", " & $abc[1] & "]" & @CRLF & _ "> _ArraySearch($abc,""Hello"") = " EndFunc ;==>txt Func shouldoutput($ans) Return "_ArraySearch Should return: " & ($ans ? 1 : -1) EndFunc ;==>shouldoutput Func out() ConsoleWrite("> " & txt() & _ArraySearch($abc, "Hello") & @CRLF & "> " & shouldoutput($abc[1] == "Hello") & @CRLF & $h) EndFunc ;==>out Exit Sorry for this messy script