Skysnake

(solved) write CSV from Array efficiently

6 posts in this topic

#1 ·  Posted (edited)

Hi there

I have searched and found many posts on READING CSV.  What I need to do is WRITE CSV.

Typically I have a Listview, I use @guinness's excellent  _GUICtrlListView_CreateArray to read that LV to Array. Currently I use _FileWriteFromArray to write the output.  It is fast and it is easy.

The benefits of using these Array functions are their inherit flexibility.  The code is portable.  Plug it into LV report and you can write to file instantly.  No formatting issues.

To be sure, my problem is not with the Listview or Array.  My problem is with the file write to CSV: it should be fast, accurate and efficient.

Part of the problem is Microsoft Excel's insistence that my CSVs are not what they appear to be... :(

 

I am inspired by the simplicity of SQLITE3.EXE's dot command options:

.headers on
.mode csv
.once output.csv

 

Problem is that I want to export the text packed in characters, such as ' " ' with fields delimited by another character, such as ' , '.

Current method using _FileWriteFromArray outputs data like this

12-04-2016|           0.00|         131.00|131|Description
12-04-2016|           0.00|         132.00|132|Description 132 rece
12-04-2016|         998.00|           0.00|998|Description Receipt
12-04-2016|           0.00|         900.00|900|Description

What I want to achieve is this:

"12-04-2016","           0.00","         131.00","131","Description"
"12-04-2016","           0.00","         132.00","132","Description 132 rece"
"12-04-2016","         998.00","           0.00","998","Description Receipt"
"12-04-2016","           0.00","         900.00","900","Description"

I have done this. The long way. I have written the output one $aResult[$i][1] item at a time, spaced with the $text & $field markers.  I have placed this in an array with additional columns just to fit in the formatting. I have also played with the idea of inserting the array into a sqlite db, then run sqlite3's dot commands on that.

Is there a different method?  Is there an efficient method to specify FileWriteFromArraytoCSV?  Does anyone have ideas?  I am sure I will know how to code this, I just need advice on the most efficient way of going about it. If I missed any threads, please enlighten  me.

Thanks for reading

Edited by Skysnake

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites

i think this

_ReplaceStringInFile('Test.csv','|','","')
_ReplaceStringInFile('Test.csv',@CRLF,'"'&@CRLF&'"')

does most of the job. Just inserting a '"' at begin and deliting last line and whole job is done.

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

@czardas you deserve 5 stars.  Excellent. 

Thank you kindly

Edited by Skysnake
1 person likes this

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites

It's old code, but it's also robust and has not yet let me down. I'm happy if it helps.

Share this post


Link to post
Share on other sites

I should perhaps post that the default code works faultlessly on Win7 with OpenOffice, and on Win10 with MS Office Excel 2013. :) 

Even Excel opened the file without complaints ;)

 

1 person likes this

Skysnake

Why is the snake in the sky?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • ShawnW
      By ShawnW
      I have a script that takes a large excel file, pulls out and reorganizes certain information I need, and spits out a trimmed down csv file which I uses to upload the information on my website. Some of this information contains characters with accents or em dashes. By default it would create a csv file in ANSI which I then uploaded but had to tell my website import system it was windows-1252 in order for it to look correct.
      This was all working fine except now I need to add in a non-breaking space and non-breaking hyphen into parts of my output. At first I tried using ChrW(0xA0) and ChrW(0x2011) as replacements. A quick test in the console looked correct, however opening the csv output in notepad++ showed the space correctly but a ? for the hyphen and the file was still encoded as ANSI. I tried to view it as UTF-8 instead but this just made the space appear as xAO and also other characters appeared that way like my em dashes appeared as x97 and another symbol as xA7 etc.
      If I instead do a convert to UTF-8 from notepad++ then those problems go away except the hyphen still displays as ?. I then noticed on the page I linked for the non-breaking hyphen it lists the UTF-8 hex as 0xE2 0x80 0x91 (e28091). I was unsure how to enter this in autoit but several things i tried all failed to get the hyphen inserted.
      I need a way to get both the space and hyphen added correctly as either ANSI or UTF-8, but if it is UTF-8 then I need a way to convert all of the other data I extracted from the excel file.
      I've included a test excel file with a single line and test script to create a csv demonstrating the problem.
      test.xlsx
      test.au3
    • Ascer
      By Ascer
      1. Description.
      Udf working with MSDN System.Collections.ArrayList. Allow you to make fast operations on huge arrays, speed is even x10 better than basic _ArrayAdd.  Not prefered for small arrays < 600 items. 2. Requirements
      .NET Framework 1.1 - 4.5 (on this version Microsoft destroy old rules) System Windows 3. Possibilities.
      ;=============================================================================================================== ; UDF Name: List.au3 ; ; Date: 2018-02-17, 10:52 ; Description: Simple udf to create System Collections as ArrayList and make multiple actions on them. ; ; Function(s): _ListCreate -> Creates a new list ; _ListCapacity -> Gets a list size in bytes ; _ListCount -> Gets items count in list ; _ListIsFixedSize -> Get bool if list if fixed size ; _ListIsReadOnly -> Get bool if list is read only ; _ListIsSynchronized -> Get bool if list is synchronized ; _ListGetItem -> Get item on index ; _ListSetItem -> Set item on index ; ; _ListAdd -> Add item at end of list ; _ListClear -> Remove all list items ; _ListClone -> Duplicate list in new var ; _ListContains -> Get bool if item is in list ; _ListGetHashCode -> Get hash code for list ; _ListGetRange -> Get list with items between indexs ; _ListIndexOf -> Get index of item ; _ListInsert -> Insert a new item on index ; _ListInsertRange -> Insert list into list on index ; _ListLastIndexOf -> Get index last of item ; _ListRemove -> Remove first found item ; _ListRemoveAt -> Remove item in index ; _ListRemoveRange -> Remove items between indexs ; _ListReverse -> Reverse all items in list ; _ListSetRange -> Set new value for items in range ; _ListSort -> Sort items in list (speed of reading) ; _ListToString -> Get list object name ; _ListTrimToSize -> Remove unused space in list ; ; Author(s): Ascer ;=============================================================================================================== 4. Downloads
      List.au3 5. Examples
      SpeedTest _ArrayAdd vs ListAdd SpeedTest ArraySearch vs ListIndexOf Basic usage - crating guild with members  
    • ronmage
      By ronmage
      So I have a loop that keeps reading data from an array and searching it for the same value. If the value is no there it does work then adds the value to the array to prevent it from doing the same work.
      If _ArraySearch($ID,$filearray[$i]) = -1 Then Work.... _ArrayAdd($ID,$filearray[$i]) EndIf This is in a for loop hence $i
      So what is happening is the code works great for several hours. After a period of time _ArraySearch($ID,$filearray[$i]) will result in -1 even if $ID = $filearray. So it ready as if there is no data in the array. Anyone have this problem? 
       
      Also I am just running in using F5 not compiling it and running it if that makes a difference.
       
    • FMS
      By FMS
      Hello,
      I'm trying to read a binary file to an array but couln't get it to work.
      Also I coul not find any help in the forum around this subject whish was helpfull.
      Is there any way it could be done?
      I tried a lot of ways but maybe somebody know's the right way?
      #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7 #include <File.au3> #include <Array.au3> #include <AutoItConstants.au3> Local $in=FileOpen("TEST_labels.idx1-ubyte",16) ; 16+0=Read binary Local $data = FileRead($in) Local $FileArray = BinaryToString($data,4) ;~ $FileArray = StringSplit($BinarydData, @CRLF, 1+2) ;~ Local $FileArray = StringRegExp($BinarydData, "[^\r\n]+", 3) FileClose($in) _ArrayDisplay($FileArray,"$FileArray","",32) MsgBox($MB_SYSTEMMODAL, "", "$FileArray = " & $FileArray )  
      TEST_labels.idx1-ubyte
    • ur
      By ur
      I am reading a CSV file which is tab seperated as below.
      Local $aArray = FileReadToArray($file) And now, I am splitting this main array record wise so that Array contains internally another arrow to represent each row.
      For $i = 0 to (UBound($aArray) - 1) ;MsgBox(0,"",$aArray[$i]) $aArray[$i] = StringSplit(StringStripCR($aArray[$i]), Chr(9),2);Chr(9) for tab ;_ArrayDisplay($aArray[$i]) Next Afther that, _ArrayDIsplay is able to see the individual internal arrays.
      _ArrayDisplay($aArray[1]) But If I try to access the individual element of it as below.It is not showing any result.
      MsgBox(0,"",$aArray[1][1]) Any suggestion, below is the sample csv file.
      New Text Document.csv