Re-write of _FileReadToArray()

[Melba23]

Hi,

_FileWriteFromArray now writes from both 1D and 2D arrays - so we thought it about time that _FileReadToArray could read both types as well. Here is a Beta version of the new function for testing and comments.

A few points before starting:

 

- 1. This function returns an array - there is no longer the need to pre-declare one and pass it as a parameter.

- 2. The $iFlags parameter determines how the function proceeds:

+ 1 (works for both 1D and 2D arrays) adds a count in the top row - this is the default setting.

+ 2 (for 2D arrays only) Creates an "array of arrays" where each element contains the split line as an array.

+ 4. (only valid of you set a delimiter) works like the $STR_ENTIRESPLIT flag in StringSplit and uses all the delimiter characters as the split point. By default splits occur for each character.

- 3. If you try to read a "non-square" file (one that does not have the same number of fields on each line) and do not set the 2 flag then the function returns an error. This is meant to act as an error check where an expected file format is not found.

Have fun:

expandcollapse popup

#include <Array.au3>
#include <File.au3>

Global $aRetArray

; Create 1D array
Global $aArray[] = ["0", "1", "2", "3", "4"]
; Write it to file
_FileWriteFromArray("Test.txt", $aArray, Default, Default, @CRLF)
Sleep(1000)

; Re-read it - with count
$aRetArray = _FileReadToArray_New("Test.txt")
ConsoleWrite("Error returned: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "1D array - count", Default, 8)

; Re-read it - without count
$aRetArray = _FileReadToArray_New("Test.txt", 0)
ConsoleWrite("Error returned: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "1D array - no count", Default, 8)

; Create "square" 2D array
Global $aArray[][] = [ _
        ["00", "01", "02", "03"], _
        ["10", "11", "12", "13"], _
        ["20", "21", "22", "23"], _
        ["30", "31", "32", "33"]]
_ArrayDisplay($aArray, "Original", Default, 8)
; Write it to file
_FileWriteFromArray("Test.txt", $aArray, Default, Default, ",")
Sleep(1000)

; Re-read it - with count
$aRetArray = _FileReadToArray_New("Test.txt", Default, ",")
ConsoleWrite("Error returned: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "2D array - count", Default, 8)

; Re-read it - without count
$aRetArray = _FileReadToArray_New("Test.txt", 0, ",")
ConsoleWrite("Error returned: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "2D array - no count", Default, 8)

; Read into "array of arrays"
$aRetArray = _FileReadToArray_New("Test.txt", 2, ",")
ConsoleWrite("Error returned: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "Array of arrays", Default, 8)
; Now look inside the arrays inside the returned array
_ArrayDisplay($aRetArray[1], "Array 1 inside RetArray", Default, 8)

; Rewrite 2D array with multiple delimiters
_FileWriteFromArray("Test.txt", $aArray, Default, Default, ":|")
Sleep(1000)

; Re-read with each delimiter acting as a split point
$aRetArray = _FileReadToArray_New("Test.txt", 0, ":|")
ConsoleWrite("Error: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "Split on each character", Default, 8)

; Re-read with whole delimiter acting as a split point
$aRetArray = _FileReadToArray_New("Test.txt", 4, ":|")
ConsoleWrite("Error: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "Split on full delimiter", Default, 8)

Func _FileReadToArray_New($sFilePath, $iFlags = 1, $sDelimiter = "")

    ; $iFlags: + 1 = Dimensions in top row (default) - 1D: [0] = rows ; 2D: [0][0]/[0][1] = rows/cols
    ;          + 2 = Create 1D "array of arrays" allowing lines to have different numbers of fields
    ;          + 4 = Use entire delimiter string as split point (default each character defines a split point)

    ; Error: 1 = Error opening specified file
    ;        2 = Empty file
    ;        3 = File not "square" and flag 2 not used
    ;        4 = No delimiters found

    If $iFlags = Default Then $iFlags = 1
    If $sDelimiter = Default Then $sDelimiter = ""

    ; Set "array of arrays" flag
    Local $fExpand = True
    If BitAnd($iFlags, 2) Then
        $fExpand = False
        $iFlags -= 2
    EndIf
    ; Set delimiter flag
    Local $iEntire = 0 ; $STR_CHRSPLIT
    If BitAnd($iFlags, 4) Then
        $iEntire = 1 ; $STR_ENTIRESPLIT
        $iFlags -= 4
    EndIf
    ; Set row count and split count flags
    Local $iNoCount = 0
    If $iFlags <> 1 Then
        $iFlags = 0
        $iNoCount = 2 ; $STR_NOCOUNT
    EndIf

    ; Read file into an array
    Local $aLines = FileReadToArray($sFilePath)
    If @error Then Return SetError(@error, 0, 0)
    ; If 1D with no count return array directly
    If Not $iFlags And Not $sDelimiter Then Return $aLines
    ; Declare variables
    Local $aArray, $aSplit_2
    ; Get first dimension and add count if required
    Local $iDim_1 = UBound($aLines) + $iFlags
    ; Check delimiter
    If $sDelimiter Then
        ; Check type of return array
        If $fExpand Then ; All lines have same number of fields
            Local $iFields
            ; Count fields in first line
            Local $iDim_2 = UBound(StringSplit($aLines[0], $sDelimiter, $iEntire + $STR_NOCOUNT))
            ; Size array
            Local $aArray[$iDim_1][$iDim_2]
            ; Loop through the lines
            For $i = 0 To $iDim_1 - $iFlags - 1
                ; Split each line as required
                $aSplit_2 = StringSplit($aLines[$i], $sDelimiter, $iEntire + $STR_NOCOUNT)
                ; Count the items
                $iFields = UBound($aSplit_2)
                If $iFields <> $iDim_2 Then
                    ; Return error
                    Return SetError(3, 0, 0)
                EndIf
                ; Fill this line of the array
                For $j = 0 To $iFields - 1
                    $aArray[$i + $iFlags][$j] = $aSplit_2[$j]
                Next
            Next
            ; Check at least 2 columns
            If $iDim_2 < 2 Then Return SetError(4, 0, 0)
            ; Set dimension count
            If $iFlags Then
                $aArray[0][0] = $iDim_1 - $iFlags
                $aArray[0][1] = $iDim_2
            EndIf
        Else ; Create "array of arrays"
            ; Size array
            Local $aArray[$iDim_1]
            ; Loop through the lines
            For $i = 0 To $iDim_1 - $iFlags - 1
                ; Split each line as required
                $aArray[$i + $iFlags] = StringSplit($aLines[$i], $sDelimiter, $iEntire + $iNoCount)
            Next
            ; Set dimension count
            If $iFlags Then
                $aArray[0] = $iDim_1 - $iFlags
            EndIf
        EndIf
    Else ; 1D
        ; Declare array of correct size and set count
        Local $aArray[UBound($aLines) + 1] = [UBound($aArray) - 1]
        ; Copy data
        For $i = 0 To UBound($aLines) - 1
            $aArray[$i + 1] = $aLines[$i]
        Next
    EndIf
    ; Return the array
    Return $aArray

EndFunc

If you break the function or the returned array is not what you expected, please post the SciTE output pane, the calling line and the file concerned so that I can debug it.

M23

Edited January 11, 2014 by Melba23
New code - 1401101000

[l3ill]

Thanks M23 !  Nice work, Very Cool !

One thing I have a question about:

It supports all kinds of delimiters; certain count of WS, combined punctuation like ":," & ":_"  Very Neat.

But if the file does not contain the chosen delimiter the script exits without any error and without ArrayDisplay.

the reason I am asking is because of this.

; Error: 1 = Error opening specified file
    ;        2 = Empty file
    ;        3 = No delimiter found (2D only)
    ;        4 = No expand flag and unequal items in lines (2D only)
    ;        5 = Only 1 column (2D only)

I was expecting the #3 error to show up.

Bill

>Running:(3.3.10.2):C:Program FilesAutoIt3autoit3.exe

p.s. is there a way to combine (use more than one) delimiter(s) ?

p.s.s. Setting Delimiter not forcing 2D array when $iFlags set to Default

$aRetArray = _FileReadToArray_New("Test.txt", Default, ":")

Edited January 9, 2014 by l3ill

[Melba23]

l3ill,

When I run the UDF on a file without any delimiters I get error 5 returned as the resultant array has only the one column because the lines were unable to be split. Can you (as requested) post your test file and calling line so I can test it myself.

In fact looking again at the code I see that error 3 no longer exists - it has been overtaken by 4 & 5 during development of the UDF. I will remedy that - thanks for pointing it out.

M23

Edit: Sorry, forgot the PS and PPS (Post Post Scriptum).

- 1. It would be fairly easy to have a range of delimiters using an SRE, but before I consider this can you give me an example of where such functionality would be useful?

- 2. What do you get returned when using that calling line and what does the file look like (I did ask for these things )? Just to be clear, a 2D array is only forced if there is a valid file with delimiters - if an error occurs, you get no return at all.

Edit 2: Reposted the code with renumbered errors - for 4/5 above now read 3/4

Edited January 9, 2014 by Melba23

[l3ill]

M23,

  1. it was just something I wondered while testing, for instance a txt that uses colons and hyphens to separate info. (see example)

  2. With the example file below the script exits with no errors when using; Default, 0 or 1 as the $iFlags

I also tried the updated still not getting any errors.

Test.txt

DELIVERY DATE:30.10.2013
ROUTE: 13
DRIVER- A
D:50
F:44
DELIVERY DATE:30.10.2013
ROUTE: 13
DRIVER- B
D:38
F:39
DELIVERY DATE:30.10.2013
ROUTE: 13
DRIVER- C
D:50
F:48
DELIVERY DATE:30.10.2013
ROUTE: 13
DRIVER- H
D:39
F:33
   --->___Drink___Totals:222__Fluoride___<---   
           210               164

Just for clarity this what I was using to test:

expandcollapse popup

#include <Array.au3>
#include <File.au3>

; Create array
;~ Global $aArray[][] = [ _
;~     ["00", "01", "02", "03"], _
;~     ["10", "11", "12", "13"], _
;~     ["20", "21", "22", "23"], _
;~     ["30", "31", "32", "33"]]
;~ _ArrayDisplay($aArray, "Original", Default, 8)

; Write it to file
;~ _FileWriteFromArray("Test.txt", $aArray, Default, Default, ",")

;~ Sleep(1000)

; Re-read it
$aRetArray = _FileReadToArray_New("Test.txt", 2, ":")
_ArrayDisplay($aRetArray)

Func _FileReadToArray_New($sFilePath, $iFlags = 1, $sDelimiter = "")

    ; $iFlags: + 1 = dimensions in top row  - either [0] or [0][0]/[0][1]
    ;          + 2 = expand array during fill if required (2D only)

    ; Error: 1 = Error opening specified file
    ;        2 = Empty file
    ;        3 = No delimiter found (2D only)
    ;        4 = No expand flag and unequal items in lines (2D only)
    ;        5 = Only 1 column (2D only)

    If $iFlags = Default Then $iFlags = 1
    If $sDelimiter = Default Then $sDelimiter = ""

    ; Set array expansion flag
    Local $fExpand = False
    If BitAnd($iFlags, 2) Then
        $fExpand = True
        $iFlags -= 2
    EndIf
    ; Set count flag to numeric 1 or 0
    If $iFlags <> 1 Then $iFlags = 0
    ; Read file into an array
    Local $aLines = FileReadToArray($sFilePath)
    If @error Then Return SetError(@error, 0, 0)
    ; If 1D with no count return array directly
    If Not $iFlags And Not $sDelimiter Then Return $aLines
    ; Declare return array
    Local $aArray, $iDim_2
    If $sDelimiter Then ; 2D
        Local $aSplit_2, $iItems
        ; Add count line if required
        Local $iDim_1 = UBound($aLines) + $iFlags
        ; Set initial dimension
        If $fExpand Then
            ; Will increase automatically
            $iDim_2 = 1
        Else
            ; Count items in first line
            StringReplace($aLines[0], $sDelimiter, "")
            $iDim_2 = @extended + 1
        EndIf
        ; Size array
        Local $aArray[$iDim_1][$iDim_2]
        ; Loop through the lines
        For $i = 0 To $iDim_1 - $iFlags - 1
            ; Split each line (use full delimiter if more than 1 char)
            $aSplit_2 = StringSplit($aLines[$i], $sDelimiter, $STR_ENTIRESPLIT + $STR_NOCOUNT)
            ; Count the items
            $iItems = UBound($aSplit_2)
            If $iItems <> $iDim_2 Then
                If $fExpand Then
                    ; Expand the array if required
                    If $iItems > $iDim_2 Then
                        $iDim_2 = $iItems
                        ReDim $aArray[$iDim_1][$iDim_2]
                    EndIf
                Else
                    ; Or return error if not expandable
                    Return SetError(4, 0, 0)
                EndIf
            EndIf
            ; Fill this line of the array
            For $j = 0 To $iItems - 1
                $aArray[$i + $iFlags][$j] = $aSplit_2[$j]
            Next
        Next
        ; Check at least 2 columns
        If $iDim_2 < 2 Then Return SetError(5, 0, 0)
        ; Set dimension count
        If $iFlags Then
            $aArray[0][0] = $iDim_1 - $iFlags
            $aArray[0][1] = $iDim_2
        EndIf
    Else ; 1D
        ; Declare array of correct size and set count
        Local $aArray[UBound($aLines) + 1] = [UBound($aArray) - 1]
        ; Copy data
        For $i = 0 To UBound($aLines) - 1
            $aArray[$i + 1] = $aLines[$i]
        Next
    EndIf
    ; Return the array
    Return $aArray

EndFunc

 

[Melba23]

l3ill,

When I run that script on that file I get @error 0 and the following array returned:

Row                  Col 0                Col 1                
[0]                  DELIVERY DATE        30.10.2013           
[1]                  ROUTE                 13                  
[2]                  DRIVER- A                                 
[3]                  D                    50                   
[4]                  F                    44                   
[5]                  DELIVERY DATE        30.10.2013           
[6]                  ROUTE                 13                  
[7]                  DRIVER- B                                 
[8]                  D                    38                   
[9]                  F                    39                   
[10]                 DELIVERY DATE        30.10.2013           
[11]                 ROUTE                 13                  
[12]                 DRIVER- C                                 
[13]                 D                    50                   
[14]                 F                    48                   
[15]                 DELIVERY DATE        30.10.2013           
[16]                 ROUTE                 13                  
[17]                 DRIVER- H                                 
[18]                 D                    39                   
[19]                 F                    33                   
[20]                    --->___Drink___To 222__Fluoride___<--- 
[21]                            210      164

Which is what I would expect. Is that not what you get?

M23

[l3ill]

Yes that one works fine.

As mentioned in post 2:

Choosing a Del that doesn't exist for instance:

$aRetArray = _FileReadToArray_New("Test.txt", 2, "/")

or using Default as the $iFlags with a working Del

$aRetArray = _FileReadToArray_New("Test.txt", Default, ":")

Both exit script without error.

But you mentioned in post 3 that:

 

if an error occurs, you get no return at all

 

which is okay with me...I was only wondering why the 3 = No delimiter found (2D only)

wasn't showing up...

[Melba23]

l3ill,

Here is a version which will split on multiple delimiters - it has probems with your file because there are hyphens in the "arrows" on the penultimate line. Look for the <<<<<<<<<<<< line to see how it is done:

<snip>

But I am still unconvinced of the need for this functionality.

M23

Edit: Our posts crossed - look at the code above.

Edit 2: Embarrassingly the functionality was already built-in and I did not realise - see new code in first post.

Edited January 9, 2014 by Melba23

[guinness]

StringSplit() can also split and multiple delimiters too, or did I misunderstand?

[Melba23]

guinness,

StringSplit can use multiple characters as the single delimiter - here we are talking about using several characters as separate delimiters in the same line:

Delimiter = ":-"
Line = 1:-2

StringSplit           : a single split on ":-" to give a 2 element array: 1|2
As requested by l3ill : 2 splits (on ":" and "-") to give a 3 element array: 1||2

M23 Edited January 9, 2014 by Melba23
Fixed tags

[guinness]

Try this Melba23.

$aArray = StringSplit('test;test,test|test', ';|,', $STR_NOCOUNT) ; $STR_ENTIRESPLIT will split where :- is, without it splits at : or -

Edited January 9, 2014 by guinness

[Mat]

Does this take into account that items in CSV files can be enclosed in quotes?

[Melba23]

guinness,

Amazing - how did I ever miss that!

M23

Mat,

No - elements are straight copies of the content between the delimiters. I do not think the function should get too complex - it is not a CSV extractor but a general purpose file reader to complement FileWriteToArray.

Edited January 9, 2014 by Melba23

[guinness]

guinness,

Amazing - how did I ever miss that!

M23

It's used in the Date.au3 quite a bit. So no need for SRE

[l3ill]

M23, 

Thanks I will test tomorrow -late here...

the multi delimiter was for a txt file like this one for instance

DELIVERY DATE:30.10.2013
ROUTE - 13
DRIVER - H
D:39
F:33

It would give you the same array as before the one posted above but to split the strings with your Function you would need both

" - " & " : "

Edited January 9, 2014 by l3ill

[guinness]

Oh, and that's why when you split at @CRLF you get blank array elements, because it's splitting at @CR or @LF.

[Melba23]

guinness,

I will add another flag (similar to the one in StringSplit) to specify whether to use each or all of the delimiter characters to split. Thanks again for pointing that out.

M23

Edit: I knew about @CRLF but the full meaning just never clicked - must be age!

Edited January 9, 2014 by Melba23

[l3ill]

M23,

  Thanks for the added flag! works great !

But now the question - what’s the syntax to use a multi character delimiter :-) 

For some reason I still can’t get it to spit out any errors. I have fed it empty files, incorrect named files, shouldn’t the error handler be printing this info to the console?

Also, not sure if it was intentional but your updated script in the OP is missing the Includes and Example.

Thanks again !!!

Bill

[Melba23]

l3ill,

 

what’s the syntax to use a multi character delimiter

Add 4 to $iFlags to use the delimiter string as a single entity for splitting - do not add 4 and it will split on each character.

 

shouldn’t the error handler be printing this info to the console?

No, the function returns a value in @error - if you want to see it, you need to use ConsoleWrite to display it.

Here is an example showing both those points:

<snip>

All clear?

M23

Edited January 11, 2014 by Melba23
Removed code

[l3ill]

 

Add 4 to $iFlags to use the delimiter string as a single entity for splitting - do not add 4 and it will split on each character.

Are you sure you don't mean 6 ?

Because 4 gives me nothing but Error 3 and a ArrayDisplay Error: No array variable passed to function

$aRetArray = _FileReadToArray_New("Test.txt", 6, ":-")
ConsoleWrite("Error returned: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "Split on each character", Default, 8)

$aRetArray = _FileReadToArray_New("Test.txt", 4, ":- ")
ConsoleWrite("Error returned: " & @error & @CRLF)
_ArrayDisplay($aRetArray, "Split on whole delimiter", Default, 8)

>Running:(3.3.10.2):C:\Program Files\AutoIt3\autoit3.exe "C:\Users\Badass\Desktop\163740.au3"    
--> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
Error returned: 0
Error returned: 4

DELIVERY DATE:30.10.2013
ROUTE: 13
DRIVER: A
D:-50
F:44
DELIVERY DATE:30.10.2013
ROUTE: 13
DRIVER: B
D:-38
F:39
DELIVERY DATE:30.10.2013
ROUTE- 13
DRIVER- C
D:-50
F:48
DELIVERY DATE:30.10.2013
ROUTE- 13
DRIVER- H
D:-39
F:33

[Melba23]

l3ill,

You have a typo in that snippet (":- " - has an added space in the second call) - and I tested on a file which had the same number of items on each line when I wrote the examples this morning. So 15-all I think!

Try this with full explanation of the flags:

<snip>

Thanks for all the testing, by the way.

M23

Edited January 11, 2014 by Melba23
Removed code