_FileReadToArray() fails. Most of the time

[fopetesl]

Attempting to extract data from CSV file to display in _ExtMsgBox, sometimes it works fine but mostly it fails with Error:

3 - File lines have different numbers of fields (only if $FRTA_INTARRAYS flag not set)

I have tried all the available $iFlags combinations without success.

The CSV file format is fixed for any data collected, sample attached. The two CSV MsgBox() show correct file name

$sText = ""

Global $aLines, $fFile, $fLine, $fFcopy

   $fFile = FileOpen ( "QuickPlotData.dat", 0 )  ; find current CSV file
   $fLine = FileReadLine ( $fFile )        ; read correct CSV file name
   MsgBox(0,"CSV file: ", $fLine)

; Read file to 2D array
  MsgBox(0,"CSV[2] file: ", $fLine)
;  _FileReadToArray($fLine, $aLines)  ; 'simple' call
  _FileReadToArray($fLine, $aLines,$FRTA_NOCOUNT+$FRTA_ENTIRESPLIT,',')
      If @error Then
         MsgBox($MB_SYSTEMMODAL, "ERROR!", "CSV read error " & @error)
         exit(0)
      EndIf

  _ArrayDisplay($aLines)

 

ZZj2_Test.Won.Two.csv

[Jos]

Would you say that all records in the attached file have the same number of columns on each record defined separated by a comma?
Can't test but guess the header records need to be stripped from the file first.

Jos

Edited July 11, 2017 by Jos

[Jos]

Ok, just tested a bit and it seems that the Function requires the exact number of columns defined in each records , which is clearly not the case in the inputfile.

Quote

When a delimiter is specified the function tries to further split each line of the file - how this is done depends on the setting of the $FRTA_INTARRAYS flag. If the flag is not set and each line has the same number of fields split by the delimiter then a 2D array is created, but if this is not the case then @error is set to 3 and no array is returned. If the $FRTA_INTARRAYS flag is set the function creates a 1D array where each element is a further array holding the fields of the line split on the delimiter - the lines do not need to have the same number of fields. See example below.

Jos

[fopetesl]

You may have a point, Jos. Opening the CSV in NP++ it's

ZZj2_Test HauteurAH V7.0.01        11/07/2017    11:16:50
Lot: Won                             Sample: Two                  User: PeterH
Ha: 69.4 mm    %<15mm 0.1   L5 % 130.8   Ba: 87.6
CvH: 51.1 mm   %<25mm 5.3   L1 % 153.2   CvB: 41.3
Ha FREQ,Ha HIST,Ba FREQ,Ba HIST,,HAUTEUR,,,,,,BARBE
100.0,0.0,100.0,0.0,Ha,69.4,,,,Ba,87.6
100.0,0.0,100.0,0.0,CvH,51.1,,,,CvB,41.3
100.0,0.0,100.0,0.0,K10,0.0,L95,24.8,,K10,0.0,L95,33.5
100.0,0.0,100.0,0.0,K15,0.1,L90,30.5,,K15,0.0,L90,41.5
100.0,0.0,100.0,0.0,K20,2.0,L75,43.5,,K20,0.5,L75,56.8
100.0,0.0,100.0,0.0,K25,5.3,L50,61.8,,K25,1.5,L50,84.0
100.0,0.0,100.0,0.0,K30,9.7,L25,93.5,,K30,3.3,L25,112.5
100.0,0.0,100.0,0.0,K35,15.0,L10,119.0,,K35,5.7,L10,131.2
100.0,0.0,100.0,0.0,K40,20.7,L5,130.8,,K40,8.8,L5,142.8
100.0,0.0,100.0,0.0,K45,26.9,L2.5,0.0,,K45,12.6,L2.5,152.2
100.0,0.0,100.0,0.0,K50,33.7,L1,153.2,,K50,17.2,L1,156.0
100.0,0.0,100.0,0.0

There are ca. 500 rows below this but all identical apart from data.

You think the first five rows are screwing up the ReadToArray()?

I actually only need rows 6 to 16 and columns 5 to 12 which, when it doesn't error, display perfectly.

So I edit out the first 5 rows:

100.0,0.0,100.0,0.0,Ha,69.4,,,,Ba,87.6
100.0,0.0,100.0,0.0,CvH,51.1,,,,CvB,41.3
100.0,0.0,100.0,0.0,K10,0.0,L95,24.8,,K10,0.0,L95,33.5
100.0,0.0,100.0,0.0,K15,0.1,L90,30.5,,K15,0.0,L90,41.5
100.0,0.0,100.0,0.0,K20,2.0,L75,43.5,,K20,0.5,L75,56.8
100.0,0.0,100.0,0.0,K25,5.3,L50,61.8,,K25,1.5,L50,84.0
100.0,0.0,100.0,0.0,K30,9.7,L25,93.5,,K30,3.3,L25,112.5
100.0,0.0,100.0,0.0,K35,15.0,L10,119.0,,K35,5.7,L10,131.2
100.0,0.0,100.0,0.0,K40,20.7,L5,130.8,,K40,8.8,L5,142.8
100.0,0.0,100.0,0.0,K45,26.9,L2.5,0.0,,K45,12.6,L2.5,152.2
100.0,0.0,100.0,0.0,K50,33.7,L1,153.2,,K50,17.2,L1,156.0
100.0,0.0,100.0,0.0
100.0,0.0,100.0,0.0

but I still get error 3.

 

However, when I revert to

_FileReadToArray($fLine, $aLines)

it works without error.

Is there any way I can get ReadToArray() to begin on row 6?

[fopetesl]

I did try $FRTA_INTARRAYS but the eventual 1D array just showed 630+ times 1{array} with no data, i.e. didn't seem to split each row.

[ViciousXUSMC]

2 hours ago, fopetesl said:

Is there any way I can get ReadToArray() to begin on row 6?

_FileReadToArray($sFilePath, $aFile)

For $i = 6 to UBound($aFile)
Next

That should get you line 6 and onward, then you can further process/parse the data in the loop.

Edited July 11, 2017 by ViciousXUSMC

[fopetesl]

1 hour ago, ViciousXUSMC said:

_FileReadToArray($sFilePath, $aFile)

For $i = 7 to UBound($aFile)
Next

That should get you line 6 and onward, then you can further process/parse the data in the loop.

Unfortunately _FileReadToArray() spits the dummy before the UBound() call so I'm still stuffed

[ViciousXUSMC]

I have this working, its crude and probably can be improved but if I get what you are trying to do (read a file to a array with delimiter) it works!

Replace test.txt with one of your files.

#Include <Array.au3>
#Include <File.au3>

If FileExists(@TempDir & "\temp.txt") Then FileDelete(@TempDir & "\Temp.txt")

Local $aFile
Local $aFileNew
Local $aTest

_FileReadToArray(@ScriptDir & "\test.txt", $aFile)

For $i = 1 To $aFile[0]
$sNew = $aFile[$i]
Do
$sNew = StringRegExpReplace($sNew, "(.*)", "${1},", 1)
;MsgBox(0, "", $sNew)
$aTest = StringRegExp($sNew, ",", 3)
$iCount = UBound($aTest)
Until $iCount = 13
$aFile[$i] = $sNew
Next

;_ArrayDisplay($aFile)

_FileWriteFromArray(@TempDir & "\temp.txt", $aFile, 1)

_FileReadToArray(@TempDir & "\temp.txt", $aFileNew, 1, ",")

_ArrayDisplay($aFileNew)

 

[Melba23]

fopetesl,

The 2D _FileReadToArray function was written (by me) to split "square" csv files - i.e. those with the same number of elements on each line. This is a design decision to warn if the array is not "square" as would be expected for the majority of csv files.

If you want to read files like yours with different numbers of elements on each line then you need to do some work yourself:

#include <Array.au3>
#include <File.au3>

$sCSV = "ZZj2_Test.Won.Two.csv"

Global $aLines
_FileReadToArray($sCSV, $aLines)

;_ArrayDisplay($aLines, "", Default, 8)

Global $aLines2D[$aLines[0]][1]

For $i = 1 To $aLines[0]

    $aSplit = StringSplit($aLines[$i], ",")

    If $aSplit[0] > UBound($aLines2D, 2) Then
        ReDim $aLines2D[UBound($aLines2D)][$aSplit[0]]
    EndIf

    ;_ArrayDisplay($aSplit, "", Default, 8)

    For $j = 1 To $aSplit[0]

        $aLines2D[$i - 1][$j - 1] = $aSplit[$j]

    Next

    ;_ArrayDisplay($aLines2D, "", Default, 8)

Next

_ArrayDisplay($aLines2D, "", Default, 8)

As to  "I did try $FRTA_INTARRAYS but the eventual 1D array just showed 630+ times 1{array} with no data, i.e. didn't seem to split each row" - that is exactly what the Help file tells you will happen - each element of the array is another array split on the delimiter. Once again it is up to you get these elements into a 2D array, using logic not dissimilar to that in the example above.

M23

[jchd]

A .csv file must adhere to the RFC standard or desist from calling itself .csv

[DXRW4E]

You will need\should use _FileReadToArrayEx ,  It must be faster and it will have to do exactly All what you want them to do

#include <Array.au3>
#include "_FileReadToArrayEx.au3"

Local $aArray
$fTimerDiff = TimerInit()
$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", $FRTA_ARRAYFIELD + 0 + $FRTA_STRIPALL + $FRTA_CHECKSINGEQUOTE)
;$aArray = _FileReadToArrayEx(@DesktopDir & '\Test.inf', ",", 0 + 1 + 16 + 12)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite("_FileReadToArrayEx() : " & $fTimerDiff & @CRLF)
_ArrayDisplay($aArray)
_ArrayDisplay(($aArray[1]))
_ArrayDisplay(($aArray[6]))

#include <Array.au3>
#include "_FileReadToArrayEx.au3"

Local $aArray, $sData

$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", BitOR($FRTA_NOCOUNT, $FRTA_ARRAYFIELD, $FRTA_STRIPALL))
If Not @Error Then
    ;;Local $iaArray = UBound($aArray) - 1   ; or $iaArray = @Extended
    For $i = 0 To UBound($aArray) - 1
        For $y = 0 To UBound($aArray[$i]) - 1
            $sData &= ($aArray[$i])[$y] & ","
        Next
        $sData = StringTrimRight($sData, 1) & @CRLF
    Next
EndIf
ConsoleWrite($sData & @LF)

;Or

$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", BitOR($FRTA_ARRAYFIELD, $FRTA_STRIPALL, $FRTA_CHECKSINGEQUOTE))
If Not @Error Then
    ;;Local $iaArray = UBound($aArray) - 1   ; or $iaArray = @Extended
    For $i = 1 To ($aArray)[0]
        For $y = 1 To ($aArray[$i])[0]
            $sData &= ($aArray[$i])[$y] & ","
        Next
        $sData = StringTrimRight($sData, 1) & @CRLF
    Next
EndIf
ConsoleWrite($sData & @LF)

Ciao.

Edited July 11, 2017 by DXRW4E

[fopetesl]

Impressive help appreciated. This seems to have resoled the issue

For $i = 7 to UBound($fLine)
Next
_FileReadToArray($fLine, $aLines)
      If @error Then
         MsgBox($MB_SYSTEMMODAL, "ERROR!", "CSV read error " & @error)
         exit(0)
      EndIf

I get to display the data I want and haven't had a failure

Thanks for your help. Grazie mille.