Jump to content

_FileReadToArray() fails. Most of the time


Recommended Posts

Attempting to extract data from CSV file to display in _ExtMsgBox, sometimes it works fine but mostly it fails with Error:

3 - File lines have different numbers of fields (only if $FRTA_INTARRAYS flag not set)

I have tried all the available $iFlags combinations without success.

The CSV file format is fixed for any data collected, sample attached. The two CSV MsgBox() show correct file name

$sText = ""

Global $aLines, $fFile, $fLine, $fFcopy

   $fFile = FileOpen ( "QuickPlotData.dat", 0 )  ; find current CSV file
   $fLine = FileReadLine ( $fFile )        ; read correct CSV file name
   MsgBox(0,"CSV file: ", $fLine)

; Read file to 2D array
  MsgBox(0,"CSV[2] file: ", $fLine)
;  _FileReadToArray($fLine, $aLines)  ; 'simple' call
  _FileReadToArray($fLine, $aLines,$FRTA_NOCOUNT+$FRTA_ENTIRESPLIT,',')
      If @error Then
         MsgBox($MB_SYSTEMMODAL, "ERROR!", "CSV read error " & @error)
         exit(0)
      EndIf

  _ArrayDisplay($aLines)

 

ZZj2_Test.Won.Two.csv

The most powerful number in the Universe.  Zero.

Link to comment
Share on other sites

  • Developers

Would you say that all records in the attached file have the same number of columns on each record defined separated by a comma?
Can't test but guess the header records need to be stripped from the file first.

Jos

Edited by Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

  • Developers

Ok, just tested a bit and it seems that the Function requires the exact number of columns defined in each records , which is clearly not the case in the inputfile.

Quote

When a delimiter is specified the function tries to further split each line of the file - how this is done depends on the setting of the $FRTA_INTARRAYS flag. If the flag is not set and each line has the same number of fields split by the delimiter then a 2D array is created, but if this is not the case then @error is set to 3 and no array is returned. If the $FRTA_INTARRAYS flag is set the function creates a 1D array where each element is a further array holding the fields of the line split on the delimiter - the lines do not need to have the same number of fields. See example below.

Jos

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Link to comment
Share on other sites

You may have a point, Jos. Opening the CSV in NP++ it's

ZZj2_Test HauteurAH V7.0.01        11/07/2017    11:16:50
Lot: Won                             Sample: Two                  User: PeterH
Ha: 69.4 mm    %<15mm 0.1   L5 % 130.8   Ba: 87.6
CvH: 51.1 mm   %<25mm 5.3   L1 % 153.2   CvB: 41.3
Ha FREQ,Ha HIST,Ba FREQ,Ba HIST,,HAUTEUR,,,,,,BARBE
100.0,0.0,100.0,0.0,Ha,69.4,,,,Ba,87.6
100.0,0.0,100.0,0.0,CvH,51.1,,,,CvB,41.3
100.0,0.0,100.0,0.0,K10,0.0,L95,24.8,,K10,0.0,L95,33.5
100.0,0.0,100.0,0.0,K15,0.1,L90,30.5,,K15,0.0,L90,41.5
100.0,0.0,100.0,0.0,K20,2.0,L75,43.5,,K20,0.5,L75,56.8
100.0,0.0,100.0,0.0,K25,5.3,L50,61.8,,K25,1.5,L50,84.0
100.0,0.0,100.0,0.0,K30,9.7,L25,93.5,,K30,3.3,L25,112.5
100.0,0.0,100.0,0.0,K35,15.0,L10,119.0,,K35,5.7,L10,131.2
100.0,0.0,100.0,0.0,K40,20.7,L5,130.8,,K40,8.8,L5,142.8
100.0,0.0,100.0,0.0,K45,26.9,L2.5,0.0,,K45,12.6,L2.5,152.2
100.0,0.0,100.0,0.0,K50,33.7,L1,153.2,,K50,17.2,L1,156.0
100.0,0.0,100.0,0.0

There are ca. 500 rows below this but all identical apart from data.

You think the first five rows are screwing up the ReadToArray()?

I actually only need rows 6 to 16 and columns 5 to 12 which, when it doesn't error, display perfectly.

So I edit out the first 5 rows:

100.0,0.0,100.0,0.0,Ha,69.4,,,,Ba,87.6
100.0,0.0,100.0,0.0,CvH,51.1,,,,CvB,41.3
100.0,0.0,100.0,0.0,K10,0.0,L95,24.8,,K10,0.0,L95,33.5
100.0,0.0,100.0,0.0,K15,0.1,L90,30.5,,K15,0.0,L90,41.5
100.0,0.0,100.0,0.0,K20,2.0,L75,43.5,,K20,0.5,L75,56.8
100.0,0.0,100.0,0.0,K25,5.3,L50,61.8,,K25,1.5,L50,84.0
100.0,0.0,100.0,0.0,K30,9.7,L25,93.5,,K30,3.3,L25,112.5
100.0,0.0,100.0,0.0,K35,15.0,L10,119.0,,K35,5.7,L10,131.2
100.0,0.0,100.0,0.0,K40,20.7,L5,130.8,,K40,8.8,L5,142.8
100.0,0.0,100.0,0.0,K45,26.9,L2.5,0.0,,K45,12.6,L2.5,152.2
100.0,0.0,100.0,0.0,K50,33.7,L1,153.2,,K50,17.2,L1,156.0
100.0,0.0,100.0,0.0
100.0,0.0,100.0,0.0

but I still get error 3.

 

However, when I revert to

_FileReadToArray($fLine, $aLines)

it works without error. :)

Is there any way I can get ReadToArray() to begin on row 6?

The most powerful number in the Universe.  Zero.

Link to comment
Share on other sites

1 hour ago, ViciousXUSMC said:
_FileReadToArray($sFilePath, $aFile)

For $i = 7 to UBound($aFile)
Next

That should get you line 6 and onward, then you can further process/parse the data in the loop.

Unfortunately _FileReadToArray() spits the dummy before the UBound() call so I'm still stuffed :'(

The most powerful number in the Universe.  Zero.

Link to comment
Share on other sites

I have this working, its crude and probably can be improved but if I get what you are trying to do (read a file to a array with delimiter) it works!

Replace test.txt with one of your files.

#Include <Array.au3>
#Include <File.au3>

If FileExists(@TempDir & "\temp.txt") Then FileDelete(@TempDir & "\Temp.txt")

Local $aFile
Local $aFileNew
Local $aTest

_FileReadToArray(@ScriptDir & "\test.txt", $aFile)

For $i = 1 To $aFile[0]
$sNew = $aFile[$i]
Do
$sNew = StringRegExpReplace($sNew, "(.*)", "${1},", 1)
;MsgBox(0, "", $sNew)
$aTest = StringRegExp($sNew, ",", 3)
$iCount = UBound($aTest)
Until $iCount = 13
$aFile[$i] = $sNew
Next

;_ArrayDisplay($aFile)

_FileWriteFromArray(@TempDir & "\temp.txt", $aFile, 1)

_FileReadToArray(@TempDir & "\temp.txt", $aFileNew, 1, ",")

_ArrayDisplay($aFileNew)

 

Link to comment
Share on other sites

  • Moderators

fopetesl,

The 2D _FileReadToArray function was written (by me) to split "square" csv files - i.e. those with the same number of elements on each line. This is a design decision to warn if the array is not "square" as would be expected for the majority of csv files.

If you want to read files like yours with different numbers of elements on each line then you need to do some work yourself:

#include <Array.au3>
#include <File.au3>

$sCSV = "ZZj2_Test.Won.Two.csv"

Global $aLines
_FileReadToArray($sCSV, $aLines)

;_ArrayDisplay($aLines, "", Default, 8)

Global $aLines2D[$aLines[0]][1]

For $i = 1 To $aLines[0]

    $aSplit = StringSplit($aLines[$i], ",")

    If $aSplit[0] > UBound($aLines2D, 2) Then
        ReDim $aLines2D[UBound($aLines2D)][$aSplit[0]]
    EndIf

    ;_ArrayDisplay($aSplit, "", Default, 8)

    For $j = 1 To $aSplit[0]

        $aLines2D[$i - 1][$j - 1] = $aSplit[$j]

    Next

    ;_ArrayDisplay($aLines2D, "", Default, 8)

Next

_ArrayDisplay($aLines2D, "", Default, 8)

As to  "I did try $FRTA_INTARRAYS but the eventual 1D array just showed 630+ times 1{array} with no data, i.e. didn't seem to split each row" - that is exactly what the Help file tells you will happen - each element of the array is another array split on the delimiter. Once again it is up to you get these elements into a 2D array, using logic not dissimilar to that in the example above.

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to comment
Share on other sites

A .csv file must adhere to the RFC standard or desist from calling itself .csv

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

You will need\should use _FileReadToArrayEx ,  It must be faster and it will have to do exactly All what you want them to do

#include <Array.au3>
#include "_FileReadToArrayEx.au3"

Local $aArray
$fTimerDiff = TimerInit()
$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", $FRTA_ARRAYFIELD + 0 + $FRTA_STRIPALL + $FRTA_CHECKSINGEQUOTE)
;$aArray = _FileReadToArrayEx(@DesktopDir & '\Test.inf', ",", 0 + 1 + 16 + 12)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite("_FileReadToArrayEx() : " & $fTimerDiff & @CRLF)
_ArrayDisplay($aArray)
_ArrayDisplay(($aArray[1]))
_ArrayDisplay(($aArray[6]))
#include <Array.au3>
#include "_FileReadToArrayEx.au3"

Local $aArray, $sData

$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", BitOR($FRTA_NOCOUNT, $FRTA_ARRAYFIELD, $FRTA_STRIPALL))
If Not @Error Then
    ;;Local $iaArray = UBound($aArray) - 1   ; or $iaArray = @Extended
    For $i = 0 To UBound($aArray) - 1
        For $y = 0 To UBound($aArray[$i]) - 1
            $sData &= ($aArray[$i])[$y] & ","
        Next
        $sData = StringTrimRight($sData, 1) & @CRLF
    Next
EndIf
ConsoleWrite($sData & @LF)

;Or

$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", BitOR($FRTA_ARRAYFIELD, $FRTA_STRIPALL, $FRTA_CHECKSINGEQUOTE))
If Not @Error Then
    ;;Local $iaArray = UBound($aArray) - 1   ; or $iaArray = @Extended
    For $i = 1 To ($aArray)[0]
        For $y = 1 To ($aArray[$i])[0]
            $sData &= ($aArray[$i])[$y] & ","
        Next
        $sData = StringTrimRight($sData, 1) & @CRLF
    Next
EndIf
ConsoleWrite($sData & @LF)

Ciao.

Edited by DXRW4E

apps-odrive.pngdrive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Link to comment
Share on other sites

Impressive help appreciated. This seems to have resoled the issue

For $i = 7 to UBound($fLine)
Next
_FileReadToArray($fLine, $aLines)
      If @error Then
         MsgBox($MB_SYSTEMMODAL, "ERROR!", "CSV read error " & @error)
         exit(0)
      EndIf

I get to display the data I want and haven't had a failure :)

Thanks for your help. Grazie mille.

The most powerful number in the Universe.  Zero.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...