fopetesl

_FileReadToArray() fails. Most of the time

12 posts in this topic

#1 ·  Posted

Attempting to extract data from CSV file to display in _ExtMsgBox, sometimes it works fine but mostly it fails with Error:

3 - File lines have different numbers of fields (only if $FRTA_INTARRAYS flag not set)

I have tried all the available $iFlags combinations without success.

The CSV file format is fixed for any data collected, sample attached. The two CSV MsgBox() show correct file name

$sText = ""

Global $aLines, $fFile, $fLine, $fFcopy

   $fFile = FileOpen ( "QuickPlotData.dat", 0 )  ; find current CSV file
   $fLine = FileReadLine ( $fFile )        ; read correct CSV file name
   MsgBox(0,"CSV file: ", $fLine)

; Read file to 2D array
  MsgBox(0,"CSV[2] file: ", $fLine)
;  _FileReadToArray($fLine, $aLines)  ; 'simple' call
  _FileReadToArray($fLine, $aLines,$FRTA_NOCOUNT+$FRTA_ENTIRESPLIT,',')
      If @error Then
         MsgBox($MB_SYSTEMMODAL, "ERROR!", "CSV read error " & @error)
         exit(0)
      EndIf

  _ArrayDisplay($aLines)

 

ZZj2_Test.Won.Two.csv


The most powerful number in the Universe.  Zero.

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Would you say that all records in the attached file have the same number of columns on each record defined separated by a comma?
Can't test but guess the header records need to be stripped from the file first.

Jos

Edited by Jos

Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

#3 ·  Posted

Ok, just tested a bit and it seems that the Function requires the exact number of columns defined in each records , which is clearly not the case in the inputfile.

Quote

When a delimiter is specified the function tries to further split each line of the file - how this is done depends on the setting of the $FRTA_INTARRAYS flag. If the flag is not set and each line has the same number of fields split by the delimiter then a 2D array is created, but if this is not the case then @error is set to 3 and no array is returned. If the $FRTA_INTARRAYS flag is set the function creates a 1D array where each element is a further array holding the fields of the line split on the delimiter - the lines do not need to have the same number of fields. See example below.

Jos


Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

#4 ·  Posted

You may have a point, Jos. Opening the CSV in NP++ it's

ZZj2_Test HauteurAH V7.0.01        11/07/2017    11:16:50
Lot: Won                             Sample: Two                  User: PeterH
Ha: 69.4 mm    %<15mm 0.1   L5 % 130.8   Ba: 87.6
CvH: 51.1 mm   %<25mm 5.3   L1 % 153.2   CvB: 41.3
Ha FREQ,Ha HIST,Ba FREQ,Ba HIST,,HAUTEUR,,,,,,BARBE
100.0,0.0,100.0,0.0,Ha,69.4,,,,Ba,87.6
100.0,0.0,100.0,0.0,CvH,51.1,,,,CvB,41.3
100.0,0.0,100.0,0.0,K10,0.0,L95,24.8,,K10,0.0,L95,33.5
100.0,0.0,100.0,0.0,K15,0.1,L90,30.5,,K15,0.0,L90,41.5
100.0,0.0,100.0,0.0,K20,2.0,L75,43.5,,K20,0.5,L75,56.8
100.0,0.0,100.0,0.0,K25,5.3,L50,61.8,,K25,1.5,L50,84.0
100.0,0.0,100.0,0.0,K30,9.7,L25,93.5,,K30,3.3,L25,112.5
100.0,0.0,100.0,0.0,K35,15.0,L10,119.0,,K35,5.7,L10,131.2
100.0,0.0,100.0,0.0,K40,20.7,L5,130.8,,K40,8.8,L5,142.8
100.0,0.0,100.0,0.0,K45,26.9,L2.5,0.0,,K45,12.6,L2.5,152.2
100.0,0.0,100.0,0.0,K50,33.7,L1,153.2,,K50,17.2,L1,156.0
100.0,0.0,100.0,0.0

There are ca. 500 rows below this but all identical apart from data.

You think the first five rows are screwing up the ReadToArray()?

I actually only need rows 6 to 16 and columns 5 to 12 which, when it doesn't error, display perfectly.

So I edit out the first 5 rows:

100.0,0.0,100.0,0.0,Ha,69.4,,,,Ba,87.6
100.0,0.0,100.0,0.0,CvH,51.1,,,,CvB,41.3
100.0,0.0,100.0,0.0,K10,0.0,L95,24.8,,K10,0.0,L95,33.5
100.0,0.0,100.0,0.0,K15,0.1,L90,30.5,,K15,0.0,L90,41.5
100.0,0.0,100.0,0.0,K20,2.0,L75,43.5,,K20,0.5,L75,56.8
100.0,0.0,100.0,0.0,K25,5.3,L50,61.8,,K25,1.5,L50,84.0
100.0,0.0,100.0,0.0,K30,9.7,L25,93.5,,K30,3.3,L25,112.5
100.0,0.0,100.0,0.0,K35,15.0,L10,119.0,,K35,5.7,L10,131.2
100.0,0.0,100.0,0.0,K40,20.7,L5,130.8,,K40,8.8,L5,142.8
100.0,0.0,100.0,0.0,K45,26.9,L2.5,0.0,,K45,12.6,L2.5,152.2
100.0,0.0,100.0,0.0,K50,33.7,L1,153.2,,K50,17.2,L1,156.0
100.0,0.0,100.0,0.0
100.0,0.0,100.0,0.0

but I still get error 3.

 

However, when I revert to

_FileReadToArray($fLine, $aLines)

it works without error. :)

Is there any way I can get ReadToArray() to begin on row 6?


The most powerful number in the Universe.  Zero.

Share this post


Link to post
Share on other sites

#5 ·  Posted

I did try $FRTA_INTARRAYS but the eventual 1D array just showed 630+ times 1{array} with no data, i.e. didn't seem to split each row.


The most powerful number in the Universe.  Zero.

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

2 hours ago, fopetesl said:

Is there any way I can get ReadToArray() to begin on row 6?

_FileReadToArray($sFilePath, $aFile)

For $i = 6 to UBound($aFile)
Next

That should get you line 6 and onward, then you can further process/parse the data in the loop.

Edited by ViciousXUSMC

Share this post


Link to post
Share on other sites

#7 ·  Posted

1 hour ago, ViciousXUSMC said:
_FileReadToArray($sFilePath, $aFile)

For $i = 7 to UBound($aFile)
Next

That should get you line 6 and onward, then you can further process/parse the data in the loop.

Unfortunately _FileReadToArray() spits the dummy before the UBound() call so I'm still stuffed :'(


The most powerful number in the Universe.  Zero.

Share this post


Link to post
Share on other sites

#8 ·  Posted

I have this working, its crude and probably can be improved but if I get what you are trying to do (read a file to a array with delimiter) it works!

Replace test.txt with one of your files.

#Include <Array.au3>
#Include <File.au3>

If FileExists(@TempDir & "\temp.txt") Then FileDelete(@TempDir & "\Temp.txt")

Local $aFile
Local $aFileNew
Local $aTest

_FileReadToArray(@ScriptDir & "\test.txt", $aFile)

For $i = 1 To $aFile[0]
$sNew = $aFile[$i]
Do
$sNew = StringRegExpReplace($sNew, "(.*)", "${1},", 1)
;MsgBox(0, "", $sNew)
$aTest = StringRegExp($sNew, ",", 3)
$iCount = UBound($aTest)
Until $iCount = 13
$aFile[$i] = $sNew
Next

;_ArrayDisplay($aFile)

_FileWriteFromArray(@TempDir & "\temp.txt", $aFile, 1)

_FileReadToArray(@TempDir & "\temp.txt", $aFileNew, 1, ",")

_ArrayDisplay($aFileNew)

 

Share this post


Link to post
Share on other sites

#9 ·  Posted

fopetesl,

The 2D _FileReadToArray function was written (by me) to split "square" csv files - i.e. those with the same number of elements on each line. This is a design decision to warn if the array is not "square" as would be expected for the majority of csv files.

If you want to read files like yours with different numbers of elements on each line then you need to do some work yourself:

#include <Array.au3>
#include <File.au3>

$sCSV = "ZZj2_Test.Won.Two.csv"

Global $aLines
_FileReadToArray($sCSV, $aLines)

;_ArrayDisplay($aLines, "", Default, 8)

Global $aLines2D[$aLines[0]][1]

For $i = 1 To $aLines[0]

    $aSplit = StringSplit($aLines[$i], ",")

    If $aSplit[0] > UBound($aLines2D, 2) Then
        ReDim $aLines2D[UBound($aLines2D)][$aSplit[0]]
    EndIf

    ;_ArrayDisplay($aSplit, "", Default, 8)

    For $j = 1 To $aSplit[0]

        $aLines2D[$i - 1][$j - 1] = $aSplit[$j]

    Next

    ;_ArrayDisplay($aLines2D, "", Default, 8)

Next

_ArrayDisplay($aLines2D, "", Default, 8)

As to  "I did try $FRTA_INTARRAYS but the eventual 1D array just showed 630+ times 1{array} with no data, i.e. didn't seem to split each row" - that is exactly what the Help file tells you will happen - each element of the array is another array split on the delimiter. Once again it is up to you get these elements into a 2D array, using logic not dissimilar to that in the example above.

M23

1 person likes this

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

#10 ·  Posted

A .csv file must adhere to the RFC standard or desist from calling itself .csv

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#11 ·  Posted (edited)

You will need\should use _FileReadToArrayEx ,  It must be faster and it will have to do exactly All what you want them to do

#include <Array.au3>
#include "_FileReadToArrayEx.au3"

Local $aArray
$fTimerDiff = TimerInit()
$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", $FRTA_ARRAYFIELD + 0 + $FRTA_STRIPALL + $FRTA_CHECKSINGEQUOTE)
;$aArray = _FileReadToArrayEx(@DesktopDir & '\Test.inf', ",", 0 + 1 + 16 + 12)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite("_FileReadToArrayEx() : " & $fTimerDiff & @CRLF)
_ArrayDisplay($aArray)
_ArrayDisplay(($aArray[1]))
_ArrayDisplay(($aArray[6]))
#include <Array.au3>
#include "_FileReadToArrayEx.au3"

Local $aArray, $sData

$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", BitOR($FRTA_NOCOUNT, $FRTA_ARRAYFIELD, $FRTA_STRIPALL))
If Not @Error Then
    ;;Local $iaArray = UBound($aArray) - 1   ; or $iaArray = @Extended
    For $i = 0 To UBound($aArray) - 1
        For $y = 0 To UBound($aArray[$i]) - 1
            $sData &= ($aArray[$i])[$y] & ","
        Next
        $sData = StringTrimRight($sData, 1) & @CRLF
    Next
EndIf
ConsoleWrite($sData & @LF)

;Or

$aArray = _FileReadToArrayEx(@DesktopDir & '\ZZj2_Test.Won.Two.csv', ",", BitOR($FRTA_ARRAYFIELD, $FRTA_STRIPALL, $FRTA_CHECKSINGEQUOTE))
If Not @Error Then
    ;;Local $iaArray = UBound($aArray) - 1   ; or $iaArray = @Extended
    For $i = 1 To ($aArray)[0]
        For $y = 1 To ($aArray[$i])[0]
            $sData &= ($aArray[$i])[$y] & ","
        Next
        $sData = StringTrimRight($sData, 1) & @CRLF
    Next
EndIf
ConsoleWrite($sData & @LF)

Ciao.

Edited by DXRW4E
1 person likes this

OneDrive_Logo.png drive_app_badge.png box-logo.png new_logo.png MEGA_Logo.png

Share this post


Link to post
Share on other sites

#12 ·  Posted

Impressive help appreciated. This seems to have resoled the issue

For $i = 7 to UBound($fLine)
Next
_FileReadToArray($fLine, $aLines)
      If @error Then
         MsgBox($MB_SYSTEMMODAL, "ERROR!", "CSV read error " & @error)
         exit(0)
      EndIf

I get to display the data I want and haven't had a failure :)

Thanks for your help. Grazie mille.


The most powerful number in the Universe.  Zero.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • timmy2
      By timmy2
      I figure there must be a reason for _FileReadToArray to exist when FileReadToArray can already read an entire text file into an array, but I don't understand the _FileReadToArray examples in the Help File.  The threads about _FileReadToArray here in the forum all seem to end up using StringSplit to work with delimiters, yet _FileReadToArray has some sort of built-in delimiter capability. 
      Here's where I was hoping _FileReadToArray might be applicable.  Let's assume the text file I wish to read into my script is like this:
      Lorem ipsum dolor sit amet, consectetur adipiscing elit. Cras hendrerit tristique purus quis eleifend. @@ Maecenas interdum dui in magna tincidunt, eget imperdiet tellus ornare. * Fusc * blandit * tellu * metus Nulla nec blandit ligula. @@ Fusce nec rhoncus enim. Cras facilisis velit velit, in fringilla felis porttitor sed. Cras vel nisi lobortis, vestibulum justo quis, lacinia est. Maecenas eu dolor ac est posuere semper a a enim. I would like to import the text file into a 1D array, with the @@'s used as delimiters.  So in this example the resulting 1D array would have 3 elements in it. Note that I want to retain the formatting and line endings (CRLF's).
      Can _FileReadToArray be used to accomplish this on its own?
      Thank you.
    • VIP
      By VIP
      CSV1:
      "Index","Name","Home Address","Last Love" "1","Rola Takizawa","Nữ hoàng, Nga Nhật","24/04/2016 22:37:00 PM" "2","Saori Hara","Hình Ảnh Nóng, Đức Nhật","21/03/2016 21:07:00 PM" "3","Ozawa","phổ biến nhất","23/04/2016 23:31:00 PM" "3","Akiho Ameri Ichinose","Kanagawa, Ameri, Ichinose","22/04/2016 21:50:00 PM"  
      CSV2:
      Index,Name,Home Address,Last Love 1,Takizawa,"Nữ hoàng, Nga Nhật","24/04/2016 22:37:00 PM" 2,"Saori Hara","Hình Ảnh Nóng, Đức Nhật","21/03/2016 21:07:00 PM" 3,Ozawa,"phổ biến nhất","23/04/2016 23:31:00 PM" 3,"Akiho Ameri Ichinose","Kanagawa, Ameri, Ichinose","22/04/2016 21:50:00 PM"  
      CSV3:
      Index,Name,Home Address,Last Love 1,Takizawa,"Nữ hoàng, Nga Nhật",24/04/2016 22:37:00 PM 2,Saori Hara,"Hình Ảnh Nóng, Đức Nhật",21/03/2016 21:07:00 PM 3,Ozawa,phổ biến nhất,23/04/2016 23:31:00 PM 3,Akiho Ameri Ichinose,"Kanagawa, Ameri, Ichinose",22/04/2016 21:50:00 PM ---------------------------------------------------------------------------------- Global $CSV3 = 'Index,Name,Home Address,Last Love' & @CRLF $CSV2 &= '1,Takizawa,"Nữ hoàng, Nga Nhật",24/04/2016 22:37:00 PM' & @CRLF $CSV2 &= '2,Saori Hara,"Hình Ảnh Nóng, Đức Nhật",21/03/2016 21:07:00 PM' & @CRLF $CSV2 &= '3,Ozawa,phổ biến nhất,23/04/2016 23:31:00 PM' & @CRLF $CSV2 &= '3,Akiho Ameri Ichinose,"Kanagawa, Ameri, Ichinose",22/04/2016 21:50:00 PM' & @CRLF  
      Does not work when have more ","
      #include <GUIConstantsEx.au3> #include <MsgBoxConstants.au3> #include <GUIListView.au3> #include <Array.au3> #include <File.au3> Global $CSV1 = '"Index","Name","Home Address","Last Love"' & @CRLF $CSV1 &= '"1","Rola Takizawa","Nữ hoàng, Nga Nhật","24/04/2016 22:37:00 PM"' & @CRLF $CSV1 &= '"2","Saori Hara","Hình Ảnh Nóng, Đức Nhật","21/03/2016 21:07:00 PM"' & @CRLF $CSV1 &= '"3","Ozawa","phổ biến nhất","23/04/2016 23:31:00 PM"' & @CRLF $CSV1 &= '"3","Akiho Ameri Ichinose","Kanagawa, Ameri, Ichinose","22/04/2016 21:50:00 PM"' & @CRLF Global $CSV2 = "Index,Name,Home Address,Last Love" & @CRLF $CSV2 &= "1,Takizawa,'Nữ hoàng, Nga Nhật','24/04/2016 22:37:00 PM'" & @CRLF $CSV2 &= "2,'Saori Hara','Hình Ảnh Nóng, Đức Nhật','21/03/2016 21:07:00 PM'" & @CRLF $CSV2 &= "3,Ozawa,'phổ biến nhất','23/04/2016 23:31:00 PM'" & @CRLF $CSV2 &= "3,'Akiho Ameri Ichinose','Kanagawa, Ameri, Ichinose','22/04/2016 21:50:00 PM'" & @CRLF Local $hOpen=FileOpen("CSV1.csv",128) FileWrite($hOpen,$CSV1) FileClose($hOpen) Local $hOpen=FileOpen("CSV2.csv",128) FileWrite($hOpen,$CSV2) FileClose($hOpen) Global $rArrayCSV1,$rArrayCSV2 Local $ArrayCSV1 = _FileReadToArray("CSV2.csv", $rArrayCSV1, Default,',') ConsoleWrite( $ArrayCSV1 & ' - ' & 'error = ' & @error & @CRLF) _ArrayDisplay($rArrayCSV1) Local $ArrayCSV2 = _FileReadToArray("CSV2.csv", $rArrayCSV2, Default,',') ConsoleWrite( $ArrayCSV2 & ' - ' & 'error = ' & @error & @CRLF) _ArrayDisplay($rArrayCSV2)  
    • TThomasson
      By TThomasson
      Hi everyone. New guy here. I'm still learning this awesome language and I'm unable to figure this one out from google searches. Heres my problem:
      I'm working on a small application to help users in my environment connect to wireless projectors. To keep this easily updated with new projectors I'm reading the room names and IP addresses from a csv file and putting them into a 2D array. (MeetingRoom1,xxx.xxx.xxx.xxx)
      So far I'm able to read the 0 column and display the room names in a combo box. Where I am stuck is how to take the user's room selection from the gui and associate it with an IP address in the array. After that point I've got things prepared to pass the address to the connection application.  
      Any help you all could provide would be greatly appreciated. 
    • simy8891
      By simy8891
      Hi guys,
      I'm either going crazy or AutoIT is..
      Basically I need to get to the parent folder of @scriptdir, to do that, I used:
      $parentFolder=StringLeft(@ScriptDir, StringInStr(@ScriptDir, "\", 0, -1))  So far, all good. When I use MsgBox to show $parentfolder, I'm indeed able to see the folder above @scriptdir. Now, I've added a subfolderfile to the string with:
      $PPID_Bulk_Big_printers_csv_path=$parentFolder&'Printers\printers_PPID_Big.csv'  And also here, all fine, MsgBox shows me the file location correctly.
      Now, for some weird reason, $PPID_Bulk_Big_printers_csv_path changes again!
      See the complete part of the script below:
      ; CSV to 2D array $parentFolder=StringLeft(@ScriptDir, StringInStr(@ScriptDir, "\", 0, -1)) ;Goes one folder up MsgBox(0,'',@ScriptDir&'----'&$parentFolder&'Printers\printers_PPID_Big.csv') $PPID_Bulk_Big_printers_csv_path=$parentFolder&'Printers\printers_PPID_Big.csv' ;CSV location Dim $PPID_Bulk_Big_printers $PPID_Bulk_Big_filereadtoarray_result = _FileReadToArray($PPID_Bulk_Big_printers_csv_path, $PPID_Bulk_Big_printers) ;$PPID_Bulk_Big_filereadtoarray_result will contain any error (if there's any) - $printers is an array which contains the list of printers MsgBox(0,'',$PPID_Bulk_Big_printers_csv_path) ; Still shows correct path ;Check if there was any error with the _FileReadToArray If $PPID_Bulk_Big_filereadtoarray_result <> 0 Then ;If no error reading the file ;Do Something Else ;If there was an error reading the CSV file MsgBox(0,'hhh',"msg="&$PPID_Bulk_Big_printers_csv_path) MsgBox(48,'Problem detected!','The file "'&$PPID_Bulk_Big_printers_csv_path&'" was not found. Software will exit!') Sleep (50) Exit EndIf So what's happening here is, I'm getting into the ELSE part of the script, so there was an error with reading the file and I'm getting indeed the message box with "Problem detected!" but NOT the message box above?? Why?
      Cheers
    • Command3r
      By Command3r
      Hello scripters i got another problem reading all lines in txt file except the last one.
      I dont want to write how many lines to read just reading all except the last whatever how many it is.

      My efforts:

      #include <file.au3> $file = @ScriptDir & "\x.txt" $ReadLines = FileReadLine($file) Local $Read _FileReadToArray($file, $Read) For $i = 1 To $Read - 1 ; with-1 or without-1 the result is the same MsgBox(0, $i, $Read - 1) Next
      Question here: "Why first msgbox title is "1" ? and how can i avoid that?"

      #include <file.au3> $file = @ScriptDir & "\x.txt" $ReadLines = FileReadLine($file) Local $Read _FileReadToArray($file, $Read) For $i = 1 To $Read MsgBox(0, $i, $Read[0] -1) ; here in msg's text resolved but in title not :D i think the problem from $i ?? Next