_JL_

Truncated read in FileReadToArrray()

15 posts in this topic

#1 ·  Posted

Hi,

I am using FileReadToArray() to read a text file with over 8000 lines:

                $updDataSet = FileReadToArray($useOutputFilename & ".txt")

Each line is about a few hundred characters and the whole file is about 1.2MB.

What I found is that the resultant array only has 6448 elements, and the last line in the array is clearly truncated halfway compared to the text file. The function call didn't return any error - it just quietly stopped reading the rest of the lines from the file.

From the documentation this should be well-within the limits of array dimension and string size so I am not sure what the problem is.

Any pointers or suggestions are most appreciated.

Thanks.

Share this post


Link to post
Share on other sites



#2 ·  Posted

Welcome to AutoIt and the forum!

You could use function _FileReadToArray from the File UDF to check if you get the same (wrong) result.
If yes, then there might be a problem with the file content.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

#3 ·  Posted

Thanks water for the suggestion.

I just tried _FileReadToArray() on the same file. It read about 200 more lines (~6600 vs ~6400) but the results are similar, i.e., incomplete read and the last line is truncated in the middle.

This suggests that the file content is not the problem as the two functions stopped at different points in the file. I also checked the file contents using notepad and found no problem.

Any other ideas?

Thanks.

Share this post


Link to post
Share on other sites

#4 ·  Posted

You could use Notepad++ or a similar tool to display invisible characters. Maybe this sheds some light onto the issue.


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

#5 ·  Posted

I believe it is not due to the file content because if that's the case the functions should always stop at the same point.

I just tried running the script in another PC with more memory (16GB vs 4 GB) and the function read ~8800 lines, still short of the 9000+ lines in the file but this strongly suggests that it is a memory-related issue. The host is far from running out of memory though (~7GB out of 16GB used with the script running) ...

Share this post


Link to post
Share on other sites

#6 ·  Posted

Would you mind to post the file in this thread so we can play with it and see if we can reproduce the problem?


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

#7 ·  Posted

BTW: Which version of AutoIt do you run?


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

#8 ·  Posted

I used the latest version of Autoit. I also just tried the beta version which is still no luck.

Attached the text file. Thanks for the help.

StockPrice-ALL.txt

Share this post


Link to post
Share on other sites

#9 ·  Posted

I saved your file to C:\temp and get 9278 entries (rows 0-9277) in the returned array when running the following code.
Last row in the array is:

Quote

VHSI HSI Volatility Index;2017/05/31;11.78;12.16;11.78;11.92;;;HKD;2.7586;;;;;;;;2.7213;;;;;

My script:

#include <Array.au3>
Global $aFile = FileReadToArray("C:\temp\StockPrice-ALL.txt")
_ArrayDisplay($aFile)

Maybe you could download the file from your post above and run my script on this file?


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

#10 ·  Posted

If you remove the two first useless lines, then you have a SSV (semicolumn separated value) file which can be easily imported into an SQLite database, building a 9275-row table. You can query this DB using SQLite support proposed by AutoIt.

Note that the last column has no explicit name in the header line. I believe it's due to an extraneous semicolumn at the end of each data line.

1 person likes this

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#11 ·  Posted

Your script works perfectly.

I found the problem - the txt file is being generated dynamically and the codes run into race condition where it may call FileReadtoArray() before the file is completely saved to disk. If I add some extra delay for the save to finish then problem solved!

Thanks to water for the help! This is a great community.

Share this post


Link to post
Share on other sites

#12 ·  Posted

Glad to be of service :) 

1 person likes this

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2017-04-18 - Version 1.4.8.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (NEW 2017-02-27 - Version 1.3.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

#13 ·  Posted

"StockPrice-ALL.txt" file into a 2D array.

#include <Array.au3>
#include <Date.au3>

Local $aFile
Local $aFile = FileReadToArray("StockPrice-ALL.txt")
;_ArrayDisplay($aFile)

Local $iUB = UBound($aFile)
Local $iMaxCols = 1 ; Must be at least 1 column and less than 50.

Local $tCur = _Date_Time_GetLocalTime()
Local $aRetArray[$iUB][50] = [[$aFile[0] & " - " & _Date_Time_SystemTimeToDateTimeStr($tCur)], [$aFile[1]]] ; First 2 lines and unknown number of columns (less than 50).

For $i = 2 To $iUB - 1 ; Start on 3rd line
    $aTemp = StringSplit($aFile[$i], ";", 2)
    If UBound($aTemp) > $iMaxCols Then $iMaxCols = UBound($aTemp) ; Find maximum number of columns.
    For $j = 0 To UBound($aTemp) - 1
        $aRetArray[$i][$j] = $aTemp[$j]
    Next
Next
ReDim $aRetArray[$iUB][$iMaxCols]

; Add extra row in array if last line in file is blank.
StringReplace(FileRead("StockPrice-ALL.txt"), @LF, "")
If @extended = $iUB Then _ArrayAdd($aRetArray, "", 0, ";", $iUB) ; Add extra row to array.

_ArrayDisplay($aRetArray)

 

1 person likes this

Share this post


Link to post
Share on other sites

#14 ·  Posted

Thanks to jchd for the SQLite pointer and Malkey for the code example. Very useful to a AutoIT noob like me.

 

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

If you want I can make an example DB with what I believe is a better design. I'll do that in the (my) morning. Done.

You can use SQLite Expert (free for home version) to play with it.

Stock.sq3

Edited by jchd
DB added

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now