Load large file to variable

[Skitty]

I don't know how to get this to work. the data returned by _Load() is flawed like my scripting.

expandcollapse popup

#include<ButtonConstants.au3>
#include<GUIConstantsEx.au3>
#include<ProgressConstants.au3>
#include<WindowsConstants.au3>

#Region ### START Koda GUI section ### Form=
$GUI = GUICreate("Form1", 395, 149, 185, 278)
$Prog = GUICtrlCreateProgress(24, 96, 345, 33)
$Button = GUICtrlCreateButton("Copy", 168, 48, 57, 25, $WS_GROUP)
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

While 1
    Switch GUIGetMsg()
        Case -3
            Exit
        Case $Button
            $File = FileOpenDialog("", @DesktopDir, "ALL(*.*)")
            If Not @error Then
            $D=_Load($File)
            EndIf
            $Fl=FileOpen(@DesktopDir&"\test.exe",2)
            FileWrite($FL,$d)
            FileClose($FL)
    EndSwitch
WEnd

Func _Loade($File); lol, oops...

    $F = FileOpen($File)
    If @error Then Exit MsgBox(16, "Error", "There was an error opening the file.", 3)

    $i = 0

    Do
        $i += 1
        FileReadLine($F)
    Until @error

    FileClose($F)

    $FO = FileOpen($File,16)
    $Data = ""
    $Datatmp = ""
    $T = Round($i/100)
    $lp = 0
    $prog2 = 1

    For $loop = 0 To $i -2
        $lp += 1
        $Datatmp = FileReadLine($FO)&@lf
        If @error Then ExitLoop
        $Data &= $Datatmp
        If $lp > $T Then
            GUICtrlSetData($Prog,$prog2)
            $lp = 0
            $prog2 += 1
        EndIf
    Next

    GUICtrlSetData($Prog,0)
    FileClose($FO)

    Return $Data

EndFunc   ;==>_Copy

I wanted to be able to create a file with the data gathered with filereadeline.

Inb4: fileread, filecopy, singlethreaded, stupid idea/example...

Edited April 18, 2011 by System238

[Mat]

There are much faster ways to get the number of lines in a file... Search the forum for that

What kinds of files are you talking about? From experience updating a progress bar really does slow things up (I know it looks nice though ) so unless it really is necessary then I wouldn't bother and just read the file in one hit.

If you want a simple solution then work with bytes rather than lines. Get the number of bytes with FileGetSize and then use ReadFile with x characters (numbers like 4096 are usually used to make it faster at a lower level). That would be the easiest way to make it faster and more reliable.

The alternative is asynchronous file reading. That way your GUI will still be responsive while you are reading. I'm not sure how easy it is to get information about how much has been read, but the worst that can happen is that you need a little animation that says loading and let people get on with doing whatever else they need to do in the GUI while it is.

Up to you how far you want to go. Using bytes is an easy solution with progress that would be easy to do though.

[UEZ]

What about $data = FileRead($sFilePath) ? Will load the full file to $data variable!

$lines = _FileCountLines($sFilePath) will count the lines of that particular file.

Br,

UEZ

[guinness]

Check out GEOSofts website for a SRE to count the lines in a File >> http://dundats.mvps.org/autoit/udf_code.aspx?udf=filex

I was referring to _File_CountLines()

Edited April 18, 2011 by guinness

[Skitty]

The gui is irrelevant as I just made this as an example.

the example was, How do I use filereadline() to load a files binary as if it was read by fileread().

[Mat]

What would you use FileReadLine to read binary files? Read my post, particularly the bit about reading bytes

[UEZ]

$hFile = FileOpen("test.exe", 16)
$bData = FileRead($hFile)
FileClose($hFile)

Br,

UEZ

[PsaltyDS]

If you can't be bothered to post a reasonable example, why should people try to help you? You are calling _Load(), but your function is called _Loade(). If the GUI is irrelevant then why is there at all, adding length and complexity to your example? Is the file a binary or not? If it is, explain why you want to "count lines", and if it is not, then why open it as a binary?

You aren't making much sense here.

[Mat]

Just to say how stupid the example is: how about FileCopy

[Skitty]

If you can't be bothered to post a reasonable example, why should people try to help you? You are calling _Load(), but your function is called _Loade(). If the GUI is irrelevant then why is there at all, adding length and complexity to your example? Is the file a binary or not? If it is, explain why you want to "count lines", and if it is not, then why open it as a binary?

You aren't making much sense here.

Basically, what Im trying to do is to load an entire file (any file) into a variable using this method.

If not possible, then are there any other ways I could go about loading a 50-100mb file into a variable w/o having the gui pause while this is happening?

Here is a more detailed example of what I was trying to do.

Every now and then my app comes across a huge file and while reading that file to a variable, I wanted another smaller gui to show up and show some activity to tell the user that the app will be available in just a moment.

expandcollapse popup

#include<File.au3>
#include<Array.au3>
#include<Crypt.au3>
#include<FTPEx.au3>
#include<Memory.au3>
#include<SQLite.au3>
#include<SQLite.dll.au3>
#include<GuiListView.au3>
#include<GuiIPAddress.au3>
#include<TabConstants.au3>
#include<editconstants.au3>
#include<ScreenCapture.au3>
#include<GUIconstantsEx.au3>
#include<staticconstants.au3>
#include<windowsconstants.au3>
#include<ListBoxConstants.au3>
#include<ProgressConstants.au3>
Global $Sound = True, $Deleteit = True,$Admin = False,$Help = False
Global $Spawn = False,$NoUpdate = False, $StopScanstate = False
Global $hInputEdit, $hOutputEdit, $hKey, $StopScan, $St, $Showstatus, $rx
Global $bToggle = 0, $iItems = 10, $iW = 575, $iH = 330
Global $Btns[$iItems], $Lbls[$iItems], $Button1, $Parent
Global $hGUI, $bToggle, $Passive = 0, $List3, $aDirs[2], $oContainer, $Dev
Global $MainEdit, $List1, $Edit1, $bHelp
Global $Checkbox1, $Label1, $Group2, $Tab1, $Group1, $Dwn
Global $DB, $TD[5], $_ListView
Global $ResFile = @SystemDir & "\shell32.dll", $B3 = ".\Media\1c.mp3"
Global $HardMode, $Checkbox4
Global $B1 = ".\Media\1b.mp3", $B2 = ".\Media\1a.mp3", $Progress1, $Icon1
Global $cnt = Random(0, 180, 1), $Checkbox1, $Checkbox2, $Checkbox3
Global $Warning, $bHelp,$Alert = ".\Media\Alert.mp3", $Stml = 100
_SQLite_Startup()
If Not FileExists("Memory.db") Then
    $DB = _SQLite_Open("Memory.db")
    _SQLite_Exec($DB, "CREATE TABLE ThemeData (Adress,Val1,Val2,Val3,Val4,Val5);")
    _SQLite_Exec($DB, "CREATE TABLE DataBase (Adress,Val1,Val2,Val3,Val4,Val5);")
    _SQLite_Exec($DB, "CREATE TABLE ScriptData (Adress,Val1,Val2,Val3,Val4,Val5);")
    _SQLite_Exec($DB, "CREATE TABLE ScanData (Adress,Val1,Val2,Val3,Val4,Val5);")
    $Help = True
Else
    $DB = _SQLite_Open("Memory.db")
EndIf

_FCR(); This is just a snippet, not the whole script...

Func _FCR()

    _Crypt_Startup()

    $Dmy = GUICreate("")
    $hGUI = GUICreate(@ScriptName, $iW, $iH, -1, -1,-1, $WS_EX_ACCEPTFILES, $Dmy)
    $Group1 = GUICtrlCreateGroup("Progress", 16, 237, 537, 41)
    $Edit1 = GUICtrlCreateEdit("", 24, 248, 521, 25, "", 0)
    GUICtrlCreateGroup("", -99, -99, 1, 1)
    $Group2 = GUICtrlCreateGroup("DataBase Creation", 16, 80, 265, 153)
    $Checkbox4 = GUICtrlCreateCheckbox("10 Mb", 32, 143, 105, 17)
    $Checkbox3 = GUICtrlCreateCheckbox("25 Mb", 32, 175, 97, 17)
    $Checkbox2 = GUICtrlCreateCheckbox("50 Mb", 32, 159, 121, 17)
    $Checkbox1 = GUICtrlCreateCheckbox("No Update", 32, 127, 121, 17)
    $Combo1 = GUICtrlCreateCombo("Ez-Mode", 32, 103, 97, 25)
    If $Help Then GUICtrlSetTip(-1, "In hard mode, all you need to do is choose how large files that do not get scanned should be." & _
            'If "No Update" is selected, files data in the database is not updated. ', "Info", 1, 1)
    $Input1 = GUICtrlCreateInput("", 88, 208, 185, 16)
    $Button4 = GUICtrlCreateButton("Add File", 184, 176, 89, 25, $WS_GROUP)
    If $Help Then GUICtrlSetTip(-1, "Add file information to the database", "Add file to db", 1, 1)
    $Icon2 = GUICtrlCreateIcon($ResFile, 246, 160, 184, 16, 16, BitOR($SS_NOTIFY, $WS_GROUP))
    $Icon3 = GUICtrlCreateIcon($ResFile, 246, 160, 152, 16, 16, BitOR($SS_NOTIFY, $WS_GROUP))
    $Button5 = GUICtrlCreateButton("Create db", 184, 144, 89, 25, $WS_GROUP)
    If $Help Then GUICtrlSetTip(-1, "Create a hash log of all files in a selected directory." & @CRLF & _
            "Selectiong this option again will add all files in new directory" & @CRLF & _
            "and update existing file hash data for file alreaddy processed.", "Create DataBase", 1, 1)
    GUICtrlCreateGroup("", -99, -99, 1, 1)
    $Button1 = GUICtrlCreateButton("&Save", 32, 208, 51, 17, $WS_GROUP)
    GUICtrlSetTip(-1, "Same as add file...", "Save file info", 1, 1)
    $Group3 = GUICtrlCreateGroup("Inspecting Station", 288, 80, 265, 153)
    $Button2 = GUICtrlCreateButton("Check File", 456, 160, 89, 25, $WS_GROUP)
    If $Help Then GUICtrlSetTip(-1, "Display information on a preprocessed file.", "File Info", 1, 1)
    $Button3 = GUICtrlCreateButton("Check All", 456, 192, 89, 25, $WS_GROUP)
    If $Help Then GUICtrlSetTip(-1, "Check to see if any files that have been processed have changed.", "File change reporter.", 1, 1)
    $List1 = GUICtrlCreateList("", 296, 96, 145, 136,$WS_VSCROLL)
    GUICtrlCreateGroup("", -99, -99, 1, 1)
    $Icon1 = GUICtrlCreateIcon($ResFile, 323, 488, 120, 32, 32, BitOR($SS_NOTIFY, $WS_GROUP))
    $Warning = GUICtrlCreateIcon($ResFile, 338, 488, 120, 32, 32, BitOR($SS_NOTIFY, $WS_GROUP))
    $Progress1 = GUICtrlCreateProgress(16, 288, 537, 25, 0x01)
    $Button6 = GUICtrlCreateButton("Return", 472, 56, 81, 25, $WS_GROUP)
    If $Help Then GUICtrlSetTip(-1, "Return to main index", "Return", 1, 1)
    $StopScan = GUICtrlCreateIcon($ResFile, 28, 232, 96, 33, 33, BitOR($SS_NOTIFY, $WS_GROUP))
    If $Help Then GUICtrlSetTip(-1, "Stop the information gathering process.", "Stop", 1, 1)

    GUICtrlSetData($Combo1, "Hard Mode")
    GUICtrlSetCursor($Combo1, 0)
    GUICtrlSetFont($Group2, 8, 400, 0, "Terminal")
    GUICtrlSetFont($Group1, 8, 400, 0, "Terminal")
    GUICtrlSetFont($Group3, 8, 400, 0, "Terminal")
    GUICtrlSetState($Icon3, $GUI_HIDE)
    GUICtrlSetState($Warning, $GUI_HIDE)
    GUICtrlSetState($Icon2, $GUI_HIDE)
    GUICtrlSetState($Input1, $GUI_DISABLE)
    GUICtrlSetState($StopScan, $GUI_HIDE)
    GUICtrlSetState($Button1, $GUI_DISABLE)
    GUICtrlSetCursor($StopScan, 0)
    GUICtrlSetCursor($Checkbox4, 0)
    GUICtrlSetCursor($Checkbox3, 0)
    GUICtrlSetCursor($Checkbox2, 0)
    GUICtrlSetCursor($Checkbox1, 0)
    GUICtrlSetState($Checkbox1, $GUI_DISABLE)
    GUICtrlSetState($Checkbox3, $GUI_DISABLE)
    GUICtrlSetState($Checkbox2, $GUI_DISABLE)
    GUICtrlSetState($Checkbox4, $GUI_DISABLE)
    GUICtrlSetState($Edit1, $GUI_DISABLE)
    GUISetState()

    While 1
        $nMsg = GUIGetMsg()
        Switch $nMsg
            Case $GUI_EVENT_CLOSE
                Exit
            Case $Combo1
                $Mode = GUICtrlRead($Combo1)
                If $Mode = "Hard Mode" Then
                    GUICtrlSetState($Input1, $GUI_ENABLE)
                    GUICtrlSetState($Button1, $GUI_ENABLE)
                    GUICtrlSetState($Checkbox1, $GUI_ENABLE)
                    GUICtrlSetState($Checkbox2, $GUI_ENABLE)
                    GUICtrlSetState($Checkbox3, $GUI_ENABLE)
                    GUICtrlSetState($Checkbox4, $GUI_ENABLE)
                    $HardMode = True
                ElseIf $Mode = "Ez-Mode" Then
                    GUICtrlSetState($Input1, $GUI_DISABLE)
                    GUICtrlSetState($Button1, $GUI_DISABLE)
                    GUICtrlSetState($Checkbox1, $GUI_DISABLE)
                    GUICtrlSetState($Checkbox2, $GUI_DISABLE)
                    GUICtrlSetState($Checkbox3, $GUI_DISABLE)
                    GUICtrlSetState($Checkbox4, $GUI_DISABLE)
                    $HardMode = True
                EndIf
            Case $List1
                $Data = GUICtrlRead($List1)
                If @error Then $Data = FileOpenDialog("Select a file to inspect", @DesktopDir, "All Files(*.*)")
                If $Data = @error Then ContinueLoop
                If Not FileExists($Data) Then ContinueLoop MsgBox(0, "File Error", "File does not exist...", 3, $hGUI)
                _CheckFile($Data)
            Case $Button1
                $File = FileOpenDialog("Select a file to inspect", @DesktopDir, "All Files(*.*)")
                If @error Then ContinueLoop
                _GenerateInformation($File)
                GUICtrlSetData($Input1, $File)
                TrayTip(@ScriptName, "File information was added to database.", 4, 1)
            Case $Button2
                $Data = GUICtrlRead($List1)
                If Not FileExists($Data) Then
                    $Data = FileOpenDialog("Select a file to inspect", @DesktopDir, "All Files(*.*)")
                    If @error Then ContinueLoop
                EndIf
                If Not FileExists($Data) Then ContinueLoop MsgBox(0, "File Error", "File does not exist...", 3, $hGUI)
                _CheckFile($Data)
            Case $Button3
                GUICtrlSetState($StopScan, $GUI_SHOW)
                GUICtrlSetState($StopScan, $GUI_ENABLE)
                GUICtrlSetState($Button3, $GUI_DISABLE)
                GUICtrlSetState($Button4, $GUI_DISABLE)
                GUICtrlSetState($Button2, $GUI_DISABLE)
                GUICtrlSetState($Button5, $GUI_DISABLE)
                _CheckAllFiles()
                GUICtrlSetState($Button3, $GUI_ENABLE)
                GUICtrlSetState($Button4, $GUI_ENABLE)
                GUICtrlSetState($Button2, $GUI_ENABLE)
                GUICtrlSetState($Button5, $GUI_ENABLE)
                GUICtrlSetState($StopScan, $GUI_HIDE)
            Case $Button4
                $Data = FileOpenDialog("Select a file to inspect", @DesktopDir, "All Files(*.*)")
                If @error Then ContinueLoop MsgBox(0, "Error", $Data)
                If Not FileExists($Data) Then ContinueLoop MsgBox(0, "File Error", "File does not exist...", 3, $hGUI)
                _GenerateInformation($Data)
                For $i = 0 To 1
                    GUICtrlSetState($Icon2, $GUI_SHOW)
                    Sleep(200)
                    GUICtrlSetState($Icon2, $GUI_HIDE)
                    Sleep(200)
                Next
            Case $Button5
                $Data = FileSelectFolder("Select Base Directory", @HomeDrive, "", @HomeDrive, $hGUI)
                If @error Then ContinueLoop
                ;MsgBox(0, "", $Data)
                GUICtrlSetState($StopScan, $GUI_SHOW)
                GUICtrlSetState($StopScan, $GUI_ENABLE)
                GUICtrlSetState($Button3, $GUI_DISABLE)
                GUICtrlSetState($Button4, $GUI_DISABLE)
                GUICtrlSetState($Button2, $GUI_DISABLE)
                GUICtrlSetState($Button5, $GUI_DISABLE)
                _DBFILL($Data)
                For $i = 0 To 1
                    GUICtrlSetState($Icon3, $GUI_SHOW)
                    Sleep(200)
                    GUICtrlSetState($Icon3, $GUI_HIDE)
                    Sleep(200)
                Next
                GUICtrlSetState($Button3, $GUI_ENABLE)
                GUICtrlSetState($Button4, $GUI_ENABLE)
                GUICtrlSetState($Button2, $GUI_ENABLE)
                GUICtrlSetState($Button5, $GUI_ENABLE)
                GUICtrlSetState($StopScan, $GUI_HIDE)
                GUICtrlSetData($Edit1, "")
            Case $Button6
                For $i = $Stml To 1 Step -1; I know...
                    $i -= 5
                    WinSetTrans($hGUI, "", $i)
                    Sleep(10)
                Next
                GUISetState(@SW_HIDE,$hGUI)
                GUICtrlDelete($Warning)
                GUICtrlDelete($StopScan)
                GUICtrlDelete($Input1)
                GUICtrlDelete($List1)
                GUICtrlDelete($Edit1)
                GUICtrlDelete($Icon1)
                GUICtrlDelete($Icon2)
                GUICtrlDelete($Icon3)
                GUICtrlDelete($Combo1)
                GUICtrlDelete($Group1)
                GUICtrlDelete($Group2)
                GUICtrlDelete($Group3)
                GUICtrlDelete($Button1)
                GUICtrlDelete($Button2)
                GUICtrlDelete($Button3)
                GUICtrlDelete($Button4)
                GUICtrlDelete($Button5)
                GUICtrlDelete($Button6)
                GUICtrlDelete($Progress1)
                GUICtrlDelete($Checkbox1)
                GUICtrlDelete($Checkbox2)
                GUICtrlDelete($Checkbox3)
                GUICtrlDelete($Checkbox4)
                GUICtrlSetState($Lbls[0], $GUI_SHOW)
                GUICtrlSetState($Btns[0], $GUI_SHOW)
                ExitLoop
            Case $Checkbox1
                If GUICtrlRead($Checkbox1) = $GUI_CHECKED Then
                    $NoUpdate = True
                Else
                    $NoUpdate = False
                EndIf
            Case $Checkbox2
                If GUICtrlRead($Checkbox2) = $GUI_CHECKED Then
                    GUICtrlSetState($Checkbox3, $GUI_DISABLE)
                    GUICtrlSetState($Checkbox4, $GUI_DISABLE)
                Else
                    GUICtrlSetState($Checkbox3, $GUI_ENABLE)
                    GUICtrlSetState($Checkbox4, $GUI_ENABLE)
                EndIf
            Case $Checkbox3
                If GUICtrlRead($Checkbox3) = $GUI_CHECKED Then
                    GUICtrlSetState($Checkbox2, $GUI_DISABLE)
                    GUICtrlSetState($Checkbox4, $GUI_DISABLE)
                Else
                    GUICtrlSetState($Checkbox2, $GUI_ENABLE)
                    GUICtrlSetState($Checkbox4, $GUI_ENABLE)
                EndIf
            Case $Checkbox4
                If GUICtrlRead($Checkbox4) = $GUI_CHECKED Then
                    GUICtrlSetState($Checkbox2, $GUI_DISABLE)
                    GUICtrlSetState($Checkbox3, $GUI_DISABLE)
                Else
                    GUICtrlSetState($Checkbox2, $GUI_ENABLE)
                    GUICtrlSetState($Checkbox3, $GUI_ENABLE)
                EndIf
        EndSwitch
        $Data = ""
        $StopScanstate = False
    WEnd
    $NoUpdate = False
    _Crypt_Shutdown()
    ;_CTRLSHOWHIDE()
    Return 1
EndFunc   ;==>_FCR


Func _DBFILL($SourceFolder)
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $GUI_EVENT_CLOSE
            Exit
        Case $StopScan
            $StopScanstate = True
    EndSwitch
    If $StopScanstate = True Then Return 1
    Local $File, $Search, $i = 0

    $Search = FileFindFirstFile($SourceFolder & "\*.*")
    If $Search = -1 Then Return

    While 1
        If $i > 100 Then $i = 0
        GUICtrlSetData($Progress1, $i)
        $nMsg = GUIGetMsg()
        Switch $nMsg
            Case $GUI_EVENT_CLOSE
                Exit
            Case $StopScan
                $StopScanstate = True
        EndSwitch
        If $StopScanstate = True Then Return 1
        $File = FileFindNextFile($Search)
        If @error = 1 Then ExitLoop
        If @extended = 1 Then
            _DBFILL($SourceFolder & "\" & $File)
        Else
            GUICtrlSetData($Edit1, FileGetShortName($SourceFolder & "\" & $File))
            _GenerateInformation($SourceFolder & "\" & $File)
        EndIf
        $i += 1
    WEnd
    FileClose($Search)
    If $StopScanstate = True Then Return 1
EndFunc   ;==>_DBFILL

Func _GenerateInformation($File)

    If $HardMode = True Then

        $2 = GUICtrlRead($Checkbox2)
        $3 = GUICtrlRead($Checkbox3)
        $4 = GUICtrlRead($Checkbox4)

        If $2 = $GUI_CHECKED Then
            If Round(FileGetSize($File) / 1048576, 2) > 50 Then Return 1
        EndIf

        If $3 = $GUI_CHECKED Then
            If Round(FileGetSize($File) / 1048576, 2) > 25 Then Return 1
        EndIf

        If $4 = $GUI_CHECKED Then
            If Round(FileGetSize($File) / 1048576, 2) > 10 Then Return 1
        EndIf
    EndIf

    Local $FO = FileOpen($File)
    If @error Then
        _DataAddEx("ScanData", FileGetShortName($File), "Error opening file")
        Return 1
    EndIf

    Local $FR = FileRead($FO)
    FileClose($FO)

    _DataAddEx("ScanData", _
            FileGetShortName($File), _
            _Crypt_HashData($FR, $CALG_MD5), _
            FileGetTime($File, 0, 1), _
            FileGetSize($File), _
            FileGetAttrib($File), _
            @YEAR & "/" & @MON & "/" & @MDAY)

    Return 1

EndFunc   ;==>_GenerateInformation

Func _CheckAllFiles()
    Local $hQuery, $aRow, $sMsg, $i = 0, $Alrt = False
    _SQLite_Query(-1, "SELECT * FROM ScanData;", $hQuery)
    While _SQLite_FetchData($hQuery, $aRow) = $SQLITE_OK
        If $i > 100 Then $i = 0
        GUICtrlSetData($Progress1, $i)
        $nMsg = GUIGetMsg()
        Switch $nMsg
            Case $GUI_EVENT_CLOSE
                Exit
            Case $StopScan
                $StopScanstate = True
        EndSwitch
        If $StopScanstate = True Then ExitLoop
        GUICtrlSetData($Edit1, $aRow[0])
        $Ar = _DataQueryEx("ScanData", $aRow[0], True)
        If IsArray($Ar) Then
            Local $FO = FileOpen($Ar[0])
            Local $FR = FileRead($FO)
            FileClose($FO)
            If _Crypt_HashData($FR, $CALG_MD5) <> $Ar[1] Then
                If $Alrt = False And $Sound Then
                    SoundPlay($Alert)
                    $Alrt = True
                EndIf
                GUICtrlSetState($Icon1, $GUI_HIDE)
                GUICtrlSetState($Warning, $GUI_SHOW)
                GUICtrlSetData($List1, FileGetShortName($aRow[0]))
            EndIf
        EndIf
        $i += 1
    WEnd
    GUICtrlSetData($Edit1, "")
    Return 1
EndFunc   ;==>_CheckAllFiles

Func _CheckFile($Data)

    Local $Ar = _DataQueryEx("ScanData", FileGetShortName($Data), True), $Data2
    If IsArray($Ar) Then
        Local $FO = FileOpen($Ar[0])
        If @error Then
            MsgBox(16, "Error", "there was an error opening the selected file", 3, $hGUI)
            Return 1
        EndIf
        Local $FR = FileRead($FO)
        FileClose($FO)
        Local $Data2 = "====================File Change Information==================" & @CRLF & _
                "File........................... " & FileGetLongName($Ar[0]) & @CRLF & _
                "------------------------------------------------------------------" & @CRLF & _
                "Previouse Hash................. " & _Crypt_HashData($FR, $CALG_MD5) & @CRLF & _
                "Current Hash................... " & $Ar[1] & @CRLF & _
                "------------------------------------------------------------------" & @CRLF & _
                "Previouse Time................. " & $Ar[2] & @CRLF & _
                "Current Time................... " & FileGetTime($Ar[0], 0, 1) & @CRLF & _
                "------------------------------------------------------------------" & @CRLF & _
                "Previouse File size............ " & $Ar[3] & "Kb" & @CRLF & _
                "Current File size.............. " & FileGetSize($Ar[0]) & "Kb" & @CRLF & _
                "Current File size.............. " & Round(FileGetSize($Ar[0]) / 1048576, 2) & "Mb" & @CRLF & _
                "------------------------------------------------------------------" & @CRLF & _
                "Previouse file Attrib.......... " & $Ar[4] & @CRLF & _
                "Current File Attrib............ " & FileGetAttrib($Ar[0]) & @CRLF & _
                "------------------------------------------------------------------" & @CRLF & _
                "Base Scane generated on date... " & $Ar[5]
    Else
        MsgBox(16, "Error", "There is no data related to this file.", 5, $hGUI)
        Return 1
    EndIf

    $Form1 = GUICreate("Form1", 568, 322, 276, 134)
    GUICtrlCreateGroup("File Information", 16, 40, 537, 273)
    GUICtrlCreateEdit("", 24, 56, 521, 249)
    GUICtrlSetData(-1, $Data2)
    GUICtrlCreateGroup("", -99, -99, 1, 1)
    $retu = GUICtrlCreateButton("&Return", 488, 16, 65, 25, $WS_GROUP)
    GUISetState(@SW_SHOW)


    While 1
        $nMsg = GUIGetMsg()
        Switch $nMsg
            Case $retu, $GUI_EVENT_CLOSE
                GUIDelete($Form1)
                GUISwitch($hGUI)
                ExitLoop
        EndSwitch
    WEnd
    Return 1
EndFunc   ;==>_CheckFile

Func _DataAddEx($Table, $Adress, $Val1, $Val2 = "", $Val3 = "", $Val4 = "", $Val5 = "")
    Local $Data
    _SQLite_QuerySingleRow($DB, "SELECT Adress FROM " & $Table & " WHERE Adress='" & $Adress & "'", $Data)
    If $Data[0] <> "" Then
        ;If Not MsgBox(17, "Data exists", "Information for this file exists." & @CRLF & @CRLF & "Continue Operation?") = 1 Then Return 1
        If $NoUpdate = True Then Return 1
        _SQLite_Exec($DB, "UPDATE " & $Table & " SET Val1='" & $Val1 & "', Val2='" & $Val2 & "', Val3='" & $Val3 & "', Val4='" & $Val4 & "', Val5='" & $Val5 & "' WHERE Adress='" & $Adress & "'")
    Else
        _SQLite_Exec($DB, "INSERT INTO " & $Table & " (Adress,Val1,Val2,Val3,Val4,Val5) VALUES ('" & $Adress & "','" & $Val1 & "','" & $Val2 & "','" & $Val3 & "','" & $Val4 & "','" & $Val5 & "');")
    EndIf
    Return 1
EndFunc   ;==>_DataAddEx

Func _DataQueryEx($Table, $Adress, $Ar = False)
    Local $Ret
    If _SQLite_QuerySingleRow($DB, "SELECT * FROM " & $Table & " WHERE Adress='" & $Adress & "'", $Ret) == $SQLITE_OK Then ; =>"*"<= means all...
        If $Ret[0] == "" Then
            Return @error
        Else
            If $Ar Then Return $Ret
            Return $Ret[1]
        EndIf
    EndIf
EndFunc   ;==>_DataQueryEx

Edited April 19, 2011 by System238

[PsaltyDS]

AutoIt is single threaded. When your script hits FileRead() on a large file your script will be blocked for a moment. That said, 50-100MB really isn't a very big file these days, and should take less time to read into a variable than it would take the user to read a popup about it.

If you are talking about more than just reading it to a file, like a loop that processes the read data and takes significant time with a 50-100MB chunk of data (i.e. _Crypt_HashData), then perhaps simply checking FileGetSize() before reading it would allow you to pop up a user notification only when the file is above a certain threshold size.

[Skitty]

AutoIt is single threaded. When your script hits FileRead() on a large file your script will be blocked for a moment. That said, 50-100MB really isn't a very big file these days, and should take less time to read into a variable than it would take the user to read a popup about it.

If you are talking about more than just reading it to a file, like a loop that processes the read data and takes significant time with a 50-100MB chunk of data (i.e. _Crypt_HashData), then perhaps simply checking FileGetSize() before reading it would allow you to pop up a user notification only when the file is above a certain threshold size.

That was my original plan (checking FileGetSize() before reading it would allow you to pop up a user notification), but I thought It might be cooler if I could use filereadline the way I used it in my first example so I could display the reading process via progressbar.

So this brings me down to the center of my problem.

Why can't read the file into a variable even after adding all the stripped "char(10)'s"?

Is it because I add one to the very last line?

Should I remove the last one?

Ps- I know about the single threaded nature of this application(AutoIt3.exe), thats why (in the original script) I have a second instance of the application monitoring the main process and displaying information related to the main scripts activity, although not perfect, but works.

I just thought it would be a neat idea to have the main app update the monitoring app that its about to read a fairly large file.

So, is there a way to identify the characters that are replaced by filereadeline? eg, save all @CR @LF characters and put them back into their place?

-thanks for the help.

Edited April 18, 2011 by System238

[PsaltyDS]

OK, so the file is a text file, and the binary thing was just a red herring to throw us off and confuse the issue.

Now for this:

Why can't read the file into a variable even after adding all the stripped "char(10)'s"?

Nothing stops you from reading the entire file into a variable, period. And regardless of if you strip @LF or not. If you want to process the file one line at a time, then read it into an array instead of a string with _FileReadToArray().

Describe the source data file. What kind of text file is it? Why do you think all this manipulation of @LF characters is required?

[Skitty]

OK, so the file is a text file, and the binary thing was just a red herring to throw us off and confuse the issue.

Now for this: Nothing stops you from reading the entire file into a variable, period. And regardless of if you strip @LF or not. If you want to process the file one line at a time, then read it into an array instead of a string with _FileReadToArray().

Describe the source data file. What kind of text file is it? Why do you think all this manipulation of @LF characters is required?

first of all, what the heck happened to the "I" after "can't" and before "Read" in my last post...

Second, by looking at the sample script, you can see that when you click "db create", the thing starts recursing every single file on your hdd.

So since every file is opened and read, I would say the script should preferably be universal. (Open and load any file to a variable in this manner if file is above 100Mb)

I think the manipulation is required because I read in the help file that the filereadeline function strip's all @LF characters "Char(10)", and the help file says it is this specific character and not "@CR" but "@LF".

All I wanted to do is have a progress bar update the process, I don't really care about performance or speed issues.

Edited April 19, 2011 by System238

[jchd]

You're mixing things, adding to confusion.

If you want to be able to "process" _any_ file, text or binary, then you need to read it as binary. There is _no_ notion of "line" in a binary file.

Using the filesize, determine if you read and process it in one small enough chunk, or if you want to read it in multiple pieces in which case your progress bar makes sense.