read big file

[JRSmile]

Hi Folks,

i have a small problem and might need some advice on how to read a file the most ram/speed efficient way.

i compared three methods and could see that a binary read to ram and then a conversion into a string is much (5 times) faster then a direct FileRead in string mode.

expandcollapse popup

#include <array.au3>
#include <file.au3>
#include <FileConstants.au3>

; Method 1
while True
$tTimer = TimerInit()
$hFile = FileOpen( @ScriptDir & "\wcl.txt", 16 ) ; 16 = binary mode
$sBinData = FileRead( $hFile )
FileClose( $hFile )
$file = BinaryToString($sBinData)
ConsoleWrite(round(TimerDiff($ttimer) /1000,2) &" Sekunden"& @CRLF)
$file = 0
WEnd ; 10-12 seconds of import time on a single hdd configuration, 6 GB of ram used during import

;Method 2
while True
$tTimer = TimerInit()
$hFile = FileOpen( @ScriptDir & "\wcl.txt", 16 ) ; 16 = binary mode
$sBinData = FileRead( $hFile )
$iBytes = @extended
FileClose( $hFile )
$tInts = DllStructCreate( "int[" & $iBytes/4 & "]" )
$pInts = DllStructGetPtr( $tInts )
$tBytes = DllStructCreate( "byte[" & $iBytes & "]", $pInts )
DllStructSetData( $tBytes, 1, $sBinData )
$sBinData = 0
;~ MsgBox(0,Default,"check ram")
$file = BinaryToString(DllStructGetData($tBytes, 1))
;~ MsgBox(0,Default,"check ram")
$tBytes = 0
$tInts = 0
;~ MsgBox(0,Default,"check ram")
ConsoleWrite(round(TimerDiff($ttimer) /1000,2) &" Sekunden"& @CRLF)
WEnd ; 12-14 seconds of import time on a single hdd configuration, 5 GB of ram used during import

;Method 3
while True
$tTimer = TimerInit()
$hFile = FileOpen( @ScriptDir & "\wcl.txt", $FO_READ )
$sStrData = FileRead($hFile)
FileClose($hFile)
ConsoleWrite(round(TimerDiff($ttimer) /1000,2) &" Sekunden"& @CRLF)
$sStrData = 0
WEnd ; 40-50 seconds of import time on a single hdd configuration, 4 GB of ram used during import

i tested this with a 1 gb log file which is round about the size of files i need to handle quickly.

if you guys know another way to load a file into memory and get quick access to its content leave me a line please.

Best Regards,

JR.

Edited August 4, 2014 by JRSmile

[JRSmile]

me again, i found memory mapped files on the forum, but they only work to an extend that the adress space of the struct runs out if it exceeds 2gb ( i wonder why because the source file is only 1gb).

#include <APIConstants.au3>
#include <WinAPIEx.au3>

$sFile = @ScriptDir & "\wcl.txt"
$hFile = _WinAPI_CreateFileEx($sFile, 3, 0x80000000, 7, 0x08000000)
$hMapping = _WinAPI_CreateFileMapping($hFile, 0, '', $PAGE_READONLY)
$iBinarySize = _WinAPI_GetFileSizeEx($hFile)
ConsoleWrite($iBinarySize)
$pAddress = _WinAPI_MapViewOfFile($hMapping, 0, 0, $FILE_MAP_READ)
$tBuffer = DllStructCreate('byte[' & $iBinarySize & ']', $pAddress)
$strFile = BinaryToString(DllStructGetData($tBuffer, 1))
MsgBox(0,Default,"check ram")
_WinAPI_UnmapViewOfFile($pAddress)
_WinAPI_CloseHandle($hMapping)
_WinAPI_CloseHandle($hFile)
ConsoleWrite($strFile & @CRLF)

Edited August 4, 2014 by JRSmile