Jump to content
Sign in to follow this  
JRSmile

read big file

Recommended Posts

JRSmile

Hi Folks,

i have a small problem and might need some advice on how to read a file the most ram/speed efficient way.

i compared three methods and could see that a binary read to ram and then a conversion into a string is much (5 times) faster then a direct FileRead in string mode.

#include <array.au3>
#include <file.au3>
#include <FileConstants.au3>

; Method 1
while True
$tTimer = TimerInit()
$hFile = FileOpen( @ScriptDir & "\wcl.txt", 16 ) ; 16 = binary mode
$sBinData = FileRead( $hFile )
FileClose( $hFile )
$file = BinaryToString($sBinData)
ConsoleWrite(round(TimerDiff($ttimer) /1000,2) &" Sekunden"& @CRLF)
$file = 0
WEnd ; 10-12 seconds of import time on a single hdd configuration, 6 GB of ram used during import

;Method 2
while True
$tTimer = TimerInit()
$hFile = FileOpen( @ScriptDir & "\wcl.txt", 16 ) ; 16 = binary mode
$sBinData = FileRead( $hFile )
$iBytes = @extended
FileClose( $hFile )
$tInts = DllStructCreate( "int[" & $iBytes/4 & "]" )
$pInts = DllStructGetPtr( $tInts )
$tBytes = DllStructCreate( "byte[" & $iBytes & "]", $pInts )
DllStructSetData( $tBytes, 1, $sBinData )
$sBinData = 0
;~ MsgBox(0,Default,"check ram")
$file = BinaryToString(DllStructGetData($tBytes, 1))
;~ MsgBox(0,Default,"check ram")
$tBytes = 0
$tInts = 0
;~ MsgBox(0,Default,"check ram")
ConsoleWrite(round(TimerDiff($ttimer) /1000,2) &" Sekunden"& @CRLF)
WEnd ; 12-14 seconds of import time on a single hdd configuration, 5 GB of ram used during import

;Method 3
while True
$tTimer = TimerInit()
$hFile = FileOpen( @ScriptDir & "\wcl.txt", $FO_READ )
$sStrData = FileRead($hFile)
FileClose($hFile)
ConsoleWrite(round(TimerDiff($ttimer) /1000,2) &" Sekunden"& @CRLF)
$sStrData = 0
WEnd ; 40-50 seconds of import time on a single hdd configuration, 4 GB of ram used during import

i tested this with a 1 gb log file which is round about the size of files i need to handle quickly.

if you guys know another way to load a file into memory and get quick access to its content leave me a line please.

Best Regards,

JR.

Edited by JRSmile

$a=StringSplit("547275737420796F757220546563686E6F6C75737421","")For $b=1 To UBound($a)+(-1*-1*-1)step(2^4/8);&$b+=1*2/40*µ&Asc(4)Assign("c",Eval("c")&Chr(Dec($a[$b]&$a[$b+1])))''Chr("a")&"HI"Next;time_U&r34d,ths,U-may=get$the&c.l.u.e;b3st-regards,JRSmile;MsgBox(0x000000,"",Eval("c"));PiEs:d0nt+*b3.s4d.4ft3r.1st-try:-)

Share this post


Link to post
Share on other sites
JRSmile

me again, i found memory mapped files on the forum, but they only work to an extend that the adress space of the struct runs out if it exceeds 2gb ( i wonder why because the source file is only 1gb).

#include <APIConstants.au3>
#include <WinAPIEx.au3>

$sFile = @ScriptDir & "\wcl.txt"
$hFile = _WinAPI_CreateFileEx($sFile, 3, 0x80000000, 7, 0x08000000)
$hMapping = _WinAPI_CreateFileMapping($hFile, 0, '', $PAGE_READONLY)
$iBinarySize = _WinAPI_GetFileSizeEx($hFile)
ConsoleWrite($iBinarySize)
$pAddress = _WinAPI_MapViewOfFile($hMapping, 0, 0, $FILE_MAP_READ)
$tBuffer = DllStructCreate('byte[' & $iBinarySize & ']', $pAddress)
$strFile = BinaryToString(DllStructGetData($tBuffer, 1))
MsgBox(0,Default,"check ram")
_WinAPI_UnmapViewOfFile($pAddress)
_WinAPI_CloseHandle($hMapping)
_WinAPI_CloseHandle($hFile)
ConsoleWrite($strFile & @CRLF)
Edited by JRSmile

$a=StringSplit("547275737420796F757220546563686E6F6C75737421","")For $b=1 To UBound($a)+(-1*-1*-1)step(2^4/8);&$b+=1*2/40*µ&Asc(4)Assign("c",Eval("c")&Chr(Dec($a[$b]&$a[$b+1])))''Chr("a")&"HI"Next;time_U&r34d,ths,U-may=get$the&c.l.u.e;b3st-regards,JRSmile;MsgBox(0x000000,"",Eval("c"));PiEs:d0nt+*b3.s4d.4ft3r.1st-try:-)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×