FileRead in binary mode question

[nekkutta]

I'm trying to work on a File Splitter app, and I have run into the AutoIt Error: Error Allocating Memory, I was wondering what the max size Fileread can read into memory is, the file that I ran into the problem is 942,537,015 bytes and the help file states that a binary can hold 2,147,483,647 bytes, so I shouldn't be running out of memory for this. anyone have any info on what this might be?

Script:

expandcollapse popup

$BaseDir = @DesktopDir & '\Filesplit'
$files = FileOpenDialog("Select Files","","All Files (*.*)" ,7)
If @error Then Exit
If StringInStr($files,'|') Then ;multiple files
    $aFiles = StringSplit($files,'|',3)
Else
    Dim $aFiles[2]
    $aFiles[0] = StringLeft($files,StringInStr($files,'\',0,-1)-1 )
    $aFiles[1] = StringTrimLeft($files,StringInStr($files,'\',0,-1) )
EndIf
If FileExists($BaseDir) Then DirRemove($BaseDir,1)
DirCreate($BaseDir)

;Const $cdSize = 736329728
Const $cdSize = 191919728
$curCdFree = $cdSize
$cdcount = 1

;first CD
$curCdDir = $BaseDir & '\CD ' & $cdcount
DirCreate($curCdDir)

;Start Manifest
$manifesthndl = FileOpen($curCdDir & '\MANIFEST',2)
$curCdFree -= 2048

;Install Stub
FileInstall('.\fsstub.exe',$curCdDir & '\',1)
$curCdFree -= FileGetSize($curCdDir & '\fsstub.exe')

;Create Autorun
$arinf = FileOpen($curCdDir & '\Autorun.inf',2)
FileWrite($arinf,"[AutoRun]" & @CRLF & "open=fsstub.exe" & @CRLF & "icon=fsstub.exe,0" & @CRLF & "action=Extract from CD(s)" & @CRLF)
FileClose($arinf)
$curCdFree -= FileGetSize($curCdDir & '\Autorun.inf')

;Start Splitter
For $i = 1 To UBound($aFiles) - 1
    $curFilehndl = FileOpen($aFiles[0] & '\' & $aFiles[$i],16)
    $curFiledata = FileRead($curFilehndl)
    FileClose($curFilehndl)
    Do
        $fspfile = Random(10000000,99999999,1) & '.fsp'
    Until Not FileExists($curCdDir & '\' & $fspfile)
    $disklist = ''
    $curDataPos = 1
    Do
        $fspdata = BinaryMid($curFiledata,$curDataPos,$curCdFree)
        $disklist &= $cdcount & ','
        $curDataPos += $curCdFree
        $curfsp = FileOpen($curCdDir & '\' & $fspfile,18)
        FileWrite($curfsp,$curFiledata)
        FileClose($curfsp)
        $curCdFree -= FileGetSize($curCdDir & '\' & $fspfile)
        If $curDataPos < BinaryLen($curFiledata) Then _NextCD()
    Until $curDataPos >= BinaryLen($curFiledata)
    $disklist = StringTrimRight($disklist,1)
    man($aFiles[$i],$disklist,$fspfile)
Next

Func _NextCD()
    $cdcount += 1
    $curCdDir = $BaseDir & '\CD ' & $cdcount
    DirCreate($curCdDir)
    $curCdFree = $cdSize
EndFunc
Func man($oringalname,$disks,$fspname)
    FileWriteLine($manifesthndl,$oringalname & '|' & $disks & '|' & $fspname)
EndFunc

[AdmiralAlkex]

There'll be some copying around in memory so you need to double stuff. Look at task manager while you run your script and you'll see. You're probably hitting the 2GB ram limit on x86 systems.

You either need a bucketload of ram on a x64 system, or even better, just read the file in parts.

Seriously, it's insane to read whole files like that. Ask anyone.

And next time, please post a reproducer, like this:

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=n
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
$hFile = FileOpen(FileOpenDialog("Select Files","","All Files (*.*)" , 3), 16)
$bFile = FileRead($hFile)
ConsoleWrite(BinaryLen($bFile) & @CRLF)
FileClose($hFile)

MsgBox(0, "", "")

A short runnable example is worth way more than a complete script that can't be run.

Edited January 15, 2011 by AdmiralAlkex

[nekkutta]

Script does run, just crashes on large files, what commands would I use to read a file in parts? haven't ever dealt with that, then again, also haven't done too much dealing with binary files, minus populating a sqlite db with jpgs.

EDIT: Nevermind, Just noticed the [, $count] Parameter for File Read, never noticed that before... D'oh

Edited January 16, 2011 by nekkutta

[nekkutta]

Ok, Now it works fine, just need to create the joiner.

Also It would be nice if it was documented in the help file that the FilePos is updated by the count in FileRead, took some wading through my code to try and figure out why my output file was about half the size of the original.

Edited January 16, 2011 by nekkutta

[AdmiralAlkex]

Yeah, Doing FileRead's after each other continues reading the file. Kinda makes sense when you remember that FileSetPos/FileGetPos is completely new and reading files is old.

[nekkutta]

Yeah, Doing FileRead's after each other continues reading the file. Kinda makes sense when you remember that FileSetPos/FileGetPos is completely new and reading files is old.

I agree, does make sense, but never having dealt with chunk reading, I was going off the info in the help file, then kinda doing a head scratch trying to figure out what was wrong, and doing the trial and error method of debugging, ie. if I comment out this line does it work?

Glad I have it working now, I'm also pretty much done with the stub executable, so.....

If anyone would be interested in a program to split large files across several cds with an autorun to rejoin them, I'll post my code for the spliter and stub.

[AdmiralAlkex]

If anyone would be interested in a program to split large files across several cds with an autorun to rejoin them, I'll post my code for the spliter and stub.

I don't know, I usually use WinRAR in store mode to split files. Why don't you post it in the examples forum, I'm sure someone somewhere will have use for it

[MvGulik]

If anyone would be interested in a program to split large files across several cds with an autorun to rejoin them, I'll post my code for the spliter and stub.

Preverbly in the Example script section ... I hope.

... reading files is old.

Edited January 16, 2011 by MvGulik

[AdmiralAlkex]