Sign in to follow this  
Followers 0
nekkutta

FileRead in binary mode question

9 posts in this topic

I'm trying to work on a File Splitter app, and I have run into the AutoIt Error: Error Allocating Memory, I was wondering what the max size Fileread can read into memory is, the file that I ran into the problem is 942,537,015 bytes and the help file states that a binary can hold 2,147,483,647 bytes, so I shouldn't be running out of memory for this. anyone have any info on what this might be?

Script:

$BaseDir = @DesktopDir & '\Filesplit'
$files = FileOpenDialog("Select Files","","All Files (*.*)" ,7)
If @error Then Exit
If StringInStr($files,'|') Then ;multiple files
    $aFiles = StringSplit($files,'|',3)
Else
    Dim $aFiles[2]
    $aFiles[0] = StringLeft($files,StringInStr($files,'\',0,-1)-1 )
    $aFiles[1] = StringTrimLeft($files,StringInStr($files,'\',0,-1) )
EndIf
If FileExists($BaseDir) Then DirRemove($BaseDir,1)
DirCreate($BaseDir)

;Const $cdSize = 736329728
Const $cdSize = 191919728
$curCdFree = $cdSize
$cdcount = 1

;first CD
$curCdDir = $BaseDir & '\CD ' & $cdcount
DirCreate($curCdDir)

;Start Manifest
$manifesthndl = FileOpen($curCdDir & '\MANIFEST',2)
$curCdFree -= 2048

;Install Stub
FileInstall('.\fsstub.exe',$curCdDir & '\',1)
$curCdFree -= FileGetSize($curCdDir & '\fsstub.exe')

;Create Autorun
$arinf = FileOpen($curCdDir & '\Autorun.inf',2)
FileWrite($arinf,"[AutoRun]" & @CRLF & "open=fsstub.exe" & @CRLF & "icon=fsstub.exe,0" & @CRLF & "action=Extract from CD(s)" & @CRLF)
FileClose($arinf)
$curCdFree -= FileGetSize($curCdDir & '\Autorun.inf')

;Start Splitter
For $i = 1 To UBound($aFiles) - 1
    $curFilehndl = FileOpen($aFiles[0] & '\' & $aFiles[$i],16)
    $curFiledata = FileRead($curFilehndl)
    FileClose($curFilehndl)
    Do
        $fspfile = Random(10000000,99999999,1) & '.fsp'
    Until Not FileExists($curCdDir & '\' & $fspfile)
    $disklist = ''
    $curDataPos = 1
    Do
        $fspdata = BinaryMid($curFiledata,$curDataPos,$curCdFree)
        $disklist &= $cdcount & ','
        $curDataPos += $curCdFree
        $curfsp = FileOpen($curCdDir & '\' & $fspfile,18)
        FileWrite($curfsp,$curFiledata)
        FileClose($curfsp)
        $curCdFree -= FileGetSize($curCdDir & '\' & $fspfile)
        If $curDataPos < BinaryLen($curFiledata) Then _NextCD()
    Until $curDataPos >= BinaryLen($curFiledata)
    $disklist = StringTrimRight($disklist,1)
    man($aFiles[$i],$disklist,$fspfile)
Next

Func _NextCD()
    $cdcount += 1
    $curCdDir = $BaseDir & '\CD ' & $cdcount
    DirCreate($curCdDir)
    $curCdFree = $cdSize
EndFunc
Func man($oringalname,$disks,$fspname)
    FileWriteLine($manifesthndl,$oringalname & '|' & $disks & '|' & $fspname)
EndFunc

[size="2"] "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian Kernighan[/size]

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

There'll be some copying around in memory so you need to double stuff. Look at task manager while you run your script and you'll see. You're probably hitting the 2GB ram limit on x86 systems.

You either need a bucketload of ram on a x64 system, or even better, just read the file in parts.

Seriously, it's insane to read whole files like that. Ask anyone.

And next time, please post a reproducer, like this:

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=n
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
$hFile = FileOpen(FileOpenDialog("Select Files","","All Files (*.*)" , 3), 16)
$bFile = FileRead($hFile)
ConsoleWrite(BinaryLen($bFile) & @CRLF)
FileClose($hFile)

MsgBox(0, "", "")

A short runnable example is worth way more than a complete script that can't be run.

Edited by AdmiralAlkex

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Script does run, just crashes on large files, what commands would I use to read a file in parts? haven't ever dealt with that, then again, also haven't done too much dealing with binary files, minus populating a sqlite db with jpgs.

EDIT: Nevermind, Just noticed the [, $count] Parameter for File Read, never noticed that before... D'oh

Edited by nekkutta

[size="2"] "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian Kernighan[/size]

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Ok, Now it works fine, just need to create the joiner.

Also It would be nice if it was documented in the help file that the FilePos is updated by the count in FileRead, took some wading through my code to try and figure out why my output file was about half the size of the original.

Edited by nekkutta

[size="2"] "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian Kernighan[/size]

Share this post


Link to post
Share on other sites

Yeah, Doing FileRead's after each other continues reading the file. Kinda makes sense when you remember that FileSetPos/FileGetPos is completely new and reading files is old.

Share this post


Link to post
Share on other sites

Yeah, Doing FileRead's after each other continues reading the file. Kinda makes sense when you remember that FileSetPos/FileGetPos is completely new and reading files is old.

I agree, does make sense, but never having dealt with chunk reading, I was going off the info in the help file, then kinda doing a head scratch trying to figure out what was wrong, and doing the trial and error method of debugging, ie. if I comment out this line does it work?

Glad I have it working now, I'm also pretty much done with the stub executable, so.....

If anyone would be interested in a program to split large files across several cds with an autorun to rejoin them, I'll post my code for the spliter and stub.


[size="2"] "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian Kernighan[/size]

Share this post


Link to post
Share on other sites

If anyone would be interested in a program to split large files across several cds with an autorun to rejoin them, I'll post my code for the spliter and stub.

I don't know, I usually use WinRAR in store mode to split files. Why don't you post it in the examples forum, I'm sure someone somewhere will have use for it :x

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

If anyone would be interested in a program to split large files across several cds with an autorun to rejoin them, I'll post my code for the spliter and stub.

Preverbly in the Example script section ... I hope. :shifty:

... reading files is old.

:P:x Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0