How to split a 100 MB binary file?

footswitch · September 6, 2007

Hello folks!

So... the title says it all...

I have, let's say, a 100 MB binary file.

How to remove (for instance) the first 512 bytes of that file? And leave the rest of the file intact.

Can someone help me on this?

Any time and/or effort is much appreciated.

Thanks in advance!

footswitch

Edited September 7, 2007 by footswitch

evilertoaster · September 6, 2007

You will need to use FileRead() to read the file in in any case.

psudo code might be somthing like-

FileWrite("output",StringTrimLeft(FileRead("input",4),512))

I'm a little rusty on the current state of autoit's binary reading/writing but something like that should work

footswitch · September 7, 2007

You will need to use FileRead() to read the file in in any case.

psudo code might be somthing like-
FileWrite("output",StringTrimLeft(FileRead("input",4),512))
I'm a little rusty on the current state of autoit's binary reading/writing but something like that should work

You're evil

If you insist so much in reading the whole file, then I must say... the file is 2GB long. Now what? Stick to the FileRead?

Lol don't take me wrong, but if I wanted to use FileRead() I'd use it and wouldn't ask how to do this without having to copy the whole file to memory.

Thanks for your attention though.

Regards

footswitch

MHz · September 7, 2007

You're evil
If you insist so much in reading the whole file, then I must say... the file is 2GB long. Now what? Stick to the FileRead?
Lol don't take me wrong, but if I wanted to use FileRead() I'd use it and wouldn't ask how to do this without having to copy the whole file to memory.
Thanks for your attention though.
Regards
footswitch

It looks like FileRead() in the example by evilertoaster only does the 1st 4 chars.

footswitch · September 7, 2007

It looks like FileRead() in the example by evilertoaster only does the 1st 4 chars.

His answer isn't a solution. His intent was, I think, to open the file in binary read mode and then extract 512 bytes from it.

You need to open the file in binary read mode first.

So the correct coding would be (please note that I could be wrong with these +1 and -$chunk_size, but it must be close to this):

; this will read a file to memory, remove the first 256 bytes from it and save the result to a new file

$file_src="sourcefile.bin" ; source file
$file_dest="destfile.bin" ; destination file (chunked)

$chunk_size=256 ; number of bytes to remove, thus the $start value for BinaryMid() (+1 to [b]start[/b], for instance, in byte no. 257)
$count=FileGetSize($file_src)-$chunk_size ; this will be the new filesize, thus the $count value for BinaryMid()

$handle_src=FileOpen($file_src,16) ; open in binary read mode
$handle_dest=FileOpen($file_dest,16+2) ; open new file in binary WRITE mode - erase previous contents if they exist

FileWrite($handle_dest,BinaryMid(FileRead($handle_src),$chunk_size+1,$count)) ; read the whole file with FileRead() and return chunked file with BinaryMid(), writing it to the new file with FileWrite()

; close files after using them
FileClose($handle_src)
FileClose($handle_dest)

Just to remind you, this can't be done with a 2GB file

EDIT: Some more explanations introduced

Edited September 7, 2007 by footswitch

SmOke_N · September 7, 2007

You're only native option then would be to read it line by line... 2GB file... Yeah, see you next week

PsaltyDS · September 7, 2007

Looked interesting, so I gave it shot with a demo. The input file was 512MB in the test and took 42 seconds to break it into six files:

Dim $sRead = "C:\Temp\InputFile.bin", $iFile = 0, $EOF = 0, $OneMeg = 1024 * 1024, $Timer = TimerInit()

$hRead = FileOpen($sRead, 0 + 16) ; Binary read

Do
    $iFile += 1
    $hWrite = FileOpen($sRead & ".Part" & $iFile, 2 + 16) ; Binary overwrite
    For $n = 1 To 100
        $binData = FileRead($hRead, $OneMeg) ; 1MB blocks
        $EOF = @error
        FileWrite($hWrite, $binData)
        If $EOF Then ExitLoop
    Next
    FileClose($hWrite)
Until $EOF

FileClose($hRead)
$Timer = TimerDiff($Timer)

$sMsg = "Completed breaking up file " & $sRead & " into 100MB parts." & @CRLF
$sMsg &= Round(FileGetSize($sRead) / $OneMeg, 2) & "MB file broken up into " & $iFile & " parts." & @CRLF
$sMsg &= "Processed in " & Round($Timer / 1000, 2) & " seconds."
MsgBox(64, "Finished", $sMsg)

ptrex · September 7, 2007

@all

Something from old box :

; File Splitter 1.0 = Origional Script
;                                      -by Andrew Dunn (Hallman)

; File Splitter 1.1 = GUI modified, added functions
;                         -smashly
;                     New Progress Bars, modified split/join function
;                         -Hallman
;                     Fixed Broken drop files & hide n show gui for join & resized/moved progress window.
;                         -smashly

#include <GUIConstants.au3>
#include <file.au3>

Opt("GUIOnEventMode", 1)    ;1 = enable

$apw = 330
$aph = 134

$MainWindow = GUICreate("File Splitter 1.1", $apw, $aph, @DesktopWidth - $apw - 10, @DesktopHeight - $aph - 65, -1, $WS_EX_ACCEPTFILES)
WinSetOnTop($MainWindow, "", 1) ; Set the gui as always on top
GUISetOnEvent($GUI_EVENT_CLOSE, "_Exit")

GUICtrlCreateGroup("Drop or Browse for a file to Split or Join", 10, 10, 310, 114)

$InFile = GUICtrlCreateInput("", 20, 30, 190, 20)
$OutDirectory = GUICtrlCreateInput("", 20, 60, 190, 20)
$InBrowse = GUICtrlCreateButton("Input File", 220, 30, 90, 20)

GUICtrlSetOnEvent($InBrowse, "_handler")

$OutBrowse = GUICtrlCreateButton("Output Directory", 220, 60, 90, 20)

GUICtrlSetOnEvent($OutBrowse, "_handler")

$SplitInto = GUICtrlCreateLabel("Split File Into:", 22, 93, 64, 20)
$InSize = GUICtrlCreateInput("100", 90, 90, 60, 20)
$MB = GUICtrlCreateCombo("", 160, 90, 50, 21)
GUICtrlSetData($MB, "MB|KB|B", "MB")

$SplitJoin = GUICtrlCreateButton("", 220, 90, 90, 20)
GUICtrlSetOnEvent($SplitJoin, "_handler")

GUICtrlSetState($InFile, $GUI_DROPACCEPTED)
GUICtrlSetState($OutDirectory, $GUI_DROPACCEPTED)
GUISetOnEvent($GUI_EVENT_DROPPED, "_filedrop")

GUISetState(@SW_SHOW,$MainWindow)


;  Old Progress Bar
;~ $progressbar = GUICtrlCreateProgress (20,121,290,17,$PBS_SMOOTH)
;~ $percent = GUICtrlCreateLabel("", 155, 123, 30, 12, $SS_CENTER)
;~ GUICtrlSetFont($percent,8,500)
;~ GUICtrlSetBkColor($percent,0xffffff)

; ----- Create Progress GUI -----------------------------------------------------------------

$Progress_Win = GUICreate("file progress ...", $apw, $aph, @DesktopWidth - $apw - 10, @DesktopHeight - $aph - 65, $WS_CAPTION)

$Total_Percent_Label = GUICtrlCreateLabel("Total Percent Done = 0%", 10, 10, 300, 15)

$Total_Percent_Progress = GUICtrlCreateProgress(10, 25, 310, 25)
GUICtrlSetLimit(-1, 100, 0)

$File_Number_Label = GUICtrlCreateLabel("File 0 Of 0", 10, 65, 300, 15)

$File_Percent_Progress = GUICtrlCreateProgress(10, 80, 310, 25)
GUICtrlSetLimit(-1, 100, 0)

$File_Percent_Label = GUICtrlCreateLabel("Percent = 0", 10, 105, 300, 15)

; havn't done anything with this yet
;~ $Cancel_Btn = GUICtrlCreateButton("Cancel", 10, 130, 100, 20)

GUISetState(@SW_HIDE, $Progress_Win)

; --------------------------------------------------------------------------------------------

_StartUpDisable()

While 1
    Sleep(10)
WEnd

Func _handler()
    Select
        Case @GUI_CtrlId = $InBrowse ;Input File browse button, same sorta filtering as dropfile, so the gui will show the right controls depending on file opened.
            $Select_File = FileOpenDialog("Open ...", "::{20D04FE0-3AEA-1069-A2D8-08002B30309D}", "All Files (*.*)", 1)
            If Not @error Then
                Dim $szDrive, $szDir, $szFName, $szExt
                $FileType = _PathSplit($Select_File, $szDrive, $szDir, $szFName, $szExt)
                If StringLeft($FileType[4], 3) <> ".FS" Then
                    GUICtrlSetData($InFile, $Select_File)
                    GUICtrlSetData($OutDirectory, $FileType[1] & StringTrimRight($FileType[2], 1))
                    EnableSplitCtrls()
                ElseIf $FileType[4] = ".FS1" And $FileType[4] <> "" Then
                    GUICtrlSetData($InFile, $Select_File)
                    $sp = StringSplit($FileType[2], "\")
                    $i = $sp[0] - 1
                    $LastString = StringLen($sp[$i])
                    $DirOut = StringTrimRight($FileType[1] & $FileType[2], $LastString + 2)
                    GUICtrlSetData($OutDirectory, $DirOut)
                    EnableJoinCtrls()
                ElseIf StringLeft($FileType[4], 3) = ".FS" And StringRight($FileType[4], 2) <> "S1" Then
                    GUICtrlSetData($InFile, "", "")
                    GUICtrlSetData($OutDirectory, "", "")
                    _StartUpDisable()
                EndIf
            EndIf
        Case @GUI_CtrlId = $OutBrowse ; Output directory browse button.
            $OutputPath = FileSelectFolder("Choose the ISO folder you want to share.", "", 1)
            If Not @error Then
                GUICtrlSetData($OutDirectory, $OutputPath)
            EndIf
        Case @GUI_CtrlId = $SplitJoin ;Split or Join button depending what file is dropped/opened
            If GUICtrlRead($SplitJoin) = "Split File" Then
                GUICtrlSetState($SplitJoin, $GUI_DISABLE)
                Dim $szDrive, $szDir, $szFName, $szExt
                $OutFile = _PathSplit(GUICtrlRead($InFile), $szDrive, $szDir, $szFName, $szExt)
                $OutFile2 = GUICtrlRead($OutDirectory)
                $n = 0
                $outpath = $OutFile2 & '\Split' & $n & "_" & $OutFile[3] & $OutFile[4]
                Do
                    $n = $n + 1
                    $exist1 = FileExists($outpath)
                    $outpath = $OutFile2 & '\Split' & $n & "_" & $OutFile[3] & $OutFile[4]
                    $exist1 = FileExists($outpath)
                Until $exist1 = 0

                GUISetState(@SW_HIDE, $MainWindow)

                $File_Split = Split_File(GUICtrlRead($InFile), $outpath & "\", $OutFile[3] & $OutFile[4])

                GUISetState(@SW_SHOW, $MainWindow)

                If $File_Split = -1 Then
                    MsgBox(0, "error", "The size you inputed for the split size is larger than the file itself!")
                    GUICtrlSetState($SplitJoin, $GUI_ENABLE)
                ElseIf $File_Split = -2 Then
                    MsgBox(0, "error", "Unable to open the source file.")
                    GUICtrlSetState($SplitJoin, $GUI_ENABLE)
                ElseIf $File_Split = -3 Then
                    MsgBox(0, "error", "Unable to to create output files.")
                    GUICtrlSetState($SplitJoin, $GUI_ENABLE)
                ElseIf $File_Split = 1 Then
                    MsgBox(0, "Done!", 'Done splitting the file. The pieces have been saved in: ' & @CRLF & @CRLF & $outpath & '\')
                    GUICtrlSetState($SplitJoin, $GUI_ENABLE)
                EndIf
            ElseIf GUICtrlRead($SplitJoin) = "Join File" Then
                GUICtrlSetState($SplitJoin, $GUI_DISABLE)
                Dim $szDrive, $szDir, $szFName, $szExt
                $Path_Split = _PathSplit(GUICtrlRead($InFile), $szDrive, $szDir, $szFName, $szExt)
                $Temp_Split = StringSplit($Path_Split[3], ".")
                $Real_File_Name = ""
                For $i = 1 To ($Temp_Split[0] - 1) Step 1
                    $Real_File_Name &= $Temp_Split[$i] & "."
                Next
                $Real_File_Name = StringTrimRight($Real_File_Name, 1)
                $OutFile2 = GUICtrlRead($OutDirectory)
                $n = 0
                $outpath = $OutFile2 & '\Join' & $n & "_" & $Real_File_Name
                Do
                    $n = $n + 1
                    $exist1 = FileExists($outpath)
                    $outpath = $OutFile2 & '\Join' & $n & "_" & $Real_File_Name
                    $exist1 = FileExists($outpath)
                Until $exist1 = 0

                GUISetState(@SW_HIDE, $MainWindow)

                $File_Join = Join_File(GUICtrlRead($InFile), $outpath & "\" & $Real_File_Name, $Temp_Split[$Temp_Split[0]])

                GUISetState(@SW_SHOW, $MainWindow)

                If $File_Join = -1 Then
                    MsgBox(0, "error", "Unable to open the output file.")
                    GUICtrlSetState($SplitJoin, $GUI_ENABLE)
                ElseIf $File_Join = -2 Then
                    MsgBox(0, "error", "Unable to open the file part one.")
                    GUICtrlSetState($SplitJoin, $GUI_ENABLE)
                ElseIf $File_Join = 1 Then
                    MsgBox(0, "Done!", 'Done joining the file. The file has been saved in: ' & @CRLF & @CRLF & $outpath & '\')
                    GUICtrlSetState($SplitJoin, $GUI_ENABLE)
                EndIf
            EndIf
    EndSelect
EndFunc   ;==>_handler

Func _filedrop() ;Just some drop file filtering so the gui will offer the right controls for the file dropped
    Dim $szDrive, $szDir, $szFName, $szExt
    $FileType = _PathSplit(@GUI_DragFile, $szDrive, $szDir, $szFName, $szExt)
    If @GUI_DropId = $InFile Then
        If $FileType[4] <> "" And StringLeft($FileType[4], 3) <> ".FS" Then
            GUICtrlSetData($OutDirectory, $FileType[1] & StringTrimRight($FileType[2], 1))
            EnableSplitCtrls()
        ElseIf $FileType[4] = ".FS1" And $FileType[4] <> "" Then
            $sp = StringSplit($FileType[2], "\")
            $i = $sp[0] - 1
            $LastString = StringLen($sp[$i])
            $DirOut = StringTrimRight($FileType[1] & $FileType[2], $LastString + 2)
            GUICtrlSetData($OutDirectory, $DirOut)
            EnableJoinCtrls()
        ElseIf $FileType[4] = "" Or StringLeft($FileType[4], 3) = ".FS" And StringRight($FileType[4], 2) <> "S1" Then
            GUICtrlSetData($InFile, "", "")
            GUICtrlSetData($OutDirectory, "", "")
            _StartUpDisable()
        EndIf
    ElseIf @GUI_DropId = $OutDirectory Then
        If $FileType[4] <> "" Then
            GUICtrlSetData($OutDirectory, "", "")
        ElseIf $FileType[4] = "" Then
            GUICtrlSetData($OutDirectory, "", "")
            GUICtrlSetData($OutDirectory, @GUI_DragFile)
        EndIf
    EndIf
EndFunc   ;==>_filedrop


Func _StartUpDisable() ;Disable n hide some controls so a user can only do things in the right order.
    GUICtrlSetState($SplitInto, $GUI_DISABLE + $GUI_HIDE)
    GUICtrlSetState($InSize, $GUI_DISABLE + $GUI_HIDE)
    GUICtrlSetState($MB, $GUI_DISABLE + $GUI_HIDE)
    GUICtrlSetState($SplitJoin, $GUI_DISABLE + $GUI_HIDE)
EndFunc   ;==>_StartUpDisable

Func EnableSplitCtrls() ;Show n Enable some controls if the user drops/opens a file to split.
    GUICtrlSetState($SplitInto, $GUI_ENABLE + $GUI_SHOW)
    GUICtrlSetState($InSize, $GUI_ENABLE + $GUI_SHOW)
    GUICtrlSetState($MB, $GUI_ENABLE + $GUI_SHOW)
    GUICtrlSetState($SplitJoin, $GUI_ENABLE + $GUI_SHOW)
    GUICtrlSetData($SplitJoin, "Split File")
EndFunc   ;==>EnableSplitCtrls

Func EnableJoinCtrls() ;Show n enable some controls if a user drops/opens .FS1 for join.
    GUICtrlSetState($SplitInto, $GUI_DISABLE + $GUI_HIDE)
    GUICtrlSetState($InSize, $GUI_DISABLE + $GUI_HIDE)
    GUICtrlSetState($MB, $GUI_DISABLE + $GUI_HIDE)
    GUICtrlSetState($SplitJoin, $GUI_ENABLE + $GUI_SHOW)
    GUICtrlSetData($SplitJoin, "Join File")
EndFunc   ;==>EnableJoinCtrls

Func Split_File($Source, $Output_Folder, $Base_Name)

    ; Reset progress to 0 in case the user has split a file before this
    GUICtrlSetData($Total_Percent_Label, "Total Percent Done = 0%")
    GUICtrlSetData($Total_Percent_Progress, 0)
    GUICtrlSetData($File_Number_Label, "File 0 Of 0")
    GUICtrlSetData($File_Percent_Progress, 0)
    GUICtrlSetData($File_Percent_Label, "Percent = 0")



    ; Check if the file is smaller than what the user inputed to be the split size
    If GUICtrlRead($MB) = "MB" And (FileGetSize($Source) / 1048576) <= GUICtrlRead($InSize) Then Return -1
    If GUICtrlRead($MB) = "KB" And (FileGetSize($Source) / 1024) <= GUICtrlRead($InSize) Then Return -1
    If GUICtrlRead($MB) = "B" And FileGetSize($Source) <= GUICtrlRead($InSize) Then Return -1

    ; open the file that will be split (source file)
    $Open_File = FileOpen($Source, 0)
    If $Open_File = -1 Then Return -2

    ; show progress bar
    GUISetState(@SW_SHOW, $Progress_Win)

    ; get the size of the source file
    $File_Size = FileGetSize($Source)

    ; Calculate the total number of files the source file will be split into
    $Number_Of_Files = ""
    If GUICtrlRead($MB) = "MB" Then
        $Number_Of_Files = Ceiling($File_Size / (GUICtrlRead($InSize) * 1048576))
    ElseIf GUICtrlRead($MB) = "KB" Then
        $Number_Of_Files = Ceiling($File_Size / (GUICtrlRead($InSize) * 1024))
    ElseIf GUICtrlRead($MB) = "B" Then
        $Number_Of_Files = Ceiling($File_Size / GUICtrlRead($InSize))
    EndIf

    ; convert the split size form to bytes
    $Split_Size = ""
    If GUICtrlRead($MB) = "MB" Then
        $Split_Size = (GUICtrlRead($InSize) * 1048576)
    ElseIf GUICtrlRead($MB) = "KB" Then
        $Split_Size = (GUICtrlRead($InSize) * 1024)
    ElseIf GUICtrlRead($MB) = "B" Then
        $Split_Size = GUICtrlRead($InSize)
    EndIf

    ; create a variable to hold the current file number being wrote
    $Cur_File_Number = 0

    Sleep(1000)

    $Last_Per_Wrote = 0

    ; loop through all the file pieces (except for the last one since it wont be the same size as the others) and write it's chunk of the source files data
    For $i = 1 To $Number_Of_Files - 1
        $Cur_File_Number = $i

        If $i = 1 Then
            $Open_Cur_File = FileOpen($Output_Folder & $Base_Name & "." & $Number_Of_Files & ".FS" & $i, 10)
        Else
            $Open_Cur_File = FileOpen($Output_Folder & $Base_Name & ".FS" & $i, 10)
        EndIf

        If $Open_Cur_File = -1 Then
            FileClose($Open_File)
            DirRemove(StringTrimRight($Output_Folder, 1), 1)
            Return -3
        EndIf

        GUICtrlSetData($File_Number_Label, "File " & $i & " Of " & $Number_Of_Files)

        If $Split_Size >= 20000 Then

            $Loop_Amount = Floor($Split_Size / 20000)

            $Loop_To_Num = ($Loop_Amount * 20000)

            For $ii = 20000 To $Loop_To_Num Step 20000
                FileWrite($Open_Cur_File, FileRead($Open_File, 20000))

                $Cur_File_Percent = Round((100 / $Loop_To_Num) * $ii, 0)

                If $Cur_File_Percent <> $Last_Per_Wrote Then
                    GUICtrlSetData($File_Percent_Progress, $Cur_File_Percent)
                    GUICtrlSetData($File_Percent_Label, "Percent = " & $Cur_File_Percent)

                    $Last_Per_Wrote = $Cur_File_Percent
                EndIf
            Next
            
            FileWrite($Open_Cur_File, FileRead($Open_File, $Split_Size - (20000 * $Loop_Amount)))

        Else
            FileWrite($Open_Cur_File, FileRead($Open_File, $Split_Size))
        EndIf
        

        GUICtrlSetData($File_Percent_Label, "Percent = 100")
        GUICtrlSetData($File_Percent_Progress, 100)

        FileClose($Open_Cur_File)


        ; calculate total percent done
        $Cur_Total_Per = Round((100 / $Number_Of_Files) * $i, 0)

        GUICtrlSetData($Total_Percent_Label, "Total Percent Done = " & $Cur_Total_Per & "%")

        GUICtrlSetData($Total_Percent_Progress, $Cur_Total_Per)
        ;-----------------------------
    Next

    ; calculate the total number of the source files data already wrote in the split files
    $Bytes_Wrote = $Cur_File_Number * $Split_Size

    ; calculate the number bytes left over that still need to be wrote
    $Left_Over_Bytes = $File_Size - $Bytes_Wrote

    ; Open the last split file
    $Open_Cur_File = FileOpen($Output_Folder & $Base_Name & ".FS" & ($Cur_File_Number + 1), 10)
    If $Open_Cur_File = -1 Then
        FileClose($Open_File)
        DirRemove(StringTrimRight($Output_Folder, 1), 1)
        Return -3
    EndIf

    GUICtrlSetData($File_Number_Label, "File " & $Number_Of_Files & " Of " & $Number_Of_Files)

    ; Write the rest of the bytes to the last file part
    If $Left_Over_Bytes > 20000 Then
        $Loop_Amount = Floor($Left_Over_Bytes / 20000)
        $Loop_To_Num = ($Loop_Amount * 20000)

        For $ii = 20000 To $Loop_To_Num Step 20000
            FileWrite($Open_Cur_File, FileRead($Open_File, 20000))

            $Cur_File_Percent = Round((100 / $Loop_To_Num) * $ii, 0)

            If $Cur_File_Percent <> $Last_Per_Wrote Then
                GUICtrlSetData($File_Percent_Progress, $Cur_File_Percent)
                GUICtrlSetData($File_Percent_Label, "Percent = " & $Cur_File_Percent)
                $Last_Per_Wrote = $Cur_File_Percent
            EndIf

        Next
        FileWrite($Open_Cur_File, FileRead($Open_File, $Left_Over_Bytes - (20000 * $Loop_Amount)))
    Else
        FileWrite($Open_Cur_File, FileRead($Open_File, $Left_Over_Bytes))
    EndIf

    GUICtrlSetData($File_Percent_Progress, 100)
    GUICtrlSetData($File_Percent_Label, "Percent = 100")

    GUICtrlSetData($Total_Percent_Label, "Total Percent Done = 100%")
    GUICtrlSetData($Total_Percent_Progress, 100)

    Sleep(1000)

    ; close files
    FileClose($Open_Cur_File)
    FileClose($Open_File)

    GUISetState(@SW_HIDE, $Progress_Win)

    ; return success
    Return 1
EndFunc   ;==>Split_File

Func Join_File($File_Part_One_Path, $Output_File, $Number_Of_Pieces)
    ; Show progress bar


    ; Open the output file
    $Save_File = FileOpen($Output_File, 10)
    If $Save_File = -1 Then Return -1

    ; Reset progress to 0 in case the user has split a file before this
    GUICtrlSetData($Total_Percent_Label, "Total Percent Done = 0%")
    GUICtrlSetData($Total_Percent_Progress, 0)
    GUICtrlSetData($File_Number_Label, "File 0 Of 0")
    GUICtrlSetData($File_Percent_Progress, 0)
    GUICtrlSetData($File_Percent_Label, "Percent = 0")

    ; Show progress bar
    GUISetState(@SW_SHOW, $Progress_Win)

    ; Open part one of the split file in read mode
    $Open_PartOne_File = FileOpen($File_Part_One_Path, 0)
    If $Open_PartOne_File = -1 Then
        FileClose($Save_File)
        GUISetState(@SW_HIDE, $Progress_Win)
        Return -2
    EndIf

    $Part_One_Size = FileGetSize($File_Part_One_Path)

    GUICtrlSetData($File_Number_Label, "File 1 Of " & $Number_Of_Pieces)

    ; Write the contents of part one to the output file
    If $Part_One_Size < 20000 Then
        FileWrite($Save_File, FileRead($Open_PartOne_File))
    Else

        $Last_Per_Wrote = 0

        $Loop_Amount = Floor($Part_One_Size / 20000)
        $Loop_To_Num = ($Loop_Amount * 20000)

        For $ii = 20000 To $Loop_To_Num Step 20000

            FileWrite($Save_File, FileRead($Open_PartOne_File, 20000))

            $Cur_File_Percent = Round((100 / $Loop_To_Num) * $ii, 0)

            If $Cur_File_Percent <> $Last_Per_Wrote Then
                GUICtrlSetData($File_Percent_Progress, $Cur_File_Percent)
                GUICtrlSetData($File_Percent_Label, "Percent = " & $Cur_File_Percent)
                $Last_Per_Wrote = $Cur_File_Percent
            EndIf
        Next

        FileWrite($Save_File, FileRead($Open_PartOne_File, $Part_One_Size - $Loop_To_Num))

        GUICtrlSetData($File_Percent_Label, "Percent = 100")
        GUICtrlSetData($File_Percent_Progress, 100)

    EndIf

    FileClose($Open_PartOne_File)

    ; calculate total percent done
    $Cur_Total_Per = Round((100 / $Number_Of_Pieces) * 1, 0)

    GUICtrlSetData($Total_Percent_Label, "Total Percent Done = " & $Cur_Total_Per & "%")

    GUICtrlSetData($Total_Percent_Progress, $Cur_Total_Per)
    ;-----------------------------

    ; Set variable to the base file name of all the file parts
    $Base_Split_File_path = StringReplace($File_Part_One_Path, "." & $Number_Of_Pieces & ".FS1", "")

    ; Loop through all the file pieces writing their contents to the output file
    For $i = 2 To $Number_Of_Pieces Step 1
        $Open_Part_File = FileOpen($Base_Split_File_path & ".FS" & $i, 0)
        If $Open_Part_File = -1 Then
            FileClose($Save_File)
            GUISetState(@SW_HIDE, $Progress_Win)
            MsgBox(0, "Error", "Unable to locate """ & $Base_Split_File_path & ".FS" & $i & """. Make sure it's in the same directory as part one.")
            Return 0
        EndIf

        $Part_Size = FileGetSize($Base_Split_File_path & ".FS" & $i)

        GUICtrlSetData($File_Number_Label, "File " & $i & " Of " & $Number_Of_Pieces)

        If $Part_Size < 20000 Then
            FileWrite($Save_File, FileRead($Open_Part_File))
        Else

            $Last_Per_Wrote = 0

            $Loop_Amount = Floor($Part_Size / 20000)
            $Loop_To_Num = ($Loop_Amount * 20000)

            For $ii = 20000 To $Loop_To_Num Step 20000

                FileWrite($Save_File, FileRead($Open_Part_File, 20000))

                $Cur_File_Percent = Round((100 / $Loop_To_Num) * $ii, 0)

                If $Cur_File_Percent <> $Last_Per_Wrote Then
                    GUICtrlSetData($File_Percent_Progress, $Cur_File_Percent)
                    GUICtrlSetData($File_Percent_Label, "Percent = " & $Cur_File_Percent)
                    $Last_Per_Wrote = $Cur_File_Percent
                EndIf
            Next

            FileWrite($Save_File, FileRead($Open_Part_File, $Part_Size - $Loop_To_Num))

            GUICtrlSetData($File_Percent_Label, "Percent = 100")
            GUICtrlSetData($File_Percent_Progress, 100)

        EndIf


        FileClose($Open_Part_File)

        ; calculate total percent done
        $Cur_Total_Per = Round((100 / $Number_Of_Pieces) * $i, 0)

        GUICtrlSetData($Total_Percent_Label, "Total Percent Done = " & $Cur_Total_Per & "%")

        GUICtrlSetData($Total_Percent_Progress, $Cur_Total_Per)
        ;-----------------------------

    Next

    ; close the output file
    FileClose($Save_File)

    GUISetState(@SW_HIDE, $Progress_Win)

    ; return Success
    Return 1
EndFunc   ;==>Join_File

Func _Exit()
    Exit
EndFunc   ;==>_Exit

I didn't make as you can see.

But nevertheless to good to stay under the carpet

Regards,

ptrex

footswitch · September 7, 2007

You're only native option then would be to read it line by line... 2GB file... Yeah, see you next week

LOL

yeah that's the point looolll

(...)

Since we're talking about binary R/W I think this translates to using some sort of buffer and progressively copy stuff from the old to the new file. Even if that was possible, it can be compared to a FileCopy(), and that's the exact opposite of what I want to do. It still would take a lot of time (because of HD Read/Write and script's memory operations) and it would be extremely stupid

I'm looking for a sort of FileMove( $file_src, $file_dest, $byte_start, $byte_end).

Yeah, basically that's it. Wow, pretty simple to tell, ain't it?

EDIT: Wow, wait let me check out those replies

Edited September 7, 2007 by footswitch

evilertoaster · September 7, 2007

since you want to copy data you kinda have to read it... I guess what you're asking is for a way to read the start bytes of a file without ever opening it...the problem there is I think most windows API/standard read/writes open the file first .

soooo..... an alterntive is then to simply raw read the bytes off the hard drive. If you look here - http://www.autoitscript.com/forum/index.php?showtopic=44841 it shows a way to data directly form the hard drive. You'll need to find the offset for your file and start reading it that way.

Really though...Opening the file in autoit is pretty fast (especially with all the speed imporvments in the latest versions).

Edited September 7, 2007 by evilertoaster

PsaltyDS · September 7, 2007

Since we're talking about binary R/W I think this translates to using some sort of buffer and progressively copy stuff from the old to the new file. Even if that was possible, it can be compared to a FileCopy(), and that's the exact opposite of what I want to do. It still would take a lot of time (because of HD Read/Write and script's memory operations) and it would be extremely stupid
I'm looking for a sort of FileMove( $file_src, $file_dest, $byte_start, $byte_end).

I guess I'm missing something about your requirements. What about the code I posted above doesn't work for you?

SmOke_N · September 7, 2007

What about the code I posted above doesn't work for you?

Everything works well for him I'm sure until he gets to Dim and probably stops failing after $sMsg)

(j/k)

footswitch · September 7, 2007

since you want to copy data you kinda have to read it... I guess what you're asking is for a way to read the start bytes of a file without ever opening it...the problem there is I think most windows API/standard read/writes open the file first .
soooo..... an alterntive is then to simply raw read the bytes off the hard drive. If you look here - http://www.autoitscript.com/forum/index.php?showtopic=44841 it shows a way to data directly form the hard drive. You'll need to find the offset for your file and start reading it that way.
Really though...Opening the file in autoit is pretty fast (especially with all the speed imporvments in the latest versions).

Alright, if there's still someone who didn't understand what I'm looking for (although I believe to have explained myself, apart from my english which isn't that perfect when I intend to explain stuff

), here it goes.

I don't know how filesystems work, but I do imagine that files have "FileStart" and "FileEnd" pointers. So:

- Name any file you want;

- Remove some bytes from the beggining of the file. That is, I believe, change the "FileStart" pointer to another location further in the file and "get rid of"/"ignore" the data left behind.

I've already tried to explain this three times, and I really believe I've made myself clear.

I DON'T CARE how the file is to be handled When I'm doing something, the thing I'm looking for is results. I'm also NOT looking for things impossible to accomplish. I only analyse the task and try to get an approximate idea of how long it might take to accomplish and how effective it should be.

I guess I'm missing something about your requirements. What about the code I posted above doesn't work for you?

I didn't fully test your script (which, by the way, isn't that useless

), but basically it copies data. If you're still wondering what's wrong with that, imagine that your file is 4GB long. It takes a long time and spends a lot of resources doing something that could be done in less than one second with really few operations. Not even to talk about data integrity issues.

When I code something, I always TRY to use these rules:

- Make it logical (why copy millions of bytes from one side to the other when all you want to do is remove a dozen of bytes from there?)

- Make it ready for the ultimate stretching of its functionality.

- Make it fool-proof (always expect a nasty user to wreck it by doing something unexpected)

So, your script might be pretty cool for a 100 MB file. But if you're dealing with 4GB... Not recommended at all

Guys, I really appreciate your efforts Gosh do I feel so tiny when I have these sort of tiny problems...

PsaltyDS · September 8, 2007

Well, I _did_ misunderstand then. I read your intention as splitting a large file into multiple files of 100MB each. Looking back at the posts, I'm not sure now how I got that impression...

As for just removing something from the beginning of a file without writing a new file out... I don't think it can be done with normal file operation APIs. The file begins at a block boundry in the disk file tables. That has a disk address. Theoretically you could just change the starting disk address of the file in the file table. But that is NOT something you access with regular Windows file operations. If the part you want to remove doesn't happen to be an exact integer number of blocks long, then it has to be rewritten anyway because the file has to start on a disk address boundary. If the file system blocks are 64KB (a common size) then you would have to change the file by an integer multiple of 64KB blocks. You just don't have that kind of low-level access to the file system with easily available API calls.

Kram3r · September 8, 2007

Use one of the usefull UDF's I have seen

APITailRW

Description: Functions that assist with reading binary files./ reading/ writing/ deleting to tail by line number fast/

writing to any line

based on "APIRW.au3" by Larry, modified Zedna; now by Randallc

Author :Write tail etc by Randallc; original binary by Larry, Zedna [binary wites by EriFash?]

Tip:

don't read to all to one var, use a read/write cycle of one byte each cycle

Edited September 8, 2007 by Kram3r

Siao · September 8, 2007

Tip:
don't read to all to one var, use a read/write cycle of one byte each cycle

I'd say that would be very inefficient.

Autoit's built in File IO functions aren't suited for working large files, that's for sure.

For writing file, DllCall the WriteFile API, of course, I think there are no other options for the task OP wants to do. That APITailRW posted above has it UDF'ed.

For reading large files, file mapping to the rescue... CreateFile, CreateFileMapping, MapViewOfFile, then write it to a new file, then unmap and close handles.

Edited September 8, 2007 by Siao

Kram3r · September 8, 2007

Siao can u be explicit on "file mapping to the rescue... CreateFile, CreateFileMapping, MapViewOfFile" where are that funcs ? UDFs ?

Siao · September 8, 2007

At MSDN.

DllCall'ing them shouldn't be too hard.

Although I haven't tested if it would really pay off speedwise, because OP problem involves changing size of file, so you can't just FlushViewOfFile. So maybe it wouldn't be much faster than doing ReadFile/WriteFile in chunks (just not one byte chunks ). Although with file mapping you can do it without a loop, passing whole of MapViewOfFile to WriteFile and letting the system handle it.

Edited September 8, 2007 by Siao

Kram3r · September 8, 2007

Yes MapViewOfFile to WriteFile would be great, I confess that i don't know the existence of that Kernel32.dll func or possibility.

___________________

As for the above:

Multi-Byte chunks would be more inteligent, if footswitch take in considerantion that in the last read

he as to get off the extra bytes that he gets.

Example:

want 10 bytes and reads 4 byte chunks

get rid of 2

Edited September 8, 2007 by Kram3r

footswitch · September 8, 2007

Thank you PsaltyDS, Kram3r and Siao.

Your replies have been educative, especially the one regarding how the filesystem works. It made sense somehow.

And ptrex... I will take a look at the script you provided. Thanks for that one as well

So I guess this is the end of our journey.

Files can't be splitted without being copied.

Well, actually, you can split a file but the only part of it that you don't have to copy is the first part.

Since I want to remove 512 bytes from the start of a 4GB file, then I'll have to practically copy all of it.

Cool. If all that was said is true, this is a nasty filesystem limitation...

EDIT: I still believe that what I wanted to do is possible (although this can sound a little low level):

- discard the first xxx bytes of the file;

(this is where I start to sound stupid; please bear in mind I'm not that familiar with filesystems)

- move (copy?) a portion of the file (until it reaches the next cluster) to the beginning of a free cluster, and set the beginning of the file to that new cluster;

- create a pointer from the new cluster to the rest of the file.

(I know this might not be exactly what happens in reality, but I think it's close )

So the file would become fragmented, but in practical terms it would still be a single file, now without its first xxx bytes.

Edited September 8, 2007 by footswitch

How to split a 100 MB binary file?

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members